PCIe Fatal Surprise Link Down

Error:
00d2 20/07/16 13:12:49 CRT Bios Critical Interrupt #04 PCIe Fatal Surprise Link Down (00:02.2) 69 [a1 00 12]
00d3 20/07/16 13:12:49 MAJ Bios Critical Interrupt #05 PCIe Warn Receiver Error (00:02.2) 70 [a0 00 12]
Cause:
This is the PCI Bridge chip.
The issue can be caused by: Hardware Failure or Older BIOS Version

 Solution:
Need to check the BIOS Version of problematic Node


[root@sdw4 ~]# syscfg -i
 System Configuration Utility Version 12.0 Build 9
 Copyright (c) 2013 Intel Corporation



System BIOS and FW Versions:

BIOS Version.............. XXXXXXX.XXX.XX.XX.XXXX

BMC Version
 Boot Code............... 01.17
 Op Code................. 1.17.4151



1. If Node is in an older BIOS version then its better to ask customer to upgrade the Node to latest one.

2. If in case the Node is already on Latest Version and still error persists, Need to collect below mentioned logs

a. If we are able to log to the Node

 1. System Event Log

 2. BIOS version

 3. dca_profiler log

b. If Node is unreachable, issue below command from mdw

 1. ipmiutil sel list -N sdw#-sp -U root -P <password> > /tmp/sdw#_sel.log



Note: Password V1 -- calvin, V2 -- sephiroth (If customer has changed the password kindly do get it from them)



After collecting logs, its better we have to replace the Node



Special Notes:

Two other recommendations:

Manually upgrade the BIOS and BMC on the failed server using instructions provided here:

https://emc--c.na5.visual.force.com/apex/KB_HowTo?id=kA0700000004W8j



If node is unreachable, power off and remove power cables. Then power on and perform the manual update.





































Comments











Post a Comment



















Popular posts from this blog









GP - Kerberos errors and resolutions





















 Issue:1.  Ticket granting has expired with the error message,   ERROR: psql: GSSAPI continuation error: Unspecified GSS failure. The Minor code may also display the error message, GSSAPI continuation error: Ticket expired.   Solution:  Create a new ticket on the Master server using the following command:  kinit -k -t /home/gpadmin/gpdb-kerberos.keytab db_user  Note:  -t is the keytab file and db_user is the user.   Issue:2.  The library required to run the kinit command does not exist in the LD_LIBRARY_PATH  set in the Greenplum binaries, producing the following error message:   ERROR: kinit: relocation error: kinit: symbol krb5_get_init_creds_opt_set_fast_ccache_name, version krb5_3_MIT not defined in file libkrb5.so.3 with link time reference   Solution:  Unset the LD_LIBRARY_PATH.   IMPORTANT NOTE:  The above fix will cause the following error when you try to run Postgres commands:    ERROR: psql: error while loading shared libraries: libpq.so.5: cannot open shared object ...








Read more










How to set Optimizer at database level in greenplum





















  How can I turn ON optimizer at Database level?   You can set the optimier ON at database level using below command;     ALTER DATABASE databasename SET OPTIMIZER = ON;     Once this is done, kindly do reconnect to Database to see the change.     How do I check if optimizer is set (ON/OFF) in a given database?   You can use the show command as below to check    psql -d databasename -c 'show optimizer'   








Read more










GP - SQL Joins




















Image







  Available Join Types?  Row Elimination  Minimize Data Movement  Join Implementation Seen in Query Plans   Join Types:  Inner Join   Resulting data set is obtained from joining two tables on a common column  Each row in left table is compared with each row in right table  Matching rows will be present in result set  Most commonly used  Also know has Simple Join or EQUI-Join      Left Outer Join    Returns all rows from the left table even if there is no matching rows in right table.  Will also get the matching rows  Rows that are not matched in the right table will not be included in the result set          Right Outer Join    This is similar to left outer join, just need to flip the join         Full Outer Join    Will return all rows from both the tables where there is a match  We will receive a NULL for rows which don’t match       Cross Join    Which is also called has Cartesian products  Result set will have all rows on the left table paired with all rows on the right table  Exam...








Read more