petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !

[FlyingBlackbird]

My Blackbird does not boot into the petitboot menu anymore (black screen) even though I can see the execution log on screen and via the ssh console with:

Code Select Expand

ssh -p 2200 root@my.ip.address

The last available console output is a failure saying:

Code Select Expand

[   90.334100762,3] PHB#0005[0:5]: eeh_freeze_clear on fenced PHB
               XE autoconfiguration failed


I have only installed a SATA-HDD drive with a bootable Ubuntu Server 19.10 (no GPU).
I have not changed/updated any firmware.
Last thing I have done was testing a PCIe GPU (NVIDIA).
After removing the GPU card from the PCIe slot I could no longer boot into petitboot...

What is going wrong?

The relevant log is (full console output is attached as file):

Code Select Expand


--== Welcome to Hostboot hostboot-3beba24/hbicore.bin ==--

  3.06480|secure|SecureROM valid - enabling functionality
  8.29259|Booting from SBE side 0 on master proc=00050000
  8.46568|ISTEP  6. 5 - host_init_fsi
  8.93808|ISTEP  6. 6 - host_set_ipl_parms
  9.49036|ISTEP  6. 7 - host_discover_targets
10.09037|HWAS|PRESENT> DIMM[03]=8080000000000000
10.09038|HWAS|PRESENT> Proc[05]=8000000000000000
10.09040|HWAS|PRESENT> Core[07]=5565000000000000
10.49937|ISTEP  6. 8 - host_update_master_tpm
10.52369|SECURE|Security Access Bit> 0x0000000000000000
10.52370|SECURE|Secure Mode Disable (via Jumper)> 0x8000000000000000
...
57.11902|ISTEP 21. 2 - host_verify_hdat
57.20205|ISTEP 21. 3 - host_start_payload
[   58.179010391,5] OPAL skiboot-c81f9d6 starting...
[   58.179013552,7] initial console log level: memory 7, driver 5
[   58.179015737,6] CPU: P9 generation processor (max 4 threads/core)
...
[   65.266762790,5] PHB#0000:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..ff SLOT=CPU1 Slot2 (16x)
Petitboot (v1.10.3-pdd2d545)
──────────────────────────────────────────────────────────────────────────────

  System information
  System configuration
  System status log
  Language
  Rescan devices
  Retrieve config from URL
  Plugins (0)
*Exit to shell           










──────────────────────────────────────────────────────────────────────────────
Enter=accept, e=edit, n=new, x=exit, l=language, g=log, h=help
Info: Waiting for device discovery[   85.086133287,3] PHB#0005[0:5]: PHB Freeze/Fence detected !
[   85.086197573,3] PHB#0005[0:5]:             PCI FIR=2000000000000000
[   85.086249297,3] PHB#0005[0:5]:         PCI FIR WOF=2000000000000000
[   85.086289203,3] PHB#0005[0:5]:            NEST FIR=0000800000000000
[   85.086354836,3] PHB#0005[0:5]:        NEST FIR WOF=0000800000000000
[   85.086394899,3] PHB#0005[0:5]:            ERR RPT0=0000000000000001
[   85.086489826,3] PHB#0005[0:5]:            ERR RPT1=0000000000000000
[   85.086534460,3] PHB#0005[0:5]:             AIB ERR=0000200000000000
[   85.086941635,3] PHB#0005[0:5]:                  brdgCtl = 00000002
[   85.087002150,3] PHB#0005[0:5]:             deviceStatus = 00200020
[   85.087036852,3] PHB#0005[0:5]:               slotStatus = 00402000
[   85.087081358,3] PHB#0005[0:5]:               linkStatus = a8120008
[   85.087137004,3] PHB#0005[0:5]:             devCmdStatus = 00100107
[   85.087181127,3] PHB#0005[0:5]:             devSecStatus = 00000000
[   85.087239088,3] PHB#0005[0:5]:          rootErrorStatus = 00000000
[   85.087285589,3] PHB#0005[0:5]:          corrErrorStatus = 00000000
[   85.087325009,3] PHB#0005[0:5]:        uncorrErrorStatus = 00000000
[   85.087370016,3] PHB#0005[0:5]:                   devctl = 00000020
[   85.087419580,3] PHB#0005[0:5]:                  devStat = 00000020
[   85.087466277,3] PHB#0005[0:5]:                  tlpHdr1 = 00000000
...
[   85.088610802,3] PHB#0005[0:5]:       phbRxeArbErrorLog1 = 0000000000000000
  [Disk: sda2 / ef49aa17-bb70-4fea-a8fc-29e235f7ab9f]
    Ubuntu, with Linux 5.3.0-26-generic (recovery mode)
    Ubuntu, with Linux 5.3.0-26-generic
    Ubuntu, with Linux 5.3.0-29-generic (recovery mode)
    Ubuntu, with Linux 5.3.0-29-generic
    Ubuntu
[   85.088655011,3] PHB#0005[0:5]:     phbRxeMrgErrorStatus = 0000000000000001
...
[   85.089573601,3] PHB#0005[0:5]:                PEST[0ff] = 3740002a01000000 0000000000000000
[enP4p1s0f2] Probing from base tftp://192.168.178.1/pxelinux.cfg/[   90.311273403,3] PHB#0005[0:5]: PHB Freeze/Fence detected !
[   90.311357669,3] PHB#0005[0:5]:             PCI FIR=2000000000000000
...
[   90.315282185,3] PHB#0005[0:5]:         phbRegbErrorLog1 = 0001020000000000
[   90.315338900,3] PHB#0005[0:5]:                PEST[000] = 8000000000000000 8000000000000000
[   90.315413179,3] PHB#0005[0:5]:                PEST[001] = 8000000000000000 8000000000000000
[   90.315491213,3] PHB#0005[0:5]:                PEST[002] = 8000000000000000 8000000000000000
...
[   90.333937493,3] PHB#0005[0:5]:                PEST[0fe] = 8000000000000000 8000000000000000
[   90.334011680,3] PHB#0005[0:5]:                PEST[0ff] = b740002a01000000 8000000000000000
[   90.334100762,3] PHB#0005[0:5]: eeh_freeze_clear on fenced PHB
               XE autoconfiguration failed


PS: I have logged in into OpenBMC via SSH and see these strange error messages that may be related:

Code Select Expand

root@blackbird:~# journalctl | grep fail
May 10 19:37:25 blackbird kernel: g_mass_storage 1e6a0000.usb-vhub:p1: failed to start g_mass_storage: -22
May 10 19:37:27 blackbird systemd-udevd[789]: Process 'mtd_probe /dev/mtd2ro' failed with exit code 1.
May 10 19:37:27 blackbird systemd-udevd[790]: Process 'mtd_probe /dev/mtd3ro' failed with exit code 1.
May 10 19:37:27 blackbird systemd-udevd[837]: Process 'mtd_probe /dev/mtd4ro' failed with exit code 1.
May 10 19:37:27 blackbird systemd-udevd[792]: Process 'mtd_probe /dev/mtd5ro' failed with exit code 1.
May 10 19:37:28 blackbird systemd-udevd[788]: Process 'mtd_probe /dev/mtd0ro' failed with exit code 1.
May 10 19:37:28 blackbird systemd-udevd[791]: Process 'mtd_probe /dev/mtd1ro' failed with exit code 1.
May 10 19:37:28 blackbird systemd-udevd[836]: Process 'mtd_probe /dev/mtd6ro' failed with exit code 1.
May 10 19:37:29 blackbird kernel: A link change request failed with some changes committed already. Interface eth0 may have been left with an inconsistent configuration, please check.
May 10 19:37:31 blackbird kernel: A link change request failed with some changes committed already. Interface sit0 may have been left with an inconsistent configuration, please check.
Feb 03 22:05:40 blackbird kernel[1052]: [    3.810720] g_mass_storage 1e6a0000.usb-vhub:p1: failed to start g_mass_storage: -22
Feb 03 22:05:43 blackbird kernel[1052]: [   22.397690] A link change request failed with some changes committed already. Interface eth0 may have been left with an inconsistent configuration, please check.
Feb 03 22:05:43 blackbird kernel[1052]: [   24.051627] A link change request failed with some changes committed already. Interface sit0 may have been left with an inconsistent configuration, please check.
Feb 07 21:45:05 blackbird systemd[1]: Starting Stop the ethernet link failover...
Feb 07 21:45:07 blackbird systemd[1]: Started Stop the ethernet link failover.


PS2: This question is more precise follow-up to that question:
         https://forums.raptorcs.com/index.php?action=post;topic=49.0;last_msg=473

[SiteAdmin]

Code Select Expand

PHB Freeze/Fence detected !

This is an unhappy planar; the ASpeed VGA controller is not functioning correctly.  Try carefully removing the system from your case and powering on in a static-free environment (not on an antistatic mat as they are conductive).  If the problem persists you will need to submit an RMA request via the "My Account" link at https://www.raptorcs.com.

[FlyingBlackbird]

Thanks a lot for your quick response, I will try it out this weekend and come back with my results

[FlyingBlackbird]

I will try it out this weekend and come back with my results

OK, I have tested my Blackbird with a minimal hardware attached and the planar unmounted (detached) from the case's mouting points and I can reproduce the error
(see the attached log captured via the serial console + the picture of the planar).

Oh unlucky day, I have to open an RMA...

Code Select Expand

...
[    4.095231] IMC PMU (null) Register failed
[    7.230872] kAFS: failed to register: -97
[    7.613717] udevd[1694]: specified group 'kvm' unknown
[    7.621590] udevd[1695]: specified group 'kvm' unknown
nvram process returned non-zero exit status
dmesg: klogctl: Operation not permitted
)0 [1;24r [m   [4l [?7h [?25l [?1c [H [J [H [J Petitboot (v1.10.3-pdd2d545)
[2d ────────────────────────────────────────────────────────────────────────────── [4;3HSystem information [5;3HSystem configuration [6;3HSystem status log [7;3HLanguage [8;3HRescan devices [9;3HRetrieve config from URL [10;3HPlugins (0)
[11d * [0;10;7m  Exit to shell            [22;2H [m  ──────────────────────────────────────────────────────────────────────────────
[23d Enter=accept, e=edit, n=new, x=exit, l=language, g=log, h=help
[24d Welcome to Petitboot
Info: Waiting for device discovery[   86.421879937,3] PHB#0005[0:5]: PHB Freeze/Fence detected !
...
[   92.219717338,3] PHB#0005[0:5]: eeh_freeze_clear on fenced PHB


[MPC7500]

That's sad. Would be interesting to know what caused this error.

[FlyingBlackbird]

That's sad. Would be interesting to know what caused this error.

I will come back here with the RMA result (but I guess it is hard to diagnose)

[FlyingBlackbird]

Just a short update:

I have opened a RMA last week and asked Raptor CS about the handling details but did not yet get an answer so far.

Meanwhile I am trying to understand how customs and shipping works for RMA cases (esp. since I plan to keep some part and send back only the planar, CPU and cooler).

Long story short:
1. I have to fill a form for duty/tax reimbursement
2. I have to fill a form to announce the export
3. I have to send the parcel back to the US together with more forms filled
4. Once I receive replacement hardware I will have to pay the duty/taxes again

Since I am located in Germany here the best link I have found describing the steps the numbers 2 + 3 (sorry, it is in German):

https://www.dhl.de/content/dam/images/Paket/GK/downloads/dhl-paket-international-merkblatt-zoll-112019.pdf

For #1 I think I have to use the form #0223 ("Antrag auf Erstattung" = application for reimbursement) which can be downloaded at
www.zoll.de by searching for "0223".

Once I am through this I will publish my experiences at the Raptor wiki (and strongly recommend to order Raptor hardware from the new EU reseller www.vikings.net
to ease customs and RMA issues ;-)

[SiteAdmin]

You won't get any Emails or acknowledgement beyond the RMA number and shipping address you received when you created the RMA on the Web site.

Send the defective product back according to those instructions, and we'll send out a replacement unit.  As far as export goes, unfortunately that is not an area we can offer advice in aside from noting that many countries have a process where you can record the outgoing shipment of a defective product so as to avoid paying VAT on the (marked as) replacement  / repaired unit.

[FlyingBlackbird]

@SiteAdmin Thanks for clarifying this. At least I can contribute my experiences about customs and duty here later ;-)

[FlyingBlackbird]

Just a (short ;-) update regarding the customs handling of my RMA from Germany/Europe to the USA due to defective hardware (motherboard):

I have shipped the package back to Raptor CS today via DHL after having done the formal requirements by the aid of a customs service (agent) that costs about 40 Euros.

The best documentation I have found are here (in German only, sorry, but sometimes you can switch to English):
https://www.dhl.de/de/geschaeftskunden/express/infos-knowhow/downloads.html
https://www.dhl.de/content/dam/images/Express/pdf/dhl-express-einfach-durch-den-zoll-062019.pdf
https://www.dhl.de/content/dam/images/Express/downloads/zollinformation/zollinformationen-und-dokumente/dhl-express-broschuere-einfach-durch-den-zoll-012018.pdf

Major lessons learned:

0. The process becomes the more complicated the higher the sum of the product values is:
     < 1000 Euro: "normal"
     1000 - 3000 Euro: complicated
     > 3000 Euro: Very complicated (eg. in case of Talos II motherboards with CPUs!)

1. You have to create a proforma invoice to declare the virtual value of the products you are sending back.
    Always use the original product names from the RaptorCS invoice.
    DHL has a good and free online service to create a proforma invoide:
    https://www.dhl.de/de/geschaeftskunden/express/infos-knowhow/hilfe-zollabwicklung/formulare-zoll.html

2. You have to announce the export via an IT system that produces an MRN number (= movement reference number?).
    This was done by my customs service agent (who I paid for as stated above).
    I could also have created a login with the IT certificate of my income tax login ("Elster IT system") but since I have no certificate this would take another week time to wait for the certificate.

3. The customs tariff code according to HS (a world-wide standardized code for goods, HS = harmonized system)
    you have to specify (during import as well in export forms) is in my case (for the motherboard and CPU):
        8473 3020 00 0 (= first 6 digits are "harmonized" world-wide, the digits 6 and 7 are EU-specific, digits 9 and 10 indicate EU measure codes like anti-dumping rules, digit 11 is a national extension)
    For Imports you use 11 digits, for exports 8 digits.
    For details see: https://www.zolltarifnummern.de/info/kombinierte-nomenklatur

    8473 3020 = Electronic assemblies of automatic data-processing machines or for other machines of heading 8471, n.e.s.
    8471          = Automatic data-processing machines and units thereof; magnetic or optical readers, machines for transcribing data onto data media in coded form and machines for processing such data, n.e.s.
    See: https://www.tariffnumber.com/2020/8471

4. If you buy and pay the parcel service online (www.dhl.de) you can fill the export declaration form CN23 online and print out everything.
    This costs another 80 Euros (36 Euros shipping + costs for additional higher insurance up to 2000 Euro)

5. Don't forget to add 4 proforma invoices into the plastic bag attached to the parcel, the paper with the MRN number and the paper stuff from DHL.

6. Duty/tax reimbursement (refunding) is another process where you have to fill two forms:
    Form 0223 "Antrag auf Erstattung"
    Form 0235 "Zusatzblatt zum Antrag auf Erstattung"
    I have asked the responsible customs officer via phone if she wants to see the defect hardware before I send it back and she said "not required" - but this may differ in your case.

Summary:
Never ever again I want to experience this waste of time and money (additional 40 + 80 Euro).
The next time I would order directly in Europe at www.vikings.net (which was not yet available when I ordered last year).

BTW: I am neither affiliated nor have any relationship to DHL or www.vikings.net - I am just reporting my personal experiences

[MPC7500]

Thank you very much. I appreciate that post.

[MPC7500]

FlyingBlackbird, in which slot did you put the graphic card? 8x or 16x?

[FlyingBlackbird]

The NVIDA Geforce GTX 1050 Ti (by MSI) requires a PCIe 16x slot (by form factor) and a supplementary 6-pin PCIe power connector

[FlyingBlackbird]

Another update regarding the duty/tax reimbursement (funding) due to my RMA:

6. Duty/tax reimbursement (refunding) is another process where you have to fill two forms:
    Form 0223 "Antrag auf Erstattung"
    Form 0235 "Zusatzblatt zum Antrag auf Erstattung"
    I have asked the responsible customs officer via phone if she wants to see the defect hardware before I send it back and she said "not required" - but this may differ in your case.

TLDR: Reimbursement worked without problems - the customs officer didn't want to see the hardware before reimbursement (as promised during a  phone call)

After having shipped the defective hardware back to RaptorCS USA (incl. customs procedures as described in a prev. post)
I have applied via the filled forms to reimburse my duty/taxes partially for the hardware that I have sent back.

16 days later (today) I received a letter from the customs office with the calculation of the reimbursement: They pay back 336 of the totally paid 356 Euros
(I have kept some hardware like the screw drawer). My personal calculation of the reimbursement was 339 Euros but the 3 Euros difference are due to currency conversion differences.

Customs did not ask to see the hardware (which I can't since I have already sent it back).

[MPC7500]

Thanks!

Has the motherboard arrived at RaptorCS yet? Did you get feedback what was defective and why?
I'm curious how long it will take until Raptor can ship again.