Author Topic: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !  (Read 14109 times)

FlyingBlackbird

  • Full Member
  • ***
  • Posts: 102
  • Karma: +3/-0
    • View Profile
petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« on: February 07, 2020, 05:28:15 pm »
My Blackbird does not boot into the petitboot menu anymore (black screen) even though I can see the execution log on screen and via the ssh console with:

Code: [Select]
ssh -p 2200 root@my.ip.address
The last available console output is a failure saying:

Code: [Select]
[   90.334100762,3] PHB#0005[0:5]: eeh_freeze_clear on fenced PHB
               XE autoconfiguration failed

I have only installed a SATA-HDD drive with a bootable Ubuntu Server 19.10 (no GPU).
I have not changed/updated any firmware.
Last thing I have done was testing a PCIe GPU (NVIDIA).
After removing the GPU card from the PCIe slot I could no longer boot into petitboot...

What is going wrong?

The relevant log is (full console output is attached as file):

Code: [Select]
--== Welcome to Hostboot hostboot-3beba24/hbicore.bin ==--

  3.06480|secure|SecureROM valid - enabling functionality
  8.29259|Booting from SBE side 0 on master proc=00050000
  8.46568|ISTEP  6. 5 - host_init_fsi
  8.93808|ISTEP  6. 6 - host_set_ipl_parms
  9.49036|ISTEP  6. 7 - host_discover_targets
 10.09037|HWAS|PRESENT> DIMM[03]=8080000000000000
 10.09038|HWAS|PRESENT> Proc[05]=8000000000000000
 10.09040|HWAS|PRESENT> Core[07]=5565000000000000
 10.49937|ISTEP  6. 8 - host_update_master_tpm
 10.52369|SECURE|Security Access Bit> 0x0000000000000000
 10.52370|SECURE|Secure Mode Disable (via Jumper)> 0x8000000000000000
...
 57.11902|ISTEP 21. 2 - host_verify_hdat
 57.20205|ISTEP 21. 3 - host_start_payload
[   58.179010391,5] OPAL skiboot-c81f9d6 starting...
[   58.179013552,7] initial console log level: memory 7, driver 5
[   58.179015737,6] CPU: P9 generation processor (max 4 threads/core)
...
[   65.266762790,5] PHB#0000:00:00.0 [ROOT] 1014 04c1 R:00 C:060400 B:01..ff SLOT=CPU1 Slot2 (16x)
 Petitboot (v1.10.3-pdd2d545)
 ──────────────────────────────────────────────────────────────────────────────

  System information
  System configuration
  System status log
  Language
  Rescan devices
  Retrieve config from URL
  Plugins (0)
 *Exit to shell           










 ──────────────────────────────────────────────────────────────────────────────
 Enter=accept, e=edit, n=new, x=exit, l=language, g=log, h=help
 Info: Waiting for device discovery[   85.086133287,3] PHB#0005[0:5]: PHB Freeze/Fence detected !
[   85.086197573,3] PHB#0005[0:5]:             PCI FIR=2000000000000000
[   85.086249297,3] PHB#0005[0:5]:         PCI FIR WOF=2000000000000000
[   85.086289203,3] PHB#0005[0:5]:            NEST FIR=0000800000000000
[   85.086354836,3] PHB#0005[0:5]:        NEST FIR WOF=0000800000000000
[   85.086394899,3] PHB#0005[0:5]:            ERR RPT0=0000000000000001
[   85.086489826,3] PHB#0005[0:5]:            ERR RPT1=0000000000000000
[   85.086534460,3] PHB#0005[0:5]:             AIB ERR=0000200000000000
[   85.086941635,3] PHB#0005[0:5]:                  brdgCtl = 00000002
[   85.087002150,3] PHB#0005[0:5]:             deviceStatus = 00200020
[   85.087036852,3] PHB#0005[0:5]:               slotStatus = 00402000
[   85.087081358,3] PHB#0005[0:5]:               linkStatus = a8120008
[   85.087137004,3] PHB#0005[0:5]:             devCmdStatus = 00100107
[   85.087181127,3] PHB#0005[0:5]:             devSecStatus = 00000000
[   85.087239088,3] PHB#0005[0:5]:          rootErrorStatus = 00000000
[   85.087285589,3] PHB#0005[0:5]:          corrErrorStatus = 00000000
[   85.087325009,3] PHB#0005[0:5]:        uncorrErrorStatus = 00000000
[   85.087370016,3] PHB#0005[0:5]:                   devctl = 00000020
[   85.087419580,3] PHB#0005[0:5]:                  devStat = 00000020
[   85.087466277,3] PHB#0005[0:5]:                  tlpHdr1 = 00000000
...
[   85.088610802,3] PHB#0005[0:5]:       phbRxeArbErrorLog1 = 0000000000000000
  [Disk: sda2 / ef49aa17-bb70-4fea-a8fc-29e235f7ab9f]
    Ubuntu, with Linux 5.3.0-26-generic (recovery mode)
    Ubuntu, with Linux 5.3.0-26-generic
    Ubuntu, with Linux 5.3.0-29-generic (recovery mode)
    Ubuntu, with Linux 5.3.0-29-generic
    Ubuntu
[   85.088655011,3] PHB#0005[0:5]:     phbRxeMrgErrorStatus = 0000000000000001
...
[   85.089573601,3] PHB#0005[0:5]:                PEST[0ff] = 3740002a01000000 0000000000000000
 [enP4p1s0f2] Probing from base tftp://192.168.178.1/pxelinux.cfg/[   90.311273403,3] PHB#0005[0:5]: PHB Freeze/Fence detected !
[   90.311357669,3] PHB#0005[0:5]:             PCI FIR=2000000000000000
...
[   90.315282185,3] PHB#0005[0:5]:         phbRegbErrorLog1 = 0001020000000000
[   90.315338900,3] PHB#0005[0:5]:                PEST[000] = 8000000000000000 8000000000000000
[   90.315413179,3] PHB#0005[0:5]:                PEST[001] = 8000000000000000 8000000000000000
[   90.315491213,3] PHB#0005[0:5]:                PEST[002] = 8000000000000000 8000000000000000
...
[   90.333937493,3] PHB#0005[0:5]:                PEST[0fe] = 8000000000000000 8000000000000000
[   90.334011680,3] PHB#0005[0:5]:                PEST[0ff] = b740002a01000000 8000000000000000
[   90.334100762,3] PHB#0005[0:5]: eeh_freeze_clear on fenced PHB
               XE autoconfiguration failed

PS: I have logged in into OpenBMC via SSH and see these strange error messages that may be related:

Code: [Select]
root@blackbird:~# journalctl | grep fail
May 10 19:37:25 blackbird kernel: g_mass_storage 1e6a0000.usb-vhub:p1: failed to start g_mass_storage: -22
May 10 19:37:27 blackbird systemd-udevd[789]: Process 'mtd_probe /dev/mtd2ro' failed with exit code 1.
May 10 19:37:27 blackbird systemd-udevd[790]: Process 'mtd_probe /dev/mtd3ro' failed with exit code 1.
May 10 19:37:27 blackbird systemd-udevd[837]: Process 'mtd_probe /dev/mtd4ro' failed with exit code 1.
May 10 19:37:27 blackbird systemd-udevd[792]: Process 'mtd_probe /dev/mtd5ro' failed with exit code 1.
May 10 19:37:28 blackbird systemd-udevd[788]: Process 'mtd_probe /dev/mtd0ro' failed with exit code 1.
May 10 19:37:28 blackbird systemd-udevd[791]: Process 'mtd_probe /dev/mtd1ro' failed with exit code 1.
May 10 19:37:28 blackbird systemd-udevd[836]: Process 'mtd_probe /dev/mtd6ro' failed with exit code 1.
May 10 19:37:29 blackbird kernel: A link change request failed with some changes committed already. Interface eth0 may have been left with an inconsistent configuration, please check.
May 10 19:37:31 blackbird kernel: A link change request failed with some changes committed already. Interface sit0 may have been left with an inconsistent configuration, please check.
Feb 03 22:05:40 blackbird kernel[1052]: [    3.810720] g_mass_storage 1e6a0000.usb-vhub:p1: failed to start g_mass_storage: -22
Feb 03 22:05:43 blackbird kernel[1052]: [   22.397690] A link change request failed with some changes committed already. Interface eth0 may have been left with an inconsistent configuration, please check.
Feb 03 22:05:43 blackbird kernel[1052]: [   24.051627] A link change request failed with some changes committed already. Interface sit0 may have been left with an inconsistent configuration, please check.
Feb 07 21:45:05 blackbird systemd[1]: Starting Stop the ethernet link failover...
Feb 07 21:45:07 blackbird systemd[1]: Started Stop the ethernet link failover.

PS2: This question is more precise follow-up to that question:
         https://forums.raptorcs.com/index.php?action=post;topic=49.0;last_msg=473
« Last Edit: February 07, 2020, 05:32:53 pm by FlyingBlackbird »

SiteAdmin

  • Administrator
  • *****
  • Posts: 41
  • Karma: +15/-0
  • RCS Staff
    • View Profile
Re: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« Reply #1 on: February 07, 2020, 06:10:08 pm »
Code: [Select]
PHB Freeze/Fence detected !
This is an unhappy planar; the ASpeed VGA controller is not functioning correctly.  Try carefully removing the system from your case and powering on in a static-free environment (not on an antistatic mat as they are conductive).  If the problem persists you will need to submit an RMA request via the "My Account" link at https://www.raptorcs.com.

FlyingBlackbird

  • Full Member
  • ***
  • Posts: 102
  • Karma: +3/-0
    • View Profile
Re: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« Reply #2 on: February 07, 2020, 06:16:24 pm »
Thanks a lot for your quick response, I will try it out this weekend and come back with my results

FlyingBlackbird

  • Full Member
  • ***
  • Posts: 102
  • Karma: +3/-0
    • View Profile
Re: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« Reply #3 on: February 10, 2020, 03:03:16 pm »
I will try it out this weekend and come back with my results

OK, I have tested my Blackbird with a minimal hardware attached and the planar unmounted (detached) from the case's mouting points and I can reproduce the error
(see the attached log captured via the serial console + the picture of the planar).

Oh unlucky day, I have to open an RMA...

Code: [Select]
...
[    4.095231] IMC PMU (null) Register failed
[    7.230872] kAFS: failed to register: -97
[    7.613717] udevd[1694]: specified group 'kvm' unknown
[    7.621590] udevd[1695]: specified group 'kvm' unknown
nvram process returned non-zero exit status
dmesg: klogctl: Operation not permitted
 )0 [1;24r [m   [4l [?7h [?25l [?1c [H [J [H [J Petitboot (v1.10.3-pdd2d545)
 [2d ────────────────────────────────────────────────────────────────────────────── [4;3HSystem information [5;3HSystem configuration [6;3HSystem status log [7;3HLanguage [8;3HRescan devices [9;3HRetrieve config from URL [10;3HPlugins (0)
 [11d * [0;10;7m  Exit to shell            [22;2H [m  ──────────────────────────────────────────────────────────────────────────────
 [23d Enter=accept, e=edit, n=new, x=exit, l=language, g=log, h=help
 [24d Welcome to Petitboot
 Info: Waiting for device discovery[   86.421879937,3] PHB#0005[0:5]: PHB Freeze/Fence detected !
...
[   92.219717338,3] PHB#0005[0:5]: eeh_freeze_clear on fenced PHB

MPC7500

  • Hero Member
  • *****
  • Posts: 588
  • Karma: +41/-1
    • View Profile
    • Twitter
Re: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« Reply #4 on: February 10, 2020, 05:41:39 pm »
That's sad. Would be interesting to know what caused this error.

FlyingBlackbird

  • Full Member
  • ***
  • Posts: 102
  • Karma: +3/-0
    • View Profile
Re: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« Reply #5 on: February 11, 2020, 01:07:39 am »
That's sad. Would be interesting to know what caused this error.

I will come back here with the RMA result (but I guess it is hard to diagnose)

FlyingBlackbird

  • Full Member
  • ***
  • Posts: 102
  • Karma: +3/-0
    • View Profile
Re: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« Reply #6 on: February 18, 2020, 05:18:08 pm »
Just a short update:

I have opened a RMA last week and asked Raptor CS about the handling details but did not yet get an answer so far.

Meanwhile I am trying to understand how customs and shipping works for RMA cases (esp. since I plan to keep some part and send back only the planar, CPU and cooler).

Long story short:
1. I have to fill a form for duty/tax reimbursement
2. I have to fill a form to announce the export
3. I have to send the parcel back to the US together with more forms filled
4. Once I receive replacement hardware I will have to pay the duty/taxes again

Since I am located in Germany here the best link I have found describing the steps the numbers 2 + 3 (sorry, it is in German):

https://www.dhl.de/content/dam/images/Paket/GK/downloads/dhl-paket-international-merkblatt-zoll-112019.pdf

For #1 I think I have to use the form #0223 ("Antrag auf Erstattung" = application for reimbursement) which can be downloaded at
www.zoll.de by searching for "0223".

Once I am through this I will publish my experiences at the Raptor wiki (and strongly recommend to order Raptor hardware from the new EU reseller www.vikings.net
to ease customs and RMA issues ;-)

SiteAdmin

  • Administrator
  • *****
  • Posts: 41
  • Karma: +15/-0
  • RCS Staff
    • View Profile
Re: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« Reply #7 on: February 18, 2020, 06:30:36 pm »
You won't get any Emails or acknowledgement beyond the RMA number and shipping address you received when you created the RMA on the Web site.

Send the defective product back according to those instructions, and we'll send out a replacement unit.  As far as export goes, unfortunately that is not an area we can offer advice in aside from noting that many countries have a process where you can record the outgoing shipment of a defective product so as to avoid paying VAT on the (marked as) replacement  / repaired unit.

FlyingBlackbird

  • Full Member
  • ***
  • Posts: 102
  • Karma: +3/-0
    • View Profile
Re: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« Reply #8 on: February 19, 2020, 01:47:15 pm »
@SiteAdmin Thanks for clarifying this. At least I can contribute my experiences about customs and duty here later ;-)

FlyingBlackbird

  • Full Member
  • ***
  • Posts: 102
  • Karma: +3/-0
    • View Profile
Re: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« Reply #9 on: March 02, 2020, 06:39:56 am »
Just a (short ;-) update regarding the customs handling of my RMA from Germany/Europe to the USA due to defective hardware (motherboard):

I have shipped the package back to Raptor CS today via DHL after having done the formal requirements by the aid of a customs service (agent) that costs about 40 Euros.

The best documentation I have found are here (in German only, sorry, but sometimes you can switch to English):
https://www.dhl.de/de/geschaeftskunden/express/infos-knowhow/downloads.html
https://www.dhl.de/content/dam/images/Express/pdf/dhl-express-einfach-durch-den-zoll-062019.pdf
https://www.dhl.de/content/dam/images/Express/downloads/zollinformation/zollinformationen-und-dokumente/dhl-express-broschuere-einfach-durch-den-zoll-012018.pdf

Major lessons learned:

0. The process becomes the more complicated the higher the sum of the product values is:
     < 1000 Euro: "normal"
     1000 - 3000 Euro: complicated
     > 3000 Euro: Very complicated (eg. in case of Talos II motherboards with CPUs!)

1. You have to create a proforma invoice to declare the virtual value of the products you are sending back.
    Always use the original product names from the RaptorCS invoice.
    DHL has a good and free online service to create a proforma invoide:
    https://www.dhl.de/de/geschaeftskunden/express/infos-knowhow/hilfe-zollabwicklung/formulare-zoll.html

2. You have to announce the export via an IT system that produces an MRN number (= movement reference number?).
    This was done by my customs service agent (who I paid for as stated above).
    I could also have created a login with the IT certificate of my income tax login ("Elster IT system") but since I have no certificate this would take another week time to wait for the certificate.

3. The customs tariff code according to HS (a world-wide standardized code for goods, HS = harmonized system)
    you have to specify (during import as well in export forms) is in my case (for the motherboard and CPU):
        8473 3020 00 0 (= first 6 digits are "harmonized" world-wide, the digits 6 and 7 are EU-specific, digits 9 and 10 indicate EU measure codes like anti-dumping rules, digit 11 is a national extension)
    For Imports you use 11 digits, for exports 8 digits.
    For details see: https://www.zolltarifnummern.de/info/kombinierte-nomenklatur

    8473 3020 = Electronic assemblies of automatic data-processing machines or for other machines of heading 8471, n.e.s.
    8471          = Automatic data-processing machines and units thereof; magnetic or optical readers, machines for transcribing data onto data media in coded form and machines for processing such data, n.e.s.
    See: https://www.tariffnumber.com/2020/8471

4. If you buy and pay the parcel service online (www.dhl.de) you can fill the export declaration form CN23 online and print out everything.
    This costs another 80 Euros (36 Euros shipping + costs for additional higher insurance up to 2000 Euro)

5. Don't forget to add 4 proforma invoices into the plastic bag attached to the parcel, the paper with the MRN number and the paper stuff from DHL.

6. Duty/tax reimbursement (refunding) is another process where you have to fill two forms:
    Form 0223 "Antrag auf Erstattung"
    Form 0235 "Zusatzblatt zum Antrag auf Erstattung"
    I have asked the responsible customs officer via phone if she wants to see the defect hardware before I send it back and she said "not required" - but this may differ in your case.

Summary:
Never ever again I want to experience this waste of time and money (additional 40 + 80 Euro).
The next time I would order directly in Europe at www.vikings.net (which was not yet available when I ordered last year).


BTW: I am neither affiliated nor have any relationship to DHL or www.vikings.net - I am just reporting my personal experiences
« Last Edit: March 02, 2020, 09:45:34 am by FlyingBlackbird »

MPC7500

  • Hero Member
  • *****
  • Posts: 588
  • Karma: +41/-1
    • View Profile
    • Twitter
Re: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« Reply #10 on: March 02, 2020, 07:54:10 pm »
Thank you very much. I appreciate that post.

MPC7500

  • Hero Member
  • *****
  • Posts: 588
  • Karma: +41/-1
    • View Profile
    • Twitter
Re: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« Reply #11 on: March 08, 2020, 12:47:12 pm »
FlyingBlackbird, in which slot did you put the graphic card? 8x or 16x?

FlyingBlackbird

  • Full Member
  • ***
  • Posts: 102
  • Karma: +3/-0
    • View Profile
Re: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« Reply #12 on: March 08, 2020, 03:24:33 pm »
The NVIDA Geforce GTX 1050 Ti (by MSI) requires a PCIe 16x slot (by form factor) and a supplementary 6-pin PCIe power connector

FlyingBlackbird

  • Full Member
  • ***
  • Posts: 102
  • Karma: +3/-0
    • View Profile
Re: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« Reply #13 on: March 21, 2020, 07:14:47 am »
Another update regarding the duty/tax reimbursement (funding) due to my RMA:


6. Duty/tax reimbursement (refunding) is another process where you have to fill two forms:
    Form 0223 "Antrag auf Erstattung"
    Form 0235 "Zusatzblatt zum Antrag auf Erstattung"
    I have asked the responsible customs officer via phone if she wants to see the defect hardware before I send it back and she said "not required" - but this may differ in your case.

TLDR: Reimbursement worked without problems - the customs officer didn't want to see the hardware before reimbursement (as promised during a  phone call)

After having shipped the defective hardware back to RaptorCS USA (incl. customs procedures as described in a prev. post)
I have applied via the filled forms to reimburse my duty/taxes partially for the hardware that I have sent back.

16 days later (today) I received a letter from the customs office with the calculation of the reimbursement: They pay back 336 of the totally paid 356 Euros
(I have kept some hardware like the screw drawer). My personal calculation of the reimbursement was 339 Euros but the 3 Euros difference are due to currency conversion differences.

Customs did not ask to see the hardware (which I can't since I have already sent it back).

MPC7500

  • Hero Member
  • *****
  • Posts: 588
  • Karma: +41/-1
    • View Profile
    • Twitter
Re: petitboot error: PHB#0005[0:5]: PHB Freeze/Fence detected !
« Reply #14 on: March 22, 2020, 09:30:45 am »
Thanks!

Has the motherboard arrived at RaptorCS yet? Did you get feedback what was defective and why?
I'm curious how long it will take until Raptor can ship again.