Updating Talos II firmware to IBM PNOR V2.18?

[JeremyRand]

According to this IBM documentation (thanks awilfox for the pointer), PNOR v2.18 contains the following change:

HIPER/Pervasive:   A problem was fixed for a processor core checkstop with SRC BC70E540 logged with Signature Description " ex(n0p1c4) (NCUFIR[11]) NCU no response to snooped TLBIE".  This problem is intermittent and random but occurs with relatively high frequency for certain workloads.  The trigger for the failure is one core of a fused core pair going into a stopped state while the other core of the pair continues running.

Is it possible to get the Talos II firmware bumped to upstream IBM v2.18? I believe I may be running into this bug.

[tle]

AFAIK no, Talos firmware is not synced with upstream however it does not stop people from using a modified version of upstream kernel. Perhaps you should have a read at https://www.flamingspork.com/blog/2020/05/25/op-build-v2-5-firmware-for-the-raptor-blackbird/ and see if you could adapt anything from his work

[tle]

Having said that, I am unsure if using custom firmware would void warranty or not. That I have to leave the answer to official response from RaptorCS

[ejfluhr]

I believe that all OpenPOWER/OPAL-based systems use SMT4 cores, not SMT8 (i.e. "fused") cores.    So that may mean you aren't seeing the same problem.

Do you get the same fault callout?

[JeremyRand]

I believe that all OpenPOWER/OPAL-based systems use SMT4 cores, not SMT8 (i.e. "fused") cores.    So that may mean you aren't seeing the same problem.

Do you get the same fault callout?

I get the same error "NCU no response to snooped TLBIE". I asked Timothy on IRC and he said that the 18/22-core Raptor CPU's use fused cores. Which tracks with the fact that this started happening to me when I upgraded from 4-core CPU's to 22-core CPU's.

Looks like Raptor only has minimal changes to the IBM repo that contains the fix, and there are no rebase conflicts. So, I'll try building a modified firmware that incorporates IBM's fix and see if it helps here.

[ejfluhr]

The 18c & 22c parts are "paired" meaning 2 SMT4 cores share the same L2 & L3, unlike the 4c and 8c which are "unpaired" meaning each core gets the full L2 and L3 to itself.    This is not the same as "fused" (i.e. SMT8 cores) but it is quite likely that the fix will also work for "paired" cores as presumably the issue is sharing cacheable/non-cacheable pathways.   Good luck!

[JeremyRand]

The 18c & 22c parts are "paired" meaning 2 SMT4 cores share the same L2 & L3, unlike the 4c and 8c which are "unpaired" meaning each core gets the full L2 and L3 to itself.    This is not the same as "fused" (i.e. SMT8 cores) but it is quite likely that the fix will also work for "paired" cores as presumably the issue is sharing cacheable/non-cacheable pathways.   Good luck!

I think you're right about the terminology, but yeah, I doubt the IBM docs draw a distinction since those docs are exclusively written for SMT8 users.

I've successfully built a PNOR with IBM's patch; it installed without major issues (I accidentally hosed things the first time I installed it due to the BMC running out of RAM -- protip, don't put multiple PNOR images in OpenBMC /tmp/ -- but power-cycling the BMC fixed that). So far it seems stable, I'll be running it for the next month or so to see if any checkstops happen.

[tle]

The 18c & 22c parts are "paired" meaning 2 SMT4 cores share the same L2 & L3, unlike the 4c and 8c which are "unpaired" meaning each core gets the full L2 and L3 to itself.    This is not the same as "fused" (i.e. SMT8 cores) but it is quite likely that the fix will also work for "paired" cores as presumably the issue is sharing cacheable/non-cacheable pathways.   Good luck!

I think you're right about the terminology, but yeah, I doubt the IBM docs draw a distinction since those docs are exclusively written for SMT8 users.

I've successfully built a PNOR with IBM's patch; it installed without major issues (I accidentally hosed things the first time I installed it due to the BMC running out of RAM -- protip, don't put multiple PNOR images in OpenBMC /tmp/ -- but power-cycling the BMC fixed that). So far it seems stable, I'll be running it for the next month or so to see if any checkstops happen.

Would you be able to provide more details on which patch? Many thanks

[JeremyRand]

The 18c & 22c parts are "paired" meaning 2 SMT4 cores share the same L2 & L3, unlike the 4c and 8c which are "unpaired" meaning each core gets the full L2 and L3 to itself.    This is not the same as "fused" (i.e. SMT8 cores) but it is quite likely that the fix will also work for "paired" cores as presumably the issue is sharing cacheable/non-cacheable pathways.   Good luck!

I think you're right about the terminology, but yeah, I doubt the IBM docs draw a distinction since those docs are exclusively written for SMT8 users.

I've successfully built a PNOR with IBM's patch; it installed without major issues (I accidentally hosed things the first time I installed it due to the BMC running out of RAM -- protip, don't put multiple PNOR images in OpenBMC /tmp/ -- but power-cycling the BMC fixed that). So far it seems stable, I'll be running it for the next month or so to see if any checkstops happen.

Would you be able to provide more details on which patch? Many thanks

This is the hcode branch I used: https://github.com/JeremyRand/hcode/tree/talos-2019-07-25-master-rebased

As you can see, it's simply a copy of Raptor's hcode, rebased against current upstream IBM hcode (there were no rebase conflicts). The specific bugfix commit is this one: https://github.com/JeremyRand/hcode/commit/ca06a0c996e3b48c02cfb3912dddf7ca23ec4202

I've been running it on my DD2.3 2x18-core Talos II for 2 months, and my DD2.2 2x22-core Talos II for 1 month, and I have had no checkstops on either since applying the patch, nor any new issues. At this point I can recommend that Raptor integrate the bugfix into a new PNOR release. Since the rebase had no conflicts, it should be trivially easy for Raptor to do this.

[tle]

The 18c & 22c parts are "paired" meaning 2 SMT4 cores share the same L2 & L3, unlike the 4c and 8c which are "unpaired" meaning each core gets the full L2 and L3 to itself.    This is not the same as "fused" (i.e. SMT8 cores) but it is quite likely that the fix will also work for "paired" cores as presumably the issue is sharing cacheable/non-cacheable pathways.   Good luck!

I think you're right about the terminology, but yeah, I doubt the IBM docs draw a distinction since those docs are exclusively written for SMT8 users.

I've successfully built a PNOR with IBM's patch; it installed without major issues (I accidentally hosed things the first time I installed it due to the BMC running out of RAM -- protip, don't put multiple PNOR images in OpenBMC /tmp/ -- but power-cycling the BMC fixed that). So far it seems stable, I'll be running it for the next month or so to see if any checkstops happen.

Would you be able to provide more details on which patch? Many thanks

This is the hcode branch I used: https://github.com/JeremyRand/hcode/tree/talos-2019-07-25-master-rebased

As you can see, it's simply a copy of Raptor's hcode, rebased against current upstream IBM hcode (there were no rebase conflicts). The specific bugfix commit is this one: https://github.com/JeremyRand/hcode/commit/ca06a0c996e3b48c02cfb3912dddf7ca23ec4202

I've been running it on my DD2.3 2x18-core Talos II for 2 months, and my DD2.2 2x22-core Talos II for 1 month, and I have had no checkstops on either since applying the patch, nor any new issues. At this point I can recommend that Raptor integrate the bugfix into a new PNOR release. Since the rebase had no conflicts, it should be trivially easy for Raptor to do this.

Thanks for the information