OpenBMC reporting critical error Proc.FSI.Error.MasterDetectionFailure

[r34per]

I checked the OpenBMC web interface to find it reported in being critical health with 200 high priority errors logged yesterday over the course of about 3 hours, and as far as I can tell it's the same error for all of them.

The error is

Code Select Expand

org.open_power.Proc.FSI.Error.MasterDetectionFailure
When I expand the entry this is what it reads-

Code Select Expand

CALLOUT_DEVICE_PATH=/sys/devices/platform/gpio-fsi/fsi0/slave@00:00/raw CALLOUT_ERRNO=0 _PID=8606

Like I said they all appear to be the same error with that same message, although the PID= number is different. My blackbird appeared to have powered off at some point as well, though I don't know when(can I check that somewhere in openbmc?). I stepped away from my pc for the evening before the first was logged and forgot to shut it down for the night, and when I went to use it this morning it was not running.

I'm running void linux as the os, and I couldn't find any logs that would shed any light on it. It seems by default void does have a syslog daemon and I never bothered installing one, oops.

Is this a cause for concern, and should I put a ticket in with RCS about it? It happened once before a few weeks or so ago but I chalked it up to a fluke, I cleared the logs and it seemed to be fine.

[Hasturtium]

You may be interested to know that Raptor weighed in on their Twitter after it was pointed out to them, with a first suggestion to try reseating the CPU.

The link:
https://nitter.poast.org/RaptorCompSys/status/1652858741635665920#m

[r34per]

I'll give that a try, thanks for the heads up!

[cy384]

I got this error in my Blackbird's BMC log when there was a brief (like one second) power outage (brownout maybe?) last week.

[MPC7500]

Then I would try this:
https://wiki.raptorcs.com/wiki/Troubleshooting/Guard_Partition

Otherwise, if that doesn't help, I would re-flash the BMC and OpenPOWER firmware:
https://wiki.raptorcs.com/wiki/Updating_Firmware

[r34per]

I got this error in my Blackbird's BMC log when there was a brief (like one second) power outage (brownout maybe?) last week.

That could be what's happening to mine too when I think about. Brief brown-outs aren't uncommon when it's windy or stormy, which it has been when it happened. I should probably invest in a UPS for it and see if it still gives me any trouble. I'll try MPC7500's suggestions too, thanks for the help!

[Borley]

These events have also been showing in the 'Server Health' section of my BMC web panel. Just four marked from 2020, and two more recent from December 2023.

Code Select Expand


org.Open_power.Proc.FSI.Error.MasterDetectionFailure
CALLOUT_DEVICE_PATH=/sys/devices/platform/gpio-fsi/fsi0/slave@00:00/raw CALLOUT_ERRNO=0_PID=4612

My system has been fine, other than occasionally booting without properly setting RTC time to the host (same issue that ClassicHasClass has been seeing?)

If this is power outage related, that would make sense since, through 2020, I unknowingly had my Blackbird on a bad uninterruptible power supply. Then later had it in a location prone to outages before replacing the UPS.

If these logs are nothing critical, they should be safe to clear from the log?

[Hasturtium]

If you believe you've determined the cause, I don't see a reason to keep the error messages around for further investigation. Good work troubleshooting it.

[ClassicHasClass]

My system has been fine, other than occasionally booting without properly setting RTC time to the host (same issue that ClassicHasClass has been seeing?)

Yeah, I'm trying to do more research on that. I'm assuming the BMC settings just got whacked given that the password was also scrambled but it still seems an odd failure mode.