PCI Power State Management

[Corvidae]

Has anyone managed to get their PCIE devices to enter a low-power state?

I've been trying to figure out if adding suspend support to blackbird / talos systems is possible, so I've been doing some digging into getting that working. The first step would be to get s2idle working, but in addition to not being able to wake the system once it enters s2idle, I also see my kernel logs get filled with lines like:

Code: [Select]

amdgpu 0000:01:00.0: refused to change power state from D0 to D3hot
That happens for a lot of other PCI devices as well, not just the GPU. I'm wondering if that has something to do with why no wakeup sources are working.

I looked into kernel sources a bit and saw that the power state is set through a PCI config space write. It seems like those writes just get ignored, though. I even tried using the setpci command to set it manually:

Code: [Select]

sudo setpci -v -s 0004:01:00.2 CAP_PM+4.b=bI also just tried 3 instead of 0x08 | 0x03 = 0xb but that still does nothing, not even anything in dmesg.

The last thing I was able to dig up was this line from the OPAL documentation:

phb4 doesn’t support ASB config space writes

I tried looking up what those acronyms mean - phb4 is the PCIE controller for POWER9, but I have no idea what an ASB is. Does anyone know if that might be the issue here?

[MPC7500]

pocock looked also in this topic:
https://forums.raptorcs.com/index.php/topic,372.0.html

The ASB is the Advanced System Bus: Page 8
https://wiki.raptorcs.com/w/images/0/04/POWER9_Registers_vol1_version1.1_pub.pdf

Here is the PHB4 specification:
https://wiki.raptorcs.com/w/images/a/ad/P9_PHB_version1.0_27July2018_pub.pdf

[Corvidae]

Thanks for pointing me towards those documents. Unfortunately, it seems the behavior I observed is intended: on page 189 of the PHB4 spec, the PM config field is set to be read-only and only ever return D0.

I also checked the PHB4 code in Skiboot and it seems they explicitly added a check to block writes just to that config field. I'm hoping it's just a case of them not having a reason to implement it, and not it actually just being impossible...