Show Posts
1
General Hardware Discussion / KVM cross-endian guest on POWER8 / IBM S822L
« on: November 19, 2024, 02:08:49 am »
Hi everyone,
I recently acquired a IBM S822L dual socket system and now am starting to get to the point of using KVM/libvirt to bring guests to it. As I was used to on my Blackbird I could run big-endian guests without much performance penalty at all running little-endian on the host.
However, on the POWER8 it seems that no matter what I do, I get up to 50% steal time inside a VM when the bare-metal and the VM are otherwise completely idle. When I put any load on the VM it gets worse and jumps up to 80% or more making any vCPU config moot. This only happens when running a big-endian guest kernel and userland, which I need for my build-bots.
A bit of background of what I'm dealing with here:
* dual POWER8E 10 core SMT8 / 256GB RAM running Opal on latest available firmware
* no entitlements beyond basic & micro LPAR and the usual AIX
* bare-metal runs ArchPOWER ppc64le with SMT switched off as required
* guests I'm testing with run ArchPOWER ppc64 or ppc64+32bit userland
* kernel version is 6.11.9 on bare-metal & guests
* Qemu 9.0.3/9.1.1 w/ libvirt 10.8.0
I've tried the following all exhibiting the same steal time issue:
* single vCPU w/o threads
* single vCPU w/o threads pinned to a physical core & NUMA zone
* dual vCPU w/ 8 threads (SMT)
* dual vCPU w/ 8 threads pinned to a physical core & NUMA zone (SMT)
What I did not try is running actual PowerKVM. It seems too outdated for my liking and I'd rather have a recent package base as close to upstream as possible.
Running ppc64le in the guest does not show the problem at all and performs splendidly.
Bit at a loss here and I would rather I not yet call it a lost cause.
I recently acquired a IBM S822L dual socket system and now am starting to get to the point of using KVM/libvirt to bring guests to it. As I was used to on my Blackbird I could run big-endian guests without much performance penalty at all running little-endian on the host.
However, on the POWER8 it seems that no matter what I do, I get up to 50% steal time inside a VM when the bare-metal and the VM are otherwise completely idle. When I put any load on the VM it gets worse and jumps up to 80% or more making any vCPU config moot. This only happens when running a big-endian guest kernel and userland, which I need for my build-bots.
A bit of background of what I'm dealing with here:
* dual POWER8E 10 core SMT8 / 256GB RAM running Opal on latest available firmware
* no entitlements beyond basic & micro LPAR and the usual AIX
* bare-metal runs ArchPOWER ppc64le with SMT switched off as required
* guests I'm testing with run ArchPOWER ppc64 or ppc64+32bit userland
* kernel version is 6.11.9 on bare-metal & guests
* Qemu 9.0.3/9.1.1 w/ libvirt 10.8.0
I've tried the following all exhibiting the same steal time issue:
* single vCPU w/o threads
* single vCPU w/o threads pinned to a physical core & NUMA zone
* dual vCPU w/ 8 threads (SMT)
* dual vCPU w/ 8 threads pinned to a physical core & NUMA zone (SMT)
What I did not try is running actual PowerKVM. It seems too outdated for my liking and I'd rather have a recent package base as close to upstream as possible.
Running ppc64le in the guest does not show the problem at all and performs splendidly.
Bit at a loss here and I would rather I not yet call it a lost cause.
