KVM cross-endian guest on POWER8 / IBM S822L

[kth5]

Hi everyone,

I recently acquired a IBM S822L dual socket system and now am starting to get to the point of using KVM/libvirt to bring guests to it. As I was used to on my Blackbird I could run big-endian guests without much performance penalty at all running little-endian on the host.

However, on the POWER8 it seems that no matter what I do, I get up to 50% steal time inside a VM when the bare-metal and the VM are otherwise completely idle. When I put any load on the VM it gets worse and jumps up to 80% or more making any vCPU config moot. This only happens when running a big-endian guest kernel and userland, which I need for my build-bots.

A bit of background of what I'm dealing with here:
* dual POWER8E 10 core SMT8 / 256GB RAM running Opal on latest available firmware
* no entitlements beyond basic & micro LPAR and the usual AIX
* bare-metal runs ArchPOWER ppc64le with SMT switched off as required
* guests I'm testing with run ArchPOWER ppc64 or ppc64+32bit userland
* kernel version is 6.11.9 on bare-metal & guests
* Qemu 9.0.3/9.1.1 w/ libvirt 10.8.0

I've tried the following all exhibiting the same steal time issue:
* single vCPU w/o threads
* single vCPU w/o threads pinned to a physical core & NUMA zone
* dual vCPU w/ 8 threads (SMT)
* dual vCPU w/ 8 threads pinned to a physical core & NUMA zone (SMT)

What I did not try is running actual PowerKVM. It seems too outdated for my liking and I'd rather have a recent package base as close to upstream as possible.

Running ppc64le in the guest does not show the problem at all and performs splendidly.

Bit at a loss here and I would rather I not yet call it a lost cause. 

[ClassicHasClass]

Just curious: not possible to run big on both host and guest? I'm not sure why that's happening to you, but I'm curious to see if it's the mismatch that's the problem, or running big that's the problem.

[kth5]

I had big-endian at first (in PowerVM) but I ultimately decided to run little-endian upon a re-install using Opal as that's also the primary build of ArchPOWER and I figured I have basically zero entitlements on the box anyway (original owner unknown, ebay steal at just 250EUR shipping included so I don't complain).

I'll have a look at installing ppc64 on a spare drive once I either get to the datacenter or get remote management set up properly. ASMI is unfortunately not providing remote serial nor virtual disks.

I want VMs for all three architectures in the end though, ppc64le, ppc64 as well as ppc 32bit which works awesome on Blackbird.