KVM cross-endian guest on POWER8 / IBM S822L

[kth5]

Hi everyone,

I recently acquired a IBM S822L dual socket system and now am starting to get to the point of using KVM/libvirt to bring guests to it. As I was used to on my Blackbird I could run big-endian guests without much performance penalty at all running little-endian on the host.

However, on the POWER8 it seems that no matter what I do, I get up to 50% steal time inside a VM when the bare-metal and the VM are otherwise completely idle. When I put any load on the VM it gets worse and jumps up to 80% or more making any vCPU config moot. This only happens when running a big-endian guest kernel and userland, which I need for my build-bots.

A bit of background of what I'm dealing with here:
* dual POWER8E 10 core SMT8 / 256GB RAM running Opal on latest available firmware
* no entitlements beyond basic & micro LPAR and the usual AIX
* bare-metal runs ArchPOWER ppc64le with SMT switched off as required
* guests I'm testing with run ArchPOWER ppc64 or ppc64+32bit userland
* kernel version is 6.11.9 on bare-metal & guests
* Qemu 9.0.3/9.1.1 w/ libvirt 10.8.0

I've tried the following all exhibiting the same steal time issue:
* single vCPU w/o threads
* single vCPU w/o threads pinned to a physical core & NUMA zone
* dual vCPU w/ 8 threads (SMT)
* dual vCPU w/ 8 threads pinned to a physical core & NUMA zone (SMT)

What I did not try is running actual PowerKVM. It seems too outdated for my liking and I'd rather have a recent package base as close to upstream as possible.

Running ppc64le in the guest does not show the problem at all and performs splendidly.

Bit at a loss here and I would rather I not yet call it a lost cause. 

[ClassicHasClass]

Just curious: not possible to run big on both host and guest? I'm not sure why that's happening to you, but I'm curious to see if it's the mismatch that's the problem, or running big that's the problem.

[kth5]

I had big-endian at first (in PowerVM) but I ultimately decided to run little-endian upon a re-install using Opal as that's also the primary build of ArchPOWER and I figured I have basically zero entitlements on the box anyway (original owner unknown, ebay steal at just 250EUR shipping included so I don't complain).

I'll have a look at installing ppc64 on a spare drive once I either get to the datacenter or get remote management set up properly. ASMI is unfortunately not providing remote serial nor virtual disks.

I want VMs for all three architectures in the end though, ppc64le, ppc64 as well as ppc 32bit which works awesome on Blackbird.

[sharkcz]

We have pretty similar P8 machine here (with FW860.B1 (SV860_243)) and do not observe such behaviour. The host is Fedora 40 for ppc64le, the ppc64 big endian VM is Fedora 28 (as this is the last supporting ppc64), little endian VMs are F-40.

[sharkcz]

btw our VM is defined with

Code: [Select]

  <os>
    <type arch='ppc64' machine='pseries-6.2'>hvm</type>
  </os>
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>POWER8</model>
  </cpu>


[kth5]

Thanks for the feedback so there must be something wrong with my distro, which is helpful to know!

Anyway, I am now having issues with Qemu refusing to run due to RMA vs VRMA:

Unable to create 512MiB RMA (VRMA only allows 128MiB

I have 512MB <memory> defined but the relevant sections of the XML (I think are):

  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='ppc64' machine='pseries-9.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <hpt>
      <maxpagesize unit='KiB'>4</maxpagesize>
    </hpt>
    <ccf-assist state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>POWER8</model>
  </cpu>[/quote}

[sharkcz]

My guess would be there is something wrong with the VM config, like 4K vs 64k page size (we are 64k only), minimum CPU level in the kernel, things like that.

[kth5]

Yes, I switched to a 64K page size kernel again and was able to start the VMs, yet no change. Steal time still there