kernel config: page size 4k vs 64k

[pocock]

Most distributions have chosen the 4k page size for their kernels on Intel architectures.

On powerpc64le, Debian (since 2014) and Fedora are using 64k

This may be good for HPC environments but troublesome for workstation users

Here are some specific observations:

Nouveau driver apparaently won't work at all on anything other than 4k

btrfs filesystems have a sectorsize that corresponds to the page size on the host where the filesystem was created.  The filesystems can't be mounted on any host with a different page size.

Example problems:

• you have a 6TB drive with btrfs from an x86 workstation, you want to move it to a Blackbird running the default Debian or Fedora kernel but it won't mount.
• you install Fedora 33 with the default kernel (64k page size) and default filesystem (btrfs from Fedora 33) and later you want to recompile your kernel for 4k.  Now you can't mount your root filesystem because mkfs was run for 64k

As Fedora is going to use btrfs by default now, btrfs volumes will be more common and users are more likely to encounter frustration from time to time.

The btrfs developers recently started a patch that allows the systems with 64k page size to read a btrfs volume with 4k sectorsize.  It only works in one direction and it is read-only.

Does anybody feel that distributions should offer different permutations of their kernel and installer based on the 4k page size?

From the perspective of workstation users, are there any other strong reasons, other than Nouveau and btrfs, for distributions to consider this?

[ClassicHasClass]

Some emulation and virtualization tools have dependencies on page size. We had some page size issues in Firefox initially, though I think we smoked most of those out.

I'd certainly prefer a 4K page size as it's more "typical" but given RHEL's prominence on HPC POWER I can understand why Fedora ppc64le defaults to 64K as well. I certainly wouldn't expect them to do two flavours, especially since they don't even support big-endian ppc64 anymore, which I think would be more significant architecturally than page size.

[FlyingBlackbird]

Most distributions have chosen the 4k page size for their kernels on Intel architectures.

The btrfs developers recently started a patch that allows the systems with 64k page size to read a btrfs volume with 4k sectorsize.  It only works in one direction and it is read-only.

You are mentioning the kernel "memory" page size (`getconf PAGESIZE") and the `btrfs` file system page size.

Just to understand this issue right:

How is this related? Does `btrfs` by default (or always?) use the same file system page size as the kernel does for the memory page size?

[pocock]

Yes, that is correct, the sectorsize parameter for a btrfs filesystem must be the same as PAGESIZE for the kernel

Therefore, btrfs filesystems created on systems with a 4k page size can only be used on systems with a 4k page size

btrfs filesystems created on systems with a 64k page size can only be used on systems with a 64k page size

[pocock]

Some emulation and virtualization tools have dependencies on page size. We had some page size issues in Firefox initially, though I think we smoked most of those out.

Maybe that is why Thunderbird is not working at all either.

Simply running a 4k kernel doesn't always fix the issues.  Sometimes it is necessary to rebuild each broken application on the host running a 4k kernel.  This is because the build scripts of some applications look at the page size during compile time and assume it will be the same for run-time and they hardcode it into the binary.

I will probably try building both the Firefox and Thunderbird packages from source on a Debian buster host running with the 4k page size.

I tried Firefox for a WebRTC call yesterday and the video was missing, I'll see if that works with the 4k page size too, with and without recompiling Firefox itself.

The two biggest concerns I heard from people about this platform are not being able to use Firefox reliably and the noise issues.  Developers are reluctant to embrace any platform that might sidetrack us from the things we are supposed to be working on.  If the noise issues can be fixed with Vikings' water cooling solution and the Firefox issues can be reined in by a 4k page size then it could make a crucial difference to the success of the platform.  It is therefore quite important to look at how to engage distributions in that solution.

[surf]

There were pictures of water cooled IBM hardware a while back.  It seems like the CPU block would be the only non-standard part.  Can Raptor get these from IBM?

[pocock]

(I modified the code last night, now it builds 64k and 4k kernels as two different flavours so you can install both on the same system and choose between them in the petitboot menu, the 4k kernel file has 4k in the filename and ABI string)

Here is my patch for building a Debian kernel package with the 4k page size, like other architectures

To use it:

Code: [Select]

mkdir -p ~/ws/kernel
cd ~/ws/kernel
wget http://deb.debian.org/debian/pool/main/l/linux/linux_4.19.132.orig.tar.xz
xzcat linux_4.19.132.orig.tar.xz | tar xf -
git clone https://gitlab.com/dpocock/linux-kernel-debian
cd linux-kernel-debian
git checkout pocock/buster-ppc64el-4k
cd ../linux-4.19.132
ln -s ../linux-kernel-debian/debian .
dpkg-buildpackage -rfakeroot -i.* --no-sign -b -j`grep -c ^processor /proc/cpuinfo`
cd ..

You should find the packages *.deb in ~/ws/kernel

You can just install it with dpkg and reboot into the kernel with 4k page size

Beware: if you have a btrfs root filesystem with sectorsize=64, the kernel with 4k page size can't mount it.  If your root filesystem is ext4 it will just work.

If people are comfortable with this change then it will be a good idea to open bug reports for the kernel packaging teams in each of the relevant distributions.

[pocock]

Today I made some tests comparing Firefox WebRTC on 64k and 4k page size

With a Logitech webcam, I found both 64k and 4k worked

With Elgato Camlink 4k, v4l2-loopback and my gstreamer script, I found it only worked on 4k.  With 64k, Firefox displayed the local monitor / self-view from the Camlink but the remote host wasn't receiving the picture.  Firefox was receiving the picture from the remote host, so this was a one-way video problem.

The gstreamer script is a very simple chroma-key example, it is not OBS but then it is less than 10 lines of code.

[ClassicHasClass]

Maybe that is why Thunderbird is not working at all either.

These were pretty low-level changes, so I don't know if they're what's blocking Thunderbird. But I run and test Firefox all the time. Perhaps, if there's interest, I can provide my binaries (I already provide my .mozconfigs).

[pocock]

I've found the relevant config in the Fedora kernel package and posted about the same issue in the Fedora PPC mailing list

[cchinicz]

Hi guys, any idea of how 4K vs 64K page size can affect phoronix gzip compression test results?

My Dell 5590 with Intel i5 8th gen with 16GB running Fedora 33 finished the test in 46 sec while my Blackbird 4 cores and 32GB also running Fedora 33 did it in 67 secs. I did not expect the i5 to outperform the 4 cores Power9.

[MauryG5]

I don't know if this is the case, but I know for sure that a lot depends on the optimization which on Power is usually much lower unfortunately and therefore often these comparisons cannot be made ...

[pocock]

For POWER9, are you reporting the speed on a 4k kernel of a 64k kernel?

The i5 will have 4k kernel

How many memory channels on each system?

Do you watch the load on the CPU cores, for example, using top (press 1) or the GNOME System Monitor?  Does the test only use 1 core or does it use all available cores?

Notice that the POWER9 architecture is faster for tasks that can be split between all the cores.  Some applications only run on a single core.

[cchinicz]

Hi pocock, please see answers below.

For POWER9 I'm running on Fedora binary I've downloaded and therefore on a 64K kernel (default for ppc64el), while on the Dell notebook on a 4K kernel (default for x86).

How many memory channels on each system? On the Dell notebook two x 8GB and on the Blackbird 2 x 16GB.

Do you watch the load on the CPU cores, for example, using top (press 1) or the GNOME System Monitor? Yes, when I run the test I see on the monitor load being distributed across the various CPU threads, on both the notebook and Blackbird.

So, the fact that the test results show that the Dell i5 outperforms the Blackbird maybe because of the different page sizes?

See attached the complete test results.

[MPC7500]

These results you could compare:
https://openbenchmarking.org/result/2102047-HA-2101296AS71

Only difference: Kernel pagesize and installed RAM and of course the harddisk.