Raptor Computing Systems Community Forums (BETA)
Software => Operating Systems and Porting => Topic started by: pocock on September 14, 2020, 05:19:55 am
-
Most distributions have chosen the 4k page size for their kernels on Intel architectures.
On powerpc64le, Debian (since 2014 (https://salsa.debian.org/kernel-team/linux/commit/aed63a56b189d771116f2d4b8fe10bbec528e6a2)) and Fedora are using 64k
This may be good for HPC environments but troublesome for workstation users
Here are some specific observations:
Nouveau driver apparaently won't work at all on anything other than 4k (https://bugs.freedesktop.org/show_bug.cgi?id=94757)
btrfs filesystems have a sectorsize that corresponds to the page size on the host where the filesystem was created (https://lore.kernel.org/linux-btrfs/01a71119-f662-e559-534b-82a6d9fc79f4@pocock.pro/). The filesystems can't be mounted on any host with a different page size.
Example problems:
- you have a 6TB drive with btrfs from an x86 workstation, you want to move it to a Blackbird running the default Debian or Fedora kernel but it won't mount.
- you install Fedora 33 with the default kernel (64k page size) and default filesystem (btrfs from Fedora 33) and later you want to recompile your kernel for 4k. Now you can't mount your root filesystem because mkfs was run for 64k
As Fedora is going to use btrfs by default now, btrfs volumes will be more common and users are more likely to encounter frustration from time to time.
The btrfs developers recently started a patch that allows the systems with 64k page size to read a btrfs volume with 4k sectorsize. It only works in one direction and it is read-only.
Does anybody feel that distributions should offer different permutations of their kernel and installer based on the 4k page size?
From the perspective of workstation users, are there any other strong reasons, other than Nouveau and btrfs, for distributions to consider this?
-
Some emulation and virtualization tools have dependencies on page size. We had some page size issues in Firefox initially, though I think we smoked most of those out.
I'd certainly prefer a 4K page size as it's more "typical" but given RHEL's prominence on HPC POWER I can understand why Fedora ppc64le defaults to 64K as well. I certainly wouldn't expect them to do two flavours, especially since they don't even support big-endian ppc64 anymore, which I think would be more significant architecturally than page size.
-
Most distributions have chosen the 4k page size for their kernels on Intel architectures.
The btrfs developers recently started a patch that allows the systems with 64k page size to read a btrfs volume with 4k sectorsize. It only works in one direction and it is read-only.
You are mentioning the kernel "memory" page size (`getconf PAGESIZE") and the `btrfs` file system page size.
Just to understand this issue right:
How is this related? Does `btrfs` by default (or always?) use the same file system page size as the kernel does for the memory page size?
-
Yes, that is correct, the sectorsize parameter for a btrfs filesystem must be the same as PAGESIZE for the kernel
Therefore, btrfs filesystems created on systems with a 4k page size can only be used on systems with a 4k page size
btrfs filesystems created on systems with a 64k page size can only be used on systems with a 64k page size
-
Some emulation and virtualization tools have dependencies on page size. We had some page size issues in Firefox initially, though I think we smoked most of those out.
Maybe that is why Thunderbird is not working at all either.
Simply running a 4k kernel doesn't always fix the issues. Sometimes it is necessary to rebuild each broken application on the host running a 4k kernel. This is because the build scripts of some applications look at the page size during compile time and assume it will be the same for run-time and they hardcode it into the binary.
I will probably try building both the Firefox and Thunderbird packages from source on a Debian buster host running with the 4k page size.
I tried Firefox for a WebRTC call yesterday and the video was missing, I'll see if that works with the 4k page size too, with and without recompiling Firefox itself.
The two biggest concerns I heard from people about this platform are not being able to use Firefox reliably and the noise issues. Developers are reluctant to embrace any platform that might sidetrack us from the things we are supposed to be working on. If the noise issues can be fixed with Vikings' water cooling solution (https://forums.raptorcs.com/index.php/topic,110.msg1408.html#msg1408) and the Firefox issues can be reined in by a 4k page size then it could make a crucial difference to the success of the platform. It is therefore quite important to look at how to engage distributions in that solution.
-
There were pictures of water cooled IBM hardware a while back. It seems like the CPU block would be the only non-standard part. Can Raptor get these from IBM?
-
(I modified the code last night, now it builds 64k and 4k kernels as two different flavours so you can install both on the same system and choose between them in the petitboot menu, the 4k kernel file has 4k in the filename and ABI string)
Here is my patch for building a Debian kernel package with the 4k page size, like other architectures (https://gitlab.com/dpocock/linux-kernel-debian/-/commits/pocock/buster-ppc64el-4k)
To use it:
mkdir -p ~/ws/kernel
cd ~/ws/kernel
wget http://deb.debian.org/debian/pool/main/l/linux/linux_4.19.132.orig.tar.xz
xzcat linux_4.19.132.orig.tar.xz | tar xf -
git clone https://gitlab.com/dpocock/linux-kernel-debian
cd linux-kernel-debian
git checkout pocock/buster-ppc64el-4k
cd ../linux-4.19.132
ln -s ../linux-kernel-debian/debian .
dpkg-buildpackage -rfakeroot -i.* --no-sign -b -j`grep -c ^processor /proc/cpuinfo`
cd ..
You should find the packages *.deb in ~/ws/kernel
You can just install it with dpkg and reboot into the kernel with 4k page size
Beware: if you have a btrfs root filesystem with sectorsize=64, the kernel with 4k page size can't mount it. If your root filesystem is ext4 it will just work.
If people are comfortable with this change then it will be a good idea to open bug reports for the kernel packaging teams in each of the relevant distributions.
-
Today I made some tests comparing Firefox WebRTC on 64k and 4k page size
With a Logitech webcam, I found both 64k and 4k worked
With Elgato Camlink 4k (https://www.elgato.com/en/gaming/cam-link-4k), v4l2-loopback and my gstreamer script (https://gitlab.com/freertc/gstreamer-chroma-key), I found it only worked on 4k. With 64k, Firefox displayed the local monitor / self-view from the Camlink but the remote host wasn't receiving the picture. Firefox was receiving the picture from the remote host, so this was a one-way video problem.
The gstreamer script is a very simple chroma-key example, it is not OBS but then it is less than 10 lines of code.
-
Maybe that is why Thunderbird is not working at all either.
These were pretty low-level changes, so I don't know if they're what's blocking Thunderbird. But I run and test Firefox all the time. Perhaps, if there's interest, I can provide my binaries (I already provide my .mozconfigs).
-
I've found the relevant config in the Fedora kernel package (https://src.fedoraproject.org/rpms/kernel/blob/master/f/kernel-ppc64le-rhel.config) and posted about the same issue in the Fedora PPC mailing list (https://lists.fedoraproject.org/archives/list/ppc@lists.fedoraproject.org/thread/OWXAJDWZ3ARBJXUYV2DLMMBIRKK2AAX2/)
-
Hi guys, any idea of how 4K vs 64K page size can affect phoronix gzip compression test results?
My Dell 5590 with Intel i5 8th gen with 16GB running Fedora 33 finished the test in 46 sec while my Blackbird 4 cores and 32GB also running Fedora 33 did it in 67 secs. I did not expect the i5 to outperform the 4 cores Power9.
-
I don't know if this is the case, but I know for sure that a lot depends on the optimization which on Power is usually much lower unfortunately and therefore often these comparisons cannot be made ...
-
For POWER9, are you reporting the speed on a 4k kernel of a 64k kernel?
The i5 will have 4k kernel
How many memory channels on each system?
Do you watch the load on the CPU cores, for example, using top (press 1) or the GNOME System Monitor? Does the test only use 1 core or does it use all available cores?
Notice that the POWER9 architecture is faster for tasks that can be split between all the cores. Some applications only run on a single core.
-
Hi pocock, please see answers below.
For POWER9 I'm running on Fedora binary I've downloaded and therefore on a 64K kernel (default for ppc64el), while on the Dell notebook on a 4K kernel (default for x86).
How many memory channels on each system? On the Dell notebook two x 8GB and on the Blackbird 2 x 16GB.
Do you watch the load on the CPU cores, for example, using top (press 1) or the GNOME System Monitor? Yes, when I run the test I see on the monitor load being distributed across the various CPU threads, on both the notebook and Blackbird.
So, the fact that the test results show that the Dell i5 outperforms the Blackbird maybe because of the different page sizes?
See attached the complete test results.
-
These results you could compare:
https://openbenchmarking.org/result/2102047-HA-2101296AS71
Only difference: Kernel pagesize and installed RAM and of course the harddisk.
-
These results you could compare:
https://openbenchmarking.org/result/2102047-HA-2101296AS71
Only difference: Kernel pagesize and installed RAM.
Interesting: this other test done with a 4K page size and 16 cores achieved a much worse performance.. see the attached image comparing a performance per core.
So, maybe the issue here is that Intel i5 outperforms Power9 per core, regardless of page size and 64K page gives actually more performance per core for this specific test.
What comes to my mind is how bad is Power9 as a desktop..
-
The graph is wrong ;D
But the Phoronix test suite can't do anything about that. You have to divide the lower result by 4 (228)
-
The graph is wrong ;D
But the Phoronix test suite can't do anything about that. You have to divide the lower result by 4 (228)
So, it's 250 secs with 4 cores vs 228 secs with 16 cores. It seems that the additional cores did not help improve performance and actually Intel cores and threads handle this specific load/test better than Power9.
The question remains: how good is Blackbird as a desktop performance wise vs Intel? I posted another topic about Thunderbird running faster on this same Dell 5580 (i5 8th gen) than on Blackbird, that means, real world desktop user experience.
Maybe Power9 is a better fit for HPC or TB HANA in memory DBs but not as a Desktop alternative, not taking into account the difficulty to find software for the architecture.
-
Both CPUs are 4-core.
At the end it's all about optimisations.
Improving performance of Phoronix benchmarks on POWER9 (https://sthbrx.github.io/index3.html)
Easier Power ISA vectorizing for fun and profit with GCC x86 intrinsics (https://www.talospace.com/2019/07/easier-power-vectorizing-for-fun-and.html)
-
Maybe Power9 is a better fit for HPC or TB HANA in memory DBs but not as a Desktop alternative, not taking into account the difficulty to find software for the architecture.
That's a bit of a stretch, don't you think? Those Intel cores (and similar AMD ones) come with a mandatory signed binary blob on a secondary CPU that has full access to all your data. In the worst case, you could simply be computing faster for an adversary by selecting the Intel solution.
Given the lack of those blobs is one of POWER's main reasons to be used on desktop, a more appropriate comparison for the specific purpose of determining POWER's usability as a desktop computer would be against other silicon that is also blob-free -- older Intel parts (Core Duo timeframe), or blob-free ARM devices (e.g. Rockchip).
If you want the fastest possible desktop with no regard for security, owner control, or privacy, then the simple fact is the latest AMD devices running Windows are your best bet. For many of us, and especially here at Raptor, that represents such a large risk that we'd rather go back to pen and paper...
...much like the Justice Department just did here in the US, actually, after the SolarWinds proprietary / signed malware problems. ;D
-
cchincz, as MPC7500 correctly points out, they are both 4-core CPUs -- the "16 CPUs" are an artifact of SMT -- and optimization is still a problem for some programs. This is improving slowly. For example, we now have better SIMD in Firefox and in other media packages.
-
I agree 100 per 100 with what MPC and Classic say and I had already written it to you before I cchincz. It is useless to make comparisons with X86, they cannot be done because we do not have the software optimizations that they have! This thing matters more than you might think, it matters a lot, it is the main reason for which we do not have the same or superior performance. Unfortunately at the moment we cannot see the goodness and performance superiority of Power because the software does not allow it. But in any case just as our SiteAdmin says, at least they are safe and fully controllable compared to X86 and I can personally say that at the level of operating system and normal use in Desktop, I personally go great in any case. If one day we are lucky enough to have optimized programs and native programs for our Power architecture, then for sure things will change a lot and the scales will move a lot ...
-
Maybe Power9 is a better fit for HPC or TB HANA in memory DBs but not as a Desktop alternative, not taking into account the difficulty to find software for the architecture.
That's a bit of a stretch, don't you think? Those Intel cores (and similar AMD ones) come with a mandatory signed binary blob on a secondary CPU that has full access to all your data. In the worst case, you could simply be computing faster for an adversary by selecting the Intel solution.
Given the lack of those blobs is one of POWER's main reasons to be used on desktop, a more appropriate comparison for the specific purpose of determining POWER's usability as a desktop computer would be against other silicon that is also blob-free -- older Intel parts (Core Duo timeframe), or blob-free ARM devices (e.g. Rockchip).
If you want the fastest possible desktop with no regard for security, owner control, or privacy, then the simple fact is the latest AMD devices running Windows are your best bet. For many of us, and especially here at Raptor, that represents such a large risk that we'd rather go back to pen and paper...
...much like the Justice Department just did here in the US, actually, after the SolarWinds proprietary / signed malware problems. ;D
Hi, I was not discussing privacy concerns. Actually I'm quite happy with my Blackbird and the community here, which is very supportive of new and less tech savvy users like myself. Also, it performs quite well for my needs from a desktop. I agree with MauryG5 and ClassicHasClass that it's pointless to compare performance.
I appreciate very much the work you guys do at Raptor developing open hardware and wish you continued sucess.
Sincerely
-
Both CPUs are 4-core.
At the end it's all about optimisations.
Improving performance of Phoronix benchmarks on POWER9 (https://sthbrx.github.io/index3.html)
Easier Power ISA vectorizing for fun and profit with GCC x86 intrinsics (https://www.talospace.com/2019/07/easier-power-vectorizing-for-fun-and.html)
This then answers my question, 4K and 64K page sizes resulted in roughly the same performance.
The discussion about Fedora desktop for PPC64el being 4K is related to compatibility aspects and not performance.
Thanks for the clarification.
-
above all it affects the GPUs that are used. The 4K dimensions have become indispensable if you want to use a new generation GPU like Le Navi10 or 14, or the Nano ...! I believe that even the Big Navi will now want the dimensions in 4K, lately it is clear that AMD in the latest drivers it has provided, has made a switch to 4K and therefore we need to adapt the Kernel and currently none of the developers of the various distros are going to use the 4K option apparently ...
-
Hi, I was not discussing privacy concerns. Actually I'm quite happy with my Blackbird and the community here, which is very supportive of new and less tech savvy users like myself. Also, it performs quite well for my needs from a desktop. I agree with MauryG5 and ClassicHasClass that it's pointless to compare performance.
I appreciate very much the work you guys do at Raptor developing open hardware and wish you continued sucess.
Sincerely
No problem. We're glad to hear you're happy with your Blackbird, and of course are always interested in where we do fall a bit short in real-world performance (and how to improve said shortcomings with software optimizations)!
-
Guys I was trying to compile and install the 5.11 kernel on Debian using the universal procedure but it doesn't let me install the openssl-devel library. In fact, as soon as I try to compile it gives me the classic error that it does not find it. Do you have any addictions in turn that I don't know? Thank you.
-
The libs and other ssl needed on your computer.
libssl-dev
libssl1.1
openssl
perl-openssl-defaults
python-openssl
python-service-identity
ssl-cert
Also, using make menuconfig, you must change the CONFIG_SYSTEM_TRUSTED_KEYS= from "debian/certs/debian-uefi-certs.pem" to simply ""
# Certificates for signature checking
#
CONFIG_SYSTEM_TRUSTED_KEYRING=y
CONFIG_SYSTEM_TRUSTED_KEYS="debian/certs/debian-uefi-certs.pem"
# CONFIG_SYSTEM_EXTRA_CERTIFICATE is not set
CONFIG_SECONDARY_TRUSTED_KEYRING=y
CONFIG_SYSTEM_BLACKLIST_KEYRING=y
CONFIG_SYSTEM_BLACKLIST_HASH_LIST=""
CONFIG_BINARY_PRINTF=y
That is in the crypto section down near the bottom.
-
Hi Xlinder, always thanks for your precious help, I installed what I was missing, only 2 at the end of your list, unfortunately I can't understand the options to be fixed in the section you tell me in the api encryption section. You tell me exactly what I have to do in that section but for how it looks in the configuration menu, I am sending you a photo to understand. I can't understand it from how you wrote it in your post sorry ... Currently mine is like this and it gives me an error in the line "debian / certain / eufi-certs.pem" written just like this including the quotes ... how should I change this? How should the other options be put? Thanks for your patience...
-
Go down to Additional X.509 keys for default system keyring.
Hit Enter.
Delete all the text, including the quotes, and Enter again.
Then EXIT, EXIT, EXIT and save the new configuration.
Please let us know how it works for you.
-
Now compile regularly, but yesterday evening I had to interrupt the procedure because I was compiling from Daniel Pocock's 5.9.6 kernel and being that the compilation in the case of Debian switches the GPU off and on several times, I could no longer continue since kernel 5.9. X on Power suffers from this bug, if you turn off the GPU or when switching to the video output of the GPU, the monitor goes black and it is no longer possible to do anything. I found the same problem on Fedora so I realized that the problem lies in the Linux Power Kernel in version 5.9.X. In fact, every time I load Debian with 5.9 I always have to switch the video output first otherwise it crashes irreversibly ... Today I try to do everything from 4.19 and I will let you know thanks Xlinder. In which line of the kernel configuration I understood that I had to put only double quotes for that I asked you to write me exactly what I had to put. Tonight I try and tell you. Thanks for everything always.
-
Actually the double quotes would only be needed if you modified the line in a Text Editor. However, doing it with Menuconfig is a better way.
You can try the Text editor later after you have a working kernel and wish to play around. ;)
-
Xlinder once again a special thanks to you for your good work with Debian, here is the complicated and correctly installed Kernel 5.11. Also on Debian the universal procedure works perfectly once you have everything installed and modify those 4 necessary parameters. So I still don't understand why Fedora doesn't let me do that and gives me that damn error due to the dracut file. I also tried to remove the 95 resume folder which is the one offending but unfortunately it does not install it anyway. Yes, the mistake does not take it out anymore but does not complete the installation ... I would like to understand what to do sincerely ...