Raptor Computing Systems Community Forums (BETA)

Software => Operating Systems and Porting => Topic started by: pocock on January 08, 2021, 04:47:38 pm

Title: Fedora 35 (was F-34): going to 4k page size? Help needed
Post by: pocock on January 08, 2021, 04:47:38 pm
We had some discussions (https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/BNZCLUM5E4JMTQHR5YLV6GK3P73WZLFA/) in fedora-devel about changing the default page size from 64k back to 4k

This could be done for Fedora 34 - I opened a change request under Fedora policy (https://fedoraproject.org/wiki/Changes/Power4kPageSizeF34)

How do people feel about this?

Does anybody want to take ownership of the Fedora change request?

Technically, only one line of code needs to change but there is some coordination to make sure everything in userland is rebuilt and to do some tests of the installer as a bare minimum.

Code: [Select]
CONFIG_PPC_64K_PAGES=n

I might be willing to take ownership of the change personally but if somebody else wants to take ownership, that will free up some of my time for other things.

It is also important to verify that other users are comfortable with this strategy: please feel free to comment through the Fedora mailing list.
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: ClassicHasClass on January 08, 2021, 05:12:28 pm
I see sharkcz is already involved in this conversation. I personally am not aware of anything that would break as a result (in fact, as you correctly point out, many things would suddenly "just work"). My only hesitancy is it would make 64K page kernels more of a pariah than they are already, sort of like another situation with big-endian, and there are performance reasons for 64K pages. Might this, perhaps, be a Workstation-only change and keep the performance benefits for Fedora Server?
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: pocock on January 08, 2021, 05:35:40 pm
I already asked this question in the same Fedora thread and it was answered there.  It looks like an extra hassle for them but maybe they can be persuaded, after all, IBM now owns Red Hat.

The page size issue impacts multiple architectures, not only ppc64el.  If proponents of bigger page sizes are willing to invest in resolving it, there are developers who will step up to help and there are tools we can use to detect signs of these problems.  It is a bit like Covid contact tracing, if the right people act with the right tools, we can solve it but we also have other battles to fight too.
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: MauryG5 on January 09, 2021, 04:00:25 am
Hi Pocock, I saw your post you tell Fedora developers about this issue that has been around for us for some time now, but do you think we could finally fix this damn crash problem on Navi 10?  You seem to have been supporting this for some time now, as you have seen I think, I was able to start Debian with Wayland's carelessness and the new firmware which obviously was only a Debian problem, but on Fedora we still have to evaluate.  I am trying to modify the Kernel but I noticed that the Fedora one, downloaded from their handouts and therefore official, does not allow to decompress and therefore I cannot try to enter to change the page size and do a test accordingly, then also disabling  Wayland ...
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: pocock on January 09, 2021, 07:18:43 am

You need to re-compile the Fedora kernel (https://fedoraproject.org/wiki/Building_a_custom_kernel)
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: MauryG5 on January 12, 2021, 12:13:26 pm
Good evening guys, I'm writing to you from my Power system with Fedora and Kernel 5.9.16! I am sending you the photos of the screen shot just taken. I don't know exactly how he did to load the Kernel because the procedure I had done last time, I was not able to complete it completely and I actually wanted to go back to making some changes but it is also true that I was able to compile before and then install the various modules. It only gave me an error in the "make install" command. But today restarting to see if I could complete the installation, I realized that the Kernel 5.916 made by me, was in plain sight in the list of Fedora Kernels. From what I saw at first boot, the Kernel behaves in a very similar way to the modified one on Debian, that made me install my friend Pocock and therefore the flaw he gives me about that I see in this. If for some reason, he puts himself on a black screen, like that he goes into energy saving or you are without a screen saver and he turns off the montitor after a set time, the system freezes and does not even let me use the monitor options in fact and I have to restart. If I use the computer regularly then it seems to work fine. This flaw also affects the moment it finishes loading the operating system, both Fedora now and Debian with 5.9.6. The moment it finishes loading and the screen would like to move to GPU mode, the screen turns black and has the same identical block and this does it with both Debian and Fedora, so I have to move the video output to where I have the GPU before he goes to black screen and at that point he works. So I think there is some flaw in these new 5.9.X Kernels, regarding this particular. Having said that, the changes I made in this Kernel were, CPU selection which by default from Power 8, I changed it to Power 9, I changed the famous page size from 64K to 4K, as I guessed l 'friend Pocock. The rest I left everything by default. Wayland is active, so the problem at this point seems to be the size of the page ... Pocock thanks for everything you do, you helped me a lot to understand many things that I hope you and others will need ... I am sending you photos of the result of my work ...

Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: tle on January 21, 2021, 06:29:13 am
 ;D I can confirm my AMD Radeon R9 now runs again on kernel 5.10.7 (4K page sizes).
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: MauryG5 on January 21, 2021, 07:27:23 am
Great job TLE, now you too have seen that the problem is only page size.  Can you tell me all the packages you installed on Fedora before doing the Kernel build and installation procedure?  Thanks
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: tle on January 21, 2021, 10:52:56 pm
 I follow the instructions in the Fedora wiki page to get the `fepkg` CLI run as expected. No magic here at all
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: MauryG5 on January 22, 2021, 01:21:39 am
I ask you because the wiki procedure has never worked for me.  I have installed the packages it asks me but giving the commands then nothing works.  I use the universal procedure that is after having compiled the Kernel, I use in sequence: make -j, make modules_install and make install.  It just doesn't work well on Fedora because some specific package seems to be missing ...
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: tle on January 22, 2021, 07:46:49 am
What error did you run into when you run `fedpkg local`. Have you made changes in `kernel-local` and `kernel.spec` file?
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: MauryG5 on January 22, 2021, 08:18:03 am
No I have not made any changes.  Are 2 items to change in the compilation do you mean?  I only changed those 2 items I was talking about in the other post, other I have not changed ... let me understand better then how to set these 2 options ... thanks
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: tle on January 23, 2021, 07:32:07 am
Regardless of those 2 aforementioned changes, `fedpkg local` should run fine. I could try to help if you could tell me where you get stuck and what errors are
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: MauryG5 on January 23, 2021, 08:12:59 am
ok thanks, I have just downloaded the new stable version of the Kernel, 5.10.9, as soon as possible I will try to do the procedure again, I have installed some other packages, when the error comes out I post the output and show you where it stops ... Time to do everything and I send it ...
Thanks for your availability in the meantime!
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: MauryG5 on January 23, 2021, 03:14:02 pm
TLE here is the error Fedora gives me when I try to do Kernel compilation. After making the changes with "make menuconfig" I execute make -j and press enter.It performs the compilation but when it seems to be finished, it gives this error, what do you think is due to? What correction should be made?
Thanks
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: xilinder on January 24, 2021, 08:27:11 am
You should find CONFIG_DEBUG_INFO_BTF in the Kernel Hacking section of the .config file. You can open the config file with a text editor and use the Search function to find it, but do not modify or save the file from the editor, use menuconfig to make the changes.
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: MauryG5 on January 24, 2021, 10:24:37 am
hi Xlinder, so I have to disable a function that is called that way in the Kernel, when I make the changes after opening with make menuconfig ... I will try to search for this entry and see if I can find it to disable it ... thanks
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: MauryG5 on January 24, 2021, 12:40:45 pm
I solved that problem by removing that option in that section you were telling me Xlinder perfect. Now I have another problem related to the final installation of the Kernel that is the last step before it becomes active and usable. When I do the command "sudo make install" he gives me this error, in practice he says he does not find that path that you see in the error and frankly I do not understand because for example on Ubuntu this problem does not give me and as soon as I run this command, he installs the Kernel in a regular way and makes it active. I wonder why on Fedora is missing this path with these directories and files... Do you have any idea what's going on? Thanks always

Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: tle on January 24, 2021, 05:16:44 pm
It's hard to tell what goes wrong because I did not do the manual way, ie `make menuconfig && make`; instead I rely on the `fedpkg` to do the job as instructed in the official Fedora custom kernel compilation guides

Using fedpkg would yield RPM files which won't cause any issue with installation
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: MauryG5 on January 25, 2021, 12:59:12 am
and then you can tell me the procedure you do with the commands you give, I can't do the driving one, if you let me understand what kind of commands you give maybe I can do it too.  In this procedure he says not to find that path, I checked in the sys directory and in fact there is just that path but obviously from where I am, which are in the Kernel folder, he cannot find that path and in fact I do not understand  how come it looks for a path that cannot find in the Kernel folder ...
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: xilinder on January 25, 2021, 08:22:00 am
Just a guess. Since these computers can not go to "Sleep" or "Hibernate" then there is no need for a "Resume". So something in the config file is set for sleep or hibernate and therefore requires resume.
I do not know what. It should not affect operation.
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: MauryG5 on January 25, 2021, 12:01:03 pm
The nice thing, you know what Xlinder is?  That in any case if you restart the system, the Kernel will find it in any case in the petit boot list and it works perfectly.  Only if you ask the system it doesn't recognize it, it says you haven't installed any 5.10.9 Kernel in this case and so if you want to remove it you have to manually do it file by file.  I would just like to understand why on other distros this procedure works, while on Fedora it encounters this problem in the last command for definitive installation of the Kernel.  Understanding where to possibly correct the defect ... At the end there is only one command to finish everything and it is a shame to stop almost at the end even if it works the same but is not recognized by the system ...
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: xilinder on January 25, 2021, 01:52:19 pm
Maury, I don't know. I gave up trying to build Fedora kernels when attempting to install the build system and it kept telling me that "pesign" is required and then says it is not available for ppc64le. Now there is a couple of weeks I'll never get back, %@#**&:(.

Anyway, if it were my system I would (as root) make a file that contains 0:0 only and save it in /sys/power
as resume.
Keep in mind....Crashing a computer does not hurt as bad as crashing a motorcycle.  :P
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: MauryG5 on January 25, 2021, 02:30:56 pm
I don't know I could try it as you say, it costs me nothing.  Create a file I think you mean with a text editor, write 0: 0 and then save it and put it in the resume directory if I understand what you are saying ...

Yes I've seen that perisign talk, but from what I understand non-X86 users don't need that file.  My problem in that procedure is that nothing works for me, I tried to install all the dependencies it asks me but still I can't do anything with that procedure, I don't know how TLE does to make it work, I didn't understand anything.  ..
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: xilinder on January 25, 2021, 02:40:36 pm
There is no resume directory. Just a bunch of files in the /sys/power directory.
Yeah, check my avatar, that is how I felt trying to get it to work.
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: MauryG5 on January 25, 2021, 03:32:36 pm
I understand, well tomorrow I'll try to do this thing you tell me and let's see what happens. Listen one more thing in the meantime, no one can answer me at the moment unfortunately, do you have any news of what happened to Chromium on Power? Fedora hasn't updated for a long time and I can't install it on Debian and Ubuntu, what the hell happened?
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: xilinder on January 26, 2021, 08:10:52 am
Google pulled the plug on Chromium.
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: pocock on January 26, 2021, 02:29:05 pm
What does that mean for the QtWebEngine support (https://forums.raptorcs.com/index.php/topic,215.msg1543.html#msg1543)?

Many other things depend on it.
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: pocock on February 16, 2021, 03:31:36 pm
The change to 4k page size has now been proposed for Fedora 35

There is a discussion thread on the Fedora devel mailing list (https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/5ODBK374RIYRSJGH5W2CFTRLWWL2AIV7/)

If you have any feedback please contribute both on the mailing list and the change wiki (https://fedoraproject.org/wiki/Changes/Power4kPageSize)

Even if you don't support the change, please feel free to contribute ideas to help make 64k more painless, I already put a few ideas in the thread
Title: Re: Fedora 35 (was F-34): going to 4k page size? Help needed
Post by: ClassicHasClass on February 16, 2021, 03:36:24 pm
I think what would be a nice interim option is a Copr for 4K kernels. I've tried to do one of my own, but it's not building with the official Fedora method yet, and I'd like to work the bugs out of that method before making a Copr for it (Dan Horak and I had some discussion about it). Is someone already doing this?

For the record, though, I wholeheartedly support this. I'm curious what packages have 64K dependencies.
Title: Re: Fedora 35 (was F-34): going to 4k page size? Help needed
Post by: pocock on February 18, 2021, 09:30:37 am
Personally, I'm curious about the possibilities with 64k but I think the platform will get more momentum with 4k over the next 2-3 years.  Any way to offer both would be good.

Notice that:

a) 4k kernel needs to be in the installer ISO image, if you build a kernel with Copr, you need to build an installer image too

b) all packages need to be recompiled using the 4k kernel, although I suspect less than 1% of packages will be sensitive to page size, doing a full rebuild is the only way to be sure

If anybody can add details to the thread in Fedora devel that is the best place to discuss it now
Title: Re: Fedora 35 (was F-34): going to 4k page size? Help needed
Post by: q66 on February 18, 2021, 07:52:11 pm
what? this makes no sense

you don't need to recompile *any* packages, since source code can't know the page size ahead of time

even if something hardcodes page size, a recompilation will not help you (not to mention that things compiled with a 64KiB page size constant will work on both 64k and 4k kernels)
Title: Re: Fedora 35 (was F-34): going to 4k page size? Help needed
Post by: pocock on February 19, 2021, 01:44:57 am

There are a small number of packages that are sensitive like this

Some comments in this thread:

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/AAXJFZNIRXK2WRHZVYD3RUGQGZEYUH74/

and here

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775788#15

Title: Re: Fedora 35 (was F-34): going to 4k page size? Help needed
Post by: q66 on February 19, 2021, 01:43:08 pm
i don't see how that's relevant

1) recompilation gives you the same binary regardless of the kernel you are running
2) therefore, if you have stuff sensitive to page size, you have to identify and patch the specific packages
3) again, things that assume 64k generally work on both 4k and 64k, so there is no problem, it's only a problem the other way around
Title: Re: Fedora 34: going to 4k page size? Help needed
Post by: q66 on February 19, 2021, 01:46:04 pm
I see sharkcz is already involved in this conversation. I personally am not aware of anything that would break as a result (in fact, as you correctly point out, many things would suddenly "just work"). My only hesitancy is it would make 64K page kernels more of a pariah than they are already, sort of like another situation with big-endian, and there are performance reasons for 64K pages. Might this, perhaps, be a Workstation-only change and keep the performance benefits for Fedora Server?

i'd rather see 64k just go away

the traditional page size for powerpc is 4KiB, large default pages just have more drawbacks than benefits and i'd be surprised if you could find me a single benchmark where larger pages give you performance benefits that you can't also get by using hugepages (ideally with madvise) while retaining all the benefits of smaller pages in the general case

also https://yarchive.net/comp/linux/page_sizes.html
Title: Re: Fedora 35 (was F-34): going to 4k page size? Help needed
Post by: pocock on February 19, 2021, 01:47:27 pm
In some cases, I think the build script (e.g. a configure script or Makefile) might call a function like "getconf(PAGESIZE)" and it saves the result

This is not good behavior but I think there are some packages like this but not very many
Title: Re: Fedora 35 (was F-34): going to 4k page size? Help needed
Post by: ClassicHasClass on February 20, 2021, 11:23:23 am
@pocock is right; I've seen such packages. It may come down to marking them outright or submitting patches for them to check at runtime.

I'm loathe to see 64K go away completely because there's a reason for its existence even though it's not relevant to many of us. More importantly, however, I guarantee IBM won't let it, because the big systems that pay the bills all use it, so whatever we can do to get it to coexist won't be fighting institutional inertia.
Title: Re: Fedora 35 (was F-34): going to 4k page size? Help needed
Post by: pocock on February 20, 2021, 02:48:53 pm
I think it is good to have both options too, I'm not anti-64k by any means

On the other hand, the reality is that most developers in the Linux world, developers of device drivers and GPU manufacturers are only testing their products on systems with a 4k page size.  If the developers at AMD and NVIDIA include some ppc64le hardware in their test routines before a release then they are unlikely to release code that completely fails, as is the case right now.
Title: Re: Fedora 35 (was F-34): going to 4k page size? Help needed
Post by: tle on March 20, 2021, 10:03:11 pm
Sticky situation. I like to enjoy a smooth experience with amdgpu but at the same time I want to keep all QEMU KVM guest hosts running. There is no guarantee that amdgpu would function correct with 4K pages btw, so whatever we are proposing are just workaround NOT the proper fix for the issue.

My only hope is Intel, if the DG2 driver could run perfectly on 64K, I would definitely ditch AMD.
Title: Re: Fedora 35 (was F-34): going to 4k page size? Help needed
Post by: sharkcz on March 31, 2021, 05:37:24 am
Due a recent issue with 5.11 kernels breaking Polaris-based cards (like my WX4100) we have now a couple of fixes to the 64k support in the amdgpu driver. We are interested in feedback from other cards as well.

Please install 5.11 kernel from my scratch build - https://koji.fedoraproject.org/koji/taskinfo?taskID=64908758 and let us know. The 3 patches are submitted for inclusion in the next Fedora kernel builds via https://gitlab.com/cki-project/kernel-ark/-/merge_requests/991, until they reach mainline. They are in the AMD's drm-next tree.
Title: Re: Fedora 35 (was F-34): going to 4k page size? Help needed
Post by: MPC7500 on April 03, 2021, 08:30:23 am
This is my dmesg (https://pastebin.com/vLvw1tVN) output. I can boot with this Kernel, but still get error messages. It seems my Navi card differs from others. Best Kernel with no issues is still 5.4.xx.
Title: Re: Fedora 35 (was F-34): going to 4k page size? Help needed
Post by: MauryG5 on April 03, 2021, 03:53:48 pm
MPC sorry but have you also tried with a Kernel compiled with 4K pages? I haven't been able to see in the output if this 5.11 you have tried is using or if you see from this output if they are enabled ...
Title: Re: Fedora 35 (was F-34): going to 4k page size? Help needed
Post by: sharkcz on April 08, 2021, 06:52:17 am
@MPC7500, thanks for testing, it's really a different backtrace.