Raptor Computing Systems Community Forums (BETA)
Software => Operating Systems and Porting => Topic started by: pocock on January 08, 2021, 04:47:38 pm
-
We had some discussions (https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/BNZCLUM5E4JMTQHR5YLV6GK3P73WZLFA/) in fedora-devel about changing the default page size from 64k back to 4k
This could be done for Fedora 34 - I opened a change request under Fedora policy (https://fedoraproject.org/wiki/Changes/Power4kPageSizeF34)
How do people feel about this?
Does anybody want to take ownership of the Fedora change request?
Technically, only one line of code needs to change but there is some coordination to make sure everything in userland is rebuilt and to do some tests of the installer as a bare minimum.
CONFIG_PPC_64K_PAGES=n
I might be willing to take ownership of the change personally but if somebody else wants to take ownership, that will free up some of my time for other things.
It is also important to verify that other users are comfortable with this strategy: please feel free to comment through the Fedora mailing list.
-
I see sharkcz is already involved in this conversation. I personally am not aware of anything that would break as a result (in fact, as you correctly point out, many things would suddenly "just work"). My only hesitancy is it would make 64K page kernels more of a pariah than they are already, sort of like another situation with big-endian, and there are performance reasons for 64K pages. Might this, perhaps, be a Workstation-only change and keep the performance benefits for Fedora Server?
-
I already asked this question in the same Fedora thread and it was answered there. It looks like an extra hassle for them but maybe they can be persuaded, after all, IBM now owns Red Hat.
The page size issue impacts multiple architectures, not only ppc64el. If proponents of bigger page sizes are willing to invest in resolving it, there are developers who will step up to help and there are tools we can use to detect signs of these problems. It is a bit like Covid contact tracing, if the right people act with the right tools, we can solve it but we also have other battles to fight too.
-
Hi Pocock, I saw your post you tell Fedora developers about this issue that has been around for us for some time now, but do you think we could finally fix this damn crash problem on Navi 10? You seem to have been supporting this for some time now, as you have seen I think, I was able to start Debian with Wayland's carelessness and the new firmware which obviously was only a Debian problem, but on Fedora we still have to evaluate. I am trying to modify the Kernel but I noticed that the Fedora one, downloaded from their handouts and therefore official, does not allow to decompress and therefore I cannot try to enter to change the page size and do a test accordingly, then also disabling Wayland ...
-
You need to re-compile the Fedora kernel (https://fedoraproject.org/wiki/Building_a_custom_kernel)
-
Good evening guys, I'm writing to you from my Power system with Fedora and Kernel 5.9.16! I am sending you the photos of the screen shot just taken. I don't know exactly how he did to load the Kernel because the procedure I had done last time, I was not able to complete it completely and I actually wanted to go back to making some changes but it is also true that I was able to compile before and then install the various modules. It only gave me an error in the "make install" command. But today restarting to see if I could complete the installation, I realized that the Kernel 5.916 made by me, was in plain sight in the list of Fedora Kernels. From what I saw at first boot, the Kernel behaves in a very similar way to the modified one on Debian, that made me install my friend Pocock and therefore the flaw he gives me about that I see in this. If for some reason, he puts himself on a black screen, like that he goes into energy saving or you are without a screen saver and he turns off the montitor after a set time, the system freezes and does not even let me use the monitor options in fact and I have to restart. If I use the computer regularly then it seems to work fine. This flaw also affects the moment it finishes loading the operating system, both Fedora now and Debian with 5.9.6. The moment it finishes loading and the screen would like to move to GPU mode, the screen turns black and has the same identical block and this does it with both Debian and Fedora, so I have to move the video output to where I have the GPU before he goes to black screen and at that point he works. So I think there is some flaw in these new 5.9.X Kernels, regarding this particular. Having said that, the changes I made in this Kernel were, CPU selection which by default from Power 8, I changed it to Power 9, I changed the famous page size from 64K to 4K, as I guessed l 'friend Pocock. The rest I left everything by default. Wayland is active, so the problem at this point seems to be the size of the page ... Pocock thanks for everything you do, you helped me a lot to understand many things that I hope you and others will need ... I am sending you photos of the result of my work ...
-
;D I can confirm my AMD Radeon R9 now runs again on kernel 5.10.7 (4K page sizes).
-
Great job TLE, now you too have seen that the problem is only page size. Can you tell me all the packages you installed on Fedora before doing the Kernel build and installation procedure? Thanks
-
I follow the instructions in the Fedora wiki page to get the `fepkg` CLI run as expected. No magic here at all
-
I ask you because the wiki procedure has never worked for me. I have installed the packages it asks me but giving the commands then nothing works. I use the universal procedure that is after having compiled the Kernel, I use in sequence: make -j, make modules_install and make install. It just doesn't work well on Fedora because some specific package seems to be missing ...
-
What error did you run into when you run `fedpkg local`. Have you made changes in `kernel-local` and `kernel.spec` file?
-
No I have not made any changes. Are 2 items to change in the compilation do you mean? I only changed those 2 items I was talking about in the other post, other I have not changed ... let me understand better then how to set these 2 options ... thanks
-
Regardless of those 2 aforementioned changes, `fedpkg local` should run fine. I could try to help if you could tell me where you get stuck and what errors are
-
ok thanks, I have just downloaded the new stable version of the Kernel, 5.10.9, as soon as possible I will try to do the procedure again, I have installed some other packages, when the error comes out I post the output and show you where it stops ... Time to do everything and I send it ...
Thanks for your availability in the meantime!
-
TLE here is the error Fedora gives me when I try to do Kernel compilation. After making the changes with "make menuconfig" I execute make -j and press enter.It performs the compilation but when it seems to be finished, it gives this error, what do you think is due to? What correction should be made?
Thanks
-
You should find CONFIG_DEBUG_INFO_BTF in the Kernel Hacking section of the .config file. You can open the config file with a text editor and use the Search function to find it, but do not modify or save the file from the editor, use menuconfig to make the changes.
-
hi Xlinder, so I have to disable a function that is called that way in the Kernel, when I make the changes after opening with make menuconfig ... I will try to search for this entry and see if I can find it to disable it ... thanks
-
I solved that problem by removing that option in that section you were telling me Xlinder perfect. Now I have another problem related to the final installation of the Kernel that is the last step before it becomes active and usable. When I do the command "sudo make install" he gives me this error, in practice he says he does not find that path that you see in the error and frankly I do not understand because for example on Ubuntu this problem does not give me and as soon as I run this command, he installs the Kernel in a regular way and makes it active. I wonder why on Fedora is missing this path with these directories and files... Do you have any idea what's going on? Thanks always
-
It's hard to tell what goes wrong because I did not do the manual way, ie `make menuconfig && make`; instead I rely on the `fedpkg` to do the job as instructed in the official Fedora custom kernel compilation guides
Using fedpkg would yield RPM files which won't cause any issue with installation
-
and then you can tell me the procedure you do with the commands you give, I can't do the driving one, if you let me understand what kind of commands you give maybe I can do it too. In this procedure he says not to find that path, I checked in the sys directory and in fact there is just that path but obviously from where I am, which are in the Kernel folder, he cannot find that path and in fact I do not understand how come it looks for a path that cannot find in the Kernel folder ...
-
Just a guess. Since these computers can not go to "Sleep" or "Hibernate" then there is no need for a "Resume". So something in the config file is set for sleep or hibernate and therefore requires resume.
I do not know what. It should not affect operation.
-
The nice thing, you know what Xlinder is? That in any case if you restart the system, the Kernel will find it in any case in the petit boot list and it works perfectly. Only if you ask the system it doesn't recognize it, it says you haven't installed any 5.10.9 Kernel in this case and so if you want to remove it you have to manually do it file by file. I would just like to understand why on other distros this procedure works, while on Fedora it encounters this problem in the last command for definitive installation of the Kernel. Understanding where to possibly correct the defect ... At the end there is only one command to finish everything and it is a shame to stop almost at the end even if it works the same but is not recognized by the system ...
-
Maury, I don't know. I gave up trying to build Fedora kernels when attempting to install the build system and it kept telling me that "pesign" is required and then says it is not available for ppc64le. Now there is a couple of weeks I'll never get back, %@#**&:(.
Anyway, if it were my system I would (as root) make a file that contains 0:0 only and save it in /sys/power
as resume.
Keep in mind....Crashing a computer does not hurt as bad as crashing a motorcycle. :P
-
I don't know I could try it as you say, it costs me nothing. Create a file I think you mean with a text editor, write 0: 0 and then save it and put it in the resume directory if I understand what you are saying ...
Yes I've seen that perisign talk, but from what I understand non-X86 users don't need that file. My problem in that procedure is that nothing works for me, I tried to install all the dependencies it asks me but still I can't do anything with that procedure, I don't know how TLE does to make it work, I didn't understand anything. ..
-
There is no resume directory. Just a bunch of files in the /sys/power directory.
Yeah, check my avatar, that is how I felt trying to get it to work.
-
I understand, well tomorrow I'll try to do this thing you tell me and let's see what happens. Listen one more thing in the meantime, no one can answer me at the moment unfortunately, do you have any news of what happened to Chromium on Power? Fedora hasn't updated for a long time and I can't install it on Debian and Ubuntu, what the hell happened?
-
Google pulled the plug on Chromium.
-
What does that mean for the QtWebEngine support (https://forums.raptorcs.com/index.php/topic,215.msg1543.html#msg1543)?
Many other things depend on it.
-
The change to 4k page size has now been proposed for Fedora 35
There is a discussion thread on the Fedora devel mailing list (https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/5ODBK374RIYRSJGH5W2CFTRLWWL2AIV7/)
If you have any feedback please contribute both on the mailing list and the change wiki (https://fedoraproject.org/wiki/Changes/Power4kPageSize)
Even if you don't support the change, please feel free to contribute ideas to help make 64k more painless, I already put a few ideas in the thread
-
I think what would be a nice interim option is a Copr for 4K kernels. I've tried to do one of my own, but it's not building with the official Fedora method yet, and I'd like to work the bugs out of that method before making a Copr for it (Dan Horak and I had some discussion about it). Is someone already doing this?
For the record, though, I wholeheartedly support this. I'm curious what packages have 64K dependencies.
-
Personally, I'm curious about the possibilities with 64k but I think the platform will get more momentum with 4k over the next 2-3 years. Any way to offer both would be good.
Notice that:
a) 4k kernel needs to be in the installer ISO image, if you build a kernel with Copr, you need to build an installer image too
b) all packages need to be recompiled using the 4k kernel, although I suspect less than 1% of packages will be sensitive to page size, doing a full rebuild is the only way to be sure
If anybody can add details to the thread in Fedora devel that is the best place to discuss it now
-
what? this makes no sense
you don't need to recompile *any* packages, since source code can't know the page size ahead of time
even if something hardcodes page size, a recompilation will not help you (not to mention that things compiled with a 64KiB page size constant will work on both 64k and 4k kernels)
-
There are a small number of packages that are sensitive like this
Some comments in this thread:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/AAXJFZNIRXK2WRHZVYD3RUGQGZEYUH74/
and here
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=775788#15
-
i don't see how that's relevant
1) recompilation gives you the same binary regardless of the kernel you are running
2) therefore, if you have stuff sensitive to page size, you have to identify and patch the specific packages
3) again, things that assume 64k generally work on both 4k and 64k, so there is no problem, it's only a problem the other way around
-
I see sharkcz is already involved in this conversation. I personally am not aware of anything that would break as a result (in fact, as you correctly point out, many things would suddenly "just work"). My only hesitancy is it would make 64K page kernels more of a pariah than they are already, sort of like another situation with big-endian, and there are performance reasons for 64K pages. Might this, perhaps, be a Workstation-only change and keep the performance benefits for Fedora Server?
i'd rather see 64k just go away
the traditional page size for powerpc is 4KiB, large default pages just have more drawbacks than benefits and i'd be surprised if you could find me a single benchmark where larger pages give you performance benefits that you can't also get by using hugepages (ideally with madvise) while retaining all the benefits of smaller pages in the general case
also https://yarchive.net/comp/linux/page_sizes.html
-
In some cases, I think the build script (e.g. a configure script or Makefile) might call a function like "getconf(PAGESIZE)" and it saves the result
This is not good behavior but I think there are some packages like this but not very many
-
@pocock is right; I've seen such packages. It may come down to marking them outright or submitting patches for them to check at runtime.
I'm loathe to see 64K go away completely because there's a reason for its existence even though it's not relevant to many of us. More importantly, however, I guarantee IBM won't let it, because the big systems that pay the bills all use it, so whatever we can do to get it to coexist won't be fighting institutional inertia.
-
I think it is good to have both options too, I'm not anti-64k by any means
On the other hand, the reality is that most developers in the Linux world, developers of device drivers and GPU manufacturers are only testing their products on systems with a 4k page size. If the developers at AMD and NVIDIA include some ppc64le hardware in their test routines before a release then they are unlikely to release code that completely fails, as is the case right now.
-
Sticky situation. I like to enjoy a smooth experience with amdgpu but at the same time I want to keep all QEMU KVM guest hosts running. There is no guarantee that amdgpu would function correct with 4K pages btw, so whatever we are proposing are just workaround NOT the proper fix for the issue.
My only hope is Intel, if the DG2 driver could run perfectly on 64K, I would definitely ditch AMD.
-
Due a recent issue with 5.11 kernels breaking Polaris-based cards (like my WX4100) we have now a couple of fixes to the 64k support in the amdgpu driver. We are interested in feedback from other cards as well.
Please install 5.11 kernel from my scratch build - https://koji.fedoraproject.org/koji/taskinfo?taskID=64908758 and let us know. The 3 patches are submitted for inclusion in the next Fedora kernel builds via https://gitlab.com/cki-project/kernel-ark/-/merge_requests/991, until they reach mainline. They are in the AMD's drm-next tree.
-
This is my dmesg (https://pastebin.com/vLvw1tVN) output. I can boot with this Kernel, but still get error messages. It seems my Navi card differs from others. Best Kernel with no issues is still 5.4.xx.
-
MPC sorry but have you also tried with a Kernel compiled with 4K pages? I haven't been able to see in the output if this 5.11 you have tried is using or if you see from this output if they are enabled ...
-
@MPC7500, thanks for testing, it's really a different backtrace.
-
Any work on this topic in Fedora has now been undermined by the politics that is gripping large free software organizations. This initiative is completely frozen and I can't say when it would resume.
During the Fedora 35 release cycle, many Red Hat employees joined the attacks on Dr Richard Stallman. I wrote a blog about the human rights issues involved in these online mobs and subsequently received a number of malicious communications that undermine my voluntary work as a Fedora developer.
I would not expect Raptor to be keen on taking sides in these issues, after all, they made a big effort to attain the FSF RYF certification but the OpenPOWER hardware is coming from Red Hat's parent, IBM. On that basis, I won't say a lot more about this but I feel that people who invested in this platform have a right to know that politics is getting ahead of important development issues.
-
Most unfortunate. :(
-
Any work on this topic in Fedora has now been undermined by the politics that is gripping large free software organizations. This initiative is completely frozen and I can't say when it would resume.
During the Fedora 35 release cycle, many Red Hat employees joined the attacks on Dr Richard Stallman. I wrote a blog about the human rights issues involved in these online mobs and subsequently received a number of malicious communications that undermine my voluntary work as a Fedora developer.
I would not expect Raptor to be keen on taking sides in these issues, after all, they made a big effort to attain the FSF RYF certification but the OpenPOWER hardware is coming from Red Hat's parent, IBM. On that basis, I won't say a lot more about this but I feel that people who invested in this platform have a right to know that politics is getting ahead of important development issues.
Wondering if there is any similar initiatives for clang?
-
Initiatives like how?
-
There is an update on the Fedora situation today
A legal panel has declared that rogue elements of Fedora committed harassment and abuse, this is what is effectively slowing down my work to help Fedora users on POWER
Here is my blog about it (https://danielpocock.com/harassment-decision-victory-for-volunteers-and-fedora-foundations/)
Here is the verdict from the panel (https://www.adrforum.com/DomainDecisions/1980642.htm)
It is interesting to note that when people immersed in the open source space make accusations of harassment, they are usually very biased and swayed by conflicts of interest and personal relationships.
The verdict on Fedora was made by an outside observer so it is more credible than a code of conduct.
-
I wouldn't care. Because the maintainers of Fedora (sharkcz is on this forum) are aware of the problem. If they don't feel the need to offer 4K kernels, so be it...
That's the blessing of open source, you can choose whatever you like.
And it seems the problem is fixed?
Commit 5234de6c797565815ece9321b1dfe2e6732b5090
https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.10.101
-
I wouldn't care. Because the maintainers of Fedora (sharkcz is on this forum) are aware of the problem. If they don't feel the need to offer 4K kernels, so be it...
That's the blessing of open source, you can choose whatever you like.
Given the POWER eco-system is smaller compared to the overall eco-system, the choices are not so wide and the POWER development teams in each distribution are relatively limited in what they can do
The commit below doesn't credit people who spent time identifying the root cause of the problem, that is not a blessing at all. It verges on plagiarism.
And it seems the problem is fixed?
Commit 5234de6c797565815ece9321b1dfe2e6732b5090
https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.10.101