Even as a desktop user, there is always a display server in your system
It is really useful to find out, for example, if it is X or Wayland:
You can run this command in Fedora and each VM:
$ echo $XDG_SESSION_TYPE
You can get more information in each system by looking at the output of
$ xdpyinfo
In the Debian system, you can look at this file and search for errors:
/var/log/Xorg.0.log
Any line with WW is a warning and EE is an error
Here is an example error from that file:
[ 98.019] (EE) Failed to load module "glx" (loader failed, 0)
The more details you share from that file, the easier it is to help you.
I suspect the problem you have is not a problem with the ppc64el architecture, it could be something related to X or Wayland and you might want to ask in the Debian IRC channels to see if anybody can help troubleshoot.