Raptor Computing Systems Community Forums (BETA)
Software => User Zone => Topic started by: Borley on January 04, 2020, 05:23:03 pm
-
This post is for documentation purposes and is not meant as a criticism of POWER or Raptor, rather, it is a place for me to think openly regarding my POWER9/Blackbird exerience so far. What is listed here may or may not necessarily be a POWER or Raptor board issue as I have yet to determine the root cause of several items. I welcome any suggestions or even fixes that you may have found in your own situations. The new platform has already forced me to learn quite a bit and hopefully this can be of use to somebody other than myself.
When rebooting sometimes restarts without onboard audio available
complete reboot removing power from the board is required to rectify this situation
System stutter with any I/O activity on the main drive (could be the PCIe-NVMe adapter?)
major I/O on the main drive (copying many large files to) makes the system nearly unusable[/s]
Retrospective: May have been combination of PCIe 3.0 storage and Gnome Shell/Nautilus thumbnailing large assets. No longer an issue.
Harddrive SMART reports an unsafe shutdown for every shutdown
This has not technically been an issue yet, I am just gambling that it won't blow up in my face - does the platform need to issue some kind of shutdown signal to the drive?[/s]
2024: It did not blow up in my face. Since replacing with a PCIe 4.0 storage device, is no longer an issue.
Unavailable dependencies needed for packages that would otherwise be installable:
mozc - that's okay anthy works just as well Just use anthy
anki - one single python package is not built for ppc Ended up writing my own flashcard program. "Resolved".
openshot - one single python package is not built for ppc. none of the other video editors will actually load any video content (missing codec? I'm too dumb to figure this out) openshot-qt
VLC playback
VLC sometimes remains "running" after closing out - cannot even kill it from terminal (kill PID) it *only* closes through system monitor when this occurs
VLC unable to stream online video content, "cannot decode h264" (even though local h264 files run fine (another codec issue?)
Disregard. I've been using mpv instead since at least 2020.
Random lockups - unable to determine the cause yet, running clamav-daemon in on-access scanning mode does seem to trigger it when nautilus begins thumbnailing new media in a directory being opened for the first time - disabled for now
also occured once when cycling through 漢字 within anthy[/s]
has not occurred only twice since thread creation - marking as non-issue
Misc missing software;
obs-studio (now available in Debian repos, as of Bullseye)
openshot-qt (now available in Debian repos, as of Bookworm)
openmw (now available in Debian repos, as of Bullseye)
0ad
dolphin-emu
mupen64plus Gaming == unimportant, trivial issue
ffmpeg botches gif creation when using palattes - standard (ugly) gif creation with ffmpeg is fine however
same exact config + script works okay on x86
ffmpeg (as of 5.1.4-0+deb12u1) botches h264 mp4 video encoding
Manually pass different codec with something like -vcodec mpeg2video or -vcodec vp9
Startup always selects another enP1s0... ethernet port for which there is no profile, must manually select the true port (might just be a GNOME bug?)
No longer an issue.
artha must be launched twice before it actually opens
Trivial issue - very unlikely to be a fault of POWER/Blackbird Options > "Show Artha's window on launch"
Large images in GIMP are much slower to work with despite having a dedicated add in GPU (a weaker GPU on x86 was fine in comparison)
Just need to be more patient with the program/use smaller images[/s]
Performance has long since been adequate. I just retested and it only slows when moving large layers in 4K+ image sizes. Non-issue, "resolved".
-
When rebooting sometimes restarts without onboard audio available
I haven't observed the lack of onboard audio on reboots, but I usually don't reboot my Blackbird much when it's on. On the other hand, I still occasionally have situations where it won't put video up on HDMI, or at least my home projector won't see it.
VLC sometimes remains "running" after closing out
This is not specific to the Blackbird; I've seen this on the Talos. I haven't figured out why this occurs. A killall -9 will nuke it.
Alternatively, go to the settings and uncheck Allow only one instance. This doesn't solve the problem but it makes it much less annoying, and you can clean up the corpses later.
Large images in GIMP are much slower to work with
I haven't been happy with GIMP on any platform. I use Krita or azpainter.
Some of your other issues sound more software-motivated, but I think that PCIe-NVMe bridge might be a little rickety.
-
The problem of the video that is sometimes missing on HDMI I mentioned this to Raptor myself a while ago, I am aware of the problem and are working to resolve it. When the BMC firmware update is released, the problem will be resolved, we just have to wait for it to come out ... I also often have this problem because in the evening I disconnect the computer from the power supply to avoid power surges or similar things.
-
System stutter with any I/O activity on the main drive (could be the PCIe-NVMe adapter?)
major I/O on the main drive (copying many large files to) makes the system nearly unusable
Sounds like an un-TRIMed drive, run fstrim on its mountpoint once in a while...
Unavailable dependencies needed for packages that would otherwise be installable:
mozc - that's okay anthy works just as well
anki - one single python package is not built for ppc
openshot - one single python package is not built for ppc. none of the other video editors will actually load any video content (missing codec? I'm too dumb to figure this out)
Have your distro build them, we're building all of those in Void and they seem just fine (on little endian anyway, on big endian mozc does not build)
VLC playback
VLC sometimes remains "running" after closing out - cannot even kill it from terminal (kill PID) it *only* closes through system monitor when this occurs
VLC unable to stream online video content, "cannot decode h264" (even though local h264 files run fine (another codec issue?)
Use mpv? :P
Misc missing software;
obs-studio
0ad
openmorrowind
dolphin-emu
mupen64plus
All of these except mupen64plus build/work on ppc64le nowadays, dolphin-emu does not have a JIT though so it's slow. 0ad needs some patches (available on Raptor wiki), the others don't really need anything.
(https://i.imgur.com/w1pV8gS.png)
(https://i.imgur.com/5mKjgmb.png)
-
-snip-
For the longest time, I was under the impression that my distro enables trim by default. Checking systemctl reveals that it is not even running. Looks like I have a project to do!
I also tried out MPV and I like how minimalist it is but didn't explore it enough to find streaming or subtitling features. Maybe it is time for a revisit.
-
Are you on Fedora? fstrim.timer will become default in F32 (hopefully).
-
-snip-
For the longest time, I was under the impression that my distro enables trim by default. Checking systemctl reveals that it is not even running. Looks like I have a project to do!
There is a bug in SSD trimming (discard) with NVMe SSDs in the kernel until version 5.4:
[FIXED in stable kernels 4.19 and 5.4] https://bugzilla.kernel.org/show_bug.cgi?id=202665 IOMMU related errors when performing discard on some NVMe devices (mainly NVMe SSDs). Current workaround is booting with the kernel parameter "'iommu=soft'', see the https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=530436c45ef2e446c12538a400e465929a0b3ade patch
I am currently using Fedora Server 31 with Kernel 5.4.13-201.fc31.ppc64le and cannot observe NVMe problems or crashes anymore like with Ubuntu Server 19.10 and Kernel 5.1.x (but I have to do more tests for that).
-
For certain devices. This machine is fine with a 5.3 kernel (haven't got around to updating it yet), and the Samsung NVMe SSDs Raptor ships appear to be unaffected.
-
ffmpeg botches gif creation when using palattes - standard (ugly) gif creation with ffmpeg is fine however
same exact config + script works okay on x86
Can you please see if this is fixed by this update (https://forums.raptorcs.com/index.php/topic,252.0.html)?
Please confirm the ffmpeg version you are using
-
ffmpeg botches gif creation when using palattes - standard (ugly) gif creation with ffmpeg is fine however
same exact config + script works okay on x86
Can you please see if this is fixed by this update (https://forums.raptorcs.com/index.php/topic,252.0.html)?
Please confirm the ffmpeg version you are using
I will tag this to check. I am currently on ffmpeg 4.1.6.
-
Spring 2022 update:
obs-studio is now available for ppc64le (Debian repository) - tested and working.
Additional improvements a Blackbird redesign could use:
Better hardware clock setting. Cannot interact via Linux, hwclock --set... must currently be done through BMC. I may need to check out fake-hwclock for now.
-
System stutter with any I/O activity on the main drive (could be the PCIe-NVMe adapter?)
major I/O on the main drive (copying many large files to) makes the system nearly unusable
ever reach a resolution with this?
-
I don't think this is an issue with current releases anymore as long as your device is supported for TRIMming.
-
I never had any issues
-
When rebooting sometimes restarts without onboard audio available
complete reboot removing power from the board is required to rectify this situation
Apologies for bumping this, but has anyone figured out why this happens? I'm running into this constantly, and everything I've tried to avoid having to do a full power cycle has failed. I tried powering off the individual ports of the hub (the only Genesys Logic hub on the board, GL852 I believe) it is connected to, powering off the entire hub that it is connected to (not just individual ports), and even powering off the TI USB Hub that pretty much the entire system uses, but none of them bring the audio device back.
I have to put the machine into some low-power state almost daily by request of people I live with, and since there is still no suspend support, the only option is turning the entire thing off...
-
Apologies for bumping this, but has anyone figured out why this happens? I'm running into this constantly, and everything I've tried to avoid having to do a full power cycle has failed. I tried powering off the individual ports of the hub (the only Genesys Logic hub on the board, GL852 I believe) it is connected to, powering off the entire hub that it is connected to (not just individual ports), and even powering off the TI USB Hub that pretty much the entire system uses, but none of them bring the audio device back.
I have found that instead of rebooting, it is possible to just restart the pulseaudio service (for those of us using pulseaudio and systemd):
systemctl --user restart pulseaudio
Always corrects the issue within seconds. Maybe a script that runs this a minute or two after login would automate the problem away.
-
Can you please see if this is fixed by this update (https://forums.raptorcs.com/index.php/topic,252.0.html)?
Please confirm the ffmpeg version you are using
I will tag this to check. I am currently on ffmpeg 4.1.6.
I'm so bad with consistency. Anyway, just checked with the same script while using ffmpeg 4.3.5. Problem persists.
Here is the script:
#!/bin/sh
if test $# -lt 6; then
cat <<-EOH
$0: Script to generate animated gifs easily from command line.
Usage:
$0 input.(mp4|avi|webm|flv|...) output.gif horizontal_resolution fps start_time duration
EOH
exit 1
fi
palette="$(mktemp /tmp/ffmpeg2gifXXXXXX.png)"
filters="fps=$4,scale=$3:-1:flags=lanczos"
ffmpeg -v warning -ss "$5" -t "$6" -i "$1" -vf "$filters,palettegen" -y "$palette"
ffmpeg -v warning -ss "$5" -t "$6" -i "$1" -i $palette -lavfi "$filters [x]; [x][1:v] paletteuse" -y "$2"
rm -f "$palette"
The file it produces, from a sample:
(https://forums.raptorcs.com/index.php?action=dlattach;topic=24.0;attach=534;image)
-
I have found that instead of rebooting, it is possible to just restart the pulseaudio service (for those of us using pulseaudio and systemd):
systemctl --user restart pulseaudio
Always corrects the issue within seconds. Maybe a script that runs this a minute or two after login would automate the problem away.
Are you saying your audio device doesn't drop off the USB bus completely? If I do a "hot reboot", i.e. OS -> Petitboot -> OS, the audio device is fine and works straight away. If I do a proper shutdown that turns off the CPU, status light turns orange, etc, then if I start the system again the audio device is completely gone from the USB subsystem until I completely remove all power from the system.
If it is working for you, I'd be curious to know what revision Blackbird board you have - I wonder if that makes a difference here (I have 1.02).
-
Are you saying your audio device doesn't drop off the USB bus completely? If I do a "hot reboot", i.e. OS -> Petitboot -> OS, the audio device is fine and works straight away. If I do a proper shutdown that turns off the CPU, status light turns orange, etc, then if I start the system again the audio device is completely gone from the USB subsystem until I completely remove all power from the system.
If it is working for you, I'd be curious to know what revision Blackbird board you have - I wonder if that makes a difference here (I have 1.02).
It is possible that our issues are just similar yet separate. What happens for me is that sometimes (a minority of bootups) GNOME will only report the existence of my HDMI audio through the GPU with no dropdown option for the usual "CM106 Like Sound Device" or it might just erroneously say "Speakers" (which provide no audio out when tested). Restarting pulseaudio appears to rectify this.
My main board is revision 1.01.
-
It is possible that our issues are just similar yet separate. What happens for me is that sometimes (a minority of bootups) GNOME will only report the existence of my HDMI audio through the GPU with no dropdown option for the usual "CM106 Like Sound Device" or it might just erroneously say "Speakers" (which provide no audio out when tested). Restarting pulseaudio appears to rectify this.
My main board is revision 1.01.
It sounds like that might be the case. The issue I am experiencing is that the audio device never even appears as a connected USB device, so it doesn't even have a chance to get to the audio system. As an example, here is the relevant output of lsusb -tv when I am booted into the host system with the audio device working (I plugged in a flash drive to the board USB port to rule out the hub being bad):
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
ID 1d6b:0002 Linux Foundation 2.0 root hub
<...>
|__ Port 4: Dev 5, If 0, Class=Hub, Driver=hub/4p, 480M
ID 05e3:0610 Genesys Logic, Inc. Hub
|__ Port 3: Dev 8, If 0, Class=Mass Storage, Driver=usb-storage, 480M
ID 154b:007a PNY Classic Attache Flash Drive
|__ Port 4: Dev 7, If 0, Class=Audio, Driver=snd-usb-audio, 12M
ID 0d8c:0102 C-Media Electronics, Inc. CM106 Like Sound Device
|__ Port 4: Dev 7, If 1, Class=Audio, Driver=snd-usb-audio, 12M
ID 0d8c:0102 C-Media Electronics, Inc. CM106 Like Sound Device
|__ Port 4: Dev 7, If 2, Class=Audio, Driver=snd-usb-audio, 12M
ID 0d8c:0102 C-Media Electronics, Inc. CM106 Like Sound Device
|__ Port 4: Dev 7, If 3, Class=Human Interface Device, Driver=usbhid, 12M
ID 0d8c:0102 C-Media Electronics, Inc. CM106 Like Sound Device
But if I shut down the system with power still connected, then boot back into the host system, this is what I get:
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
ID 1d6b:0002 Linux Foundation 2.0 root hub
<...>
|__ Port 4: Dev 5, If 0, Class=Hub, Driver=hub/4p, 480M
ID 05e3:0610 Genesys Logic, Inc. Hub
|__ Port 3: Dev 7, If 0, Class=Mass Storage, Driver=usb-storage, 480M
ID 154b:007a PNY Classic Attache Flash Drive
The flash drive I plugged into the lone USB port on the board is still present, so it's (probably) not the hub that is having issues. Checking dmesg output also doesn't have any useful information beyond it just not saying anything relating to the audio device when it is not being detected. Other than the board revision, the only other major difference between our setups is that I'm using pipewire instead of pulseaudio, but given that the USB device is not there in the first place I doubt that is the issue here.
-
Now that you describe it, yes my system does the same thing. I've just trained myself into always fully powering off the system whenever I reboot. So I haven't had to deal with the issue since that is now normal operating procedure for me. So no, nothing is wrong with your particular board. I think all Blackbirds are affected.
If Raptor products ever do get support for suspend to RAM, this audio adapter disappearing is probably something that both development and users will run into. Here's a quick check against my kernel boot logs:
1 Time(s): usb 1-4.4: New USB device found, idVendor=0d8c, idProduct=0102, bcdDevice= 0.10
1 Time(s): usb 1-4.4: New USB device strings: Mfr=0, Product=2, SerialNumber=0
1 Time(s): usb 1-4.4: Product: USB Sound Device
3 Time(s): usb 1-4.4: Warning! Unlikely big volume range (=6928), cval->res is probably wrong.
2 Time(s): usb 1-4.4: Warning! Unlikely big volume range (=8065), cval->res is probably wrong.
1 Time(s): usb 1-4.4: [11] FU [Line Playback Volume] ch = 2, val = -6144/1921/1
1 Time(s): usb 1-4.4: [15] FU [Line Capture Volume] ch = 2, val = -4096/2832/1
1 Time(s): usb 1-4.4: [2] FU [PCM Capture Volume] ch = 2, val = -4096/2832/1
1 Time(s): usb 1-4.4: [8] FU [Mic Capture Volume] ch = 2, val = -4096/2832/1
1 Time(s): usb 1-4.4: [9] FU [Mic Playback Volume] ch = 2, val = -6144/1921/1
1 Time(s): usb 1-4.4: current rate 30464 is different from the runtime rate 96000
1 Time(s): usb 1-4.4: new full-speed USB device number 7 using xhci_hcd
If you reboot (remaining on mains power), I wonder what, or if any, output for this device is.
-
My logs when rebooting straight to Petitboot are pretty much the same as when freshly booting after reconnecting power (both cases the audio device works), and I get no related log output at all when booting from shutdown with power still connected (where the audio device is gone). As an experiment, I tried resetting power on the USB port the audio device is on when it is already working, and it came back and worked again as normal. The issue doesn't seem to be as simple as it breaking when losing power. If anyone wants to experiment themselves, I found that the "uhubctl" program worked well for me. Just make sure to carefully read the help output and select the right port(s), as otherwise you might cut power to all USB devices and lose keyboard input.
-
Debian Bookworm ffmpeg 5.1.4-0+deb12u1 introduces an issue where default h264 encoding for mp4 videos creates artifacting. Green patches on frame-by-frame changes.
It can be worked around by manually specifying a different video codec.
ffmpeg -i sample.mp4 -vcodec mpeg2video output.mp4
-
It seems like video encoding on Power9 has been a rough edge on the platform since the beginning... I know the VP9 encoder was getting some decent ppc64le optimizations several years back; did those ever get merged into trunk? Short of using a hardware-assisted encoder I don't even know what my options are that don't require encoding to 480p.