Topics

1

Operating Systems and Porting / Unexpected suspend of Fedora 39 on Talos II

« on: January 15, 2024, 01:42:26 am »

After switching to the 6.6.9-200 kernel, I started getting unexpected system transitions to the SUSPEND state and then I could not restore the system to normal operation other than by shutting it down and turning it back on.

I found the solution quite easily, this affliction happens on Fedora on different platforms. The problem is solved by executing the following commands:

Code: [Select]

$ sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target


2

Firmware / Adventures in reverse engineering broadcom nic firmware

« on: December 28, 2023, 12:11:59 pm »

Unlocking a system with 100% open source firmware

In an era where vendors increasingly seek to use proprietary software in the devices around us to exert control over their users, the desire for open source software has expanded to the firmware that allows our machines to function, and platforms which individuals can trust and control have never been more important. However, changes to hardware platforms in recent years such as the Intel ME, vendor-supplied binary blobs and vendor-signed firmware images have repeatedly set back efforts to create open source firmware for the computers we use. The release of Power servers with 99% open source firmware excited many who had been searching for a computer they could trust, but one proprietary firmware blob remained: that of the Ethernet controller. This is the story of how that blob was reverse engineered and replaced with an open source replacement, delivering the first machine with desktop-class performance and 100% open source firmware in many years.

This talk is about how I reverse engineered the final remaining firmware blob on the Talos II/Blackbird POWER9 systems, enabling it to be replaced with an open source replacement, in an intensive reverse engineering effort that spanned several years.

The talk will begin by introducing the open source firmware movement and its practical and ethical motivations, and note the obstacles to delivering fully open source firmware for contemporary x86 and other platforms and explaining the motive behind the project, before moving onto a more technical discussion of the adventure of firmware reverse engineering and the obstacles encountered.

Subjects I intend to cover include: how the original proprietary firmware was reverse engineered from scratch with only limited knowledge of device internals; the long history of Broadcom NIC architecture and its evolution over time; the tools that had to be developed to enable the device probing, testing and reversing process; the story of a horrifying but necessary detour into reversing x86 real mode code and the novel methodology used to aid reversing; how modern NICs allow BMCs in servers to share network ports with the host, and the security hazards this creates; and how fully open source firmware was created legally using a clean room process.

This talk will be accessible to audiences unfamiliar with POWER9 or the open source firmware community, but is also intended to cover some new ground and be of interest to those familiar with the project. The talk will mainly be of interest to those interested in open source firmware and issues such as owner control and the security and auditability issues caused by proprietary firmware, and to those interested in reverse engineering.

3

Talos II / Talos II reboots itself

« on: December 01, 2023, 03:23:34 pm »

Since September I have been having problems with spontaneous reboots of my machine, it has happened four times. Recently it was very annoying, the machine rebooted over and over again without loading the operating system. Only removing the power plug helped.

I managed to take a picture of what the report looks like after such a fall:

https://www.dropbox.com/scl/fi/hk0zqvfjpidexnw7m1xln/Talos-II-2023-11-27-Crash.jpeg?rlkey=0jj34pjpv708zigt08kzfk3zl&dl=0

Any ideas what could cause this behavior?

4

Applications and Porting / [DEV] The Blend2D library partially runs on ppc64le.

« on: April 08, 2023, 12:54:04 pm »

I've been playing with the Blend2D library for a while now. It is a high performance 2D vector
graphics engine written in C++ and released under the Zlib license https://blend2d.com/.
On the x86_64 platform everything compiles and works quite well but on ppc64le it is no
longer so ideal.

What works:

• single-threaded mode without boosters;
• multi-threaded mode without boosters but in a limited way.

What doesn't work:

• SIMD operation support;
• asmJit compiler tool to generate JIT code https://github.com/asmjit/asmjit;
• multithreading support under heavy load.

I tested this on my Talos II with POWER9/18c. I downloaded the sources for the library
https://github.com/blend2d/blend2d and the bl_bench test program https://github.com/blend2d/blend2d-bench.
After downloading the library code and bl_bench, I compiled this program with the following
command:

Code: [Select]

build]$ cmake .. -DCMAKE_BUILD_TYPE=Release -DBLEND2D_DIR=../../Source/ -DBLEND2D_NO_JIT=TRUE -DBLEND2D_NO_STDCXX=false

In single-threaded mode everything works correctly but in for multi-threaded mode you should
run bl_bench with the parameter --quantity=N where N = <1, 15>. For N=16 it sometimes crashes
and for N=17+ it crashes always.

By default multithreaded mode works for 2T and 4T, to add more threads I added the following
piece of code to bl_bench/src/app.cpp in line 359:

Code: [Select]

    {
      Blend2DModule mod(8);
      runModule(mod, params);
    }

    {
      Blend2DModule mod(16);
      runModule(mod, params);
    }

After these modifications, I obtained the following results:

• bl_bench-gcc-Release-quantity-1000.txt: results for single-threaded mode with default parameters;
• bl_bench-gcc-Release-quantity-16.txt: results for single- and multi-threaded mode with quantity = 16;

I'm also attaching screenshots showing the performance of selected modes, images can be obtained from
the library's website after entering your own performance measurements using bl_bench.

The conclusion of an incorrigible dreamer:

Haiku on x86 uses the AGG engine and its UI is incredibly responsive, if on ppc64le one had such a Blend2D-based
UI then there would be no complaints about the graphical performance of Gnome or KDE ;-)

5

Operating Systems and Porting / Fedora 38 (Workstation Edition Prerelease) on Talos II works just fine

« on: January 06, 2023, 01:16:21 pm »

Today I had a day off and decided to install Fedora Rawhide from 2022/12./23, which I had downloaded two days earlier,
on a trial basis. The installation from the usb key went without any major problems, remembering my previous problems
with XFS this time I chose ext2 for the /boot directory and ext4 for the rest.

After the installation I was positively surprised, GDM booted by default with Wayland enabled and ran desktop on both cards:
ASPEED and Radeon Pro WX7100. I have two monitors connected to the Radeon with resolutions of 5120x2160 and 1920x1200.
To the ASPEED card I have a monitor connected with a resolution of 1600x1200. Wayland ran at maximum resolution on the two
smaller monitors and at 2560x1080 on the largest monitor. Gnome 43.1 ran quite smoothly on this virtual screen. Even moving
windows from tab to tab caused no noticeable problems. I tried to increase the resolution on the largest monitor from within Gnome
but was unsuccessful. Unfortunately, due to Wayland's serious shortcomings when it comes to the ability to configure details,
gnome-tweaks, xset, etc., I went back to the standard X11 installation with two monitors.

From my point of view, the most interesting new features in Fedora 38 are:

• kernel 6.2.0 series, in this case 0.rc2.20230103;
• very good support for Samsung SSDs, reding above 6.2 GB/s;
• Gnome 43.1, which, despite the developers' efforts, can be personalized with extensions such as WhiteSur ;-)
• NetBeans 16 and Java 17 work flawlessly.

6

Applications and Porting / Problem with std::numeric_limits<char>::max()

« on: November 03, 2022, 02:37:51 pm »

I'm porting my software to ppc64le and ran into a problem in the std::numeric_limits<char>::max() function. According to
the documentation, this operation should return the value CHAR_MAX which is 127. This is the case on x86_64 with gcc 11.3.1
compiler. Unfortunately on ppc64le the code compiled with gcc 12.2.1 returns the value UCHAR_MAX which is 255.

Test code:

Code: [Select]

printf("std::numeric_limits<char>::max() == %d\n", static_cast<int>(std::numeric_limits<char>::max()));

Clang 14.0.5 and IBM xlc 16.01.0001.0003 return the same value.

Have any of you encountered such a problem on distributions other than Fedora 36?

 

7

Talos II / Unsatisfactory performance of SSD drives

« on: November 01, 2022, 08:57:16 am »

I switched to the Talos II machine from a Dell Precision 7730 laptop. In the laptop I had an older-generation Samsung SSD,
a 970 Pro, connected via an M.2 connector to PCI 3.0. In the Talos II I used a 980 Pro connected to PCI 4.0 via an ICY BOX
PCIe 4.0 x4 - M.2 PCIe NVMe controller (up to 64 Gbit/s). I expected a read transfer of 5-6 GB/s and got 0.9 GB/s.
That's more than three times slower than on a laptop. Screenshots attached.

Is there any way to improve this?

8

Talos II / Temperatures and rotational speed of fans

« on: September 28, 2022, 07:55:05 am »

I have a bit of a weird situation with the temperature and speed of the fans in my Talos II:

https://www.dropbox.com/s/iyzo0dllnmclwtx/Talos-II-Sensors-2022-09-28.png?dl=0

Should the control system not increase the speed to lower the temperature on this one sensor?

9

Talos II / Petitboot lock after Fedora 36 update

« on: September 26, 2022, 12:07:45 pm »

Today, after the Fedora 36 update, I had an unpleasant surprise. The first phase of the system update went as usual without any problems.
Then, as recommended, I restarted the machine and the firmware update phase began. After this process was completed, it was rebooted
again and here are the problems:

• Petitboot did not detect the disk, there is no list of possible kernels to load;
• USB keyboard stopped working, locked on all USB sockets
• the "Power" button stopped working, I couldn't turn off the machine
• At the bottom of the screen, Petitboot wrote the message "Info: Waiting for device discovery"

Any ideas?

Minor fix, after the next restart the "Power" key works.

I made a video of how the boot process works: Petiboot after F36 firmware update

10

Operating Systems and Porting / Installing FreeBSD 14.0-CURRENT powerpc64 BE on Talos II

« on: July 19, 2022, 02:44:45 am »

I have tried several times to install FreeBSD 14.0-CURRENT powerpc64 BE on my Talos II,
unfortunately without a positive ending. I have tried several versions 220617, 220701, 220708
and each time the system installed without error messages but Petitboot did not see the system.
Yesterday I tried the 220715 version and also did not have success. So I decided to take a closer
look at it and found that the installer incorrectly fills /boot/etc/kboot.conf:

Code: [Select]

default=
='/kernel/kernel kernelname=/boot/kernel/kernel vfs.root.mountfrom=ufs:/dev/nda0p2'

After changing the entry to the following:

Code: [Select]

default=FreeBSD
FreeBSD='/kernel/kernel kernelname=/boot/kernel/kernel vfs.root.mountfrom=ufs:/dev/nda0p2'

I made the changes manually from the Petitbnoot shell. Just keep in mind that Petitboot mounts
read-only /boot by default. You have to unmount it and mount it with write permission. After the fix,
vi is helpful here, you need to unmount the partition and re-mount it read-only. Without this last
step and after direct loading of the system, the changes will not be saved. You have to repeat
them from the system level.

One final note about ZFS, by default FreeBSD on ppc64 does not support this filesystem. This
is due to Petitboot limitations. You have to compile the system yourself. The recipe can be found
here "FreeBSD/powerpc64 (POWER9) with ZFS on /".

Currently my Talos II is compiling the kernel on a single core since yesterday. I did it consciously
to have a benchmark 

11

General OpenPOWER Discussion / Forum signature size.

« on: July 16, 2022, 05:58:11 am »

Can you change the size of the signature from 300 to 1024 characters? Manipulating the colors of characters quickly exhausts this limit ;-)

12

Operating Systems and Porting / Fedora 36: Kernel upgrade problem.

« on: July 11, 2022, 02:16:48 pm »

I recently upgraded Fedora 36. DNF showed me that he has installed a new kernel 5.18.10. Unfortunately, petitboot still only sees kernel 5.18.7 after a reboot. Do I have to do anything else special?

13

Operating Systems and Porting / I am looking for a dnetc to count OGR-28 on PPC64LE

« on: July 08, 2022, 05:19:52 pm »

I tried the [PPC] /v2.9112.521/2016-06-13 version for Linux but it it doesn't work ;-)

14

Talos II / Installing the system on a Samsung 980 Pro SSD 1TB

« on: June 24, 2022, 04:33:11 pm »

Heya everyone,

At the outset, I would like to mention that I waited almost four and a half years for my own Talos II station. This whole story deserves a separate article
and I even started to prepare for it. But I figured I needed a working base first to be able to write anything on my new toy without going back to x86 too
much.

For my first attempt, I decided to install FreeBSD 13.1 on two Samsung 980 Pro SSDs. I put the disks into the Highpoint SSD7502 controller. As expected,
I saw two separate drives as a possible installation location. Unfortunately I couldn't use ZFS to do RAID although it was stated on the FreeBSD website
that ZFS runs on the powerpc64 platform. I marked the first disk as the target and decided to partition it. Partitioning worked, I chose UFS as the root
filesystem and FAT as the system for the kernel partition. I started the installation. This operation did not produce the expected results, a process crashed
while trying to unpack base.tar.xz.

So I thought that the problem may be the raid controller, because FreeBSD does not have a dedicated driver for it. So I replaced the controller with a simple
ICY BOX M.2 NVME to PCIe 4.0 adapter: IB-PCI208-HS. Unfortunately, this operation did not help. I still couldn't install the system. So I decided to try the
powerpc64le version. Surprisingly, this version installed and I was even able to boot the system. However, my joy lasted only a short time, when I tried to install
packages, the system crashed.

IMG_1376.jpeg

Second try:

IMG_1379.jpeg

I found information on the network that there were performance problems with the first Samsung 980 Pro disks for PCIe 4.0. I decided to replace
the firmware. I moved the adapter to Dell OptiPlex 7040 and updated the firmware:

IMG_1380.jpeg

IMG_1381.jpeg

IMG_1382.jpeg

IMG_1383.jpeg

After this operation, I decided to reduce the system installation time and focus on disk tests. I have successfully installed FreeBSD on a USB key.
With the key, the system worked flawlessly. Unfortunately, attempts to access the SSD drive always end up in the disappearance of the device.
Fortunately, the system did not lock up anymore.

IMG_1389.jpeg

IMG_1390.jpeg

I was disappointed. I tell myself I'll try Linux. It's probably much better here. I have installed the distribution that I use professionally on x86, AlmaLinux 9:

IMG_1387.jpeg

Installation seemed to go smoothly, but there were momentary system freezes. I looked at the logs and my worst expectations came true:

IMG_1395.jpeg

After five such errors, the system stops operating the device and further normal operation becomes impossible. Usually, full freezing of work occurs
within a quarter of an hour.

Of course, I have no more ideas what to do with it. Any help is appreciated...

PS. Sorry for the lack of direct images, but I'm having trouble providing the correct DropBox preview links.