Topics

46

Operating Systems and Porting / any test servers for developer access?

« on: September 30, 2020, 02:27:56 pm »

I've been discussing a rendering bug with the Blender developers

I suspect the most efficient way for them to investigate would be having their own hardware but if that is not possible, remote access to a demo workstation at Raptor or any other user who has a spare Talos II or Blackbird right now.

Is there anybody who can help out in cases like this?

The vast majority of things I tried are working fine though.  This Blender bug is one of the few cases where something did not work.

47

Talos II / CPU alignment / slots not working

« on: September 15, 2020, 03:12:48 pm »

After my success with the NVIDIA card, I decided to try it in each slot

I found that it wouldn't work in slot 3, that is the middle 16x slot.

I tried removing and re-inserting it a couple of times and I inspected the slot with a torch and couldn't see any problem

Next I tried removing and re-seating CPU #2.  After doing this, the NVIDIA card in slot 3 is working.

Are there any diagnostics to confirm CPUs are correctly inserted?  Everything else I did on the CPU appeared normal, for example, I had exercised all cores and RAM without problems.

Is there any technique to follow when inserting the CPUs to avoid issues like this?

48

Operating Systems and Porting / kernel config: page size 4k vs 64k

« on: September 14, 2020, 05:19:55 am »

Most distributions have chosen the 4k page size for their kernels on Intel architectures.

On powerpc64le, Debian (since 2014) and Fedora are using 64k

This may be good for HPC environments but troublesome for workstation users

Here are some specific observations:

Nouveau driver apparaently won't work at all on anything other than 4k

btrfs filesystems have a sectorsize that corresponds to the page size on the host where the filesystem was created.  The filesystems can't be mounted on any host with a different page size.

Example problems:

• you have a 6TB drive with btrfs from an x86 workstation, you want to move it to a Blackbird running the default Debian or Fedora kernel but it won't mount.
• you install Fedora 33 with the default kernel (64k page size) and default filesystem (btrfs from Fedora 33) and later you want to recompile your kernel for 4k.  Now you can't mount your root filesystem because mkfs was run for 64k

As Fedora is going to use btrfs by default now, btrfs volumes will be more common and users are more likely to encounter frustration from time to time.

The btrfs developers recently started a patch that allows the systems with 64k page size to read a btrfs volume with 4k sectorsize.  It only works in one direction and it is read-only.

Does anybody feel that distributions should offer different permutations of their kernel and installer based on the 4k page size?

From the perspective of workstation users, are there any other strong reasons, other than Nouveau and btrfs, for distributions to consider this?

49

Talos II / power consumption observations

« on: September 12, 2020, 01:48:31 pm »

I made the following observations using the meter in my UPS, it measures units of approximately 0.1A

Measurements were made with Debian buster in the idle state, at the GNOME login screen.  It actually uses less power there than at the petitboot menu.

Voltage = 240V



Unplugged: 0

BMC powered: ~ 0.1A

Single 8 core CPU installed with 4 memory chips: add 0.3A - 0.4A above the BMC power draw (total, 0.4A = 96W)

Second 8 core CPU installed with 4 more memory chips: add 0.2A - 0.3A above the single 8 core (total, 0.6A = 144W)

These are bare minimum figures in an idle system with RX 580, LSI 9207-4i4e and one SATA SSD

The system doesn't have the SAS option on the board, that would use more watts if present.

HP suggests a comparable single CPU Z8 G4 workstations can use 74W at idle:
https://www8.hp.com/h20195/v2/GetPDF.aspx/c05527763.pdf (page 31)

Their Z4 workstations can go down to about 40W at idle although the spec is more like a Blackbird with 4 core:
https://www8.hp.com/h20195/v2/getpdf.aspx/c05527757.pdf

I feel that for people who are mostly idle and just need an occasional boost of power to compile something from time to time, it may be better to stay on a single CPU and Talos II Lite but for people who need the power regularly for different types of work, the HP and Talos II would end up using similar amounts of power.

For people who leave their workstation on all the time or use it as a file server, the second CPU is wasting a lot of power when idle.  As discussed in another thread, it would be useful to have options to power down some cores

50

Operating Systems and Porting / petitboot doesn't show the OpenBSD partition

« on: September 11, 2020, 01:52:36 pm »

To workaround the issue with my HBA, I removed the HBA and did an install to an SSD in a USB 3 dock (Sharkoon)

The OpenBSD installer completed successfully

petitboot doesn't show the OpenBSD partition in the boot menu

If I exit to the OpenBSD shell, I can use fdisk to inspect the SSD and I can see the partition is there.  It is /dev/sda4 and type = a6 (OpenBSD)

Is there any way I can get petitboot to boot this partition or does it need some newer version of petitboot or some change in the OpenBSD partitioning scheme?

51

General Discussion / U.2 vs U.3 disks, PCIe 4.0 HBAs

« on: September 07, 2020, 12:29:18 pm »

I was thinking about putting U.2 disks in my workstation to get optimum speed.  However, reading about the subject, I found a few gotchas:

The U.2 disks will not work in future U.3 slots, but U.3 drives will work on U.2 controllers, according to this article.  That is despite all the effort for interoperability promised by tri-mode HBAs.  The implication is that it is better to avoid U.2 now and wait for more U.3 products in the months ahead.

Supermicro's HBA list does not include any PCIe 4.0 product now, using only 50% of the speed of the slots in Talos II.  The PCIe 4.0 cards appear to be arriving in the market very slowly.

It seems that the PCIe 4.0 products that do exist now are not fully supported yet on every OS, e.g. the LSI 9500-8i

For people running Fedora, they will go with Btrfs as the default filesystem soon.  As far as I know, Btrfs mirrored configurations require two copies of each write to go through the bus from CPU to HBA, compared to mirroring in hardware RAID, where the OS only has to send a single write to the RAID controller.  This appears to be another good reason to have the fastest possible storage path, PCIe 4.0

The conclusion: I can live with SATA SSDs running on my LSI 9207-4i4e for 6-12 months while the U.3 drives and PCIe 4.0 Tri-mode controllers become more certain.

52

Operating Systems and Porting / OpenBSD 6.8, first official release, testing, outstanding issues

« on: September 07, 2020, 03:22:08 am »

According to the PPC page on the OpenBSD web site, OpenBSD 6.8, which is imminent, will be the first official release to support PPC64

Has anybody else tried their PPC64 installer snapshots recently?

After my attempt to run it last week, they included a patch to support machines with larger amounts of memory and the installer progresses further but it still has a limit somewhere.

The next problem I found was that it becomes stuck when I have the LSI 9207-4i4e HBA in the machine.  If I remove the HBA, the OpenBSD installer boots all the way to the first question:

Code: [Select]

Welcome to the OpenBSD/powerpc64 6.8 installation program.
(I)nstall, (U)pgrade, (A)utoinstall or (S)hell?

If anybody else has time to test it, especially if you have accessories like a HBA or NIC, that can help the OpenBSD 6.8 release gain traction or at least help qualify the list of supported hardware on the wiki.

53

Firmware / swapping flash chips, ordering spares

« on: September 06, 2020, 03:43:52 am »

I looked at the wiki pages Updating Firmware and Debricking the BMC

The latter suggests "Remove the BMC SPI flash chip and reflash it with a flash programmer"

Rather than reflashing the existing chip, it may be safer to keep one additional flash chip on site.  Any new images can be flashed to the spare flash chip, it is swapped into the socket and if it doesn't work, the old chip is swapped back in.

Can anybody clarify the exact type of chip to order?  Are both flash chips identical or different part numbers?

Has anybody seen any useful video that shows the correct way to remove the chip?

I have a CH341A flash programmer, what type of cable or adapter is needed to insert the flash chip, or does it fit directly into one of these programmers?

54

Operating Systems and Porting / testing a new machine, memory test, etc

« on: September 05, 2020, 02:53:07 pm »

What burn-in test routines do people use when making a new machine or modifying the machine substantially?

For example, on x86 machines, I frequently run memtest86 from a USB stick before installing the OS.  It is not available for OpenPOWER.  It would be really useful to have something like this accessible in petitboot.

55

Operating Systems and Porting / installers not booting

« on: September 04, 2020, 09:22:43 am »

petitboot starts up and displays the menu

I've tried both the Debian 10 and OpenBSD USB sticks.  Neither of them appears to be working.  The install options (e.g. Debian/Expert mode) appear in petitboot, I select the option I want, the screen goes blank and then displays the message "SIGTERM received, booting..."

Leaving the machine like this for 5 or 10 minutes, nothing more happens, it just appears stuck like that.

56

Operating Systems and Porting / testing different operating systems next week, any ideas/requests?

« on: September 03, 2020, 02:46:16 pm »

Before putting my Talos II into proper use, I might spend a week or so testing different operating systems on it.

Can anybody make any suggestions about things that might be worthwhile to explore?

57

General OpenPOWER Discussion / Talos II arrived

« on: September 03, 2020, 02:12:14 pm »

My Talos II arrived yesterday, I'll be building it gradually over the next few days.

I already shared some observations about success with the Fractal 7 XL case, it would be really helpful to get some feedback about their fan hub connections

I'll use a legacy SATA HBA as a stop-gap measure until I get some U.2 disks in a few weeks.  Long term, can anybody comment on what will be a good choice of HBA to combine at least two U.2 disks and a handful of legacy SATA disks?

Can anybody make any other tips about getting started?

58

General CPU Discussion / max memory speeds

« on: July 31, 2020, 03:46:40 pm »

AbstractConcept made this edit in the wiki

I read somewhere that RAM operates at different speeds:

1 channel filled: 2666MHz

2 channels filled: 2400MHz

4 channels filled: 2133MHz

but the edit by AbstractConcept implies that RAM is always clocked at 2666MHz

Can anybody clarify?

59

General CPU Discussion / v2 CPU availability?

« on: July 29, 2020, 11:25:49 am »

I notice that Raptor is only offering the 4 core and 8 core version of the v2 (stepping DD2.3) chips.  The 22 core chip is only available in v1.

Is there any reason for this?

I decided to order the 8 core v2 CPUs with the intention that I will use them on a dual-CPU board until the 22 core v2.3 CPU is available.  Then I could move the 8 core CPUs onto Talos II Lite boards and upgrade to 22 core v2.3 chips on my primary workstation.

60

General Discussion / optimum add-on cards for 2-slot systems (Talos II Lite and Blackbird)

« on: July 26, 2020, 01:11:54 pm »

I've had some discussions with other developers about the POWER9 platform and one observation I've made is that people are looking at a choice between Talos II Lite or Blackbird.  Many don't need a dual CPU system and can't afford the full Talos II.

To iron out bugs in software, it is really vital to have a critical mass of developers using the platform and I suspect most of them are going to use either Talos II Lite or Blackbird.  When they narrow their choice down to those products, the next thing that comes to mind is: what should they put in the PCIe slots to get best use out of their workstation?

The wiki contains a large list of compatible products but what I'm really thinking about here is making a shortlist of five to ten really interesting products and putting them on the comparison page

Here is a summary of the problem:
 

• most people will want to put a GPU in the x16 slot - some GPUs, like the Radeon Pro W5700 throw in a USB-C socket or some other bonus which is really useful on a system like this with limited expansion slots
• with the x8 slot, most people will probably want a card that can give them some combination of SSDs (U.2 NVMe), legacy SATA disks and hopefully some other ports like OCuLink or ethernet on the same board as a bonus.  Can anybody comment on which boards are optimal in this scenario?

Some other permutations come to mind: for example, people who don't do a lot of graphics may be happy with a GPU in the x8 slot and use the x16 slot for OCuLink.  Hopefully we will also see some external OCuLink GPUs in the future but none seem to be available right now.

Feel free to add comments here or at the bottom of this wiki page