Messages

1

Firmware / Re: Messing with WOF Tables

« on: March 24, 2025, 11:03:43 am »

Got my login working.  I used these two easy scripts to watch freq, temp, and power:

cat ./watch_cpu_freq.ksh
watch -n2 "cat /proc/cpuinfo | grep MHz"

cat ./watch_sensors.ksh
# Watch thermal sensors
watch -n 1 -d sensors

My 4-core CPU idles at ~2170MHz, then when I run 1 thread of the aforementioned mersenne just one set of cores jumps to 3783MHz which is ~= 3800MHz Fmax.   So apparently frequency is managed per quad. 

When I run 2 threads, sometimes both sets jump to max, sometimes only 1 set...I guess it depends on which quads the OS allocates the threads to. 

When I run 4 threads, frequency maxes at 3400MHz.   At 8 threads, frequency maxes at 3250MHz.   So it looks like WOF is optimizing the frequency to the amount of power used by the set of active workloads on my system.

2

Firmware / Re: Messing with WOF Tables

« on: March 21, 2025, 05:24:04 pm »

Ah-ha. So that's why VRATIO_INDEX is always 0-23 and VRATIO_STEP is always exactly 1/24 to 3 figures. VRATIO_STEP is always 0.0409 to account for rounding in VRATIO_STEP to ensure that 23*VRATIO_STEP + VRATIO_START == 1. So really, VRATIO_INDEX is the easier to follow number since it's just the number of active cores. And for a 16 core CPU, one should be able to ignore any VRATIO_INDEX > 15, as 0-15 cover 1-16 active cores, right?

I don't know for sure, but I don't think it works that way.  Rather the index is set up to handle 24 cores, but if the particular processor has < 24 cores, the denominator of that ratio becomes the number of valid cores.  So a 16-core part would be 16/16=1.0, but if 1 core was stopped then the ratio would be 15/16=0.9375 which would then get rounded up the nearest VRATIO point so 23/24.  If 14 cores were active, the ratio 14/16=0.875 would math out to exactly the same as 21/24=0.875 so no rounding needed.

I had thought it was the case where if 2 active cores in a quad had requested frequencies of 2200 and 1800 then the quad (and thus both cores) would run at 2200MHz.

It is true that all cores in the same quad have to run at the same frequency, and I believe you are correct that the highest-requested frequency wins.  Different quads have the capability to run at different frequencies if implemented by that Linux so, if it's reporting different frequencies aligned with different quads, that could be real.

What are you using to monitor frequencies?   Something like watch -n2 "cat /proc/cpuinfo | grep MHz" or turbostat?
The Sforza datasheet indicates 02CY231 to have a core base frequency of 2.50GHz and max boost of 3.8GHz with Tj Maximum of 85C.  So the cores should not go below 2.50GHz unless some power-saving mode is kicking in, e.g. OS governor settings (e.g. detecting idle and lowering frequency for energy savings). 

I like to use this Mersenne-prime code to push the core power and processor temperatures pretty hard.  I'm pretty sure I compiled & ran on the Blackbird system here, but I cannot seem to log into that right now.   Maybe you can give it a go and see what happens?   I think if you run just 1 or a few threads, the power stress is low but hopefully enough to kick the governor into performance state and it should run close to 3.8GHz.   Then run max-threads and it should drive frequency down toward 2.5GHz, maybe get you closer to that 160W TDP spec?

#include <stdio.h>
#include <stdlib.h>
#include <gmp.h>
int main(int argc, char *argv[]) {
   char *endptr;
   unsigned long int p = strtoul(argv[1],&endptr,10);
   mpz_t M, powerof2, one, two;
   mpz_init(M); mpz_init(powerof2);
   mpz_init_set_str(one,"1",10);
   mpz_init_set_str(two,"2",10);
   mpz_pow_ui(powerof2,two,p);
   mpz_sub(M,powerof2,one);
   gmp_printf("%Zd",M);
   return 0;
}

E.g. I take my Intel lapdown down by running 8 copies of
num=82589933; time ./mersenne $num > M48.1 &

3

Firmware / Re: Messing with WOF Tables

« on: March 14, 2025, 12:05:51 am »

>That link doesn't seem to work, do you happen to still know how to navigate to it or know the name of the paper.

Sorry, I don't.  As I recall, it was an educational presentation/paper on how processor power is modeled.  It is pretty well known material, maybe you can find other references.

Nominal mode is just a lower power TDP.  Turbo mode is a higher power TDP.   WOF should be able to boost from both, the proc will just start and get to higher frequencies sooner with the TM table.

It's been a long time, but I believe VRATIO just means the number of ON cores (a.k.a. active) relative to the maximum available.  As the # of active cores goes down, the power from those cores is applied to the remaining cores, allowing them to boost to higher frequencies.   Depending on the system & power limit, the proc could pretty quickly apply so much power credit from offline cores that it flat-lines at the maximum possible frequency, i.e. it becomes technology limited not power limited.  It sounds like, for that table, the processor is only power-limited when >= 12 cores ON.    Technically VRATIO is "voltage ratio" intended to handle quads using the internal voltage regulator at some % below the input voltage, but it ended up not getting supported and devolved to tracking # of cores.   A core in a stopped state with the power headers off has voltage of 0v hance VRATIO=0 for that core, and you add up the # of ON vs OFF cores to get he VRATIO.

FRATIO, or "frequency ratio" was intended to handle some cores operating at lower frequencies than processor-level frequency target, since each quad has it's own clock generation.  I don't believe Linux ever implemented support for different quad frequencies so they all just run at the same frequency?   If true, WOF only uses FRATIO=1.0 indices.


>And I think the number 1 question is "how does one know which CSV gets selected for a given CPU"?

WOF has a bunch of cross-checks between the data table and the module VPD data....I'm not exactly sure which, but possibly core count, nominal/turbo/max frequencies, & the sort power target?    E.g. each table is designed to manage to a certain power target, and the processor code is trying to find a table appropriate for that processor

Cool 3D plot!   For reference, CORE_CEFF is the ratio of the workload switching power relative to TDP, where TDP=1.0.   So 0.5 means the workload has half the switching power of TDP.     If the workload is using less power, the WOF table should attempt to raise frequency, up to the maximum allowed, where it flatlines.   Similarly, if there are fewer cores active, the frequency will go up, and if both are true, the frequency will go up more.   So that explains the shape of that plot.

4

General OpenPOWER Discussion / Re: POWER11 on the horizon?

« on: January 23, 2025, 07:24:24 pm »

Video interview with Bill Starke, current chief architect for Power CPUs, here:  https://www.youtube.com/watch?v=v9S_w_Bj3oo

Interesting points:
   - Power11 is evolutionary from Power10 architecture
   - Core is same POWER ISA version as Power10, will run faster -> presume that means higher frequencies
   - more cores active than Power10
   - future CPU after Power11, neither confirmed nor denied as Power12, will be chiplet architecture similar to AMD because that was deemed better for scaling up to 16-socket computer
   - continued use of OMI memory now and into the future due to performance and RAS benefits

It doesn't look like Power11 will be any different than Power10 as far as open computing goes.

Here's hoping that S1 succeeds...

5

Firmware / Re: Adventures in reverse engineering broadcom nic firmware

« on: January 03, 2024, 04:32:58 pm »

Wow, quite an entertaining talk, and nice contribution to increasing openness of the platform...well done Hugo.

It would be quite interesting to hear from any Broadcom engineers on the hilarity of The Great Broadcom BitBang.  I would not be suprised if that was a hack invented to solve some early problem with thye design that nobody bothered to go back and fix.

6

Talos II / Re: Talos II reboots itself

« on: December 13, 2023, 06:11:40 pm »

Is the error always on c4?   If yes, can you disable c4?   

7

Mod Zone / Re: Custom cooler mount

« on: October 20, 2023, 09:49:36 am »

That looks interesting but
>We therefore use highly optimized assembly routines that take the specific properties of a given processor microarchitecture into account.

Is that what is being done for
>POWER9 support is planned
??


I'm running a mersenne-prime calculator which seems to push the CPU pretty hard.  At least, it runs down near the "base" frequency of 3.2GHz (I have a 4-core CPU).

>cat mersenne.c
#include <stdio.h>
#include <stdlib.h>
#include <gmp.h>
int main(int argc, char *argv[]) {
   char *endptr;
   unsigned long int p = strtoul(argv[1],&endptr,10);
   mpz_t M, powerof2, one, two;
   mpz_init(M); mpz_init(powerof2);
   mpz_init_set_str(one,"1",10);
   mpz_init_set_str(two,"2",10);
   mpz_pow_ui(powerof2,two,p);
   mpz_sub(M,powerof2,one);
   gmp_printf("%Zd",M);
   return 0;
}


Run with:

>cat mersenne16.ksh
num=82589933
thread=0
while (( thread < 16 ))
do
   echo $thread
   echo time ./mersenne $num > M48.$thread
   time ./mersenne $num > M48.$thread &
   (( thread += 1 ))
done

8

General OpenPOWER Discussion / Re: POWER11 on the horizon?

« on: October 20, 2023, 09:28:01 am »

>POWER8 came in 2014, POWER9 was introduced in 2017 and POWER10 ended up being introduced in 2021.

Past processor:  2021-2017 = 4 years
Next processor?  2021+4 years = 2025

9

Mod Zone / Re: Custom cooler mount

« on: October 09, 2023, 06:43:39 pm »

>with the big benefit being keeping the temps down keeps the clocks high and the power consumption down.

If using IBM's WOF, the algorithm does not work that way.   It should boost to "the same" frequency regardless of CPU temp until it exceeds the temp limit at which point it will lower frequency to protect temperatures.  The TDP is conservative and very few workloads would exceed that.   The bigger factor affecting the temp protection mechanism is ambient temp....> 30C is more likely to throttle than < 30C.

What workload are you testing with?   I just got a Blackbird running Ubuntu and am stress testing it to see how it responds.  It's quite fun.

10

General CPU Discussion / Re: Safe operating temps of a 10 core power8 cpu?

« on: October 04, 2023, 06:56:09 pm »

IBM rated all the POWER8 - POWER9 processors to long-term reliability temp of 85C.   >85C may be a tad hot for long life...

11

Mod Zone / Re: Custom cooler mount

« on: October 04, 2023, 06:52:40 pm »

Seems like you have an 18c POWER90 rated at 190W?   https://raptorcs.com/content/CP9M36/intro.html

POWER9 TDP long-term max temperature rating is 85C.  If your system runs ~70C, you have decent margin to the reliability limit.  How much is the improved cooling and how much is your workload would probably need to come from power data vs. that 190W spec.

12

Mod Zone / Re: Custom cooler mount

« on: September 11, 2023, 09:12:02 pm »

>For stability testing I ran this machine hard for multi-day cycles (one of my own tools that 100% loads the CPU cores with lots of maths).

Do you know how "hot" that workload is compared to the TDP rating of the processor? 

Regards, Eric

13

Mod Zone / Re: Custom cooler mount

« on: August 17, 2023, 11:02:38 am »

Do you have any idea how much force you are applying down through the module to the socket pins?   IBM has a target pressure to ensure even and reliable contacting across all pins for the life of the processor given certain assumptions about electrical loads (e.g. amps thru pins) and thermal cycling.   It would be interesting to know if your solutions are approximately the same, or much higher or lower.

14

Firmware / Re: Updating Talos II firmware to IBM PNOR V2.18?

« on: April 17, 2023, 05:44:57 pm »

The 18c & 22c parts are "paired" meaning 2 SMT4 cores share the same L2 & L3, unlike the 4c and 8c which are "unpaired" meaning each core gets the full L2 and L3 to itself.    This is not the same as "fused" (i.e. SMT8 cores) but it is quite likely that the fix will also work for "paired" cores as presumably the issue is sharing cacheable/non-cacheable pathways.   Good luck!

15

Mod Zone / Re: Initial findings running on water cooling

« on: March 30, 2023, 07:59:11 pm »

That is awesome.    Do you know what the power difference is running the same benchmark air-cooled vs. water-cooled?

Are you thinking of boosting frequency higher on the CPU since it is running cooler & lower power?