Raptor Computing Systems Community Forums (BETA)
General OpenPOWER Hardware => General CPU Discussion => Topic started by: pocock on May 08, 2020, 03:06:30 am
-
This article has a photo of a POWER9 with a water cooling kit
https://www.forbes.com/sites/tiriasresearch/2017/12/08/why-ibm-launched-instead-of-power9/
Water cooling could be a useful solution for people who are trying to minimize noise levels.
It can also be useful for moving heat further away from the place where equipment is used, if sufficient pipework can be installed.
-
Custom POWER8 watercooling option (https://twitter.com/octaforge/status/1101683774356705280?s=20)
-
Thanks for the link. I dug through the junk on there and found the cheese:
(https://blogs-images.forbes.com/tiriasresearch/files/2017/12/IBM-AC922-air-and-water-cooled-1200x731.jpg)
That's neat! I wonder where/how we could get those water blocks shown?
-
Vikings is working on a silent water cooling solution for POWER systems. We should have something presentable in ~one to ~two month(s) from now.
-
Outstanding!
-
nice, but a 4U air cooler with 120mm fan(s) would likely be both easier to design and more useful for most users...
-
This is really, really, really great. Is it a complete new development or an adapter for existing cooling solutions for e.g. LGA-2066 water cooling sets?
Anyway, I really appreciate your work.
I am looking forward to your OpenPOWER store (https://store.vikings.net/openpower).
-
Instead of using radiator fans on the water cooling circuit, are there any external chiller or heat pump options that could reduce overall heat emission, even if they don't reduce net noise emissions a lot? This type of option requires a way to dump the heat, much the same way a split-system air conditioner moves heat to the outdoor component.
-
Here's a draft for a POWER9 water cooler. The base plate is made of nickel-plated copper.
(https://i.ibb.co/DWFF6kz/vikings-cooler.jpg)
-
Looks really interesting.
Will the connections rotate? It could be helpful for getting the best pipe routing.
I had been looking for companies that do this, I found this company, Optimus Water Cooling (https://optimuspc.com/pages/company) are close to Raptor in Chicago.
Then there is Ice Giant (https://www.icegiantcooling.com/) in Texas, maybe not far from the location where the motherboards are manufactured.
-
Looks really interesting.
Will the connections rotate? It could be helpful for getting the best pipe routing.
Generally that's a feature of the fitting installed, not so much the cooler. Since standard dimensions are used, the cooler would be interoperable with other standard cooling systems. The top fittings in the draft are an example of rotating fittings.
We're also checking if this would work for 1U machines - would be a huge advantage for anyone looking for hosting these machines in a data center where rack space is expensive.
I had been looking for companies that do this, I found this company, Optimus Water Cooling (https://optimuspc.com/pages/company) are close to Raptor in Chicago.
Then there is Ice Giant (https://www.icegiantcooling.com/) in Texas, maybe not far from the location where the motherboards are manufactured.
While it is generally interesting to check all options available, we're based in Germany so it makes more sense to look for a partner on this side of the pond, also keeping indirect costs in check and potential legal issues manageable.
-
I use a lot of German products without any hesitation and given my location, I'd probably try your product first.
Some of the US users may prefer a supplier local to them, it could also be good publicity for the platform if some of these other vendors list Raptor and POWER9 on their web sites.
Personally, I've found that the dual POWER9 system is so fast that my projects are finished compiling in less than 30 seconds. In that time period, the CPU doesn't get hot enough for the fans to make a crazy noise.
That said, as I put more jobs on the workstation, I suspect I'll need the watercooling.
I went looking for more adventurous solutions too, for example, external chillers and heat exchangers. This could be interesting for people who can dump the heat into a thermal store. For example, you can basically rack a couple of Talos II servers in your basement and 90% of the electricity will heat water for your shower.
https://koolance.com/cooling-systems-liquid-to-liquid
https://www.performance-pcs.com/water-cooling/water-chillers/koolance-exc-450-ultra-compact-450w-recirculating-liquid-chiller-exc-450.html
-
Some of the US users may prefer a supplier local to them, it could also be good publicity for the platform if some of these other vendors list Raptor and POWER9 on their web sites.
We can wholesale this system to an U.S. location on request. Not sure if RaptorCS would be interested.
We really chose the low-cost route here to check out demand. We're using an existing high-quality cooler (with different connectors where needed) with a custom-made high-pressure mount. If there is a market big enough, we'll invest in making a completely new cooler for 1U solutions (this only would fit 2U).
Personally, I've found that the dual POWER9 system is so fast that my projects are finished compiling in less than 30 seconds. In that time period, the CPU doesn't get hot enough for the fans to make a crazy noise.
There are use-cases that require a more quiet setup, of course if you don't need it like with your use-case there's no need to spend extra $ for a water cooler.
I found the IBM HSFs quiet enough when idle as well, for everything else, like office or living room situations, those HSFs are unusable IMO.
The reason for using water cooling for POWER in a data center, at least for us, is space and using a water/glycol heat exchanger for heating nearby offices, especially with energy being very expensive in our area.
I went looking for more adventurous solutions too, for example, external chillers and heat exchangers. This could be interesting for people who can dump the heat into a thermal store. For example, you can basically rack a couple of Talos II servers in your basement and 90% of the electricity will heat water for your shower.
This could be a fun side project indeed! In order for this to be economically feasible I'd think you'd have to have more than one computer though :)
-
I found the IBM HSFs quiet enough when idle as well, for everything else, like office or living room situations, those HSFs are unusable IMO.
I agree a lot of people will have trouble with the HSFs
Another thing that comes to my mind is the performance of the case fans: if somebody has some very large and quiet case fans, e.g. 140mm and they move a lot of air across the HSF will the motherboard fan logic be able to reduce the HSF speeds?
Looking at the Noctua fans, for example, they are offering fans with different maximum speeds, for example, 800, 1200, 1500, I even saw one with 3000. They will be noisy as well but will the overall noise be lower?
-
...
We're using an existing high-quality cooler (with different connectors where needed) with a custom-made high-pressure mount.
...
Hi, can I by one of these blocks with a custom mount now?
Do you ship to Norway?
I am more or less going "all out" with this build. And custom water cooling loops is on the list of what I want to do :-)
-
At risk of going off-topic, I'd like to emphasize that water-cooling is only 50% of the problem when house-training the POWER9
The thread about powering-down unused cores (https://forums.raptorcs.com/index.php/topic,135.0.html) is equally important
The CPU cores, especially in a dual-CPU configuration, using over 100W when idle, will waste electricity and generate unwanted heat.
Electric heating is particularly inefficient, 60-70% of electricity is lost in the energy network. The only exception to this is the use of a local generator (renewable, cogeneration, etc), as it is generated locally, very little is lost in transmission lines.
In my own case, I'm using solar PV and the excess heat is therefore useful and efficient for 7-8 months of the year, it simply offsets the underfloor heating.
But for most people this could be expensive, wasteful and uncomfortable.
Solving both of these issues together (the water cooling and making some cores sleep when idle) will eliminate most concerns from buyers.
-
The lack of STR is a big one, IMO.
-
Hello there,
We were finally able to give this a quick run today and prepare some comparisons :)
Specs: Talos 2 with 1x 02CY296 (22-core, 88 threads)
Setup 1: HSF, 3U IBM HSF
Setup 2: Water cooling system heatsink, 120mm / 4.7″ radiator, 120mm / 4,7″ PWM fan, Laing DC pump, compression fittings and TPV (EPDM/PP) tubes.
Setup 3: Identical to setup 2, with MX5 thermal paste
Stressing the CPU: $ sudo sysbench --test=cpu --cpu-max-prime=990000000 --threads=90 run
We stressed each setup for 15 minutes / 9e+11 nanoseconds, both fans (HSF and the fan attached to the radiator) are PWM fans, controlled by Talos 2.
Findings:
Compared to setup 1, setup 2 showed similar core temperatures, though quieter. Which makes sense since this mechanism can't deliver the high pressure of 90 kg (200 lbs) the IBM HSF can.
Setup 3 showed the best results which were roughly 10 °C / 50 °F below the present temperatures at the 15 minute / / 9e+11 nanoseconds mark and roughly 20 °C / 68 °F below the "highest" temperature of setup 1 and 2. Quite honestly this is as positive surprise.
I'd expect even better results with a larger 240mm / 9.45″ radiator.
Unfortunately, the upper plate of the mechanism had warped during the first attempts (which ist a quality issue and shouldn't have happened) and the mechanism as a whole is too tight against the compression fittings (perhaps that's not an issue with angled fittings). It's easy to overtighten the screw of the mechanism because it's unclear how much force is necessary to hold the heat sink in place and provide enough pressure for a sufficient cooling result.
All in all this seems to be very usable in principle, but there are some improvements to be made before we're happy enough with the quality and if we want to use standard components (esp. a standard water cooled heat sink) that doesn't break the bank.
(https://i.ibb.co/JvmT67f/P9-XPX1-UPro.jpg) (https://ibb.co/JvmT67f)
(https://i.ibb.co/M5HQqMW/IMG-20210622-155612.jpg) (https://ibb.co/M5HQqMW)
(https://i.ibb.co/0q7fqxw/IMG-20210622-161120.jpg) (https://ibb.co/0q7fqxw)
(https://i.ibb.co/FmrSV3z/IMG-20210622-155320.jpg) (https://ibb.co/FmrSV3z)
(https://i.ibb.co/nrs9DvN/IMG-20210622-155346.jpg) (https://ibb.co/nrs9DvN)
(https://i.ibb.co/VBFJZh6/watercooled-HS-thermal-paste.png) (https://ibb.co/VBFJZh6) (https://i.ibb.co/6X8pskj/watercooled-HS.png) (https://ibb.co/6X8pskj) (https://i.ibb.co/nRmGc6X/3U-HSF.png) (https://ibb.co/nRmGc6X)
From left to right: watercooled with thermal paste / watercooled without thermal paste / IBM HSF
-
Nice temperature reduction! I'm pretty confident all those problems are solvable.
-
Also, what fluid did you use?
-
Nice temperature reduction! I'm pretty confident all those problems are solvable.
I'm pretty confident as well, the manufacturer has been helpful so far.
Also, what fluid did you use?
We use DM or osmosis water (better avoid distilled water from the supermarket, it's often a bit on the acidic side) and a glycol without additives somewhere around a ratio of 1:20 (or even less), I've never ran into problems with that but this also depends on what tubing you use (PVC tubing can be problematic with very high glycol contents). There are also ready-to-use solutions available for PC watercooling enthusiasts, AFAIK most of those also use glycol/water but are more expensive, some contain glutaral (even though very little) which I would be a bit scared of to be honest. Correct me if I'm wrong :)
Cooling solutions that contains glycol must be disposed of properly. In Germany for example municipalities accept it free of charge as hazardous waste in small quantities.
-
Glide, O sparkling waves and murmur softly...
We now have a second revision of our improvized mounting mechanism in place, made of stainless steel and a couple of other small improvements. The mounting mechanism is in production and will be available in approx. 2 to 4 weeks together with a water cooling kit for all RaptorCS mainboards. We'll also make it available with just the cooler (because the mechanism is made for a specific cooler), just in case people want to use existing pumps/radiators.
I'll send another update here once we have them in stock.
-
Are you planning to sell fluid as well, or if not, what's your specific recommendation? How good is the mounting pressure on these?
-
Are you planning to sell fluid as well, or if not,
Yes, though it should be pretty easy to make a solution yourself.
what's your specific recommendation?
Please see reply #20 from a while ago above, perhaps you've missed that.
How good is the mounting pressure on these?
Hand-tight.
We try to prevent over-tightening by choosing a short screw correspondingly, so that it shouldn't be possible to tighten it too much. The mounting mechanism is somewhat improvised to be low-cost, so high pressure, e.g. with IBM's 90kg high-pressure system, would bend the mechanism and probably destroy the water cooler as well.
-
No, I saw that; just checking it didn't change. Thanks!
-
No, I saw that; just checking it didn't change. Thanks!
What changed in fact is that this isn't a high-pressure mount. This was unrealistic for our budget because it would've required a custom made water cooler. I would've preferred to do just that, but... Perhaps next year :-)