anyone running local LLMs?

[power9mm]

Looking for advice on what hardware I should go with if I wanted to run a local llm on raptor hardware. Would an 8 core blackbird with a 48gb vram gpu be sufficient? or should I be considering one of the talos 2 options..? unsure how ROCm support is or if I should consider nvidia for the computer drives.

[atomicdog]

I don't think NVIDIA supports Power anymore.
I have an NVIDIA A2 and I tried to get it working but the last supported CUDA version is very old.
I tried to get Claude Code to port it over to the newer one but it was so much different it would require a whole lot of work.

[MPC7500]

You could give this a try: https://github.com/Scottcjn/nvidia-power8-patches

[power9mm]

You could give this a try: https://github.com/Scottcjn/nvidia-power8-patches

Yeah I saw that in the other thread, I suppose I'll have to give it a go if I get another power9 system before any new systems roll out whenever that will be but I don't think it'll be for awhile.. Surprised there hasn't been much interest running it on open hardware. How does claude and other LLMs work on POWER?

[atomicdog]

Claude Code doesn't run on POWER, you need to use SSH or NFS from an x64.

[daedalao]

Could've fooled me, I've been running claude code on power.

[rheaplex]

From memory, llama.cpp with Vulkan should work with Nvidia GPUs but I haven't tried that recently.

[DKnoto]

Looking for advice on what hardware I should go with if I wanted to run a local llm on raptor hardware. Would an 8 core blackbird with a 48gb vram gpu be sufficient? or should I be considering one of the talos 2 options..? unsure how ROCm support is or if I should consider nvidia for the computer drives.

I ran a few models as a test without GPU support on my machine; see the footer for details. It works quite well—models with ~32B parameters in MoE architecture, such as Qwen_Qwen3.6-35B-A3B-Q8_0.gguf, run at a speed of about 8 tokens per second. They take up about 36 GB of memory.

[r34per]

I run llama.cpp on my power8 server with a radeon v340L(technically 2 gpu's on one card but has a total vram of 16gb) using vulkan. It averages 10-12 tokens/sec, I tend to use smaller models that fit all in vram though