anyone running local LLMs?

[power9mm]

Looking for advice on what hardware I should go with if I wanted to run a local llm on raptor hardware. Would an 8 core blackbird with a 48gb vram gpu be sufficient? or should I be considering one of the talos 2 options..? unsure how ROCm support is or if I should consider nvidia for the computer drives.

[atomicdog]

I don't think NVIDIA supports Power anymore.
I have an NVIDIA A2 and I tried to get it working but the last supported CUDA version is very old.
I tried to get Claude Code to port it over to the newer one but it was so much different it would require a whole lot of work.

[MPC7500]

You could give this a try: https://github.com/Scottcjn/nvidia-power8-patches

[power9mm]

You could give this a try: https://github.com/Scottcjn/nvidia-power8-patches

Yeah I saw that in the other thread, I suppose I'll have to give it a go if I get another power9 system before any new systems roll out whenever that will be but I don't think it'll be for awhile.. Surprised there hasn't been much interest running it on open hardware. How does claude and other LLMs work on POWER?

[atomicdog]

Claude Code doesn't run on POWER, you need to use SSH or NFS from an x64.