AMD GPU for local inference

John6666
Sun, 31 Aug 2025 00:04:08 +0000

a 7900XT 24GB or a 9070 16GB

Both GPUs seem to have decent ROCm support. For generative AI tasks, while clock speed matters, insufficient VRAM makes running things difficult in the first place. Since VRAM is generally more critical, the 24GB model would likely be easier to work with.

When using ROCm-compatible GPUs on Linux, installing PyTorch with ROCm support (basically, most Hugging Face libraries work if PyTorch works) seems to allow usage with minimal code changes.

For non-Hugging Face tools, using vLLM and Ollama for LLMs, or ComfyUI and A1111 WebUI for image generation, should also work without issues.
Fine-tuning LLMs should also be possible.

However, I currently don’t own any AMD GPUs other than the APU, so please take this as general advice… Also, while it’s not much of an issue on Linux, Windows often has various incompatibilities. This is true even with nVidia. Since you’re using Linux, you’ll probably be fine.