I can't disagree tbh. I have some experience with the new Radeon pro R9700s, and they can be had for about the same price with 32gb of vram.
Something that's really important on nvidia systems with both vllm and llama-cpp is pcie bus transfer speeds when using tensor parallelism. Id suggest a host system that libnccl supports. That means if you run multiple cards, that the pcie slots share the same root complex. That means a "good" cpu and motherboard. Many modern consumer processors simply don't have enough pcie lanes and have to multiplex lanes and lbnccl can't build a usable transport (or it's limited) becoming your bottleneck. Nvidia also has a bunch of strange kernel apis like peermem and uvm and they all apply to different host systems.
Finding that out myself with dual processor systems. My numbers are still amazing for only having ~1gb/s link between cards with no nccl support.
ChipTuner
npub1qd…3fqm7
2026-09-12 15:46:25 UTC
in reply to nevent1q…l860
Author Public Key
npub1qdjn8j4gwgmkj3k5un775nq6q3q7mguv5tvajstmkdsqdja2havq03fqm7Seen on
wss://nos.lolPublished at
2026-09-12 15:46:25 UTCEvent JSON
{
"id": "6c4d66e94a526e91fe0c70c0b855bdd5815d8eb144a12b98175e7fb7e5c96373",
"pubkey": "036533caa872376946d4e4fdea4c1a0441eda38ca2d9d9417bb36006cbaabf58",
"created_at": 1789227985,
"kind": 1,
"tags": [
[
"e",
"00007d7e8b498f9f8a32906aa30f800f9942738079df8d0883515c3ce707b1ea",
"wss://relay.primal.net/",
"root"
],
[
"e",
"00001474052a4bf445d984427ca5f13a4dca37d4877b926950375aec1c3dbb8a",
"wss://theforest.nostr1.com/",
"reply",
"e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"
],
[
"p",
"01ddee289b1a2e90874ca3428a7a414764a6cad1abfaa985c201e7aada16d38c"
],
[
"p",
"e2ccf7cf20403f3f2a4a55b328f0de3be38558a7d5f33632fdaaefc726c1c8eb"
]
],
"content": "I can't disagree tbh. I have some experience with the new Radeon pro R9700s, and they can be had for about the same price with 32gb of vram. \n\nSomething that's really important on nvidia systems with both vllm and llama-cpp is pcie bus transfer speeds when using tensor parallelism. Id suggest a host system that libnccl supports. That means if you run multiple cards, that the pcie slots share the same root complex. That means a \"good\" cpu and motherboard. Many modern consumer processors simply don't have enough pcie lanes and have to multiplex lanes and lbnccl can't build a usable transport (or it's limited) becoming your bottleneck. Nvidia also has a bunch of strange kernel apis like peermem and uvm and they all apply to different host systems. \n\nFinding that out myself with dual processor systems. My numbers are still amazing for only having ~1gb/s link between cards with no nccl support. ",
"sig": "2bdfa429f771fc06eeaf428e2a37e095db446f456f3d823856d7f047c0228784e75e5caf2b062b0be08f896f43b37f21e0fa794f3c035bd62ee7af140dc8cad4"
}