RTX 3080 20GB got reBAR and P2P unlock! +70% prompt processing speed
/u/No-Refrigerator-1672
Abstract
P2P is finally unlocked for RTX 3080 20GB - this leads to massive performance improvements in tensor parallel mode prompt processing speed. First, i want to acknowledge that this tremendous achievement belongs to u/nathannguyennng - he can't post to r/locallama due to karma requirements, and gave me permission to post it instead. Please follow his original post for installation instructions, and give it an upvote! Here, I'm sharing my tests results. Test system: Proxmox Virtual Environment 9.2.21, linux kernel 6.14.11-9-pve, nvidia driver 615.71.09 Hardware: Threadripper 1920x on x399 Zenith extreme motherboard GPUs: dual RTX 3080 20GB, both installed into PCIe 3.0 x16 slots, limited to 240W power test environment: vllm 0.32.0 with cyankiwi/Swift-1.5-Qwen3.8-27B-AWQ-INT4 , MTP enabled, FP8 KV cache, tensor parallel, full launch command: vllm serve cyankiwi/Swift-1.5-Qwen3.8-27B-AWQ-INT4 --max-num-seqs 4 --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --tensor-parallel-size 2 --enable-prefix-caching --kv-cache-memory-bytes 6G --kv-cache-dtype fp8_e4m3 --enable-prompt-tokens-details --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":3}' --enable-chunked-prefill --attention-backend FLASHINFER --default-chat-template-kwargs '{"reasoning_effort": "medium"}' Both GPUs are limited to 240W of power. I have tested 3 configurations: 1) completely stock; 2) gpu-unlock ; 3) gpu-unlock + ReBAR + P2P Single stream prefill (prompt processing) speed was measured as prompt_length / time_to_first_token. In both cases, gain is defined as percentage increase over stock configuration. Prompt Tokens TTFT stock ms TTFT gpu_unlock ms TTFT gpu_unlock+p2p ms Prefill stock tok/s Prefill gpu_unlock tok/s Gain Prefill gpu_unlock+p2p tok/s Gain 8192 8471.2 7012.7 3938.5 967.1 1168.2 21% 2080 115% 16384 17572.5 14417.8 8283.4 932.4 1136.4 22% 1978 112% 32768 36753.8 29970 18179.3 891.6 1093.4 23% 1802.5 102% 65536 79432 64173.5 41010.9 825.1 1021.2 24% 1598 94% 131072 183593.1 146010 100237.2 713.9 897.7 26% 1307.6 83% 261900 465890.2 362192.5 272345.5 562.1 723.1 29% 961.6 71% As you can see, enabling both gpu_unlock and P2P gains you massive prompt processing improvement, anywhere between 70% and 115%. P2P alone gives up to 70% improvement over gpu_unlock. Measuring decode (token generation) when MTP enabled is tricky, because tok/s output metric relies heavy on MTP acceptance rate, which always changes. Therefore, instead of tok/s output rate, I measure inter-token latency (ITL) - this is time passed between two batches of tokens generated by inference engine. With MTP3, each batch contains anywhere between 1 and 4 tokens; so you token output rate is 1x to 4x the step rate I report. Prompt Tokens ITL Stock ms ITL gpu_unlock ms ITL gpu_unlock+p2p ms Decode stock step/s Decode gpu_unlock step/s Gain Decode gpu_unlock+p2p step/s Gain 8192 64.8057 62.5193 62.5695 15.43074 15.99506 4% 15.98223 4% 16384 62.9359 63.873 59.8 15.88918 15.65607 -1% 16.72241 5% 32768 60.2028 62.016 62.6955 16.61052 16.12487 -3% 15.95011 -4% 65536 66.0282 61.425 64.8648 15.14504 16.28002 7% 15.41668 2% 131072 62.5518 64.4644 60.561 15.98675 15.51244 -3% 16.51228 3% 261900 66.7368 60.26 60.3504 14.98424 16.59476 11% 16.5699 11% As real-life MTP acceptance length for agentic coding is floating around 2, 15 steps/s decode rate corresponds to 45 tok/s generation. I'm sorry if this sounds confusing, but this is the only way for you can isolate variations in MTP acceptance during tests and extract raw performance numbers. As you can see, decode results are all over the place, but within noise, except 261k long prompt test; in which stabl;e measurable performance is delivered by gpu_unlock, not p2p. Verdict: in tensor parallel mode, both recent unlocks for 3080 20GB give unimaginably large prompt processing speedup, but basically no improvement for token generation. So, if you're running long prompt tasks (that includes agentic coding too), you should check out those prompts. Also, expect far lesser improvements on systems that don't have full PCIe connectivity to the second gpu. Again, I want to express my massive thanks to u/nathannguyennng , author of ReBAR and P2P unlock - what a day for 3080 owners! submitted by /u/No-Refrigerator-1672 to r/LocalLLaMA [link] [comments]