Abstract
v0.31.0 Highlights This release features 717 commits from 307 contributors (96 new)! DeepSeek-V4.1-Flash performance : FlashMLA mega attention with the V4.1 NVFP4 compressed KV cache is now the SM100 default ( #56935 ); DeepGEMM sparse MQA logits for the indexer ( #56254 ) and Mega-Gate fusing the gate GEMM with expert selection ( #56266 ); decoder boundaries fuse the TP all-reduce, mHC input preparation ( #57643 ) and the MoE finalize ( #58586 ); a fused small-batch WO-A with inverse RoPE and MXFP8 quant on SM100/SM103 ( #58634 ); the MXFP8 wo_b GEMM fused with the sequence-parallel reduce-scatter ( #57428 ); Engram wkv sharded across TP ranks ( #58678 ) and Engram host tables shared across co-located DP replicas by default ( #57651 ); encoder CUDA graphs for the vision tower ( #56625 ); and SWA bounded replay that keeps the sliding-window KV out of prefix caching ( #56227 ). Fast restart : the new vllm preload CLI launches the weight-cache daemon that keeps post-quantized weights resident in GPU memory across engine restarts ( #56680 ), now with data parallelism ( #57386 ), MTP draft models ( #57312 ), a /health endpoint ( #58552 ) and a readiness wait ( #58370 ). Experimental initialized-engine snapshots ( vllm snapshot create/restore ) use CRIU to restore a fully initialized TP1 engine ( #51360 ). Model Runner V2 and speculative decoding : draft-model speculative decoding ( #43091 ) and custom logits processors ( #56497 ) on Model Runner V2; the new LiLiCorr drafter ( #57934 ); async scheduling for DFlash ( #58065 ) with the context K/V precompute captured in the draft CUDA graph ( #57632 ); DSpark adaptive verification for Gemma4 ( #57263 ) and variable-length decode for Kimi-K3 ( #52988 ); a DCP target with a non-DCP DSpark draft ( #56723 ); and MoE memory now counted during MRV2 profiling, avoiding OOMs on WideEP deployments ( #57270 , #58411 ). Large scale serving : MoonEP balanced EP all2all backend via --all2all-backend moonep ( #52101 ), prefill context parallelism with data parallelism ( #57075 ), a low-SM multimem reduce-scatter for SM100/SM103 ( #55072 ), DeepEPv2 with sequence parallelism ( #57210 ) and EPLB with shared-expert overlap ( #57236 ), a sharding-aware NCCL M2N weight-transfer backend for RL ( #51520 ), and KV offloading back-pressure detection ( #50045 ). Scheduling controls : --max-num-active-seqs caps RUNNING admission independently of max_num_seqs ( #56758 ), --long-prefill-token-threshold now adapts to the number of waiting prefills instead of chunking a lone request ( #57951 , #58459 ), the waiting queue was reworked so requests already holding KV blocks are scheduled first ( #58947 ), and the KV connector + MTP deadlock under KV pressure was fixed ( #57104 ). HiSparse hardening : MTP verification rows resolved with a union residency kernel ( #59235 ), MTP acceptance collapse under FULL graphs fixed ( #59309 ), no GPU pages without host backing ( #59036 ), a chunked-prefill preemption livelock fixed ( #59494 ), KV cache sized from the groups HiSparse allocates ( #59450 ), and host prefix publication and GPU prefix adoption fixes ( #59007 , #59282 ). Security : per-request mm_processor_kwargs and media_io_kwargs are rejected unless --trust-request-mm-kwargs is set ( #58830 ); prefix-cache extra keys are tagged by source so a LoRA name and a cache_salt can no longer collide ( #51899 ), and the LoRA path is part of the block hash ( #59335 ); stale multimodal receiver-cache entries can no longer replace fresh payloads ( #57833 ). Breaking changes : per-request multimodal kwargs gated ( #58830 ); tokenizer_mode="slow" removed ( #58545 ); --enable-mamba-fine-grained-prefix-cache renamed to --enable-mamba-shared-prefix-checkpoint ( #57382 ); online quantization through quantization="fp8" replaced by the fp8_per_tensor shorthand ( #53585 ) and Quark silent online quantization removed ( #51800 ); the AllSpark INT8 W8A16 backend removed ( #58001 ); --enforce-eager now also disables JIT kernel warmup ( #58197 ); XPU graphs enabled by default with VLLM_XPU_ENABLE_XPU_GRAPH removed ( #51600 ). Release Artifacts Python Wheels Platform Install PyPI (CUDA 13.0) pip install vllm PyPI (CUDA 13.0, uv) uv pip install vllm --torch-backend=auto ROCm pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.31.0/rocm723 XPU uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.31.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match Docker Images Platform Docker Image CUDA 13.0 (Default) docker pull vllm/vllm-openai:v0.31.0 CUDA 12.9 docker pull vllm/vllm-openai:v0.31.0-cu129 ROCm docker pull vllm/vllm-openai-rocm:v0.31.0 CPU docker pull vllm/vllm-openai-cpu:v0.31.0 XPU docker pull vllm/vllm-openai-xpu:v0.31.0 Other Artifacts Pre-built release artifacts are available in the Assets section at the bottom of this page, including: Source distribution tarball CUDA 12.9 Python wheels for x86_64 and arm64 CUDA 13.0 Python wheels for x86_64 and arm64 CPU Python wheels for x86_64, arm64, and macOS XPU Python wheel for x86_64 Model Support New model capabilities : DiffusionGemma structured generation mode with bounded single-token choices ( #57250 ), MiMo V2 MXFP4 MoE, BF16 MoE router and DFlash drafts ( #57784 ), Cohere2MoE auxiliary hidden states for EAGLE3/DFlash drafters ( #49819 ), GLM-5.2-MXFP4 on the ROCm DeepSeek-V3.2 path ( #51915 ), AMD-Quark mixed-precision DeepSeek-V4.1-Flash-MXFP4 ( #57071 ) and GLM-5.3-Flash Quark MXFP4 ( #56176 ) checkpoints, and a built-in granite_thinking_parser for Granite 4.2 ( #55957 ). DeepSeek-V4.1-Flash : FlashMLA mega attention with the NVFP4 compressed KV cache as the SM100 default ( #56935 ), DeepGEMM sparse MQA logits in the indexer ( #56254 ), Mega-Gate ( #56266 ), fused TP all-reduce + mHC input preparation ( #57643 ) with the MoE finalize folded in ( #58586 ), fused small-batch WO-A ( #58634 ), MXFP8 wo_b GEMM + reduce-scatter ( #57428 ), overlapped mHC coefficients for small TP batches ( #57603 ) limited to FULL CUDA graphs ( #57874 ), Engram wkv sharded across TP ( #58678 ), serialized Engram lookups with huge-page host tables ( #56926 ), Engram tables shared across DP replicas by default ( #57651 , #57914 , #59068 ), native shared-expert MegaMoE fusion without padding ( #56568 , #57204 ), faster MegaMoE staging and NVFP4 cache gathers ( #57604 ), the fused query RMSNorm + MXFP8 path restored ( #57679 ), KV-only DSpark context insertion ( #56441 ), vision encoder CUDA graphs ( #56625 , #58499 ), SWA bounded replay ( #56227 ), causal image SWA restored to match the reference ( #57152 ), updated reasoning effort mappings ( #58316 ), and startup fixes for NaN-scored candidate blocks ( #57454 ), DeepSelect sentinels ( #58215 ), DSpark non-causal attention on FlashInfer ( #57432 ), runtime JIT of offset candidate buffers ( #57667 ) and imports without Triton ( #57654 ). DeepSeek V4 : FlashInfer sparse MLA with fused inverse RoPE + FP8 quant ( #58621 ), stacked DSpark context WKV projections ( #54674 ), fused MoE expert distribution computed from the EP group ( #57465 ), FIM completion via suffix ( #44229 ), missing string= in tool calls parsed ( #56271 ), request tools attached to an existing system message ( #51856 ), image block spacing preserved ( #56882 ), and DSpark width separated from MTP stage validation ( #54631 ). GLM-5.3-Flash : opt-in FlashAttention and FlashMLA sparse backends on SM90 ( #55385 ), NoPE sparse MLA on the FlashInfer SM120 backend ( #55277 ), cooperative top-k for small decode batches ( #57327 ), indexer decode workspace sized by pooled length (3 GiB saved, #57701 ), metadata ops 1.6-4.8x faster ( #58450 ), fused kpool tail slot mapping ( #57534 ), kpool top-k through the shared dispatcher ( #57546 ), lower sparse MLA preparation overhead ( #57458 ), no D2H sync in the SM90 sparse MLA plan under async scheduling ( #58684 ), FlashKDA keeping the recurrent state in FP32 for long prefills ( #58846 ), and correctness fixes for kpool corruption with speculative decoding ( #58454 ), SM90 index_kpool mismatch ( #58704 ), kpool tail strides ( #57477 ), 500k-token prompts ( #57317 ), dense MLP layers under sequence parallelism ( #58061 ), indexer top-k backend selection ( #58594 ) and varlen paged MQA launches ( #55270 ). Qwen3.8-Flash-Next (Qwen4Exp) : FP8 main KV cache on the QSA path ( #55557 ), FP8 TP with FlashInfer TRTLLM MoE ( #55867 ), SM90 QSA tuning ( #57273 ), fused HC down projection + SiLU ( #58957 ), lower PLE metadata overhead ( #58114 ), QSA indexer workspace fragmentation fixed ( #57105 ), profiling KV cache released ( #58961 ), pinned PLE prefetch ids kept out of the CUDA graph pool ( #58489 ), and indexed expert-mapping lookups saving about 25 s of weight loading on DGX Spark ( #58720 ). Kimi K3 and MiniMax-M3 : Kimi-K3 vision patch embedder as a GEMM ( #58527 ), fused KimiViT QK RoPE (up to 29x, #58651 ), routed expert quantization ( #57430 ), reasoning parser ( #57098 ) and reasoning token counting ( #58372 ) fixes, DSpark context KV pointers refreshed after re-binding ( #58814 ), stateless first chunks no longer classified as decodes ( #51483 ), and Mamba block estimates that no longer stall admission on external prefix hits ( #57050 ); MiniMax-M3 encoder CUDA graphs ( #58673 ), Conv3dLayer patch embedding (about 62x, #58512 ), triton_mrope in the vision tower ( #58526 ), fused MiniMax2 routing with non-unit scaling ( #58880 ), and processor fixes ( #58460 , #59613 ). DiffusionGemma : one-pass sampler statistics kernel ( #58226 ), diffusion_constrained reads over logprob_token_ids (about 25% faster, #58216 ), fewer logit rows for prefill-only batches ( #57416 ), logprob_token_ids support ( #57417 ), and fixes for concurrent logprobs ( #57414 ), eager fallback dtype ( #57462 ), multimodal inputs ( #57589 ), quantized LM heads ( #48521 ) and CPU execution ( #58964 ). Multimodal : Triton mm_input_norm kernel ( #56798 , #56711 ), raw pixels kept through the DP-sharded ViT path ( #56872 ), Qwen2.5-VL video fps honored for temporal M-RoPE ( #47736 ), Whisper and Qwen2-Audio clips longer than 30s ( #57769 , #56912 ), Mistral3 placeholder grid ( #53758 ), Idefics3 unsplit patches ( #48760 ), Ovis2.5 tokens ( #52623 ), Aria expert weights ( #57487 ), MiMo-V2.5 fused FP8 qkv_proj sharding ( #57508 ), Gemma4 AutoWeightsLoader ( #55911 ) and buffer scalars ( #54213 ), Gemma4 FP8 KV with FA4 at head dim 512 ( #53175 ), Laguna RoPE ( #57189 ), Mistral-Large-3 accuracy regression ( #57563 ), malformed EXIF ( #56527 , #57234 ), JinaVL labels ( #57347 ), Sarvam MLA routing with FP32 router logits ( #56034 ), fused CohereASR attention scores ( #55190 ), and fused DFlash2 grouped convolution ( #55960 ). LoRA : Nemotron VL language models ( #56231 ), ModernBert ( #57148 ), VoyageQwen3 embeddings ( #57708 ), RoBERTa sequence classification ( #58884 ), and modules_to_save sequence-classification heads ( #53555 ) with per-adapter num_labels ( #57766 ). Engine Core Model Runner V2 : draft-model speculative decoding ( #43091 ), custom logits processors ( #56497 ) validated at admission ( #57728 ), randomized dummy inputs ( #58411 ), dummy tokens routed to MoE experts during profiling ( #57270 ), FULL decode graphs for one-token prompt tails ( #58400 ), token-to-request mappings shared ( #57102 ) and Mamba/GDN metadata reused across KV cache groups ( #58762 ), encoder-only ViT CUDA graphs ( #56922 ), weight offloader ( #57834 ) and pooling model ( #57737 ) released on shutdown, and fixes for multi-layer MTP KV in P/D ( #55055 ), fast-prefill with LoRA ( #56456 ), stale block-table writes from dummy draft steps ( #56734 ), padded prompt tails in hybrid models ( #58434 ), never-proposed draft slots ( #58784 ), auto-fit max_model_len ( #58149 ) and intermediate_tensors during capture ( #57745 ). Speculative decoding : LiLiCorr drafter ( #57934 ), DFlash async scheduling ( #58065 ) and context K/V in the draft CUDA graph ( #57632 ), Gemma4 DSpark adaptive verification ( #57263 ), Kimi-K3 variable-length adaptive verification ( #52988 ), CPU-GPU sync removed for heterogeneous vocabularies ( #57396 ), Triton recompiles avoided in the acceptance estimator ( #57107 ), and fixes for EAGLE/dense drafts with EP ( #56930 ), DFlash/DSpark profiling batches ( #56448 ), GLM MTP head memory ( #55442 ), prompt embeddings with drafts ( #57356 ), and TritonMLA causal multi-token decode ( #51065 ). Scheduler : --max-num-active-seqs ( #56758 ), adaptive --long-prefill-token-threshold ( #57951 , #58459 ), skipped_waiting replaced by a KV-holding waiting queue ( #58947 ), atomic admission of n > 1 requests ( #53936 ), KV connector + MTP deadlock ( #57104 ), a throughput cliff when max_num_seqs is not a multiple of 8 ( #57355 ), streaming continuations keeping logprobs and refreshing max tokens ( #57447 , #57676 ), a resumable request + async scheduling race ( #58259 ), and a non-blocking structured-output grammar poll ( #55931 ). Prefix caching and hybrid models : extra keys tagged by source ( #51899 ) and LoRA paths hashed ( #59335 ), --enable-mamba-shared-prefix-checkpoint ( #57382 ), prompt-tail hits with MTP restored ( #58368 ), prompt-end checkpoints kept under sparse retention ( #59146 ), align-mode checkpoint reservation ( #59175 ), stateless GDN first chunks ( #51565 ), MTP draft KV cache groups annotated on the hybrid grouping path ( #55390 ), a generalized prefill checkpoint builder ( #57783 ), batched Mamba2 prefill state saves without GPU-CPU syncs ( #49371 ), skip_reading_prefix_cache honored for connector hits ( #57269 ), incremental multimodal block hashing ( #51694 ), a KV block size every attention backend supports ( #49845 ) with clearer errors ( #58557 ), and CacheConfig.effective_attention_block_size for DCP ( #56538 ). Startup and memory : parallel Triton warmup compilation ( #58582 ), JIT warmup disabled under --enforce-eager ( #58197 ) unless fault tolerance is on ( #58593 ), DeepGEMM warmup reusing the MoE workspace ( #57268 ), allocator fragmentation no longer shrinking the KV cache during profiling ( #58430 ), sparse prefill buffers reserved before KV sizing ( #57575 ), stale FlashMLA workspace views released ( #56902 ), DeepGEMM FP8 workspace halved ( #53914 ), shared Marlin/Humming workspaces ( #57421 ), FlashInfer BF16 MoE weights converted in place ( #54699 ), UniProc startup threads bounded by the CPU quota ( #58946 ), and runtime threads set before profiling ( #55891 ). Kernels : sampled filtering for persistent top-k ( #56346 ), Murmur3 RNG for Gumbel sampling ( #51367 ), register-resident per-token-group 8-bit quant ( #55330 ), vectorized per-tensor FP8 abs-max ( #58194 ), a Triton kernel dispatcher for platform-specific overrides ( #43048 ), GateLinear for all MoE models ( #58234 ), non-local expert slots skipped in TritonExperts ( #58051 ), deferred TRT-LLM-Gen top-k finalize on the modular path ( #58635 ), SM120 batch-invariant matmul configs ( #57456 ), FlashInfer prefill dequant scratch bounded ( #57918 ), and fixes for Triton softcap NaNs ( #56579 ), QuIP Hadamard transforms ( #43462 ), CUTLASS FP8 linear on A100 ( #55884 ), FlashInfer sampling on unsupported GPUs ( #48956 ), SM100 FP8 blockwise scale padding ( #57377 ), mixed FULL graph prefill capture ( #58275 ) Inductor custom-op pattern matching ( #58189 ), CuteDSL BF16 GDN prefill diverging from FLA on Qwen3.5 ( #53864 ), SM100 fp8_ds_mla cache scales ( #49435 ), ragged decode batches in the sparse indexer ( #52500 ), stale allowed_token_ids masks after batch reordering ( #48419 , #43931 ), and prompt_embeds tensors held after requests finish ( #57988 ). Batch invariance : breakable CUDA graphs without torch.compile by default under VLLM_BATCH_INVARIANT ( #57586 ), sequence parallelism and async TP disabled ( #56377 ), and NCCL 2.31 collectives kept enabled ( #58179 ). Sleep and RL : release_kv_cache_memory() frees only KV cache memory ( #44890 ), --sleep-preserve-parameter-names retains frozen weights across level-2 sleep ( #57891 ), NCCL M2N weight transfer ( #51520 ), dense DP weight updates by DP index ( #56950 ), allocator config kept when toggling expandable segments ( #57982 ), and cache reset failures propagated during sleep ( #54581 ). Fast restart : vllm preload ( #56680 ), DP ( #57386 ), MTP drafts ( #57312 ), readiness wait ( #58370 ), /health ( #58552 ), ModelOpt MXFP8 pre-processed weights ( #57316 ), and initialized-engine snapshots ( #51360 ). Logging and observability : LoggingConfig via --logging-config and --log-level ( #57205 ), JSON logging fixes ( #57957 , #58747 ), startup log suppression fixed ( #51366 ), unified platform-aware torch profiling ( #57460 ), no 0.0% prefix cache hit rate before any query ( #54990 ), KV transfer metrics formatting ( #57068 ), MFU activation sizing from the model dtype ( #57070 ), platform-overridable env var checks ( #48599 ), VLLM_TARGET_DEVICE=empty pip install vllm for out-of-tree backends ( #41074 ), and the correct vLLM version reported when installing from source ( #57295 , #57744 ). Large Scale Serving MoE communication : MoonEP backend ( #52101 ), DeepEPv2 with sequence parallelism ( #57210 ) and EPLB + shared-expert overlap ( #57236 ), low-SM multimem reduce-scatter ( #55072 ), EPLB load statistics during Elastic EP scaling ( #58473 ), SP padded rows skipped in grouped routing so rank 0 is no longer a prefill straggler ( #56079 ), hash routing rejected on unsupported monolithic backends ( #57867 ), sampled-token broadcasts skipped under PP for requests leaving the engine ( #58542 ), DBO with DeepEP low-latency profiling ( #57502 ), external LB with replicas sharing nodes ( #53743 ), and no blocking RPC during the engine handshake ( #57226 ) or unbounded draft-token waits ( #58779 ). Context parallelism : PCP with DP, EP and MTP ( #57075 ), a DCP target with non-DCP DSpark/DFlash drafts ( #56723 ), DCP sequence lengths without a CPU-GPU sync ( #58169 ), and NIXL DCP pulls across MLA cache regions ( #57389 ). KV connectors : NIXL pipeline-parallel push prefill for attention-HMA ( #50494 ) and packed MLA layouts ( #50499 ), transport-failure metrics split from KV expiry ( #55854 ), dead peer state released without waiting for TTL ( #50047 ), expired leases reaped behind a heartbeated head ( #58292 ), push completion restored ( #58188 ), D-side activity recorded ( #52245 ); Mooncake CUSTOM_MEM_POOL ( #49300 ), request-level load failures under HMA ( #56855 , #57174 ) and bootstrap retries ( #58919 ); MoRIIO hybrid Mamba/KDA state in READ mode ( #51052 ) and a multi-decode routing race ( #51681 ); prefill cache hits in prompt_tokens_details ( #54222 ); KV-event publishers bound at port 0 ( #55844 ); KV cache metadata GET restored for external consumers ( #56925 ); partial-block KV events keep every multimodal feature ( #58288 ); saves finalized on steps without a forward ( #57775 ); abort-safe ExampleHiddenStatesConnector ( #56841 ); DecodeBench FP8 fills ( #58472 ); a KvHints request envelope ( #53423 ); and the AuxOutput connector for routed-expert outputs ( #45635 , #58150 , #58205 ). KV offloading : per-request max_load_tokens ( #55885 ), back-pressure detection ( #50045 ), SimpleCPUOffloadConnector Prometheus metrics ( #57251 ), non-prefix-cacheable ( #56810 ) and scratch ( #57145 ) groups skipped, replicated layouts for multi-group MLA ( #57652 ), regions of 512 GiB or more registered in chunks ( #51081 ), cgroup memory checked before SHM allocation ( #54014 ), and fixes for cache recency ( #51787 ), MTP-retained sliding windows ( #56709 ), canonical MLA rows ( #56799 ), event metadata ( #57453 ) and ROCm pinned memory ( #57160 ). HiSparse : union residency kernel for MTP rows ( #59235 ), MTP acceptance under FULL graphs ( #59309 ), host-backed allocation ( #59036 ), preemption livelock ( #59494 ), KV cache sizing ( #59450 ), host prefix publication ( #59007 ), GPU prefix adoption ( #59282 ), and residency metrics ( #58725 ). Encoder disaggregation (EPD) : dynamic EPD proxy with launcher-managed registration ( #54176 ), image requests batched per encoder ( #57095 ), cross-encoder caching through Mooncake ( #56242 ), metadata-only audio inputs ( #57887 ), language-model shards skipped for --mm-encoder-only ( #58086 ), EC connector metrics ( #54960 ), and encoder-only fixes ( #58490 , #58287 , #57696 ). Hardware & Performance NVIDIA : FlashInfer 0.7.0.post1 ( #58069 , #59323 ), DeepGEMM pin bump with SM120 fixes ( #57218 ), Rubin CUDA 13.4 nightly images ( #55953 ), and QuTLASS builds with PyTorch 2.13 ( #58173 ). AMD ROCm : AITER v0.1.23 ( #56885 , #58867 ), torch 2.13 and Triton 3.8 ( #50605 , #58006 ), triton_kernels 3.8 MXFP4 MoE for gpt-oss and DeepSeek-V4 ( #55934 ), a ROCR host segfault fixed ( #57328 ); DeepSeek-V4/V4.1 HCA dual-stream ( #56853 ) and layer-aware CSA2 overlap ( #57407 ), FP8 wo_a ( #54894 ), inverse RoPE fused into the sparse decode reduce ( #57435 , #57451 ) which now emits MXFP8 for a grouped FP8 wo_a ( #58456 ), MXFP8 GEMM on native 32x32 scales on gfx950 ( #58510 ), reused top-k ragged metadata ( #57434 ), faster candidate block selection ( #58208 ), Engram tables in host memory ( #57491 ) and Qwen3.8-Flash-Next PLE tables offloaded to host memory ( #57497 ), an opt-in AITER ASM decode route for DCP + speculative decoding ( #56861 ), get_top_tokens() on the DeepSeek V4 MTP drafter ( #57568 ), DSpark adaptive verification ( #52362 ), opt-in VLLM_DSV4_LOGITS_FIX for sparse-indexer logits on gfx950/gfx942 ( #50455 ), an accuracy revert of #56433 and #51692 ( #57132 ), and clear errors for FSE with DPA+ETP ( #57919 ); Kimi-K3 a4w4 FlyDSL kernels ( #53940 ) with VLLM_ROCM_USE_AITER_MOE_SITUV2=a16w4|a8w4|a4w4 ( #58201 ), low-concurrency speculative KDA ( #58045 ), sharded latent MoE under EP ( #54956 ) and fewer projection copies ( #50592 ); a ROCm Hy4 path with torch.compile (13x lower decode latency, #57526 ); MiniMax-M3 AITER QK-norm fusion ( #54535 ), copy-free K/V insert ( #56849 ) and MXFP8 fixes ( #53674 , #58089 ); GLM-5.3-Flash boot fixes ( #57192 , #57252 , #57425 ); AITER QuickReduce + RMSNorm ( #48249 ), BF16 AsyncTP ( #58098 ), static FP8 attention output fusion ( #58099 ), QK-norm/RoPE/KV-cache fusion for MRoPE ( #50212 ), AITER GDN decode for flat layouts ( #53623 ), wvSplitK for single-output GEMMs ( #53283 ), 69 fewer copies per decode step on the skinny GEMM path ( #58566 ), tuned GEMM lookup through AITER ( #55001 ), a narrower Triton prefill KV tile on RDNA3/RDNA4 ( #58225 ), staged large pageable H2D copies ( #56343 ), and fixes for MXFP4 MoE padding starving the KV cache ( #56359 ), AITER MLA FP8 prefill OOM ( #57923 ), unquantized cache descales ( #56726 ) and AITER MoE fallbacks ( #56590 , #57866 , #57426 ). VLLM_ROCM_USE_AITER_FP4_ASM_GEMM is restored and off by default ( #57055 ); SWA bounded replay is disabled on ROCm ( #57906 ). Intel XPU : PyTorch 2.14 ( #56013 ), XPU graphs on by default ( #51600 ), EPLB ( #44987 ), int8 W8A8 MoE on Triton ( #53162 ), batch invariance ( #55881 ), SYCL rotary embedding ( #55721 ), fused QK RMSNorm + RoPE + gate (decode region 55% faster, #56096 ), Model Runner V2 sampler ( #57277 ) and PP microbatch control ( #55145 ), --device-ids honored ( #56015 ), and device pointer overflow fixed ( #54514 ). CPU : FP8 W8A8 linear and MoE for Intel Diamond Rapids ( #49942 ), Arm paged attention up to 25% faster ( #56045 ), Zen DA8W4 int4 for dense and MoE layers ( #54024 ), zentorch SDPA for encoder attention ( #54508 ) and MLA prefill ( #54967 ), FP32 attention sinks ( #56252 ), W8A8 INT8 MoE on POWER ( #55316 ), W4A16 Whisper ( #58268 ), wheels built on Ubuntu 22.04 (glibc 2.34) with AMX-FP8 ( #58515 ), AVX10.2 gated on compiler support ( #58133 ), pre-built Triton CPU ( #58140 ), --device-memory-utilization alias ( #56547 ), vLLM Recipes in the CPU image ( #58796 , #57306 ), s390x protobuf pin ( #54978 ) and torchcodec video ( #58693 ), and fixes for Ministral FP8 ( #56985 ), FP32 router weights ( #56168 ), zentorch import failures ( #54923 ), macOS multimodal SHM ( #57142 ), CPU affinity per local rank ( #53636 ) and NIXL GDN state layout ( #53300 ). Quantization Humming : Hadamard transforms and NVFP4/MXFP4/MXFP8 online quantization ( #56685 ), asymmetric wNaM through compressed-tensors ( #46528 ), humming-kernels 0.1.16 ( #58054 ), and Humming in the W4A8 (INT4xFP8) MoE oracle ( #58427 ). New capabilities : native Quark W4A16 INT4/UINT4 exports ( #48606 ), opt-in load-time MXFP4 dequantization ( #50814 ), explicit per-token NVFP4 MoE backends ( #57176 ), and a canonical N-first layout for compressed-tensors WNA16 MoE ( #52798 ). Fixes : MXFP8 on layers below mm_mxfp8 shape limits ( #54223 ), online NVFP4 scales on reload ( #57954 ), LM head linear metadata ( #58444 ), and the fused SiLU-mul block-quant path skipped under a SwiGLU clamp ( #57984 ). API & Frontend New options and endpoints : --tool-strict-level ( #56268 ), response_format with tool_choice=auto ( #56086 ), DeepSeek-V4 FIM completions ( #44229 ), release_kv_cache_memory() and POST /release_kv_cache_memory ( #44890 ), fixed-token prefill scoring via prompt_logprob_token_ids ( #54335 ), per-request speculative decoding metrics in /inference/v1/generate ( #43310 ), per-request metrics ( #55084 ) and cache_write_tokens ( #57222 ) in the Responses API, Anthropic thinking in /v1/messages ( #58613 ), streaming reasoning and tool calls from the derender endpoint ( #50550 ) with offloaded detokenization ( #57528 ), vllm chat streaming thinking output ( #57045 ), request body debug logging with --enable-log-requests ( #58163 ), and only the summary line of config docstrings in --help ( #57357 ). Structured output and parsers : native Lark grammars in the xgrammar backend ( #58321 ), XGrammar 0.2.7 ( #57272 ), Granite migrated to the streaming Parser Engine ( #49648 ), and fixes for reasoning boundaries ( #56635 ), xgrammar choices with control characters ( #48115 ), list-typed JSON Schema ( #48416 ), empty structural_tag ( #47450 ), outlines EOS handling ( #58612 , #57743 ), parser-suppressed streaming logprobs ( #58583 ), Inkling tool names after reasoning ( #58792 ), and length finish_reason for truncated streaming tool calls ( #46303 ). OpenAI, Anthropic and Harmony compatibility : reasoning token counts for Harmony, DeepSeek-V3 and Step3 (#58626) and per Responses tool round (#58927), Harmony max_output_tokens in the tool loop (#58551), batched chat completions using the adjusted requests (#58929, #58958), a fresh parser per choice (#58939), deferred reasoning recounts (#56067), Responses MCP cleanup (#56988) and image detail default (#57241), Anthropic inline system detection (#58754) and disabled thinking with P/D (#58786), stop strings rejected on --tokens-only servers (#57058), invalid prompt_embeds returning 400 (#55451, #57006), prompts bounded after multimodal expansion (#57076), --override-generation-config penalties (#50769), Hub revisions (#56092, #57461), full logprobs in token-in/token-out responses (#58488), generative scoring cancellation (#57729, #58788), gRPC keepalive pings (#55102), and run-batch diarized transcriptions (#57948). Pooling : chunked embedding padding (#56505) and normalization (#57498), reranker tokenization with document limits (#57666), and BERT-family heads kept for raw logits (#57664). Rust frontend : --hf-overrides (#56931), --sse-keep-alive-interval (#58306), custom chat roles (#58311), MiMo V2.5 parsers (#57933), normalized reasoning controls (#56998), parser-owned output grammars (#55269, #57340), sampling masks over gRPC (#56777), local DP size in gRPC metadata with vllm-proto 0.3.0 (#57116, #57233), a per-request preemption histogram (#57033), lock-free histograms (#58574), Nemotron-H vision context (#57634), model-owned vision processors (#58109), an mm-processor benchmark (#51922, #58084, #58378), unsupported serve args recognized (#58330), NaN logprobs no longer killing the engine client (#51026), appended EngineCoreOutput fields accepted (#56533), and vllm-rs on PATH in the CUDA image (#57606). Benchmarks : an openai-responses backend for vllm bench serve (#54628) and model_id in latency/throughput JSON (#58112). Security Per-request mm_processor_kwargs and media_io_kwargs are rejected by default; trusted deployments opt in with --trust-request-mm-kwargs (#58830). Prefix-cache block hashes tag extra keys by source (#51899) and include the LoRA path (#59335); multimodal hash input is framed (#54283) and incremental block hashing covers every overlapping feature (#51694). Fresh multimodal payloads take precedence over a stale receiver cache (#57833), and encoder-cache hits with mismatched embedding counts are rejected (#57696). min_tokens above the filled max_tokens default is rejected instead of wedging the engine (#57731); message sanitization filters upper-case memory addresses (#58832); LoRA adapters named after a served model are rejected (#59286). Dependencies FlashInfer 0.7.0.post1 (#58069, #59323), Transformers 5.17.0 (#56108) with an upper bound in requirements (#59614), XGrammar 0.2.7 (#57272), oss-harmony replacing openai-harmony (#55128), DeepGEMM fork pin bump (#57218), FlashKDA bump (#58846), and humming-kernels 0.1.16 (#58054). CUDA 12 images use LMCache 0.4.4 and CuPy CUDA 12 (#57945); a separately tagged zstd Docker Hub image ( -x86_64-zstd ) is published (#55608); Rubin CUDA 13.4 nightly images (#55953). ROCm: AITER v0.1.23 (#58867), torch 2.13, Triton 3.8 (#50605, #58006), patched ROCR (#57328), LMCache OpenTelemetry pins (#59056). XPU: PyTorch 2.14 (#56013). CPU: wheels built on Ubuntu 22.04 (#58515), pre-built Triton CPU (#58140). Breaking Changes & Deprecations Per-request mm_processor_kwargs and media_io_kwargs now return an error unless the server is started with --trust-request-mm-kwargs ; server-level --mm-processor-kwargs / --media-io-kwargs and offline LLM are unchanged (#58830). tokenizer_mode="slow" was removed; it already behaved like "hf" under Transformers v5 (#58545). --enable-mamba-fine-grained-prefix-cache was renamed to --enable-mamba-shared-prefix-checkpoint (#57382). Online quantization through quantization="fp8" now redirects to the fp8_per_tensor online shorthand (#53585); Quark-specific silent online MXFP4 quantization was removed in favor of the online quantization API (#51800). The AllSpark INT8 W8A16 GEMM backend was removed (#58001). The return_assistant_tokens_mask option of /render and the assistant_tokens_mask response field were removed (#57520). VLLM_PLE_CPU_OFFLOAD was removed; use --engram-config (#57937). VLLM_XPU_ENABLE_XPU_GRAPH was removed and XPU graphs are on by default (#51600). The comma-separated form of --collect-detailed-traces was removed; use the list syntax (#55702). New defaults: --enforce-eager also disables JIT kernel warmup unless fault tolerance is enabled (#58197, #58593); VLLM_BATCH_INVARIANT=1 uses breakable CUDA graphs without torch.compile (#57586) and disables sequence parallelism and async TP (#56377); DeepSeek-V4.1 SWA bounded replay is on (#56227) except on ROCm (#57906); Engram host tables are shared across co-located DP replicas when possible (#57651); FlashMLA mega attention is the DeepSeek-V4.1 default on SM100 (#56935); the AITER w4a4 ASM GEMM is off by default on ROCm (#57055). Model Runner V1 + PP > 1 + async scheduling + structured output is now rejected at startup (#56250); json_object is rejected at validation with the outlines backend (#57743). transformers now has an upper bound in requirements (#59614). New Contributors @0z5a made their first contribution in https://github.com/vllm-project/vllm/pull/56441 @200lz made their first contribution in https://github.com/vllm-project/vllm/pull/55528 @AARONKANG04 made their first contribution in https://github.com/vllm-project/vllm/pull/48521 @acsoto made their first contribution in https://github.com/vllm-project/vllm/pull/57314 @adenzhou1350 made their first contribution in https://github.com/vllm-project/vllm/pull/56882 @adtygan made their first contribution in https://github.com/vllm-project/vllm/pull/56325 @amasen02 made their first contribution in https://github.com/vllm-project/vllm/pull/56576 @amd-sriram made their first contribution in https://github.com/vllm-project/vllm/pull/53792 @andrewor14 made their first contribution in https://github.com/vllm-project/vllm/pull/52956 @Ankit-Jaiswal-AMD made their first contribution in https://github.com/vllm-project/vllm/pull/56252 @baljinderhothi-cohere made their first contribution in https://github.com/vllm-project/vllm/pull/58792 @BaoYunkai made their first contribution in https://github.com/vllm-project/vllm/pull/57568 @blipbyte made their first contribution in https://github.com/vllm-project/vllm/pull/55612 @BPbruce made their first contribution in https://github.com/vllm-project/vllm/pull/46466 @chuan932 made their first contribution in https://github.com/vllm-project/vllm/pull/55330 @coderfornow made their first contribution in https://github.com/vllm-project/vllm/pull/54978 @CZT0 made their first contribution in https://github.com/vllm-project/vllm/pull/51694 @devtyagi3909 made their first contribution in https://github.com/vllm-project/vllm/pull/56635 @dilberx made their first contribution in https://github.com/vllm-project/vllm/pull/57076 @divyvasal made their first contribution in https://github.com/vllm-project/vllm/pull/49845 @equiluxe made their first contribution in https://github.com/vllm-project/vllm/pull/48115 @errmakov made their first contribution in https://github.com/vllm-project/vllm/pull/58583 @farzad-elastix made their first contribution in https://github.com/vllm-project/vllm/pull/56168 @freyfwt made their first contribution in https://github.com/vllm-project/vllm/pull/51367 @garrett361 made their first contribution in https://github.com/vllm-project/vllm/pull/57984 @git-jxj made their first contribution in https://github.com/vllm-project/vllm/pull/55702 @gokay-ai made their first contribution in https://github.com/vllm-project/vllm/pull/58292 @gongwei-130 made their first contribution in https://github.com/vllm-project/vllm/pull/55102 @haosenwang1018 made their first contribution in https://github.com/vllm-project/vllm/pull/58288 @harshit-sarvam made their first contribution in https://github.com/vllm-project/vllm/pull/56034 @HieDean made their first contribution in https://github.com/vllm-project/vllm/pull/57356 @huthvincent made their first contribution in https://github.com/vllm-project/vllm/pull/48419 @i-m-aditya made their first contribution in https://github.com/vllm-project/vllm/pull/57447 @jayzuccarelli made their first contribution in https://github.com/vllm-project/vllm/pull/58557 @jiakangkangfuzhe made their first contribution in https://github.com/vllm-project/vllm/pull/55146 @jiangLLM made their first contribution in https://github.com/vllm-project/vllm/pull/57487 @jiaran-king made their first contribution in https://github.com/vllm-project/vllm/pull/56242 @jz-yolo made their first contribution in https://github.com/vllm-project/vllm/pull/58884 @kaijunli-infr made their first contribution in https://github.com/vllm-project/vllm/pull/52101 @karya0 made their first contribution in https://github.com/vllm-project/vllm/pull/57453 @KEYS-A15 made their first contribution in https://github.com/vllm-project/vllm/pull/57528 @kwen2501 made their first contribution in https://github.com/vllm-project/vllm/pull/51520 @laulopezreal made their first contribution in https://github.com/vllm-project/vllm/pull/57189 @lijipeng787 made their first contribution in https://github.com/vllm-project/vllm/pull/58225 @limitmhw made their first contribution in https://github.com/vllm-project/vllm/pull/48606 @linnea-lin-00638949 made their first contribution in https://github.com/vllm-project/vllm/pull/47450 @LinzeShi made their first contribution in https://github.com/vllm-project/vllm/pull/57347 @mahird3 made their first contribution in https://github.com/vllm-project/vllm/pull/57708 @melcheikh made their first contribution in https://github.com/vllm-project/vllm/pull/48956 @MichaelLapshin made their first contribution in https://github.com/vllm-project/vllm/pull/57396 @mjkvaak-amd made their first contribution in https://github.com/vllm-project/vllm/pull/48249 @mkunredd made their first contribution in https://github.com/vllm-project/vllm/pull/58244 @mmastrac made their first contribution in https://github.com/vllm-project/vllm/pull/57414 @mustafayildirim made their first contribution in https://github.com/vllm-project/vllm/pull/57252 @Navjot10 made their first contribution in https://github.com/vllm-project/vllm/pull/55390 @olka-amd made their first contribution in https://github.com/vllm-project/vllm/pull/51065 @PeganovAnton made their first contribution in https://github.com/vllm-project/vllm/pull/55128 @Rakul-Chauhan made their first contribution in https://github.com/vllm-project/vllm/pull/54967 @Ricardo-M-L made their first contribution in https://github.com/vllm-project/vllm/pull/52580 @Ronnie-Rui made their first contribution in https://github.com/vllm-project/vllm/pull/54581 @RyanMa29 made their first contribution in https://github.com/vllm-project/vllm/pull/57295 @sammaji made their first contribution in https://github.com/vllm-project/vllm/pull/58626 @sawsa307 made their first contribution in https://github.com/vllm-project/vllm/pull/56497 @ScarWar made their first contribution in https://github.com/vllm-project/vllm/pull/49435 @sdougbrown made their first contribution in https://github.com/vllm-project/vllm/pull/49819 @semerandre made their first contribution in https://github.com/vllm-project/vllm/pull/55557 @sergiofigueras made their first contribution in https://github.com/vllm-project/vllm/pull/57948 @shallow10 made their first contribution in https://github.com/vllm-project/vllm/pull/58336 @SIDDARTHAREDDY8 made their first contribution in https://github.com/vllm-project/vllm/pull/57743 @simpleqt made their first contribution in https://github.com/vllm-project/vllm/pull/55936 @tangzzycc made their first contribution in https://github.com/vllm-project/vllm/pull/53283 @touch869 made their first contribution in https://github.com/vllm-project/vllm/pull/55844 @tripathiarpan20 made their first contribution in https://github.com/vllm-project/vllm/pull/57140 @twu3202 made their first contribution in https://github.com/vllm-project/vllm/pull/57769 @ubwzwd made their first contribution in https://github.com/vllm-project/vllm/pull/55931 @UNIDY2002 made their first contribution in https://github.com/vllm-project/vllm/pull/58370 @valarLip made their first contribution in https://github.com/vllm-project/vllm/pull/58659 @vcave made their first contribution in https://github.com/vllm-project/vllm/pull/51681 @voidxb made their first contribution in https://github.com/vllm-project/vllm/pull/49300 @vorapolsiloai made their first contribution in https://github.com/vllm-project/vllm/pull/50212 @vschandramourya made their first contribution in https://github.com/vllm-project/vllm/pull/58779 @weitliao made their first contribution in https://github.com/vllm-project/vllm/pull/54535 @wenjinhust made their first contribution in https://github.com/vllm-project/vllm/pull/51366 @Willian-Zhang made their first contribution in https://github.com/vllm-project/vllm/pull/58720 @wtdcode made their first contribution in https://github.com/vllm-project/vllm/pull/56268 @xinnywinne made their first contribution in https://github.com/vllm-project/vllm/pull/55084 @YannikHinteregger made their first contribution in https://github.com/vllm-project/vllm/pull/50047 @YashasviChaurasia made their first contribution in https://github.com/vllm-project/vllm/pull/58112 @Yatimai made their first contribution in https://github.com/vllm-project/vllm/pull/43462 @YCH188 made their first contribution in https://github.com/vllm-project/vllm/pull/57190 @yousafshah made their first contribution in https://github.com/vllm-project/vllm/pull/55957 @ys2025-AI made their first contribution in https://github.com/vllm-project/vllm/pull/56841 @yuchenwang3 made their first contribution in https://github.com/vllm-project/vllm/pull/54699 @zhecfy made their first contribution in https://github.com/vllm-project/vllm/pull/58316 @Zoe923 made their first contribution in https://github.com/vllm-project/vllm/pull/53864 @Zyann7 made their first contribution in https://github.com/vllm-project/vllm/pull/57784 Contributors @AndreasKaratzas , @khluu , @mgoin , @njhill , @BugenZhao , @stefankoncarevic , @robertgshaw2-redhat , @taneem-ibrahim , @Thangnguyenvn98 , @hmellor , @Juntian777 , @WoosukKwon , @yewentao256 , @gau-nernst , @mmastrac , @LucasWilkinson , @Fangzhou-Ai , @NickLucche , @aoshen02 , @JaredforReal , @mawong-amd , @DarkLight1337 , @zyongye , @sfeng33 , @vllm-agent , @okorzh-amd , @ZJY0516 , @shen-shanshan , @gty111 , @Isotr0py , @djramic , @zixi-qi , @wzhao18 , @alec-flowers , @aarushjain29 , @Rohan138 , @yma11 , @mjkvaak-amd , @ivanium , @yisustc , @MatthewBonanni , @wangxiyuan , @reidliu41 , @chaojun-zhang , @shaohuaxi , @gcanlin , @ganeshr10 , @linitra24 , @yzong-rh , @divakar-amd , @atalman , @ShuoleiWang , @simondanielsson , @rasmith , @chaunceyjiang , @micah-wil , @liusy58 , @LioEinaudi , @louie-tsai , @LiuYinfeng01 , @jeejeelee , @zhenwei-intel , @jperezdealgaba , @hlin99 , @zxd1997066 , @lucamotz , @lucifer1004 , @eopXD , @ashraf-bhuiyan , @TheEpicDolphin , @JulienDarve , @mayuyuace , @danisereb , @bigPYJ1151 , @Etelis , @JohnQinAMD , @jiangkuaixue123 , @tianmu-li , @akii96 , @vllmellm , @RyanMa29 , @afriedri , @elvircrn , @mustafayildirim , @yuzhouo7 , @wtdcode , @biswapanda , @adtygan , @hickeyma , @itayalroy , @jiangLLM , @Hotragn , @faaany , @xhx1022 , @jinzhen-lin , @sheralskumar , @Wauplin , @matteso1 , @wjabbour , @Sunt-ing , @arpera , @hclsys , @S1ro1 , @HDCharles , @fxmarty-amd , @liuzijing2014 , @UNIDY2002 , @samuelkim7 , @markmc , @errmakov , @zhejiangxiaomai , @KernelClint , @i-m-aditya , @ColinZ22 , @ppalanga, @franciscojavierarceo, @maithilijoshi20, @mfylcek, @almersawi, @albertoperdomo2, @0z5a , @jacklin78911-collab, @lzhan011, @Alex-ai-future, @freyfwt , @ZhengGong-amd, @mindungil, @amasen02 , @devtyagi3909 , @wenjinhust , @jbyczkow, @AdaAibaby, @harshit-sarvam , @vineethsaivs, @tripathiarpan20 , @sashko-zakharchuk, @semerandre , @rebklee, @git-jxj , @cjackal, @xinnywinne , @andrewor14 , @kyleliang-nv, @thillai-c, @dongluw, @ChuanLi1101, @omerpaz95, @jiaran-king , @liuyao0322, @coderfornow , @drakosha, @laulopezreal , @melcheikh , @shantipriya-amd, @Rukhaiya2004, @yuchenwang3 , @lxy-alexander, @tlrmchlsmth, @HieDean , @Zoe923 , @YCH188 , @amd-sriram , @andylolu2, @MicheleCampi, @sdougbrown , @vMaroon, @shimib, @andakai, @thegoldenflow, @Levius-Fubuki, @olka-amd , @limitmhw , @yuwenzho, @adenzhou1350 , @Josephasafg, @kylesayrs, @jdebache, @kaijunli-infr , @afierka-intel, @frida-andersson, @bnellnm, @ys2025-AI , @waizuichougou, @touch869 , @rjrock, @sawsa307 , @pavelzak, @ScarWar , @fadara01, @Mi-Jiazhi, @dilberx , @mahird3 , @Zyann7 , @roikoren755, @YukioZzz, @Ronnie-Rui , @acsoto , @czhu-cohere, @twu3202 , @oliverholworthy, @weitliao , @sergiofigueras , @tuukkjs, @Sip4818, @karen-sy, @garrett361 , @GirasoleY, @Navjot10 , @taking-lying-flat, @wangyicong52, @majunze2001, @gangula-karthik, @ubwzwd , @linnea-lin-00638949 , @kushaldabbe, @Yatimai , @guanxingithub, @jiakangkangfuzhe , @MichaelLapshin , @jhu960213, @tpopp, @mganczarenko, @qiching, @tangzzycc , @lijipeng787 , @nikhilkulkarni1755, @QwertyJack, @Ankit-Jaiswal-AMD , @vorapolsiloai , @Dao007forever, @BPbruce , @divyvasal , @simpleqt , @blipbyte , @jhaotingc, @karya0 , @farzad-elastix , @grYe99, @talorabr, @jackLei0901, @Yejing-Lai, @Priyjain-amd, @KEYS-A15 , @LinzeShi , @bohnstingl, @nightcityblade, @vcave , @AARONKANG04 , @CZT0 , @snadampal, @microslaw, @haosenwang1018 , @netanel-haber, @khushali9, @sammaji , @SIDDARTHAREDDY8 , @100milliongold, @gongwei-130 , @fululi12, @mkunredd , @mrodden, @noooop, @valarLip , @shallow10 , @200lz , @askliar, @chuan932 , @Monishver11, @YashasviChaurasia , @gokay-ai , @jiacao-amd, @yousafshah , @positive666, @TQCB, @lk-chen, @Ricardo-M-L , @huthvincent , @baljinderhothi-cohere , @vschandramourya , @kwen2501 , @xiao-llm, @zhecfy , @Willian-Zhang , @simon-veitner-redhat, @mevince, @voidxb , @hongxiayang, @YannikHinteregger , @V-3604, @Rakul-Chauhan , @frankwang28, @LCAIZJ, @PeganovAnton , @jz-yolo , @QHarshil, @R3hankhan123, @jayzuccarelli , @yannicks1, @BaoYunkai , @xaguilar-amd, @xiaohuguo2023, @wxsIcey, @harshaladhav-amd, @varun-sundar-rabindranath, @kliuae, @LostFox11