2026-10-02

Edit: I just realised that my comparisons are not portrait friendly. If on mobile, please view them in landscape mode. Last week, I posted custom node for replicating the Qwen-Image 2.1 official prompt enhancer in ComfyUI. I had made the node because I hated my previous workflow, having to switch to LM Studio to use the official prompt enhancer, unloading the model in LM Studio else Qwen-Image 2.1 would run very slowly, then running the workflow after copy/pasting the prompt in ComfyUI and then having to unload models in ComfyUI because I need to use prompt enhancement again, and then endless repeat of the same process. I wanted to do everything end to end in ComfyUI but Native Generate Text node was too slow, so adding a MTP head improved the speed a bit, which made it 1.4–1.7× faster than ComfyUI's native text generation with the Comfy-Org PE checkpoints. After that, I wanted to see how much faster it could go, and whether it could run on smaller GPUs. Few comments mentioned llama.cpp, so I added it as a second backend. I ran a few benchmarks and I was quite happy with the result: it's much faster, especially for editing, and minimum VRAM requirement went down with smaller quants at marginal quality cost. Below are my benchmarks on RTX 4090 Laptop, 16 GB, official max_length (16256 for t2i, 24000 for editing), 2 input images for editing. Each step adds one change to the one before: Text-to-image Step Tok/s Step gain Total vs baseline ComfyUI, MTP off (baseline) 23.5 – 1.00× + MTP 33.9 1.44× 1.44× + llama.cpp (Q8_0) 46.8 1.38× 1.99× + Q4_K_M (small quality cost) 63.2 1.35× 2.69× Editing (two input images) Step Tok/s Step gain Total vs baseline ComfyUI, MTP off (baseline) 19.4 – 1.00× + MTP 32.6 1.68× 1.68× + llama.cpp (Q8_0) 67.8 2.08× 3.49× + Q4_K_M (small quality cost) 85.2 1.26× 4.39× The previous release reduced the max_length to 8192 to gain some speed at the risk of truncated result but with llama.cpp, changing max_length has very little impact on generation speed. Strictly comparing the speed of the previous version to new one, the gain from llama.cpp (Q8_0) is smaller but still significant: 1.53× for t2i and 2.43× for editing. In my tests Q8_0 didn't change the quality. This time, I also wanted to show what the prompt enhancer actually does, so, I've added some comparisons. Each pair uses the same seed and settings, with the prompt going straight to Qwen-Image (left) or through the enhancer first (right). In my tests, the prompt enhancer helped most with text in images, busy scenes, and edits with several changes. If someone wants to check it out, full details are available at: https://github.com/mozophe/ComfyUI-Qwen-Image-2.1-PromptEnhancer-MTP GGUF models: https://huggingface.co/mozophe/Qwen-Image-2.1-PE-MTP-GGUF Requirements: ComfyUI v0.37.0 or newer llama.cpp backend: NVIDIA GPU on Windows or Linux, 8 GB VRAM or more. Q4_K_M for 8–12 GB, Q8_0 for 16 GB or more ComfyUI backend: any GPU ComfyUI supports, 16 GB recommended System RAM: 32 GB recommended Disk space: 9.8 GB (Q8_0) or 6.0 GB (Q4_K_M) per model, plus 0.9 GB for editing's vision part and 0.7 GB once for llama-server What's new: llama.cpp backend with MTP, picked on the loader. The ComfyUI backend is still there for AMD, Intel and Mac Runs on 8 GB VRAM with Q4_K_M llama-server and the GGUF model download automatically the first time you use them Uncensored (Heretic) versions are available as GGUFs too, so they work on both backends What the node does (same as before): Uses Qwen's official system prompts and sampling settings The rewritten prompt comes out separately from the reasoning Edit with up to 10 input images Install: ComfyUI Manager: open Manager → Custom Nodes Manager, search for Qwen-Image 2.1 Prompt Enhancer (MTP) , install, and restart ComfyUI. Manual: clone it into your custom_nodes folder and restart ComfyUI: Nothing else needs installing for either backend. The first run downloads the model (and llama-server for the llama.cpp backend), with progress shown on the node. If the download gets interrupted, just run the workflow again and it resumes. To get started, drag one of the sample workflow PNGs from the repo's workflows folder into ComfyUI. If generation speed is too slow, set quant to Q4_K_M. Not on an NVIDIA GPU? Set the loader's backend to ComfyUI. I am now doing everything end-to-end in ComfyUI, without having to switch to other apps, which just simplifies everything and I can spend more time on thinking about what I want to generate. I am thinking about what should I do next, so feel free to request for something that you find missing with the current nodes. Edit 2: Workflows llama.cpp backend (recommended on NVIDIA GPUs) - Text-to-image: https://github.com/mozophe/ComfyUI-Qwen-Image-2.1-PromptEnhancer-MTP/blob/main/workflows/qwen_image_2.1_t2i_prompt_enhancer.json - Edit: https://github.com/mozophe/ComfyUI-Qwen-Image-2.1-PromptEnhancer-MTP/blob/main/workflows/qwen_image_2.1_edit_prompt_enhancer.json ComfyUI backend - Text-to-image: https://github.com/mozophe/ComfyUI-Qwen-Image-2.1-PromptEnhancer-MTP/blob/main/workflows/qwen_image_2.1_t2i_prompt_enhancer_comfyui.json - Edit: https://github.com/mozophe/ComfyUI-Qwen-Image-2.1-PromptEnhancer-MTP/blob/main/workflows/qwen_image_2.1_edit_prompt_enhancer_comfyui.json Models (selected model in node automatically downloaded on first run): llama.cpp backend, GGUFs: https://huggingface.co/mozophe/Qwen-Image-2.1-PE-MTP-GGUF/tree/main ComfyUI backend, int8 convrot: https://huggingface.co/Comfy-Org/Qwen-Image-2.1/tree/main/text_encoders submitted by /u/mozophe to r/StableDiffusion [link] [comments]

cuda : route sm70 to the Turing MMVQ nwarps table ( #29753 ) cuda : route sm70 to the Turing MMVQ nwarps table Volta (sm_70) has no MMVQ parameter table of its own and falls through to GENERIC, which launches K-quant batch-1 decode (ncols_dst == 1) at nwarps=4. sm_70 shares TURING's tuning: the K-quant vec_dot prefers nwarps=2 there. Route sm_70 to the existing MMVQ_PARAMETERS_TURING table in both the device and the host table selector. Measured on one Tesla V100 32GB PCIe (PG500-216, driver 580.178.04, CUDA 12.0.140) with Qwen3.8-27B Q4_K_M, tg128, interleaved A/B in 6 ABBA blocks with paired per-block deltas: +1.091 t/s = +3.17 % (t = +49.0, all six per-block deltas positive); perplexity bit-identical (6.3697 +/- 0.04066 both builds, wiki.test.raw). The patched build's K-quant mul_mat_vec_q kernels launch at nwarps=2 (cubin EIATTR_MAX_THREADS) while Q4_0/Q8_0 stay at nwarps=4, and the same measurement on the September master base gave +3.84 % (t = 85). The tuning originates from the V100-focused fork anyei/llamacpp-v100 (MIT), commit b912d1b1e, which carries a dedicated MMVQ_PARAMETERS_VOLTA table; a cubin-level comparison confirmed that routing sm_70 to the existing TURING table is equivalent for the K-quant batch-1 path this change affects, so this is the minimal 2-line form. anyei/llamacpp-v100@ b912d1b1e Original-patch-by: anyei angelyoelroblesmercedes@gmail.com Update ggml/src/ggml-cuda/mmvq.cu Co-authored-by: tkittich tkittich@gmail.com Co-authored-by: Johannes Gäßler johannesg@5d6.de Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52032834 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries Ubuntu arm64 (CUDA 13) - CUDA 13.4 libraries Ubuntu x64 (ROCm 10.0) Ubuntu x64 (OpenVINO) Ubuntu x64 (SYCL FP32) Ubuntu x64 (SYCL FP16) Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide Android: Android arm64 (CPU) Android arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide Windows: Windows x64 (CPU) Windows arm64 (CPU) Windows arm64 (OpenCL Adreno) Windows x64 (CUDA 12) - CUDA 12.4 DLLs Windows x64 (CUDA 13) - CUDA 13.4 DLLs Windows arm64 (CUDA 13) - CUDA 13.4 DLLs Windows x64 (Vulkan) Windows x64 (OpenVINO) Windows x64 (SYCL) Windows x64 (ROCm 10.0) openEuler: DISABLED openEuler x86 (310p) openEuler x86 (910b, ACL Graph) openEuler aarch64 (310p) openEuler aarch64 (910b, ACL Graph) UI: UI

My comment on S3 Is the Future, S3 Is the Past — Hacker News. One thing I find notable about S3 today is that, while it used to drop in price reasonably often, there hasn't been a price drop in a full decade : 2006-03-14 $0.150/GB-month 2010-11-01 $0.140/GB-month 2012-02-01 $0.125/GB-month 2012-12-01 $0.095/GB-month 2014-02-01 $0.085/GB-month 2014-04-01 $0.030/GB-month 2016-12-01 $0.023/GB-month Today it's still $0.023/GB-month. Tags: amazon-web-services , s3

We evaluate several models on 100 tasks from the [internal Binary Exploitation benchmark] (selected at random), and find that GLM-5.3 develops full control flow hijacks in 4% of the trials; Claude Mythos Preview did so in 6%. Although GLM-5.3 performs below Claude Mythos Preview here, a meaningful threshold has clearly been crossed: earlier models, like Claude Opus 4.6 and GLM-5.2, do not succeed in any of them. — Anthropic Frontier Red Team , GLM-5.3 and the spread of advanced cyber capabilities Tags: anthropic , generative-ai , ai-security-research , glm , ai , ai-in-china , llms

Tool: Gemini 3.8 TTS Playground Google released two new Gemini text-to-speech models today - gemini-3.8-flash-tts and gemini-3.8-flash-lite-tts . They come with a library of over 2,000 voices, plus the ability to create a custom voice with "just a 30-second audio sample of your voice or a voice you have the rights to use". I vibe coded this bring-your-own-key playground interface with GPT-6 Astra, taking advantage of the open CORS policy of the underlying Gemini API. A notable feature of the API is that it makes it easy to define a full conversation between multiple characters, each with different voices and voice style instructions. Here's a short demo clip of a conversation between two pelicans debating if they should move to the Pacifica Pier . I had Claude 4.5 Opus write the script and generate a URL to render it using the tool . Your browser does not support the audio element. It took ~20 seconds to generate 1m 18s of audio using Gemini 3.8 Flash TTS (not the cheaper Flash-Lite), at a cost of 2.74 cents. Tags: text-to-speech , gemini

48% of people who talked to it live thought it was a real human. Previous systems have had a pass rate <3%. It is #1 on NVIDIA's benchmark for full-duplex AI video. This week's been so crazy!! submitted by /u/EuphoricTomorrow2440 to r/accelerate [link] [comments]

I made a little gallery of the Claude Code mascot, drawn live in the browser with code: no image assets, no dependencies. It spans 28 art styles, from cave paintings and heraldry all the way to Bauhaus, pixel art, teletext, ASCII and Maze War. The mascot watches your cursor, and you can click to spin it or press W/D/H to wave, dance and hop. It builds on ChetasLua's original Claude Code mascot in Twenty Styles (their 20 styles and the 3D engine), and I added eight experimental coding/terminal styles. How I used Claude The eight new styles (Opus 5.5 in Claude Code): [e.g. I described each look I wanted, such as glyph tiles, phosphor streams or a teletext page, along with the reference it's based on. Claude wrote each one as a CM.style({...}) drawing function on top of the existing engine, and I iterated on the screenshots until they looked right.] One favourite is Code Poem : Claude had the picture build itself out of letters taken from its own source code. Free to try: it's MIT-licensed and free. Clone the repo, run python -m http.server , and open it in your browser. All 28 styles are animated in the README. GitHub: https://github.com/henrik-thevibe/Claude-Mascot-Style-Gallery Feel free to fork it, mess around with it, or add your own style. Go wild with it. :) submitted by /u/dagerika to r/ClaudeAI [link] [comments]

Article URL: https://svelte.dev/blog/sveltekit-3-is-here Comments URL: https://news.ycombinator.com/item?id=49926536 Points: 355 # Comments: 146

v0.30.0 Highlights This release features 762 commits from 315 contributors (104 new)! New models : DeepSeek-V4.1-Flash ( #56214 , #56228 , #56208 ) with the whole KV stored in MXFP8 through the FlashMLA V4.1 record on SM100 ( #56893 ), DeepGEMM Mega-mHC ( #56962 ), and async Engram prefetch with Engram DP sharding ( #56512 ); DeepSeek-V4-Flash-Vision-Exp ( #54566 ), also on ROCm ( #55107 ) and with LoRA ( #55897 ); GLM-5.3-Flash ( #53906 ) with EPLB ( #55119 ); K2-Horizon ( #55063 ); Cohere Compass ( #54774 ); Bailing V3 VL ( #55921 ); Nanbeige4.2 via the Transformers backend ( #56071 ); and a DeepSeek-V4 CPU backend with AVX512/AMX sparse MLA, indexer, mHC and compressor kernels ( #55355 ). Fast Start : a persistent per-GPU weight-cache daemon holds post-quantized, TP-sharded weights in GPU memory so restarting engines map them over CUDA IPC with --load-format ipc_cache instead of reloading from disk ( #54921 ), now covering FP4 checkpoints ( #55465 ) and multi-node TP ( #55468 ). Watermarking : Gumbel-max watermarked generation and detection with a keyed PRF, per-request opt-out and an example detection endpoint ( #54053 ); dual-key Gumbel-max makes it compatible with speculative decoding ( #56122 ); the Rust frontend forwards the per-request controls ( #56338 ). HiSparse : a host-resident tier for sparse-MLA decode that spills KV pages to pinned host memory under GPU pressure and serves top-k misses from a per-request GPU hot buffer, enabled through HiSparseConnector ( #53781 ), with Prometheus counters ( #56061 ), a host cache shared across TP ranks ( #56629 ), and the attention config inferred from the connector ( #57041 ). Model Runner V2 : dual-batch overlap in eager mode ( #50945 ) and with FULL CUDA graphs for microbatched steps ( #51700 ); MTP ( #46994 ) and EAGLE3/DFlash/DSpark ( #50514 ) speculative decoding under pipeline parallelism; adaptive verification for every draft-model speculator through an online acceptance estimator ( #52228 ); gc frozen during graph capture, cutting capture from 12s to 2s and engine init from 28.9s to 8.2s on H200 ( #54646 ); --return-sampling-mask compacted on GPU, fixing an about 2x RL step-time regression ( #54901 ). Qwen3.8-Flash-Next performance : separate prefill and decode QSA indexer kernels ( #54513 ), fused PLE kernels ( #54517 ), FP8 indexer cache ( #54890 ), padded-index skipping in sparse GQA ( #54873 ), fused PLE residual and QSA output gate ( #55309 ), UVA PLE offload and Engram tensor parallelism via --engram-config ( #54371 ), and torch.compile removed from the NVIDIA implementation so FP8 fits on a single GB300 ( #55272 ). Kimi K3 performance : native CUDA AttnRes default on SM100 ( #54261 ), KDA mixed-batch gather/scatter removed (5.2-7.7% E2E throughput, #56159 ), grouped FP8 MLA cache insertion (4-6x kernel speedup at small batch, #55356 ), DSV3 low-latency GEMM on strided tensors (12-81% kernel speedup, #54565 ), overlapped TP8 KDA projections ( #54697 ), FlashInfer KDA kernels ( #55364 ), internal prefix checkpoints with partial prefix caching and speculative decoding ( #53614 ), and symmetric DCP disaggregation for hybrid Mamba models ( #55531 ). Large scale serving : PCP+DCP on sparse-MLA models ( #56157 ), PCP with single-module MTP and replicated DSpark ( #56107 ) and decode-only FULL CUDA graphs ( #53867 ), Elastic EP reusing CUDA graphs across reconfiguration ( #54985 ), an opt-in FlashInfer PCIe IPC all-reduce for NVLink-less boxes ( #53576 ), DeepEP v2 async finalize overlapping shared experts with combine ( #52781 ), Mooncake Store heterogeneous TP sharing ( #53129 ), a KVCR secondary-tier adapter ( #53624 ), and encoder-cache sharing over NIXL ( #47941 ) and Mooncake ( #41567 ). Quantization : targeted online quantization through quantization_config.targets ( #51285 ) and on partially pre-quantized checkpoints from any quant method ( #51392 ), W4A16 DSA with the nvfp4_fp8_ds_mla KV cache ( #51724 ), FlashInfer CuTeDSL NVFP4 W4A16 default over Marlin on SM100/103 ( #53014 ), NVFP4 in the torch linear backend ( #53319 ), per-quantization linear backend overrides ( #51204 ), AutoRound 2/3/5/6/7-bit on CUDA ( #52890 ), and DeepSelect top-k for the DSA sparse indexer ( #56464 ). Breaking changes : scale-out endpoints are opt-in on plain vllm serve via --enable-scale-out , replacing VLLM_ENABLE_SCALE_OUT_ENDPOINTS ( #54579 , #55176 ); GPTQ activation ordering ( g_idx ) removed ( #54809 ); items deprecated for 0.29 removed, including the VLLM_PREFIX_CACHE_RETENTION_INTERVAL and VLLM_MM_HASHER_ALGORITHM env vars ( #55353 ); the all Mamba cache mode deprecated ( #55041 ); python -m vllm.entrypoints.grpc_server deprecated in favor of vllm serve --grpc ( #56746 ); YaRN aligned with Transformers so vendor YaRN aliases no longer re-scale max_model_len ( #56446 ). Release Artifacts Python Wheels Platform Install PyPI (CUDA 13.0) pip install vllm PyPI (CUDA 13.0, uv) uv pip install vllm --torch-backend=auto ROCm pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.30.0/rocm723 XPU uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.30.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match Docker Images Platform Docker Image CUDA 13.0 (Default) docker pull vllm/vllm-openai:v0.30.0 CUDA 12.9 docker pull vllm/vllm-openai:v0.30.0-cu129 CUDA 13.0 + Ubuntu 24.04 docker pull vllm/vllm-openai:v0.30.0-ubuntu2404 CUDA 12.9 + Ubuntu 24.04 docker pull vllm/vllm-openai:v0.30.0-cu129-ubuntu2404 ROCm docker pull vllm/vllm-openai-rocm:v0.30.0 CPU docker pull vllm/vllm-openai-cpu:v0.30.0 XPU docker pull vllm/vllm-openai-xpu:v0.30.0 Other Artifacts Pre-built release artifacts are available in the Assets section at the bottom of this page, including: Source distribution tarball CUDA 12.9 Python wheels for x86_64 and arm64 CUDA 13.0 Python wheels for x86_64 and arm64 CPU Python wheels for x86_64, arm64, and macOS XPU Python wheel for x86_64 Model Support New models : DeepSeek-V4.1-Flash ( #56214 , #56228 , #56208 ), DeepSeek-V4-Flash-Vision-Exp ( #54566 ) with ROCm support ( #55107 ) and LoRA ( #55897 ), GLM-5.3-Flash ( #53906 ), K2-Horizon with its reasoning and tool parsers ( #55063 ), Cohere Compass ( #54774 , #55949 ), Bailing V3 VL with MTP ( #55921 ), Nanbeige4.2 via the Transformers backend ( #56071 ), and a DeepSeek-V4 CPU backend ( #55355 ). DeepSeek-V4.1-Flash : whole KV in MXFP8 through the FlashMLA V4.1 record on SM100 ( #56893 ), Mega-mHC from the DeepGEMM fork ( #56962 ), mHC post block folded into the delayed pre projection ( #56633 ), Triton-fused input metadata preparation ( #56562 ), async prefetch for CPU-offloaded Engram lookups and Engram DP sharding ( #56512 ), DSpark draft states collapsed before the SP all-gather ( #56903 ), DSpark drafter no longer inheriting uninitialized EPLB state ( #56387 ), XGrammar V4.1 schema constraints for strict tool parameters ( #56408 ), Responses text parts ( #56299 ), and image sentinel padding scoped to the Vision-Exp checkpoint ( #56554 ). DeepSeek V4 : plain RoPE without YaRN on sparse SWA layers ( #54815 ), --kv-cache-dtype auto resolving to fp8_ds_mla with FlashMLA ( #45091 ), prefill sparse index workspace sentinel seeded ( #55299 ), dequant gather grid sized by rows ( #55061 ), optional Q-norm in the fused MLA epilogue and group_size=32 packed FP8 quant ( #56215 ), sparse settings read from the text config for composite models ( #56160 ), MegaMoE startup without EP fixed ( #55914 ), and a Triton iHC pre/post fallback for HY V4 ( #55059 ). GLM-5.3-Flash : EPLB support ( #55119 ), FlashKDA for KDA chunked prefill (1.7-3.8x faster than the Triton chunk path, #55737 ), dense and masked-MHA sparse prefill for the NoPE (256, 0, 256) layout ( #55738 ), decode hot-path cleanups removing per-layer copies and a duplicate router GEMM ( #55736 ), and the glm5next NVIDIA subtree now packaged in wheels ( #55214 ). Qwen3.8-Flash-Next : separate prefill and decode QSA indexer kernels with VLLM_SPARSE_INDEXER_MAX_LOGITS_MB ( #54513 ), fused PLE kernels and merged K/V projections ( #54517 ), sparse GQA skipping padded indices ( #54873 ), FP8 indexer cache ( #54890 ), fused PLE residual and QSA output gate ( #55309 ), UVA PLE offload and Engram tensor parallelism via --engram-config ( #54371 ), torch.compile removed from the NVIDIA path ( #55272 ), compact indexer logits workspace ( #54915 ), HC combine-norm reused for MTP input ( #54687 ), PLE MTP metadata transfers ( #55054 ), Hopper LL-GEMM tuning table ( #54560 ), B200 FP8 MoE tuning ( #55890 ), and fixes for FP8 PLE weight scales ( #54722 , #54882 ) and fused PLE conv strides ( #55375 ). Kimi K3 : native CUDA AttnRes default on SM100 ( #54261 ), ll_bf16 router weight prefetch ( #53524 ), overlapped TP8 KDA projections ( #54697 ), strided DSV3 GEMM ( #54565 ), MLA decode epilogue latency ( #54896 ), NVFP4 in_proj alignment routing to DeepGEMM ( #55242 ), mixed-batch KDA without gather/scatter ( #56159 ), grouped FP8 MLA cache insertion ( #55356 ), FlashInfer KDA kernels ( #55364 ), FlashKDA bf16 checkpoint state ( #56485 ) and a stability bump ( #54859 ), SiTU on the FlashInfer CuTeDSL MoE backend ( #54606 ), internal prefix checkpoints with partial prefix caching and spec decoding ( #53614 ), and fixes for RecoverSSM startup ( #54634 , #55774 ), GDN metadata ( #54636 ), mixed-batch masks ( #54781 ), checkpoint indexing ( #55747 , #56794 ), KDA illegal memory access ( #55924 ), interleaved weight streams ( #53379 ), NVFP4 MoE conversion OOM ( #55407 ), and KDA projection overlap on Hopper ( #55426 ). Transformers backend : attention found with a fuser and vLLM's layer attached to it ( #54941 ), generic merged-column linear fusion ( #55301 ), QKV fuser for Gemma4 ( #55690 ), all vocab embeddings replaced including Gemma 3n/4 per-layer tables ( #54760 , #55083 ), and attention sinks applied ( #52156 ). Speculative decoding for models : EAGLE3 for Sarvam MLA ( #53052 ), Nemotron-H MTP with a separate, possibly quantized lm_head ( #54574 ), NVFP4 DSpark gathered top-k projection for quantized Markov heads ( #55713 ), Qwen3.5/3.6 multimodal MTP resolving n_predict from the text config ( #55369 ), Qwen3 DSpark padded-vocab drafts ( #55133 ), and GLM-OCR MTP weight loading ( #49869 ) and position masking under CUDA graphs ( #56447 ). Multimodal : torchcodec audio decoding and a selectable audio_backend in --media-io-kwargs ( #51826 ) with automatic decoding kept soundfile-first ( #55642 ), torchaudio as the default resampler ( #52598 ), media_io_kwargs in multimodal hashes ( #54241 ), cache hash kwargs scoped by modality ( #54918 ), empty video URLs with multimodal UUIDs ( #54220 ), cached audio inputs with UUIDs ( #56310 ), encoder CUDA graphs for MiniCPM-V 2.5/2.6/4.0 ( #42785 ), Voxtral Realtime with FULL_DECODE_ONLY graphs ( #51167 ), Triton/FlashInfer composite attention for multimodal prefixes ( #56305 ), pruned sliding-window tiles for Gemma 4 multimodal prompts (up to 3-4x E2E, #53147 ), SDPA for BLIP-2 Q-Former ( #55285 ), baddbmm Conformer scores ( #55062 ), fused DeepEncoder relative bias ( #55629 ), Gemma3n sparse GELU Triton kernel ( #48498 ), and no duplicate text embedding in Qwen2.5-Omni ( #55415 ). Correctness : Gemma 4 / Gemma 3n fine-tuned checkpoints without KV projections on shared layers ( #54917 , #55864 ), Gemma4 video preprocessing without image kwargs ( #56652 ) and unified audio profiling ( #56721 ), MiniCPM-V 4.6 loading ( #49417 ) and video embeds ( #54633 ), InternVL video parser ( #55779 ), Nano-Nemotron precomputed embeddings ( #56398 ), OpenPangu multimodal merge ( #55941 ) and sleep mode ( #53696 ), Qwen3-VL and Cosmos3-Edge under CPU and PP ( #53699 , #43272 ), Qwen VL processors without a tokenizer ( #54886 ), Molmo2 fps guard ( #55893 ), CohereASR audio-token estimate ( #53829 ), double BOS in LLM.chat() for multimodal models ( #55288 ), decoded frame lists parsed as one video ( #55326 ), swapped H/W in dummy video profiling ( #56385 ), base64 video validation ( #54323 ), Sentence Transformers 5.4+ configs ( #54533 ), ColQwen3.5 pooler projector ( #54847 ), torch.compile enabled for StableLM ( #54969 ) and Sarvam MLA ( #55817 ), and Nemotron latent-MoE skipping a redundant all-reduce at TP>1 (about 13% decode win, #52301 ). LoRA : rsLoRA scaling in MoE expert packing ( #55548 ), a log line when an adapter applies no weights ( #55310 ), and --lora-modules name=path with = in the path ( #53353 ). Engine Core Model Runner V2 : DBO in eager mode ( #50945 ) and with FULL CUDA graphs ( #51700 ), MTP ( #46994 ) and EAGLE3-style drafts ( #50514 ) under pipeline parallelism, adaptive verification for all draft-model speculators ( #52228 ), gc frozen during capture ( #54646 ), GPU-compacted sampling masks ( #54901 ), UVA-backed apply_write ( #55819 ), fast-prefill support ( #56145 ), DP sync skipped for all speculator uniform decodes ( #54856 ), prompt embeddings in decoder pooling ( #56639 ), pooling post-processing on non-final PP ranks ( #56666 ), kernels warmed before graph capture ( #55341 ), CUDAGraphStat ( #52358 ), Proton CUDA graph attribution ( #51084 ), platforms without pinned memory such as WSL ( #56908 ), draft moe_backend ( #54788 ) and attention_backend ( #54826 ) overrides honored, DCP metadata initialized after batch partitioning ( #55212 ), and a record_stream fix in the PP draft broadcast ( #55745 ). The MRV2-specific Triton JIT warmup migration was reverted ( #56654 ). Speculative decoding : opt-in disable_eagle_block_drop ( #53388 ), adaptive verification on FLASHINFER_MLA_SPARSE_DSV4 ( #52724 ) and deferred until after kernel warmup ( #55455 ), full-vocab EAGLE3 drafters skipping vocab remapping ( #53458 ), DSpark preserving the target DCP config ( #55472 ), DFlash drafters dropping FlashAttention's AOT schedule ( #54374 ), DP token padding excluded from draft metadata ( #55458 , #56181 ), fastsafetensors deadlock avoided for PP drafts ( #54416 ), Mamba state cached at the EAGLE resume position ( #53945 ) and preserved for padded prompt tails ( #55178 ), resumed requests padded in P/D ( #55126 ), and warmup device selection ( #55245 ). Prefix caching and scheduling : both replay boundaries retained so an EAGLE resend of a block-aligned prompt still hits ( #54713 ), zero-progress preemption cascades stopped for deferred KV frees ( #49675 ), SWA layers taking the primary block size to avoid inflating the KV block LCM ( #53007 ), align-mode Mamba states retired across null gaps ( #55450 ), the all Mamba cache mode deprecated with MRV1 fallback ( #55041 ), and vllm:request_num_preemptions histogram ( #49984 ). Kernels : FlashInfer ReplaySSM backend for Mamba2 ( #52506 ), B12X causal paged attention for SM120/SM121 via --attention-backend B12X_ATTN ( #52017 ), FlashInfer XQA decode on SM90 ( #50439 ), DeepSelect top-k with --sparse-indexer-topk-backend ( #56464 ), BF16x3 router GEMM default on SM100 for FP32-router models ( #55899 ), split-row Triton top-p for small batches ( #54651 ), manual ActivationQuantFusionPass producer for Llama ( #51415 ), fused add-RMSNorm + NVFP4 ( #51925 ), prefix-prefill tiling independent of KV page size ( #54194 ), FlashInfer GDN prefill on SM12x ( #55715 ), fused GDN MTP decode for SM110 ( #53835 ), masked MoE activation for padded layouts ( #51217 ), cooperative batched_moe_align_block_size ( #53280 ), PDL in fusedQKNormRopeKernel ( #55755 ) and weight prefetch in fused q/kv RMSNorm ( #55020 ), XD-RoPE folded into M-RoPE ( #56078 ), and the FlashAttention pin synced with upstream ( #54819 ). Startup and warmup : renderer warmup overlapped with engine core init ( #54557 ), FlashInfer autotune no longer rerun on each source change ( #54794 ), Qwen3.5/Qwen3-Next Triton kernels ( #54797 ) and GDN gated RMSNorm ( #54251 ) warmed at startup, Gemma 4 de-JITified ( #55768 ), DeepSeek V4 and shared attention, sampling, DFlash and mHC kernels migrated to the JIT warmup registry ( #50175 , #53564 , #53565 , #50176 , #53566 , #56323 , #50178 ) with parallel mHC pre-norm compilation ( #56683 ), FlashInfer deferred MoE decode kernels autotuned before capture ( #55377 ), and supplemental BF16 autotuning isolated ( #57285 ). Fast Start : weight-cache daemon with --load-format ipc_cache ( #54921 ), FP4 checkpoints ( #55465 ), multi-node TP ( #55468 ), GPU UUID socket folders ( #56669 ), per-client IPC tensor export so a copy-mode client cannot release zero-copy weights ( #56472 ), and a clear error on non-CUDA/ROCm platforms ( #56010 ). Watermarking : Gumbel-max watermarked generation and detection ( #54053 ) and dual-key Gumbel-max for speculative decoding with allow_target_only_watermarking and context deduplication ( #56122 ). Sleep and RL : NCCL communicator memory released in sleep mode ( #51485 ), DP state synced on the first step of a wave so pause and sleep stop burning 32 dummy forwards per rank ( #52957 ), pre-shared ncclUniqueId rendezvous for torch-free trainers ( #53784 ), stale HPC QK-norm weights after refit fixed ( #56706 ), and session_id on GPU BlockStored events ( #51381 ). Robustness : attention implementations must explicitly declare DCP support ( #55780 ), unavailable piecewise CUDA graphs raise instead of producing garbled output ( #54782 ), non-compiled models fall back to FULL_DECODE_ONLY graphs ( #55095 ), MRV1 breakable graphs scoped to the piecewise path ( #56312 ), sync checks cover paged async copies ( #53491 ) and torch.compile ( #56904 ), more H2D copies pinned ( #54660 , #55202 , #56138 ), GDN capture metadata built without a device sync ( #55404 ), queued work carried over when materializing the dedicated stream ( #56382 ), FlexAttention recompiles avoided ( #55888 ), persistent top-k falling back on low-shared-memory GPUs ( #54110 ), KV page size scaled for hidden-state extraction with TP ( #50894 ), out-of-bounds access in SM90 sparse MLA mixed batches ( #56969 ), sparse-MLA piecewise capture crash ( #56825 ), TRT-LLM ragged prefill regression ( #55499 ), MoE fused sum int32 overflow ( #50220 ), odd-row per-token-group quant cliff ( #56478 ), dummy-load re-initialization ( #56682 , #56688 ), warning_once retaining the LLM instance ( #54098 ), DSpark cache-group capability under python -O ( #55234 ), sparse-MLA DCP stabilized for GLM PCP configurations ( #55879 ), DeepGEMM FP8 warmup coverage ( #56452 ), Triton autotune inspection skipped without Triton ( #56676 ), mm_device_do_normalize under encoder CUDA graphs ( #55370 ), VidCom2 frame budgets read once ( #55331 ), the CUDA profiler start fix ( #55237 ), FlashInfer all-reduce workspace failures logged as errors ( #55127 ), and backend-neutral KV cache and MFU log lines ( #55650 ). Hardware & Performance NVIDIA : FlashInfer PCIe IPC all-reduce ( #53576 ), DeepGEMM pinned to the vLLM fork 2.8.0 with SM120 and SM90 paged-MQA ports ( #56876 ), public CUDA 13.4 Rubin build path ( #54640 , #56545 ), SM12x blockwise FP8 CTA raster swizzle for GB10/DGX Spark ( #55180 ), W4A4 NVFP4 preferred on SM120/121 ( #55170 ), H20 block-FP8 MoE tuning (+21%, #54668 ), an A100 PCIe fused-MoE config for Qwen3.5-122B TP2 ( #55511 ), ReLU2 fused with static FP8 quant for Nemotron-H ( #53793 ), and NVFP4 KV cache FMHA sped up ( #55031 ). AMD ROCm : TheRock base bumped to ROCm 10.0 ( #55246 ) with nightly TheRock images ( #55014 ), AITER 0.1.21.post2 ( #52826 , #55968 ), AITER custom AG/RS on the DP group (about 3% TPOT, #48247 ), multi-stream performance and rocprofiler fixes ( #55099 , #56098 ), bpreshuffled blockscaled FP8 GEMM (+4-8% QPS on DSv3, #51692 ), low-M FP32 router GEMM on gfx950 ( #54845 ), DeepSeek V4 FHMoE shared-expert fusion ( #53161 ), CSA multi-stream overlap ( #51794 ), AITER OPUS for large sparse prefill ( #54855 ), DSv4.1 mHC through AITER ( #56503 , #56513 ), MXFP8 dequant-once ( #56560 ), strided DSA decode mask ( #56628 ) and K=512 top-k tuning ( #56743 ), MiniMax-M3 indexer and top-k work ( #54682 , #52664 , #55235 , #56170 ) with fused allreduce+GemmaRMSNorm ( #54787 ), Kimi-K3 fused KDA prefill relanded ( #54038 ) and fused MLA norms ( #52494 ), GLM-5.2 kept on MRV1 ( #53155 ), GLM-5.3-Flash MTP through ragged sparse MLA ( #55239 ), W4A16 packed zero-points (-26% TPOT on Gemma 4 AWQ, #54965 ), Triton W4A16 GEMM as a custom op ( #51453 ), split AITER CK/Triton MXFP4 backends ( #50622 , #54773 ), RDNA3 W4A16 MoE on the oracle path ( #55522 ), Quark per-block FP8 MoE ( #52263 ), KV connectors with ROCM_AITER_UNIFIED_ATTN ( #53695 ), MoRI-IO SWA+HMA ( #53721 ), attention sinks in AITER sparse MLA ( #54404 ), non-causal DSpark drafts in AITER MLA decode ( #55966 ), shared KV prefill ( #55887 ), V-cache padding masked ( #53856 ), elastic EP deadlock ( #56610 ), profiler fixes ( #54171 , #56190 ), breakable CUDA graphs auto-enabled for DeepSeek-V4.1 ( #56349 ), the AITER paged-MQA outputs guard removed for DeepSeek-V4 ( #55808 ), DSV4 indexer gather cold-compile churn removed ( #56153 ), PDL JIT warmup skipped where unsupported ( #56035 ), Kimi-K3 fused prefill state_indices crash ( #56526 ), Ray NIXL agents for sharded RDT ( #56687 ), Qwen3-Omni audio encoder TP with non-divisible heads ( #45900 ), Quark MXFP4 Qwen3.8-Flash-Next shared experts on gfx950 ( #55213 ), AITER preshuffled FP8 block-scale kernel ( #56433 ), linear biases with requires_grad=False ( #56161 ), a clearer unsupported KV dtype error ( #55236 ), Mooncake in the ROCm image ( #55002 ), and AINIC pins ( #54112 , #56459 ). Intel XPU : activation CustomOps on SYCL kernels ( #53734 ), batch-invariant matmul and linear kernels ( #49209 ), fused GemmaRMSNorm ( #53678 ), fused_input_norm ( #52945 ) and fused grouped_topk ( #53580 ), LoRA for DeepSeek V4 ( #53689 ), DP external LB device assignment ( #53037 ), triton-xpu 3.8.0 shim ( #56014 ), auto-round-lib 0.15.0 ( #54515 ), and fixes for FalconH1 PP ( #55913 ), Ernie4.5-VL RoPE ( #55942 ), gated RMSNorm ops ( #54968 ), Qwen DFlash context keys ( #56431 ), INC quant selection ( #47237 ), and moe_wna16 loading ( #52651 ). CPU : DeepSeek-V4 backend ( #55355 ), native AMX-FP8 attention for Diamond Rapids with the Docker base moved to Ubuntu 25.04 and GCC 15 ( #49410 ), faster LM head on Arm (about 6% on Llama-3.1-8B W8A8, #55352 ), DeepSeek-R1 FP8 MLA + MoE correctness ( #56773 ), profiler tables grouped by input shape ( #56016 ), a dedicated Zen5 image ( #50314 ), and nightly CPU images on Docker Hub ( #55163 ). Large Scale Serving Context parallelism : PCP+DCP on sparse-MLA models ( #56157 ), PCP with single-module MTP and replicated DSpark ( #56107 ), PCP decode-only FULL CUDA graphs ( #53867 ), FlashMLASparse MTP at CP interleave > 1 ( #56722 ), symmetric DCP disaggregation for hybrid Mamba models ( #55531 ), DCP empty-shard LSE mask fused into the A2A pack kernel ( #54889 ), DCP prefill keys materialized on non-owner ranks ( #54908 ), DCP indexer interleave guard removed ( #55802 ) and respected in the PCP gather ( #56715 ), PCP producer shards as NIXL transfer ranks ( #56645 ), replicated-PCP ranks reporting done ( #53903 ), the DCP interleave adjustment scoped to NixlConnector ( #54803 ), PCP counted in multi-node world size ( #55111 ), PCP-DP validation moved to the GPU manager ( #54523 ), and one-sided EP at DP=1 ( #54016 ). Elastic EP and MoE communication : CUDA graphs reused across reconfiguration ( #54985 ), routing only to surviving engines on scale-down ( #55772 ), DeepEP v2 async finalize ( #52781 ), FlashInfer PCIe IPC all-reduce ( #53576 , #54869 ), batched CUTLASS workspace sizing ( #55579 ), and PP never dropping a decoding request from the sampled-token broadcast ( #54436 ). KV connectors : Mooncake Store heterogeneous TP sharing ( #53129 ), heterogeneous PP completion ( #56033 ), replicated GQA heads ( #52516 ), physical-block transfer length ( #54272 ), producer partial Mamba tails ( #52832 ), hybrid-model finish-time saves ( #54643 ), failed remote loads reported to the scheduler ( #50984 ), QSA ring groups excluded ( #55027 ); NIXL per-region transfer geometry ( #53780 ), int32 indices ( #51952 ), multi-handle race ( #56104 ), no eviction mid-transfer ( #54689 ), full prefix hits reported finished ( #56640 ), notification-only requests ( #57570 ), and double cleanup ( #54518 ); DecodeBench HMA, DCP, prefix and circular-buffer fixes ( #54647 , #54679 , #54878 , #54879 ); connector block tables resolved for every scheduled request ( #54853 ), hybrid KV load failures ( #50388 ), and lmcache_mp_connector state transitions guarded ( #47505 ); HiSparse per-layer offsets ( #57027 ), P/D region-mapped pulls ( #57077 ) and NIXL full-block imports ( #57049 ); NixlPush remote-agent reads guarded ( #56317 ). KV offloading : KVCR secondary-tier adapter ( #53624 ), P2P tier unbound_store_timeout_s and late-fetch rejection ( #53453 ), OffloadingConnector retention interval ( #51886 ) and offload hits under MTP/EAGLE ( #52771 ), SimpleCPUOffload fine-grained hybrid prefix hits ( #54736 ), eager registration and final flush ( #53532 ), BlockStored metadata ( #54325 ), mixed page sizes for DSA models ( #54756 ), prefix-cache bypass ( #54998 ), no-forward-step stores ( #56621 ); UniformTypeKVCacheSpecs portability ( #51690 ) and DCP scaling ( #50883 ); and fixes for async lookups ( #54872 , #55075 , #55823 ), SWA reachability and coverage ( #54362 , #55712 , #56486 ), final-token slot ( #54288 ), oversized offers ( #54759 ), ARC fallback ( #55461 ), disk alignment ( #55424 ), DiskBackend buffer race ( #51667 ), offload key ordering ( #52923 ), prefetch slot ownership ( #54975 ), load boundaries ( #52807 ), and tiering shutdown ( #52290 ). Encoder cache and EPD : P2P NIXL + CPU EC connector ( #47941 ), ECMooncakeConnector ( #41567 ), embedding inputs auto-enabled on EC/KV consumers ( #56379 ), GPU NVDEC for encoder-only instances ( #53675 ), lightweight multimodal metadata from render ( #54659 ) as JSON arrays ( #56090 ), reduced proxy serialization overhead ( #56657 ), media options and UUIDs preserved ( #56786 , #56432 ), per-item EC checks ( #56070 ), ECCPUWorker backend extension points ( #54033 ), and fixes for old kernels ( #53190 ), remote encoding failures ( #55290 ), and TP>1 load devices ( #40416 ). Quantization New capabilities : targeted online quantization ( #51285 ) and online quantization of partially pre-quantized checkpoints ( #51392 ), W4A16 DSA ( #51724 , #55538 ), FlashInfer CuTeDSL NVFP4 W4A16 ( #53014 ), NVFP4 torch backend (#53319), linear_backend_per_quant (#51204), AutoRound 2/3/5/6/7-bit (#52890), TRT-LLM FP8 block-scale MoE with SwiGLU clamp (#55069), and Quark per-block FP8 MoE on ROCm (#52263). Refactors : Quark dispatch on QuantKey (#52958) and ModelOpt linear methods unified into one QuantKey-driven method (#49381). Fixes : NVFP4 padding initialized in-kernel (#53568), unloaded NVFP4 scales detected with a NaN sentinel (#52501), NVFP4 fused SiLU+mul scales (#55643), Quark per-block scales registered as weight_scale (#54770), GPT-OSS FP8 MoE with Quark (#54824), FSE detection for Quark models (#54573), DSv4 MXFP4 backend alias selection (#53586), unset group_size on compressed-tensors WNA16 MoE (#53163), block FP8 MTP in ModelOpt mixed checkpoints (#55513), and unquantized MoE weight storage preserved on ROCm for RL updates (#46009). API & Frontend New endpoints and options : stateless /v1/responses/render (#50195), scale-out endpoints behind --enable-scale-out (#54579, #55176), count_reasoning_tokens so usage reports reasoning_tokens (#54982), prompt metadata from /inference/v1/generate (#53187), vllm:request_num_preemptions (#49984), MFU/MBU for SWA and hybrid layers (#55624), consolidated Prometheus bucket defaults (#48866), site-packages reasoning and tool parser plugins (#45241), and max_num_queued_reqs shared across API server processes (#54746). Structured output and parsers : XGrammar patternProperties / propertyNames / unevaluatedProperties (#42904), full-history reasoning scans eliminated (#55223) with is_reasoning_end derived from the grammar (#56200), invalid structured-output requests no longer stopping the engine (#51450), a unified Cohere Command parser (#56392), MCP SDK 2.x schemas (#53870), incomplete tool-call markup dropped consistently (#47562), DSML tool calls without the wrapper (#55954), misspelled wrappers (#56141) and implicit parameter closes (#54838), Gemma4 malformed openers (#53444), Seed-OSS turn boundaries (#54264), Step-3.5 with structured outputs (#53174), reasoning parser plugins loaded before headless config (#53124), and a warning for removed guided_* fields (#54285). OpenAI, Anthropic and Cohere compatibility : unsupported Responses input items return 400 (#55974), browser.find action type (#55305), Responses validation boundary restored (#55761), validation errors migrated to VLLMValidationError (#50254, #50257, #55701, #55735), unsupported reasoning_effort as a client error (#54022), content format detected through macro parameters (#53824), Anthropic stop_sequence in message_delta (#55325), Cohere priorities bounded (#56146), translation rejecting string files (#55618), tokenless chat and audio streams rejected (#54708), beam search respecting skip_special_tokens (#56211) and aborts (#56249), nested cancellation handlers removed (#55602, #57024), extension integers validated (#55606), routed-expert prompt offsets validated (#56844), MistralCommonBackend avoided for HF tokenizers (#54192), chat template fallbacks packaged (#53762), and the content format warning restored (#54622). Render and pooling : assistant_tokens_mask (#54539) and token offsets (#54692) kept aligned with truncated prompts, skip_decoder_start_token honored (#54799), renderer warmup bounded (#55448), LoRA adapter names in pooling responses (#56004), max_embed_len for chunked embeddings (#55551), body request_id honored (#55665), actual input usage for scoring (#56573), STEP token pooling (#55307), offline Jina token limits (#56415), and a warning for Qwen3 rerankers without a chat template (#56017). Rust frontend : TLS for the render server (#54999), --lora-modules (#54837), HTTP RL weight synchronization (#56567), HF revisions, offline mode and cache directory (#56386), {% generation %} chat template blocks (#56378), the vllm-proto crate on crates.io (#56365), reasoning_tokens in usage (#54883) built on token-attributed text (#52910, #54884), Mooncake/NIXL connector metrics (#52755) and vllm:iteration_tokens_total (#56990), gRPC preprocessed multimodal features (#55047), remote-prefill multimodal metadata (#54814) and engine errors surfaced as INTERNAL (#56405), Qwen4-exp multimodal (#54813), optional --max-model-len for render (#55328), per-request vocabulary counting removed (#54449), decoded chunks coalesced per update (#55012), bounded recursive parsers (#54303), tekken 0.2.0 (#53056), strict whitespace framing (#55417), consistent parser resolution (#56018), empty structured-output values rejected (#54821), selected-token logprobs (#56406), undefined token ids skipped (#55240), DeepSeek renderer fixes (#53281, #54854, #56593, #56260, #56366), and HF ShareGPT multi-turn datasets in vllm bench serve (#51104, #56915). CLI and benchmarks : vllm bench serve accounts for client queue time (#54136) with consistent streaming TTFT/E2E accounting (#55508), non-positive --max-concurrency rejected (#54887), mm-processor crash fixed (#56300), complete pooling responses measured (#56760), explicit empty and zero arguments honored in vllm chat / complete (#55710), vllm launch render shutdown hang fixed (#54913), double /stop_profile crash fixed (#51678), and remaining non-OpenAI content moved out of the openai folder (#56369). Security Validation-error response bodies bounded, closing an about 5,300x response amplification (#54684); client-supplied sparse embeddings bounded before densification (#54632). Request-controlled video sampling capped for GLMGA (#54935) and Qwen-VL (#56729) backends. cache_salt validated before reaching LMCache so one request cannot take down the engine (#51444); scale-out multimodal features validated before engine handoff (#51898); late-interaction query caches keyed by server-generated ids instead of X-Request-Id (#51445). Credentials redacted from benchmark logs (#56662); Rust frontend HTTP method labels normalized to prevent unbounded Prometheus series (#56058). Dependencies CUTLASS 4.7.1 (#54190) and CUTLASS DSL 4.7 (#54927), Transformers 5.16.1 (#53905), tpu-inference v0.29.0 (#56388), openai >= 2.25.0 (#49104), huggingface_hub >= 1.31.0 with httpx imported through it (#56460). DeepGEMM pinned to the vllm-project fork 2.8.0 (#56876) with CUDA 12.9 release builds fixed (#57554); FlashMLA V4.1 record pin (#56893); FlashKDA bumps (#54859, #56485); FlashAttention synced with upstream (#54819); _USE_MATH_DEFINES defined for FlashMLA targets (#54007). Public CUDA 13.4 Rubin build path and dependency pins (#54640, #56545). ROCm: TheRock base on ROCm 10.0 (#55246), AITER 0.1.21.post2 (#52826, #55968), Mooncake from public wheels (#55002), AINIC libionic 54.0-187-1 pinned to a snapshot (#54112, #56459). XPU: triton-xpu 3.8.0 shim (#56014), auto-round-lib 0.15.0 (#54515). CPU: Docker base on Ubuntu 25.04 with GCC 15 (#49410), Zen5 image (#50314), nightly CPU images (#55163). Obsolete TPU Dockerfile removed (#55376). Breaking Changes & Deprecations Scale-out endpoints ( /render , /derender , /inference/v1/generate ) are no longer registered on plain vllm serve unless --enable-scale-out is passed; VLLM_ENABLE_SCALE_OUT_ENDPOINTS was removed. vllm launch render and vllm serve --tokens-only always register them (#54579, #55176). GPTQ group/dynamic activation ordering removed: g_idx is ignored and the related Marlin, GPTQ, CPU and RDNA3 kernels are gone (#54809). Items deprecated for 0.29 removed: VLLM_PREFIX_CACHE_RETENTION_INTERVAL and VLLM_MM_HASHER_ALGORITHM env vars (use the config fields), the use_fp4_indexer_cache alias (use indexer_kv_dtype ), the CUDA_VISIBLE_DEVICES fallback on ROCm (use HIP_VISIBLE_DEVICES ), and the seq_lens_cpu / num_computed_tokens_cpu attention metadata properties (#55353). The all Mamba cache mode is deprecated and falls back to Model Runner V1 (#55041). python -m vllm.entrypoints.grpc_server is deprecated; use vllm serve --grpc (#56746). sse_keep_alive moved to vllm.entrypoints.serve.utils (#56369). YaRN aligned with Transformers: vendor YaRN aliases no longer multiply max_position_embeddings by the factor a second time, so derived max_model_len drops for some models (TeleChat3-36B-Thinking 131072 to 32768, sarvam-105b 5242880 to 131072); attn_factor and extrapolation_factor are ignored in favor of mscale / attention_factor (#56446). Attention implementations must explicitly declare DCP support; DCP with ROCm standard attention, Triton, FlexAttention or TurboQuant now fails at backend selection (#55780). MoRI-IO connector WRITE mode with hybrid KV cache groups now requires --disable-hybrid-kv-cache-manager ; use READ mode otherwise (#53721). Default audio resampler switched from PyAV to torchaudio (#52598); audio decoding is backend-selectable with soundfile first in auto (#51826, #55642). New defaults: FlashInfer CuTeDSL NVFP4 W4A16 over Marlin on SM100/103 (#53014), W4A4 NVFP4 over weight-only kernels on SM120/121 (#55170), BF16x3 router GEMM on SM100 (#55899), DeepEP v2 combine overlap on with VLLM_DEEPEP_V2_COMBINE_OVERLAP=0 to disable (#52781), AITER custom AG/RS on for DP attention + TP experts with VLLM_ROCM_USE_AITER_CUSTOM_AR=0 to disable (#48247), native CUDA AttnRes for Kimi-K3 on SM100 (#54261), and VLLM_USE_BREAKABLE_CUDAGRAPH=1 no longer overriding FULL graphs on MRV1 (#56312). New Contributors @100milliongold made their first contribution in https://github.com/vllm-project/vllm/pull/56017 @404mario made their first contribution in https://github.com/vllm-project/vllm/pull/52781 @982945902 made their first contribution in https://github.com/vllm-project/vllm/pull/52923 @abrahamzewoudie made their first contribution in https://github.com/vllm-project/vllm/pull/45900 @aijanai made their first contribution in https://github.com/vllm-project/vllm/pull/51678 @Aj2280 made their first contribution in https://github.com/vllm-project/vllm/pull/55728 @Akine-Ko made their first contribution in https://github.com/vllm-project/vllm/pull/54033 @akshaver made their first contribution in https://github.com/vllm-project/vllm/pull/53945 @alexliluz made their first contribution in https://github.com/vllm-project/vllm/pull/53124 @am-cohere made their first contribution in https://github.com/vllm-project/vllm/pull/54774 @amd-xavierwang made their first contribution in https://github.com/vllm-project/vllm/pull/56153 @andy-neuma made their first contribution in https://github.com/vllm-project/vllm/pull/50314 @AnshulDesai made their first contribution in https://github.com/vllm-project/vllm/pull/55271 @ArcheyChen made their first contribution in https://github.com/vllm-project/vllm/pull/54436 @arcusbuilds made their first contribution in https://github.com/vllm-project/vllm/pull/54007 @averma12 made their first contribution in https://github.com/vllm-project/vllm/pull/56721 @bakiburakogun made their first contribution in https://github.com/vllm-project/vllm/pull/55511 @BANANASJIM made their first contribution in https://github.com/vllm-project/vllm/pull/55301 @Big2Wheel made their first contribution in https://github.com/vllm-project/vllm/pull/54975 @bjf-frz made their first contribution in https://github.com/vllm-project/vllm/pull/55819 @bojiang3 made their first contribution in https://github.com/vllm-project/vllm/pull/54306 @BolinSNLHM made their first contribution in https://github.com/vllm-project/vllm/pull/54606 @BruceLoveDecimal made their first contribution in https://github.com/vllm-project/vllm/pull/50984 @byshiue made their first contribution in https://github.com/vllm-project/vllm/pull/55054 @cenab made their first contribution in https://github.com/vllm-project/vllm/pull/55710 @circlepen made their first contribution in https://github.com/vllm-project/vllm/pull/56388 @colinmcnamara made their first contribution in https://github.com/vllm-project/vllm/pull/55325 @DebugSy made their first contribution in https://github.com/vllm-project/vllm/pull/55111 @dharak-cohere made their first contribution in https://github.com/vllm-project/vllm/pull/53784 @djw8605 made their first contribution in https://github.com/vllm-project/vllm/pull/54962 @eastwood-c made their first contribution in https://github.com/vllm-project/vllm/pull/55745 @erdholion made their first contribution in https://github.com/vllm-project/vllm/pull/54285 @gc-fu made their first contribution in https://github.com/vllm-project/vllm/pull/55310 @happyyzy made their first contribution in https://github.com/vllm-project/vllm/pull/50220 @heliubj18 made their first contribution in https://github.com/vllm-project/vllm/pull/54756 @Ianniu123 made their first contribution in https://github.com/vllm-project/vllm/pull/55650 @ItsRoy69 made their first contribution in https://github.com/vllm-project/vllm/pull/55283 @jackLei0901 made their first contribution in https://github.com/vllm-project/vllm/pull/49869 @Jankwi made their first contribution in https://github.com/vllm-project/vllm/pull/54220 @JaynouOliver made their first contribution in https://github.com/vllm-project/vllm/pull/47562 @jcayab made their first contribution in https://github.com/vllm-project/vllm/pull/54584 @jorge-menjivar made their first contribution in https://github.com/vllm-project/vllm/pull/53056 @josiahdavis made their first contribution in https://github.com/vllm-project/vllm/pull/56286 @jschmied made their first contribution in https://github.com/vllm-project/vllm/pull/55180 @jyizheng made their first contribution in https://github.com/vllm-project/vllm/pull/54518 @kamb-code made their first contribution in https://github.com/vllm-project/vllm/pull/52771 @krsish made their first contribution in https://github.com/vllm-project/vllm/pull/55307 @kushaldabbe made their first contribution in https://github.com/vllm-project/vllm/pull/55548 @LauraGPT made their first contribution in https://github.com/vllm-project/vllm/pull/54944 @LearningMachine621 made their first contribution in https://github.com/vllm-project/vllm/pull/49675 @linamy85 made their first contribution in https://github.com/vllm-project/vllm/pull/49984 @lucamotz made their first contribution in https://github.com/vllm-project/vllm/pull/54110 @luoyuctl made their first contribution in https://github.com/vllm-project/vllm/pull/56387 @luyixiao95 made their first contribution in https://github.com/vllm-project/vllm/pull/52285 @lvliang-intel made their first contribution in https://github.com/vllm-project/vllm/pull/47237 @lzzzzzc made their first contribution in https://github.com/vllm-project/vllm/pull/55864 @maireneu made their first contribution in https://github.com/vllm-project/vllm/pull/54533 @mevince made their first contribution in https://github.com/vllm-project/vllm/pull/54325 @migarci2 made their first contribution in https://github.com/vllm-project/vllm/pull/55602 @mingg26 made their first contribution in https://github.com/vllm-project/vllm/pull/53524 @miyakido made their first contribution in https://github.com/vllm-project/vllm/pull/40416 @mlazos made their first contribution in https://github.com/vllm-project/vllm/pull/53319 @mohit-sarvam made their first contribution in https://github.com/vllm-project/vllm/pull/53052 @natsala13 made their first contribution in https://github.com/vllm-project/vllm/pull/55178 @null-Exception1 made their first contribution in https://github.com/vllm-project/vllm/pull/54265 @odashi made their first contribution in https://github.com/vllm-project/vllm/pull/45241 @Oskii made their first contribution in https://github.com/vllm-project/vllm/pull/55618 @pra2107tham made their first contribution in https://github.com/vllm-project/vllm/pull/54622 @Roderick-Wu made their first contribution in https://github.com/vllm-project/vllm/pull/54809 @SalikovAlex made their first contribution in https://github.com/vllm-project/vllm/pull/56414 @seanwestfall made their first contribution in https://github.com/vllm-project/vllm/pull/55691 @SherifWaly made their first contribution in https://github.com/vllm-project/vllm/pull/53379 @somuai made their first contribution in https://github.com/vllm-project/vllm/pull/55369 @specture724 made their first contribution in https://github.com/vllm-project/vllm/pull/50945 @StevenWang-CY made their first contribution in https://github.com/vllm-project/vllm/pull/54819 @stu-cao made their first contribution in https://github.com/vllm-project/vllm/pull/55802 @subhashpolisetti made their first contribution in https://github.com/vllm-project/vllm/pull/54633 @surajm20061998 made their first contribution in https://github.com/vllm-project/vllm/pull/55508 @Sy0307 made their first contribution in https://github.com/vllm-project/vllm/pull/55212 @tanyuqian made their first contribution in https://github.com/vllm-project/vllm/pull/55063 @TensorRaya made their first contribution in https://github.com/vllm-project/vllm/pull/55908 @Ubospica made their first contribution in https://github.com/vllm-project/vllm/pull/56408 @wei-core made their first contribution in https://github.com/vllm-project/vllm/pull/53835 @Whamp made their first contribution in https://github.com/vllm-project/vllm/pull/54362 @Woolgathererer made their first contribution in https://github.com/vllm-project/vllm/pull/56652 @wuhangxian made their first contribution in https://github.com/vllm-project/vllm/pull/53824 @xiaozhenbi made their first contribution in https://github.com/vllm-project/vllm/pull/56671 @xonder made their first contribution in https://github.com/vllm-project/vllm/pull/43272 @xuhuan51 made their first contribution in https://github.com/vllm-project/vllm/pull/51381 @YeonwooSung made their first contribution in https://github.com/vllm-project/vllm/pull/56593 @ykamiset made their first contribution in https://github.com/vllm-project/vllm/pull/52664 @YoavMiron made their first contribution in https://github.com/vllm-project/vllm/pull/54574 @yongqinwang-cmd made their first contribution in https://github.com/vllm-project/vllm/pull/50514 @Yuzu23 made their first contribution in https://github.com/vllm-project/vllm/pull/45091 @yzeyu71 made their first contribution in https://github.com/vllm-project/vllm/pull/50504 @z-zanez made their first contribution in https://github.com/vllm-project/vllm/pull/53129 @zdtsw made their first contribution in https://github.com/vllm-project/vllm/pull/54999 @zhang-keliang made their first contribution in https://github.com/vllm-project/vllm/pull/55288 @zhaoguochun1995 made their first contribution in https://github.com/vllm-project/vllm/pull/56382 @zhouyou9505 made their first contribution in https://github.com/vllm-project/vllm/pull/54659 @zigzagcai made their first contribution in https://github.com/vllm-project/vllm/pull/54560 @zqlcode made their first contribution in https://github.com/vllm-project/vllm/pull/56071 @zvier made their first contribution in https://github.com/vllm-project/vllm/pull/56103 @zwang86 made their first contribution in https://github.com/vllm-project/vllm/pull/54098 Contributors @khluu , @AndreasKaratzas , @taneem-ibrahim , @BugenZhao , @yewentao256 , @njhill , @NickLucche , @aoshen02 , @LopezCastroRoberto , @WoosukKwon , @LucasWilkinson , @zyongye , @hmellor , @Fangzhou-Ai , @sfeng33 , @Isotr0py , @gau-nernst , @JaredforReal , @wzhao18 , @fxmarty-amd , @stefankoncarevic , @Alex-ai-future , @mgoin , @micah-wil , @mayuyuace , @djramic , @ZJY0516 , @mawong-amd , @Rohan138 , @sychen52 , @waizuichougou , @chaunceyjiang , @simondanielsson , @KernelClint , @AdaAibaby , @gty111 , @zhenwei-intel , @zxd1997066 , @majunze2001 , @divakar-amd , @zhewenl , @giuseppegrossi , @Levius-Fubuki , @gcanlin , @jperezdealgaba , @bigPYJ1151 , @stecasta , @linitra24 , @chaojun-zhang , @shaohuaxi , @reidliu41 , @atalman , @mfylcek , @lucifer1004 , @ZeldaHuang , @askliar , @peakcrosser7 , @yzeyu71 , @hungnnvidia , @zupengwang , @TheEpicDolphin , @hlin99 , @liusy58 , @jyizheng , @jimmy-adams , @lucamotz , @jeejeelee , @shen-shanshan , @MatthewBonanni , @cjackal , @aarushjain29 , @noooop , @Juntian777 , @FeathBow , @Hotragn , @Jankwi , @lzhan011 , @subhashpolisetti , @maithilijoshi20 , @ilmarkov , @lukealonso , @mingg26 , @franciscojavierarceo , @mevince , @zdtsw , @bnellnm , @omerpaz95 , @jikunshang , @rbrugaro-amd , @bohnstingl , @specture724 , @afierka-intel , @mohit-sarvam , @andylolu2 , @pisceskkk , @starkwj , @wangyicong52 , @eastwood-c , @wuhangxian , @yzong-rh , @yma11 , @TQCB , @Sy0307 , @jbyczkow , @itayalroy , @xiaguan , @alec-flowers , @ayush1399 , @orestis-z , @mobicham, @ArcheyChen , @Zhenzhong1, @ZhengGong-amd, @jorge-menjivar , @IwakuraRein, @linamy85 , @wentian-byte, @afriedri, @dharak-cohere , @DarkLight1337, @lvliang-intel , @zigzagcai , @harjothkhara, @bojiang3 , @maireneu , @z-zanez , @JaynouOliver , @pra2107tham , @jdebache, @aijanai , @Dao007forever, @juhi10071998, @odashi , @Etelis, @fjosw, @jackLei0901 , @tarukumar, @positive666, @tzielinski-habana, @drakosha, @iyastreb, @Harry-Chen, @sheralskumar, @ccrhx4, @aaab8b, @982945902 , @tanyuqian , @DebugSy , @jasonozuzu-cohere, @byshiue , @yifjiang, @BolinSNLHM , @wseaton, @Yuzu23 , @andyxning, @andy-neuma , @foraxe, @xuhuan51 , @AnshulDesai , @luyixiao95 , @null-Exception1 , @yiz-liu, @zhang-keliang , @erdholion , @happyyzy , @GirasoleY, @SubSir, @natsala13 , @yongqinwang-cmd , @StevenWang-CY , @LauraGPT , @wei-core , @zhejiangxiaomai, @bakiburakogun , @Whamp , @almogtavor, @zhouyou9505 , @jschmied , @wenhuach21, @Big2Wheel , @vhagor, @frankie-ys, @zufangzhu, @stmatengss, @seanwestfall , @kamb-code , @somuai , @Josephasafg, @Aj2280 , @am-cohere , @Sip4818, @liuyao0322, @heliubj18 , @krsish , @Oskii , @xijiaat, @benenzhu, @tdoublep, @akshaver , @SherifWaly , @TensorRaya , @shikamd123, @Roderick-Wu , @lcskrishna, @BANANASJIM , @soodoshll, @mlazos , @Akine-Ko , @miyakido , @meena-at-work, @nikhilkulkarni1755, @jcayab , @wincent8, @abrahamzewoudie , @ronensc, @haic0, @gc-fu , @ykamiset , @Xarbirus, @zvier , @avininjamay8, @ColinZ22, @YunzhuLu, @jiacao-amd, @kliuae, @tobymao, @zexplorerhj, @ywang96, @YoavMiron , @xonder , @bjf-frz , @cenab , @josiahdavis , @juliendenize, @JiataiWang, @colinmcnamara , @tlrmchlsmth, @LearningMachine621 , @jamesETsmith, @YukioZzz, @kushaldabbe , @lzzzzzc , @SalikovAlex , @pmanczak, @fadara01, @JulienDarve, @nicole-lihui, @zRzRzRzRzRzRzR, @100milliongold , @labAxiaoming, @GuyStone, @Wauplin, @SorenDreano, @chengchengpei, @circlepen , @amd-xavierwang , @tahsintunan, @biswapanda, @junuxyz, @walterbm, @arcusbuilds , @JartX, @arpera, @jcotant-inferact, @PatrykSaffer, @maeehart, @surajm20061998 , @Woolgathererer , @kebe7jun, @ShengleiFu, @pavelzak, @stu-cao , @CedricHwong, @lxy-alexander, @BWAAEEEK, @Chinmay-Kulkarni-AMD, @mgehre-amd, @BruceLoveDecimal , @hickeyma, @jiangkuaixue123, @xiaozhenbi , @ganeshr10, @Ronald1995, @arkohut, @russellb, @zqlcode , @tomasruizt, @thillai-c, @samnordmann, @Ianniu123 , @Luosuu, @netanel-haber, @MicheleCampi, @zhaoguochun1995 , @JohnQinAMD, @liranschour, @jacklin78911-collab, @migarci2 , @404mario , @ItsRoy69 , @mkhazraee, @Ubospica , @Thangnguyenvn98, @YeonwooSung , @alexliluz , @ECMGit, @luoyuctl , @averma12 , @drslark, @ppalanga, @zwang86 , @wangshangsam, @jiahanc, @vllm-agent, @connorcarpenter15

Cloudflare Python Workers are now generally available After a two year preview, Cloudflare's support for running Python code in their server-side Workers platform is now stable: "Python is now a first-class, fully supported language on the Cloudflare Developer Platform". A neat thing about this is how it works. Cloudflare are running Python compiled to WebAssembly via Pyodide in their V8-based workerd runtime. This comes with some limitations, documented here - most notably both multiprocessing and threading are non-functional in the WebAssembly VM. One particularly interesting detail of this is the local development environment story - their pywrangler development tool (confusingly packaged as workers-py on PyPI) runs a full local simulation of their stack, including executing code with Pyodide in WebAssembly in V8 in a 123MB workerd binary, which for me ended up in node_modules/@cloudflare/workerd-darwin-arm64/bin/workerd . Python Workers represent a significant investment in the wider Python ecosystem by Cloudflare. The release announcement is credited to Gyeongjae Choi, Dominik Picheta, and Hood Chatham - Gyeongjae and Hood are both Pyodide core maintainers. Via Hacker News Tags: python , cloudflare , webassembly , pyodide

submitted by /u/Distinct-Question-16 to r/singularity [link] [comments]

Hi r/LocalLLaMA! We’re researchers at the Institute of Foundation Models (IFM), an AI research lab dedicated to open and independent development of frontier-class foundation models. We recently released K2 Horizon a connected fleet of six fully open models with size ranging from 0.9B to 375B. In addition to weights, we also open-sourced training data and recipes, training code, intermediate checkpoints, fine-grained training logs and evals. Ask us anything about pre-training and data mixes, post-training, small models on-device, MoVA and sparse attention, deployment, what’s out now, and what’s coming next. Participating in the AMA: Hector Liu u/hunterhector Alexander Moreno u/IFMAlex Mikhail Yurochkin u/my-moonfolk Rupesh Srivastava u/k2pt Junlin Chen u/Junlin_Chen110 Haonan Li u/East-Career9147 We'll be live Mon, Oct 5, 8–10 PM PT. Questions are open now, so drop yours anytime! https://preview.redd.it/awu0r6fbowsh1.png?width=3240&format=png&auto=webp&s=860018e2d8aeb7575a1ddd11ed2c2f735948938d submitted by /u/aya-ifm to r/LocalLLaMA [link] [comments]

Decision models Ollama now supports decision models through /v1/systemone , based on TypeSafe’s Jev API . Decision models return choices, probabilities, and scores instead of text. Use them for tasks such as ticket triage, model routing, and content classification. Available models: Nimble from Bespoke Labs Tev1 from Together AI ollama pull nimble Send context and one or more questions: curl http://localhost:11434/v1/systemone \ -H ' Content-Type: application/json ' \ -d ' { "model": "nimble", "state": "Our checkout has returned 500 errors since 9am.", "questions": { "label": { "type": "choice", "instructions": "Which label fits this ticket?", "criteria": { "billing": "Payments and refunds", "bug": "Software errors", "account": "Login and account access" } } } } ' Example response: { "model" : " nimble " , "answers" : { "label" : { "type" : " choice " , "choice" : " bug " , "probabilities" : { "billing" : 0.0125 , "bug" : 0.9781 , "account" : 0.0093 }, "confidence" : 0.8906 } }, "usage" : { "input_tokens" : 174 , "output_tokens" : 1 } } The API supports three question types: choice : Select an option and return probabilities for each. noul : Return the probability that a condition is true. score : Return a score across an ordered set of criteria What's Changed Settings now opens without waiting for model discovery. Fixed the macOS update menu and icon not reflecting an available update at startup. Fixed stalled MLX model downloads hanging indefinitely. Requests containing the deprecated typical_p parameter now log a warning instead of failing. Full Changelog: v0.34.4...v0.35.0

Claude Sonnet 5.5 New Sonnet model from Anthropic today. They say it "runs 30%+ faster, and costs up to 30% less for most work" - it's priced the same as Sonnet 5 but appears to beat it on every benchmark, and should be cheaper to run as well. Here are some pelicans riding bicycles . Sonnet 5.5 suffered from the same bug as Opus 5.5 : the "max" thinking effort pelican thought for 128,000 tokens (at a cost of $1.28) before running out of tokens and failing to produce an SVG. Here's the pelican it gave me for thinking effort "xhigh", at a cost of 5.74 cents and taking 41 seconds: Sonnet 5.5 appears to be almost as good as Opus 5.5 on some coding tasks, including various viral 3D animation tricks . The most interesting thing about Sonnet 5.5 is that it's now the model used for the free tier on claude.ai . OpenAI's ChatGPT free tier uses Luna 5.6, which means Anthropic currently have a much more capable free offering. I ran this prompt against that free tier: build me an HTML page that renders a three-dimensional pelican riding a bicycle using WebGL And got back this page , which is a solid effort. Anthropic's announcement reiterates that Haiku 5.5 will be available "in the coming weeks". I really hope that one is price-competitive with GPT-6 Luna! Tags: ai , generative-ai , llms , anthropic , claude , pelican-riding-a-bicycle , llm-release

A few notes on Anthropic's new Claude Fable 5.1 - with Max thinking level I got the best SVG pelican I've had from any Anthropic model (at a hefty cost of $3.30!), which I then had it animate simonwillison.net/2026/Sep/1/c...

What's Changed Structured outputs on thinking models now apply in a single pass, making them faster and more reliable. Fixed intermittent "model not found" errors with a large local library Fixed the macOS app becoming unresponsive when checking if ChatGPT or Codex is running. Qwen 3.8 prompt processing is faster on Apple Silicon. Gemma 4 on Apple Silicon now picks the best image resolution per image, keeping more detail in high-resolution images. Updated llama.cpp, MLX, and XGrammar. Full Changelog : v0.34.3...v0.34.4

What's Changed Models run on MLX on Apple Silicon by default In this release, on Apple Silicon devices, model architectures supported by the MLX runtime will automatically run on MLX. ollama pull qwen3.8 ollama run qwen3.8 During the pre-release we will be testing and enabling additional models. Full Changelog : v0.34.4...v0.40.0-rc0

hexagon: install rebuilt HTP skels ( #29828 ) hexagon: install rebuilt HTP skels Assisted-by: OpenCode hexagon: fix HTP skel catalog dependencies Assisted-by: OpenCode Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52179886 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries Ubuntu arm64 (CUDA 13) - CUDA 13.4 libraries Ubuntu x64 (ROCm 10.0) Ubuntu x64 (OpenVINO) Ubuntu x64 (SYCL FP32) Ubuntu x64 (SYCL FP16) Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide Android: Android arm64 (CPU) Android arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide Windows: Windows x64 (CPU) Windows arm64 (CPU) Windows arm64 (OpenCL Adreno) Windows x64 (CUDA 12) - CUDA 12.4 DLLs Windows x64 (CUDA 13) - CUDA 13.4 DLLs Windows arm64 (CUDA 13) - CUDA 13.4 DLLs Windows x64 (Vulkan) Windows x64 (OpenVINO) Windows x64 (SYCL) Windows x64 (ROCm 10.0) openEuler: DISABLED openEuler x86 (310p) openEuler x86 (910b, ACL Graph) openEuler aarch64 (310p) openEuler aarch64 (910b, ACL Graph) UI: UI

Release v5.17.0 New Model additions HYV4 Hy4-Preview is a 780B-parameter mixture-of-experts language model that activates 49B parameters per token. Each MoE layer holds 256 routed experts plus one always-active shared expert and routes every token to 8 of them. The context window is 1M tokens. The architecture combines four features: Multi-head Latent Attention (MLA) compresses keys and values into a low-rank latent ( kv_lora_rank ) that kv_b_proj expands back to one key/value per query head. DeepSeek Sparse Attention (DSA) selects index_topk keys per query with a lightweight indexer. Following IndexShare , only the layers marked "full" in indexer_types run an indexer; "shared" layers reuse the previous full layer's selection. Gated MLA with learnable attention sinks , where each head owns a sink logit that participates in the softmax and contributes no value, as in GPT-OSS . Independent Hyper-Connections (iHC) replace the plain residual path with hc_mult parallel residual streams that are collapsed before, and redistributed after, every sublayer. The implementation does not execute the multi-token prediction (MTP) layers. Released checkpoints keep those weights so that other runtimes can use them for speculative decoding; they are ignored at load time. Links: Documentation Add h4 ( #48473 ) by @ArthurZucker in #48473 VibeVoice VibeVoice is a novel framework for synthesizing high-fidelity, long-form speech with multiple speakers by employing a next-token diffusion approach within a Large Language Model (LLM) structure. It's designed to capture the authentic conversational "vibe" and is particularly suited for generating audio content like podcasts and multi-participant audiobooks. Links: Documentation Implement VibeVoice ( #40546 ) by @pengzhiliang in #40546 NeoMME NeoMME is a family of efficient 260M and 800M parameter multimodal-native multilingual foundation encoders from H Company. It processes multilingual text tokens and raw image patches in a single bidirectional Transformer encoder, without a separately pretrained vision tower or causal language model. NeoMME-Retriever is a model fine-tuned from the NeoMME backbone for visual document retrieval with joint late-interaction and dense objectives. It takes text queries and documents (text or page screenshots) and produces multi-vector embeddings for MeanMaxSim scoring (late-interaction) and mean-pooled embeddings for cosine similarity (dense). Links: Documentation Add NeoMME and NeoMME-Retriever ( #47992 ) by @tonywu71 in #47992 Fun-ASR-Nano Fun-ASR-Nano is an 800M-parameter end-to-end speech recognition model developed by Alibaba DAMO Academy's FunAudioLLM team. It achieves state-of-the-art performance on Chinese, English, and Japanese ASR benchmarks while being significantly smaller than comparable models. Key features are Chinese, English, and Japanese , including 7 Chinese dialects and 26 regional accents Hotword customization for domain-specific vocabulary Native punctuation output (no separate punctuation model needed) Links: Documentation Add Fun-ASR-Nano model ( #46180 ) by @LauraGPT in #46180 KimiLinear Kimi Linear is a hybrid linear attention architecture from Moonshot AI, introduced in Kimi Linear: An Expressive, Efficient Attention Architecture . At its core is Kimi Delta Attention (KDA) , a refinement of Gated DeltaNet that gives each key channel its own forget gate, so the recurrent state decays per channel instead of per head. KDA is used in most layers; every fourth layer keeps a full-attention block that reuses DeepSeek-V3's Multi-head Latent Attention (MLA), and the feed-forward blocks are DeepSeek-V3-style MoE with a shared expert. Links: Documentation Kimi linear ( #48250 ) by @remi-or in #48250 Canary Canary-1B-v2, a fast, robust multilingual model for Automatic Speech Recognition (ASR) and Speech-to-Text Translation (AST): Canary reuses the Fast Conformer encoder from Parakeet (loaded through [ ParakeetEncoder ] / [ ParakeetEncoderConfig ]) and pairs it with a Transformer decoder that uses fixed sinusoidal positional embeddings, cross-attention to the encoder outputs and tied input/output embeddings. The task is selected through a decoder prompt prefix built by [ CanaryProcessor ] of the form <|startofcontext|> <|startoftranscript|> <|emo:undefined|> <source_lang> <target_lang> <pnc|nopnc> <|noitn|> <|notimestamp|> <|nodiarize|> , where source_lang == target_lang selects transcription and otherwise selects translation. Links: Documentation model: Add NVIDIA Canary-1B-v2 to Transformers ( #46825 ) by @harshaljanjani in #46825 NeuCodec The NeuCodec model was proposed in Finite Scalar Quantization Enables Redundant and Transmission-Robust Neural Audio Compression at Low Bit-rates . NeuCodec is a neural audio codec extending on XCodec2. It takes advantage of the following features: Finite Scalar Quantization (FSQ) quantisation resulting in a single codebook , making it ideal for downstream modeling with Speech Language Models. Trained with CC data such that there are no Non-Commercial data restrictions . At 50 tokens/sec and 16 bits per token, the overall bit-rate is 0.8kbps . The codec takes in 16kHz input and outputs 24kHz using an upsampling decoder . The FSQ encoding scheme allows for bit-level error resistance suitable for unreliable and noisy channels. Links: Documentation Add support for NeuCodec ( #47143 ) by @harryjulian in #47143 Breaking changes Vision rotary embeddings (2D/3D) have been standardized into a unified RoPE frequency computation module, so users with custom vision models relying on attention-layer-level or model-specific RoPE grid interleaving logic must migrate to the new centralized modeling_rope_utils.py implementation. 🚨 Vision (2d/3d) rotary embeddings ( #48105 ) by @zucchini-nlp Generation Generation improvements include a performance optimization that avoids unnecessary accelerator synchronization on every decode step (reducing per-step overhead), and a fix to prevent unconditional downloading of remote hub files during generation. Several correctness fixes were also applied, including enforcing auto-compile cache checks for encoder-decoder models, standardizing past_key_values naming in AfMoE, and resolving flaky export and integration test failures. [ Generate ] Avoid unconditionally downloading remote hub file ( #48620 ) by @vasqu in [ #48620 ] [generate] stop synchronizing the accelerator on every decode step ( #47975 ) by @SunMarc in [ #47975 ] [AfMoE] Standardize past_key_values argument naming across forward and generate ( #48430 ) by @shenhuaqingshi in [ #48430 ] Fix MTP generation test regex gate for escaped layer ignore keys ( #48003 ) ( #48262 ) by @Noxtimo in [ #48262 ] fix(generation): Enforce the auto-compile cache check for encoder-decoder models ( #48364 ) by @harshaljanjani in [ #48364 ] [serge] Fix 4 integration tests for model generation failing with output_mismatch (list output differs (4)) ( #48133 ) by @sergereview[bot] in [ #48133 ] [VibeVoice] Skip generate export tests (flaky) ( #48396 ) by @ydshieh in [ #48396 ] Cache Fixed several cache-related bugs, including a quantized cache issue in VibeVoice, incorrect rejection of non-static cache implementations in VoxtralRealtime, missing auto-compile cache checks for encoder-decoder models, and a silent failure when paged attention is called without a cache. Documentation was also updated to clarify ContinuousBatchingConfig usage and sliding window model limitations. vibevoice: fix bug for quant cache ( #48487 ) by @kaixuanliu in [ #48487 ] Fix VoxtralRealtime rejecting non-static cache implementations ( #48082 ) by @jiqing-feng in [ #48082 ] Raise when a paged attention forward is called with no cache ( #48297 ) by @qgallouedec in [ #48297 ] [docs] Pass ContinuousBatchingConfig and sliding window models ( #48381 ) by @stevhliu in [ #48381 ] Retry get_daily_ci_runs on stale GitHub API cache ( #48374 ) by @ydshieh in [ #48374 ] Kernels Kernel support was improved with fixes for nested FLA kernel imports when only fla-core is installed, a warning when hub-kernel functions silently fall back to slower pure-PyTorch reference implementations, and the ability to register standalone functions (e.g., RoPE) in KernelConfig with optional non-inheritance of default mappings. Additional fixes include corrected repository paths for ESMFold2 kernels and updated documentation for KernelConfig customization. Support nested FLA kernel imports for fla-core ( #48221 ) by @DimensionSTP in [ #48221 ] Warn once when a hub-kernel function falls back to its reference PyTorch path ( #48185 ) by @qgallouedec in [ #48185 ] [docs] Kernel updates ( #48465 ) by @stevhliu in [ #48465 ] [ Kernels ] Enable functions into kernels registry and allow non inheritance ( #48443 ) by @vasqu in [ #48443 ] Fix kernel commit and repo paths for ESMFold2 ( #48186 ) by @Rocketknight1 in [ #48186 ] Quantization Fixed several quantization bugs, including a quant cache issue in VibeVoice, incorrect FP8 embedding handling for Qwen models, missing FP8 tensor parallelism layer overrides, and unnecessary MXFP4 weight dequantization on XPU devices. fix qwen4exp-fp8 ple embedding ( #48368 ) by @JJJYmmm in [ #48368 ] Keep MXFP4 weights quantized on XPU when use_kernels is set ( #47923 ) by @jiqing-feng in [ #47923 ] Fix missing FP8 TP layer overrides ( #48343 ) by @changwangss in [ #48343 ] Bugfixes and improvements MRoPE continued ( #48594 ) by @zucchini-nlp in [ #48594 ] [fix] Update stale expected strings in HunYuanVL integration tests ( #48646 ) by @ydshieh in [ #48646 ] [Quantizaiton]support 5/6/7 bits in AutoRound ( #48481 ) by @wenhuach21 in [ #48481 ] [fix] Update stale golden values and fix expected_logits shape in FlavaForPreTraining integration tests ( #48639 ) by @ydshieh in [ #48639 ] Fix YOLOS device mismatch with device_map="auto" ( #46886 ) by @swankystark in [ #46886 ] Honor shift_labels in decoder-only LLM/VLM losses ( #48493 ) by @qgallouedec in [ #48493 ] [KimiLinear] Fix test_cpu_offload: set num_local_experts=4 in model tester ( #48624 ) by @ydshieh in [ #48624 ] [docs] Per-layer config ( #48601 ) by @stevhliu in [ #48601 ] Fix generate_flags parsing in transformers chat ( #48597 ) by @SunMarc in [ #48597 ] [fix] Fix how we read package versions - triggered by torch 2.14+ ( #48615 ) by @ydshieh in [ #48615 ] [Docker] Upgrade CPU torch to <=2.14.0, torchcodec to <=0.16.0 ( #48614 ) by @ydshieh in [ #48614 ] Fix AttributeError in gradient_checkpointing_enable(offload=True) ( #48590 ) by @tarekziade in [ #48590 ] Another day fixing CI ( #48591 ) by @zucchini-nlp in [ #48591 ] esmfold2: keep distogram_head in fp32 as well ( #48488 ) by @kaixuanliu in [ #48488 ] Add supports_context_parallel to PreTrainedModel ( #48442 ) by @qgallouedec in [ #48442 ] [docs] mlinter reference ( #48460 ) by @stevhliu in [ #48460 ] [docs] Add a LiteRT page under community integrations ( #48540 ) by @john-rocky in [ #48540 ] [GLM 5.3 Flash] Fix NaN gradients in chunked KDA ( #48455 ) by @imvladikon in [ #48455 ] [docs] Fix [[autodoc]] directives in ALBERT model documentation ( #48593 ) by @samyuktahegde in [ #48593 ] [Fix] Fix A10 expectations for a test ( #48454 ) by @remi-or in [ #48454 ] Add PR comment CI for AMD (MI300) ( #48065 ) by @ydshieh in [ #48065 ] docs: fix docstring parameter names that do not match signatures ( #48575 ) by @simpleqt in [ #48575 ] docs: remove phantom parameters from docstrings ( #48576 ) by @simpleqt in [ #48576 ] extend some case to xpu as well ( #48502 ) by @sywangyi in [ #48502 ] [serge] Fix 2 integration tests for model glm4_moe failing with OOM (other (2)) ( #48551 ) by @sergereview[bot] in [ #48551 ] [serge] Fix 2 integration tests for model nemotron failing with import_or_config (other (2)) ( #48582 ) by @sergereview[bot] in [ #48582 ] Compress the agent conventions file and document two modular pitfalls ( #48586 ) by @tarekziade in [ #48586 ] Guard against a None video processor class when the backend is unavailable ( #48557 ) by @caiotheodoro in [ #48557 ] [serge] Fix 2 integration tests regressed by commit 83d46aa (PR #47625 ) ( #48580 ) by @sergereview[bot] in [ #48580 ] [nit] use requires_backends ( #47576 ) by @eustlb in [ #47576 ] [serge] Fix 2 integration tests for model kosmos2 failing with import_or_config (other (2)) ( #48552 ) by @sergereview[bot] in [ #48552 ] Fix failing tests for cohere_compass ( #48005 ) by @kaixuanliu in [ #48005 ] [Fix] Use dedicated helpers for DeepGEMM and SonicMoE tests ( #48523 ) by @remi-or in [ #48523 ] CI: Point test fixtures at hf-internal-testing copies we already host ( #48521 ) by @tarekziade in [ #48521 ] fix failed test cases for glm5_next ( #48497 ) by @kaixuanliu in [ #48497 ] Pass kwargs to the Mamba2 mixer in Nemotron-H, Falcon-H1 and Mamba2 ( #48490 ) by @kfastino in [ #48490 ] Retire test_multi_gpu_data_parallel_forward ( #48508 ) by @tarekziade in [ #48508 ] Processing tests [part 2] ( #47922 ) by @zucchini-nlp in [ #47922 ] QA: Add noisy comment checker ( #48484 ) by @tarekziade in [ #48484 ] Allow nested rope params for tiny models ( #48435 ) by @zucchini-nlp in [ #48435 ] Fix sliding-window mask layer_idx in Gemma3/Gemma4 create_masks_for_vision_model ( #48482 ) by @jiqing-feng in [ #48482 ] add xpu expectations for hunyuan_vl model tests ( #48504 ) by @kaixuanliu in [ #48504 ] [ Qwen 3.5 Moe ] Fix decorators ( #48436 ) by @vasqu in [ #48436 ] Infinite loop in dependency search ( #48393 ) by @zucchini-nlp in [ #48393 ] [serge] Fix 2 integration tests for model fsmt failing with output_mismatch (tensor values differ (2)) ( #48496 ) by @sergereview[bot] in [ #48496 ] Fix some tests by removing the deprecation cycle ( #48503 ) by @Cyrilvallez in [ #48503 ] Remove deprecation ( #48500 ) by @Cyrilvallez in [ #48500 ] Fix Pix2StructTextAttention init using hidden_size instead of d_kv ( #47558 ) by @ in [ #47558 ] Fix pre patch release utility ( #48499 ) by @Cyrilvallez in [ #48499 ] Update dev version ( #48498 ) by @Cyrilvallez in [ #48498 ] Fix Inkling inputs_embeds and add more tests ( #47827 ) by @Cyrilvallez in [ #47827 ] Simplify and fix qwen4 tests ( #48340 ) by @Cyrilvallez in [ #48340 ] [docs] Partial checkpointing and group_by_length ( #48463 ) by @stevhliu in [ #48463 ] doc: fix syntax error and typos in VibeVoice documentation ( #48489 ) by @VimalN2005 in [ #48489 ] [ Qwen4 Exp ] Use partial to avoid skipping mask more easily ( #48456 ) by @vasqu in [ #48456 ] Add support for NeuCodec ( #47143 ) by @harryjulian in [ #47143 ] [serge] Fix 1 integration tests regressed by commit bd95093 (PR #47493 ) ( #48426 ) by @sergereview[bot] in [ #48426 ] Fix rotary embedding regression ( #48477 ) by @Cyrilvallez in [ #48477 ] Remove deprecated mask functions ( #48476 ) by @Cyrilvallez in [ #48476 ] [MTP] Save memory by only capturing the last layer's hidden_states ( #48475 ) by @Cyrilvallez in [ #48475 ] Allow capturing only necessary hidden_states with capture_outputs ( #48081 ) by @sywangyi in [ #48081 ] Support per-layer MTP configuration ( #48264 ) by @eladsegal in [ #48264 ] [docs] Fix code snippets ( #47772 ) by @stevhliu in [ #47772 ] [Fix] Sparse TikToken tokenizers silently fail ( #48446 ) by @remi-or in [ #48446 ] No inherit decorator for NeoMME ( #48457 ) by @zucchini-nlp in [ #48457 ] Batch Rebalance Data Sampler ( #47340 ) by @delock in [ #47340 ] [serge] Fix 2 integration tests for model cwm failing with import_or_config (other (2)) ( #48414 ) by @sergereview[bot] in [ #48414 ] fix: Add DEIMv2 attribution ( #48448 ) by @harshaljanjani in [ #48448 ] [fix] inkling: mps + cuda mel spec extraction ( #47432 ) by @eustlb in [ #47432 ] fix: decode() batch path respects self.clean_up_tokenization_spaces ( #47793 ) by @lorenzozanee in [ #47793 ] Grounding dino fp16 dtype [backlog] ( #48438 ) by @molbap in [ #48438 ] Init the process group with a load-scaled timeout for sharded loading ( #48228 ) by @qgallouedec in [ #48228 ] Raise a clear error when a token is both forced and suppressed ( #47511 ) by @qgallouedec in [ #47511 ] Clarify device placement in pipelines ( #47367 ) by @LysandreJik in [ #47367 ] Add offload to gradient checkpointing ( #48444 ) by @qgallouedec in [ #48444 ] [MiniCPMV4_6] Update test_small_model_vision_generation_batch expected output (value drift) ( #48406 ) by @ydshieh in [ #48406 ] [serge] Fix 2 integration tests for model hyperclovax failing with other (other (2)) ( #48440 ) by @sergereview[bot] in [ #48440 ] fix(models): Drop the position-indexed token type lookup in RoPE encoders ( #48407 ) by @harshaljanjani in [ #48407 ] Document image_hidden_states/pixel_values mutual exclusivity for SmolVLM/Idefics2/Idefics3 ( #47714 ) by @verma8076 in [ #47714 ] Re-order a bit for easier navigation ( #48434 ) by @zucchini-nlp in [ #48434 ] Deprecated stuff gone ( #48367 ) by @zucchini-nlp in [ #48367 ] [serge] Fix 6 integration tests for model seamless_m4t_v2 failing with other (other (6)) ( #48425 ) by @sergereview[bot] in [ #48425 ] [Docs]: Update GLM 5.3 ( #48401 ) by @Dovis01 in [ #48401 ] Avoid print to stdout that fails the job check_failed_tests job ( #48391 ) by @ydshieh in [ #48391 ] Fix incorrect tuple return annotations on forward methods returning a Tensor ( #48359 ) by @Gronoxx in [ #48359 ] Fix interval merge invariant in _find_disjoint ( #47860 ) by @sharmax-vikas in [ #47860 ] skip mtp slow tests for now ( #48328 ) ( #48329 ) by @tarekziade in [ #48329 ] Update Tailscale action version in workflow ( #48394 ) by @glegendre01 in [ #48394 ] fix some failure in xpu ( #48252 ) by @sywangyi in [ #48252 ] [Improvement] Make gated delta rule more explicit ( #47625 ) by @remi-or in [ #47625 ] [CB] Fix wrong device scoping ( #48370 ) by @remi-or in [ #48370 ] Bump transformers-mlinter to 0.1.5 and clear the new findings ( #48259 ) by @tarekziade in [ #48259 ] fix: flash-attn fallback failing on torch2.13 ( #48388 ) by @NanoCode012 in [ #48388 ] [LongcatFlash] Fix test_longcat_generation_cpu: use device_map="cpu" to avoid MoE disk offload issue ( #48377 ) by @ydshieh in [ #48377 ] [Qwen3VLMoe] Update test_small_model_integration_test_batch expected output (value drift) ( #48376 ) by @ydshieh in [ #48376 ] [ONNX] Skip affected models on torch 2.13 (two dynamo regressions) ( #48191 ) by @ydshieh in [ #48191 ] Fix safe_open mmap memory exhaustion on Windows by using pread backend ( #48341 ) by @eryk-roch in [ #48341 ] Fix Zamba2 construction for num_mem_blocks > 1 checkpoints ( #48325 ) by @john-rocky in [ #48325 ] [Docs] Change 5.3 Flash pos in toc ( #48366 ) by @Dovis01 in [ #48366 ] [conftest] Use get_cpu_ram_total_gib for psutil patch (cgroup-aware) ( #48290 ) by @ydshieh in [ #48290 ] Fix flaky test_training_gradient_checkpointing for BigBirdPegasus (fp noise filter) ( #48332 ) by @ydshieh in [ #48332 ] Quiet continuous batching at default verbosity ( #48314 ) by @qgallouedec in [ #48314 ] Wait for the first request in the async continuous batching bootstrap ( #48304 ) by @qgallouedec in [ #48304 ] Ignore a stale best checkpoint recorded in a resumed trainer state ( #48319 ) by @VaggelisGian in [ #48319 ] [docs] Fix links and remove TokenizerFast ( #47748 ) by @stevhliu in [ #47748 ] Fix incorrect token classification prefix for ESMC ( #48348 ) by @Rocketknight1 in [ #48348 ] Create the continuous batching CPU group with local synchronization ( #48302 ) by @qgallouedec in [ #48302 ] Resolve continuous batching config against the text config for composite models ( #48299 ) by @qgallouedec in [ #48299 ] [ CI ] Unblock fast CI for now (failing tests) ( #48344 ) by @vasqu in [ #48344 ] [qwen4_exp] disable torch/onnx export tests due to data-dependent control flow ( #48345 ) by @ydshieh in [ #48345 ] [debug] Trace previous CI run selection in get_previous_daily_ci.py ( #48338 ) by @ydshieh in [ #48338 ] Normalize HunYuanVL's legacy field aliases via attribute_map ( #48261 ) by @hmellor in [ #48261 ] Significant community contributions The following contributors have made significant changes to the library over the last release: @ydshieh [fix] Update stale expected strings in HunYuanVL integration tests ( #48646 ) [fix] Update stale golden values and fix expected_logits shape in FlavaForPreTraining integration tests ( #48639 ) [tests] Fix integration test golden values broken by fast image processor default (PR #41388 ) ( #48637 ) [KimiLinear] Fix test_cpu_offload: set num_local_experts=4 in model tester ( #48624 ) Fix GPU memory teardown in CLI serve tests ( #48618 ) [fix] Fix how we read package versions - triggered by torch 2.14+ ( #48615 ) [Docker] Upgrade CPU torch to <=2.14.0, torchcodec to <=0.16.0 ( #48614 ) Add PR comment CI for AMD (MI300) ( #48065 ) [MiniCPMV4_6] Update test_small_model_vision_generation_batch expected output (value drift) ( #48406 ) Avoid print to stdout that fails the job check_failed_tests job ( #48391 ) [VibeVoice] Skip generate export tests (flaky) ( #48396 ) [LongcatFlash] Fix test_longcat_generation_cpu: use device_map="cpu" to avoid MoE disk offload issue ( #48377 ) [Qwen3VLMoe] Update test_small_model_integration_test_batch expected output (value drift) ( #48376 ) [ONNX] Skip affected models on torch 2.13 (two dynamo regressions) ( #48191 ) Retry get_daily_ci_runs on stale GitHub API cache ( #48374 ) [conftest] Use get_cpu_ram_total_gib for psutil patch (cgroup-aware) ( #48290 ) Fix flaky test_training_gradient_checkpointing for BigBirdPegasus (fp noise filter) ( #48332 ) [qwen4_exp] disable torch/onnx export tests due to data-dependent control flow ( #48345 ) [debug] Trace previous CI run selection in get_previous_daily_ci.py ( #48338 ) @LauraGPT Add Fun-ASR-Nano model ( #46180 ) @tarekziade Fix AttributeError in gradient_checkpointing_enable(offload=True) ( #48590 ) Compress the agent conventions file and document two modular pitfalls ( #48586 ) CI: Point test fixtures at hf-internal-testing copies we already host ( #48521 ) Retire test_multi_gpu_data_parallel_forward ( #48508 ) QA: Add noisy comment checker ( #48484 ) skip mtp slow tests for now ( #48328 ) ( #48329 ) Bump transformers-mlinter to 0.1.5 and clear the new findings ( #48259 ) @remi-or [Fix] Fix A10 expectations for a test ( #48454 ) Kimi linear ( #48250 ) [Fix] Use dedicated helpers for DeepGEMM and SonicMoE tests ( #48523 ) [Fix] Sparse TikToken tokenizers silently fail ( #48446 ) [Improvement] Make gated delta rule more explicit ( #47625 ) [CB] Fix wrong device scoping ( #48370 ) [CB] Fail faster ( #48334 ) @ArthurZucker Add h4 ( #48473 ) @harryjulian Add support for NeuCodec ( #47143 ) @delock Batch Rebalance Data Sampler ( #47340 ) @harshaljanjani fix: Add DEIMv2 attribution ( #48448 ) model: Add NVIDIA Canary-1B-v2 to Transformers ( #46825 ) fix(generation): Enforce the auto-compile cache check for encoder-decoder models ( #48364 ) fix(models): Drop the position-indexed token type lookup in RoPE encoders ( #48407 ) @tonywu71 Add NeoMME and NeoMME-Retriever ( #47992 ) @Dovis01 [Docs]: Update GLM 5.3 ( #48401 ) [Docs] Change 5.3 Flash pos in toc ( #48366 ) [Glm 5.3 Flash] GLM 5.3 Flash Support ( #48342 ) @pengzhiliang Implement VibeVoice ( #40546 )

Release v5.13.0 New Model additions KimiK 2.5, 2.6, and 2.7 This release includes the architecture for Kimi 2.5 which is used by 2.5-2.7: Kimi K2.5 is an open-source, native multimodal agentic model that advances practical capabilities in long-horizon coding, coding-driven design, proactive autonomous execution, and swarm-based task orchestration. The model was proposed in Kimi K2.5: Visual Agentic Intelligence and further improved in [Kimi K2.6: Advancing Open-Source Coding](Kimi K2.5: Visual Agentic Intelligence). Kimi K2.5 achieves significant improvements on complex, end-to-end coding tasks, generalizing robustly across programming languages (Rust, Go, Python) and domains spanning front-end, DevOps, and performance optimization. The model is capable of transforming simple prompts and visual inputs into production-ready interfaces and lightweight full-stack workflows, generating structured layouts, interactive elements, and rich animations with deliberate aesthetic precision. Links: Documentation Add new model: Kimi2-6 ( #45630 ) by @zucchini-nlp in #45630 MiMo-V2-Flash MiMo-V2-Flash is a Mixture-of-Experts (MoE) language model developed by the Xiaomi MiMo team. Designed to establish a new balance between long-context modeling capabilities and inference efficiency, the model is built for strong performance in complex reasoning and agentic tasks. Trained on 27T tokens with native 32k sequence lengths, MiMo-V2-Flash seamlessly supports an extended 256K context window while significantly reducing KV-cache storage compared to standard global attention models. Links: Documentation Add Xiaomi MiMo-V2 ( #45144 ) by @casinca in #45144 Nemotron 3.5 ASR Nemotron 3.5 ASR is a 600M-parameter multilingual speech recognition model from NVIDIA, built for high-quality transcription in both low-latency streaming and high-throughput batch settings, with native punctuation and capitalization. For streaming, it offers configurable chunk sizes—80ms, 160ms, 560ms, and 1120ms, letting users trade off latency against accuracy to suit their application. Its cache-aware FastConformer-RNNT architecture is central to this capability: unlike traditional buffered streaming, which repeatedly reprocesses overlapping audio windows, the model processes only each new incoming chunk while reusing cached encoder context from prior chunks. This eliminates redundant computation, significantly improves efficiency, and minimizes end-to-end delay without sacrificing accuracy, making it well suited to real-time transcription workloads. Links: Documentation Add Nemotron 3.5 ASR Streaming ( #46565 ) by @eustlb in #46565 NemotronAsrStreaming Nemotron ASR Streaming is a 600M-parameter English speech recognition model from NVIDIA, built for high-quality transcription in both low-latency streaming and high-throughput batch settings, with native punctuation and capitalization. For streaming, it offers configurable chunk sizes—80ms, 160ms, 560ms, and 1120ms, letting users trade off latency against accuracy to suit their application. Its cache-aware FastConformer-RNNT architecture is central to this capability: unlike traditional buffered streaming, which repeatedly reprocesses overlapping audio windows, the model processes only each new incoming chunk while reusing cached encoder context from prior chunks. This eliminates redundant computation, significantly improves efficiency, and minimizes end-to-end delay without sacrificing accuracy, making it well suited to real-time transcription workloads. Links: Documentation Add Nemotron ASR Streaming ( #46332 ) by @eustlb in #46332 Qwen3 ASR Qwen3 ASR is an automatic speech recognition model from Alibaba's Qwen team that combines a Whisper-style audio encoder with a Qwen3 language model decoder for speech-to-text transcription. The model supports automatic language detection and multilingual transcription. A forced aligner model is also included. It can be used to timestamp a provided transcript and its audio. It uses the same audio encoder model with a classification head that predicts a word's length. This model can be used with the transcript from any ASR model (see the example below with Parakeet CTC). Links: Documentation Qwen3 ASR and Forced Aligner ( #43838 ) by @mbtariq82 in #43838 ZAYA ZAYA1 is a 760M active / 8.4B total parameter MoE language model trained by Zyphra. It combines Compressed Convolutional Attention (CCA), a nonlinear ZAYA1 router, and residual scaling. Links: Documentation [new model] Add Zyphra/ZAYA1-8B ( #45862 ) by @JJJYmmm in #45862 VideoPrism The VideoPrism model was proposed in the paper VideoPrism: A Foundational Visual Encoder for Video Understanding by Google DeepMind ( blog post ). VideoPrism is a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. The model is pretrained on a large-scale heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding through global-local distillation of semantic video embeddings and a token shuffling scheme, enabling the model to focus primarily on the video modality while leveraging text associated with videos. VideoPrism achieves state-of-the-art performance on 31 out of 33 video understanding benchmarks across four broad task groups, from web video question answering to computer vision for science. Links: Documentation Add Videoprism ( #39895 ) by @MHRDYN7 in #39895 RADIO RADIO (Reduce All Domains Into One) is a family of vision foundation models from NVIDIA trained by multi-teacher distillation (e.g. CLIP, DINOv2, SAM) into a single ViT backbone. It produces both an image-level summary embedding and dense spatial features , and supports variable input resolutions through a Cropped Position Embedding (CPE) patch generator. Links: Documentation Add support for RADIO models ( #46425 ) by @meatybobby in #46425 MiniCPM3 MiniCPM3 is the third-generation MiniCPM dense language model from OpenBMB. The 4B variant ( openbmb/MiniCPM3-4B ) outperforms many 7B–9B open models on standard benchmarks while remaining lightweight enough for on-device usage. MiniCPM3 combines several architectural ideas: Multi-head Latent Attention (MLA) from DeepSeek-V2, which compresses the key/value cache into a low-rank latent representation while still using rotary embeddings on a portion of the query/key heads. A standard SwiGLU MLP (no MoE). Three scalar scaling factors that govern signal flow: scale_emb — scales input embeddings. scale_depth / sqrt(num_hidden_layers) — scales residual connections. hidden_size / dim_model_base — scales hidden states before the language model head. Links: Documentation Add MiniCPM3 ( #41116 ) by @bzantium in #41116 Breaking changes A broad set of modeling changes have been made to standardize layer declarations, mask/cache construction, and hybrid-attention handling, making many models cleanly exportable (ONNX, torch.export , ExecuTorch) and fullgraph-compilable — users relying on internal modeling APIs may need to update their code accordingly. 🚨 Modeling changes for export, compile, and hybrid-attention standardization ( #46738 ) by @IlyasMoutawwakil Attention masking for image tokens in Gemma 3/4 models has been fixed to correctly respect sliding window boundaries in local layers, which changes model behavior and may affect reproducibility of previous results. 🚨 [gemma 3/4] Fix bidirectional attention masking crossing sliding window boundaries ( #46850 ) by @douglas-reid The Expert Parallelism (EP) router contract has been corrected across many models and FP8 scale format handling has been fixed, requiring users of EP or FP8 quantization with affected models to verify their configurations and potentially update conversion mappings. 🚨 EP: fix EP router contract for many models + honor FP8 scale format ( #46818 ) by @IlyasMoutawwakil The Kernels integration has been synced to the latest version, which includes a breaking change where model-type repositories are no longer accepted by the kernels interface — users must migrate to the updated kernel repository format as shown in the updated tests. 🚨 [ Kernels ] Sync to latest version ( #46039 ) by @vasqu HfExporters: Native, Unified export for PyTorch / ONNX / ExecuTorch A native, in-Transformers export pipeline — one base class ( HfExporter ), three subclasses for the runtimes we care about, one unified API: Exporter Output Runtime DynamoExporter ExportedProgram Any PyTorch runtime, AOT compilation OnnxExporter ONNXProgram Any ONNX runtime (ORT, TensorRT, OpenVINO, …) ExecutorchExporter ExecutorchProgramManager Mobile and edge (ExecuTorch) Same call shape across all three. Dynamic shapes by default. Generation-style models split automatically into prefill + decode (+ vision/audio sub-encoders for VLMs). from transformers import AutoModelForMaskedLM , AutoTokenizer from transformers . exporters import OnnxExporter , OnnxConfig model_id = "hf-internal-testing/tiny-random-BertForMaskedLM" tokenizer = AutoTokenizer . from_pretrained ( model_id ) model = AutoModelForMaskedLM . from_pretrained ( model_id ). eval () inputs = tokenizer ([ "Hello, my dog is cute" ] * 2 , return_tensors = "pt" ) onnx_program = OnnxExporter (). export ( model , inputs , config = OnnxConfig ( dynamic = True )) new_input = tokenizer ( "Hello, my cat is so adorable!" , return_tensors = "pt" ) torch . testing . assert_close ( onnx_program . call_reference ( ** new_input )[ 0 ], # numpy reference onnx_program ( ** new_input )[ 0 ], # onnxruntime rtol = 1e-4 , atol = 1e-4 , ) Swap one line for another runtime — DynamoExporter() / DynamoConfig or ExecutorchExporter() / ExecutorchConfig(backend=...) . For generative models the prefill/decode split is captured automatically: from transformers import AutoModelForCausalLM , AutoTokenizer from transformers . exporters import OnnxExporter , OnnxConfig model_id = "hf-internal-testing/tiny-random-LlamaForCausalLM" tokenizer = AutoTokenizer . from_pretrained ( model_id ) model = AutoModelForCausalLM . from_pretrained ( model_id ). eval () inputs = tokenizer ([ "Hello, my dog is cute" ] * 2 , return_tensors = "pt" ) artifacts = OnnxExporter (). export_for_generation ( model , inputs , config = OnnxConfig ( dynamic = True )) # {"prefill": ONNXProgram, "decode": ONNXProgram} # For VLMs: also vision_encoder, audio_encoder, multi_modal_projector, language_model, lm_head Kernels Kernels: Fixed a silent SDPA math-kernel fallback for GQA models with head_dim > 256 (e.g., Gemma4) that caused O(S²) memory materialization, and resolved a regression where use_kernels=True failed to apply kernel mappings. Additional improvements include lazy loading of the default kernel mapping to prevent import failures with incompatible kernel versions, ROCm routing to AITER Triton kernels for AMD GPUs, GB10/SM121 Hub-kernel support for Qwen3.6 Gated DeltaNet, and expanded documentation for the kernel API. Fix silent SDPA math-kernel fallback for GQA when key/value head_dim > 256 or differ ( #46960 ) by @Butterfingrz in [ #46960 ] [docs] AITER kernels ( #46871 ) by @stevhliu in [ #46871 ] Documentation for the kernel API ( #46754 ) by @michaelbenayoun in [ #46754 ] update kernels-community/aiter-rope version ( #46810 ) by @Abdennacer-Badaoui in [ #46810 ] Add GB10/SM121 Hub-kernel path for Qwen3.6 Gated DeltaNet ( #46423 ) by @AzeezIsh in [ #46423 ] [ Kernels ] Trigger proper kernelization on use_kernels=True ( #46755 ) by @vasqu in [ #46755 ] Lazily build the default kernel mapping to decouple kernels from normal transformers usage ( #46681 ) by @jiqing-feng in [ #46681 ] Add some AITER kernel routing for ROCm ( #46268 ) by @Abdennacer-Badaoui in [ #46268 ] fix: position ids does not exist in upstream rotary kernel ( #46619 ) by @NanoCode012 in [ #46619 ] docs(zh): add Chinese translation of kernels.md ( #46621 ) by @shoushinya123 in [ #46621 ] Generation Several generation bugs were fixed, including Mamba2 chunked-prefill and speculative decoding for hybrid models (Zamba2, Nemotron-H, Bamba, FalconH1, GraniteMoeHybrid), beam search for Mamba models, prompt lookup decoding crashes with no EOS token, and incorrect stateful model handling for LFM2. Additional improvements include reduced unnecessary generation warnings, a fix for continuous batching output mutation, and a new option to keep input tensors on CPU during generation to avoid retracing on Neuron/TPU devices. Fix Mamba2 chunked-prefill / speculative decoding for Zamba2, Nemotron-H, Bamba, FalconH1 and GraniteMoeHybrid ( #46741 ) by @Sunt-ing in [ #46741 ] Remove some unnecessary generate warnings ( #46955 ) by @Cyrilvallez in [ #46955 ] Reject assisted generation for LFM2 and LFM2-MoE (set _is_stateful) ( #46937 ) by @Sunt-ing in [ #46937 ] Fix beam search for mamba models ( #46819 ) by @Cyrilvallez in [ #46819 ] Fix prompt lookup decoding crash when no EOS token is configured ( #46790 ) by @Sunt-ing in [ #46790 ] [Continuous Batching] Snapshot generation outputs without mutating request state ( #46670 ) by @Incheonkirin in [ #46670 ] [docs] keep generation tensors on cpu ( #46675 ) by @stevhliu in [ #46675 ] feat(generation): allow user to keep input tensors on cpu ( #46590 ) by @dacorvo in [ #46590 ] Attention Several attention-related bugs were fixed in this release, including silent SDPA math-kernel fallbacks for GQA with large head dimensions, broken Flash Attention with StaticCache , incorrect causal masking in Xcodec2, a cross-attention reshape regression in Blip2, and eager GQA support in Evolla. Accelerate hook handling was also corrected for models using linear attention to prevent silently wrong results during offloading. Fix accelerate hooks for all models using linear attention ( #46978 ) by @Cyrilvallez in [ #46978 ] Fix Xcodec2 attention to be non-causal. ( #46963 ) by @ebezzam in [ #46963 ] Fix flash attention with StaticCache ( #46914 ) by @Cyrilvallez in [ #46914 ] Fix Evolla eager attention for the GQA text decoder ( #46860 ) by @jiqing-feng in [ #46860 ] [docs] metal flash attention ( #46349 ) by @stevhliu in [ #46349 ] [ Blip2 ] Fix cross attention reshape ( #46695 ) by @vasqu in [ #46695 ] Cache Cache APIs were improved by consolidating redundant getters into a cleaner get_max_length method and updating documentation accordingly. Several bug fixes were also applied, including correcting mask generation beyond sliding windows, fixing a dimension issue in cumulative length tracking, resolving device mismatches in offloaded cache for hybrid models, and fixing crashes when loading trust_remote_code models from symlinked local caches. [docs] update cache apis ( #46892 ) by @stevhliu in [ #46892 ] Rework some old cache getters/properties ( #46862 ) by @Cyrilvallez in [ #46862 ] Fix expanded dim in the cache's cumulative length ( #46856 ) by @Cyrilvallez in [ #46856 ] Fix mask when generating beyond sliding window ( #46839 ) by @zucchini-nlp in [ #46839 ] Fix offloaded cache device mismatch on hybrid models ( #46748 ) by @Sunt-ing in [ #46748 ] Fix dynamic module symlinked cache on trust_remote_code models ( #46618 ) by @ldkhang1201 in [ #46618 ] Serve Several fixes and improvements were made to the Serve functionality, including lazy imports to prevent CLI crashes when the optional serve extra is not installed, a fix for dropped attributes during serialization of subclassed Pydantic models, and added documentation for the kernel API. fix(cli/serve): import serve handlers lazily so the CLI works without the serve extra ( #46473 ) by @ in [ #46473 ] [Fix] Serve drops some attributes at serialization ( #46680 ) by @remi-or in [ #46680 ] Reduce per_page from 100 to 50 in GitHub API calls to avoid server errors ( #46678 ) by @ydshieh in [ #46678 ] Quantization Fixed dtype casting bugs in Gemma4's vision and audio multimodal embedders when using BitsAndBytes quantization, where inputs were incorrectly cast to integer storage dtypes ( uint8 / int8 ) instead of the actual compute dtype. Also corrected FP8 quantization to round block scales before quantizing weights, ensuring dequantization produces correct values for ue8m0 (DeepSeek-V4 style) format. [Gemma4] Fix dtype casting for quantized vision/audio embedders ( #46933 ) by @sharmax-vikas in [ #46933 ] Fix dtype casting for quantized multimodal embedders ( #46904 ) by @praful-srinivasan-027 in [ #46904 ] Round the ue8m0 FP8 scale before quantizing so dequant matches the stored inverse ( #46763 ) by @Incheonkirin in [ #46763 ] Bugfixes and improvements Update workflow callers to use transformers-ci ( #47040 ) by @ydshieh in [ #47040 ] Add HunYuan VL model ( #46417 ) by @Mi-Jiazhi in [ #46417 ] Add tiny_model_id support to ProcessorTesterMixin for memory-sensitive tests ( #47005 ) by @ydshieh in [ #47005 ] chore(linter): add TRF018 modeling rule ( #46259 ) by @tarekziade in [ #46259 ] [PoC] HF exporters ( #41992 ) by @IlyasMoutawwakil in [ #41992 ] TST Skip PEFT tests if PEFT version is too low ( #47027 ) by @BenjaminBossan in [ #47027 ] CI Add PEFT integration tests ( #47021 ) by @BenjaminBossan in [ #47021 ] [glm-mode-dsa] Indexer uses interleaved rope ( #46842 ) by @pcuenca in [ #46842 ] Use standard arg names in Mllama ( #46977 ) by @zucchini-nlp in [ #46977 ] Bump min peft 0.19.1 remove weight conversion duplicate code ( #46442 ) by @BenjaminBossan in [ #46442 ] Raise a loud error for missing prefix ( #46980 ) by @Rocketknight1 in [ #46980 ] Fix typo in Qwen3 ASR no_split_module ( #47002 ) by @ebezzam in [ #47002 ] only in the original repo ( #46982 ) by @tarekziade in [ #46982 ] Fix typos in Gemma 4 Assistant documentation ( #46975 ) by @RaunaqDavidNath in [ #46975 ] the CI status should be a comment ( #46976 ) by @tarekziade in [ #46976 ] QwenVL model conversion ( #46881 ) by @zucchini-nlp in [ #46881 ] Remove default dtype in FusedRMSNormGated modules ( #46953 ) by @Cyrilvallez in [ #46953 ] FIX PEFT test changed error type ( #46959 ) by @BenjaminBossan in [ #46959 ] Fix path traversal via vocab-file arguments in tokenizer_config.json ( #46279 ) by @LinZiyuu in [ #46279 ] docs(conditional_detr): fix num_queries default in docstring (100 -> 300) ( #46939 ) by @Kropiunig in [ #46939 ] Use common floats_list method for feature extractor tests. ( #46956 ) by @ebezzam in [ #46956 ] Fix RT-DETR indexing error when num_feature_levels exceeds backbone o… ( #46833 ) by @c1prk in [ #46833 ] Fix Florence2 training-loss double-shift (same pattern as Moonshine #… ( #46898 ) by @sharmax-vikas in [ #46898 ] [Olmo3] different RoPE per layer type ( #46911 ) by @zucchini-nlp in [ #46911 ] Use inspect.getsource instead of open() for source-reading in can_set *_implementation ( #46207 ) by @rasmi in [ #46207 ] Don't pin the gated delta net norm to cuda:0 with a hardcoded device ( #46817 ) by @Sunt-ing in [ #46817 ] Fix auto-mappings registration for remote code & fixes a few custom code issues ( #46876 ) by @Cyrilvallez in [ #46876 ] Fix broken internal documentation links ( #46945 ) by @sezer-muhammed in [ #46945 ] Insert a Grafana badge in the PR ( #46774 ) by @tarekziade in [ #46774 ] [NemotronAsrStreaming] fix pipeline ( #46870 ) by @eustlb in [ #46870 ] [NemotronAsrStreaming] processor without modular ( #46865 ) by @eustlb in [ #46865 ] [ Dia ] Fix docs ( #46923 ) by @vasqu in [ #46923 ] [Docs] Fix full disk offloading docs ( #46905 ) by @kylesayrs in [ #46905 ] [CB] Changes to increase max_batch_tokens ( #46712 ) by @remi-or in [ #46712 ] Redirect to diffusers pipe in docs for experimental features ( #46875 ) by @zucchini-nlp in [ #46875 ] Install in docker ( #46910 ) by @ydshieh in [ #46910 ] [CI] Use pre-computed _OLD_MODELS in test_new_models_require_torchvision_backend ( #46882 ) by @ydshieh in [ #46882 ] call transformers-ci in a nightly run ( #46811 ) by @tarekziade in [ #46811 ] [docs] full disk offloading ( #46893 ) by @stevhliu in [ #46893 ] TST Run fast PEFT tests in normal CI ( #45679 ) by @BenjaminBossan in [ #45679 ] nemotron_asr_streaming: set _supports_flex_attn to False ( #46878 ) by @kaixuanliu in [ #46878 ] Add native masked MSE loss for Sapiens2ForPoseEstimation ( #46764 ) by @Sainava in [ #46764 ] blip 2 fix ( #46816 ) by @itazap in [ #46816 ] Use meshgrid for brevity ( #46861 ) by @zucchini-nlp in [ #46861 ] Add xcodec2 model ( #44178 ) by @ebezzam in [ #44178 ] Prevent auto-class from being modified for all models ( #46844 ) by @zucchini-nlp in [ #46844 ] Add Spanish translation of the torch.compile page ( #46852 ) by @delcenjo in [ #46852 ] docs: Update NeMo AutoModel doc examples ( #46857 ) by @adil-a in [ #46857 ] [docs] distributed training ( #44420 ) by @stevhliu in [ #44420 ] [docs] require trust_remote_code for custom_generate ( #46677 ) by @stevhliu in [ #46677 ] add distributed config ( #46705 ) by @3outeille in [ #46705 ] [Offloading] [Bugfix] Fix disk offloading of models with explicit tensor dtypes ( #46849 ) by @kylesayrs in [ #46849 ] Streamable chat parsing ( #45847 ) by @Rocketknight1 in [ #45847 ] Fix BitNet packed-weight unpacking dtype ( F.linear dtype mismatch) ( #46808 ) by @jiqing-feng in [ #46808 ] Fix typos in code ( #46579 ) by @cyyever in [ #46579 ] Fix Moonshine training-loss double-shift (train against labels, not labels[..., 1:]) ( #46784 ) by @Incheonkirin in [ #46784 ] [CB] Fix issues with FA read / writes ( #46765 ) by @remi-or in [ #46765 ] Switch decorator order ( #46853 ) by @Cyrilvallez in [ #46853 ] docs(trainer): add JIT checkpointing to trainer recipes ( #46826 ) by @efazal in [ #46826 ] Import diffusion_gemma in models init ( #46841 ) by @boringcrypto in [ #46841 ] [skills] help your agent get started ( #45732 ) by @stevhliu in [ #45732 ] Fix use_cache with seq_len > 1 ( #46032 ) ( #46084 ) by @Ramshankar07 in [ #46084 ] [Offloading] Support full disk offloading ( #46749 ) by @kylesayrs in [ #46749 ] fix: raise ValueError for empty conversation in apply_chat_template ( #46753 ) by @sharmax-vikas in [ #46753 ] Fix VideoPrismForVideoClassification returning last_hidden_state as h… ( #46830 ) by @sharmax-vikas in [ #46830 ] Avoid NumPy 2.0 __array__ copy-keyword deprecation in create_mm_token_type_ids ( #46827 ) by @qgallouedec in [ #46827 ] docs: update apple silicon doc with safetensors 0.8.0 benefits ( #46744 ) by @McPatate in [ #46744 ] [ CB ] Add FA2 to the fast path ( #46729 ) by @vasqu in [ #46729 ] Fix flex_attention block mask creation when get_seq_length returns a tensor ( #46802 ) by @jiqing-feng in [ #46802 ] Fix left-padding token selection in BioGptForSequenceClassification ( #46782 ) by @Sunt-ing in [ #46782 ] Fix broken internal links in model documentation ( #46807 ) by @ShamSaleem in [ #46807 ] DiffusionGemma: mask layout and CI ( #46654 ) by @zucchini-nlp in [ #46654 ] Use cached added-token dicts in per-token decode loops ( #46535 ) by @ishan-1010 in [ #46535 ] fix another flaky test ( #46767 ) by @zucchini-nlp in [ #46767 ] Fix secondary rate limit when downloading artifacts in slack report ( #46796 ) by @ydshieh in [ #46796 ] docs: move SmolLM3 to Text models category in _toctree.yml ( #46770 ) by @yyouretoast in [ #46770 ] Fix several bugs in cache_implementation=static ( #46446 ) by @dacorvo in [ #46446 ] [CI] Fix artifact download path in self-comment-ci workflow ( #46769 ) by @ydshieh in [ #46769 ] fixes per head minimaxm3 ( #46719 ) by @ArthurZucker in [ #46719 ] [ CI ] Fix some failures introduced by myself 😬 ( #46751 ) by @vasqu in [ #46751 ] Fix regression in ProcessorMixin._load_tokenizer_from_pretrained for tokenizers at root ( #46592 ) by @ in [ #46592 ] fix(aria): use math.ceil in get_number_of_image_patches to match actual patch count ( #46732 ) by @arnavkewalram in [ #46732 ] Return logits from semantic segmentation post-process ( #46163 ) by @guarin in [ #46163 ] Fall back to the for-loop grouped_mm on CPU ( #46743 ) by @Sunt-ing in [ #46743 ] Kernelize refactor ( #46520 ) by @michaelbenayoun in [ #46520 ] ci: add comment explaining why secrets are not inherited in security gate ( #46750 ) by @ydshieh in [ #46750 ] ci: trigger PR CI on ci-* branches ( #46746 ) by @ydshieh in [ #46746 ] finegrained v3 ( #46742 ) by @IlyasMoutawwakil in [ #46742 ] Improve AutoImageProcessor error for unavailable backends ( #46727 ) by @sisaman in [ #46727 ] skip decorators must appear after @parameterized.expand in pytest ( #46737 ) by @rasmi in [ #46737 ] [RecurrentGemma] Support attn_implementation dispatch ( #46320 ) by @YangKai0616 in [ #46320 ] [docs] clarify initialization module usage ( #46698 ) by @stevhliu in [ #46698 ] feat: bump safetensors to 0.8.0 ( #46523 ) by @McPatate in [ #46523 ] ci: disable CircleCI by replacing config with no-op ( #46721 ) by @ydshieh in [ #46721 ] [CB] Fix offloading ( #46587 ) by @remi-or in [ #46587 ] [ Templates ] Update members ( #46720 ) by @vasqu in [ #46720 ] feat[vLLM x v5]: Expose max_source_positions on VibeVoiceAsrConfig ( #46472 ) by @harshaljanjani in [ #46472 ] Laguna: support per-element output gating ( #46690 ) by @joerowell in [ #46690 ] ci: grant pull-requests:write to the security gate caller ( #46715 ) by @ydshieh in [ #46715 ] Multi-gpu loading when the whole backbone is tied ( #46625 ) by @zucchini-nlp in [ #46625 ] Delete docstring if same as in auto-doc ( #46284 ) by @zucchini-nlp in [ #46284 ] Update GLM-5.2 docs ( #46703 ) by @Dovis01 in [ #46703 ] add conversion scripts for EUPE ( #46691 ) by @molbap in [ #46691 ] [docs] compile level and batch/scheduling limits ( #46676 ) by @stevhliu in [ #46676 ] [blip_2] Support attn_implementation dispatch ( #46401 ) by @YangKai0616 in [ #46401 ] [CTRL] Support attn_implementation dispatch ( #46073 ) by @YangKai0616 in [ #46073 ] Lfm2: also thread seq_idx through ShortConv.slow_forward (non-fast-path) ( #46633 ) by @ChangyiYang in [ #46633 ] feat(pipelines): accept numpy arrays and tensors in ImageClassificationPipeline ( #39607 ) ( #46573 ) by @kamran-nizamani in [ #46573 ] Smovlm: pad videos up to max frames ( #46662 ) by @zucchini-nlp in [ #46662 ] mistral common backend fix ( #46667 ) by @itazap in [ #46667 ] [pr template] update ( #46606 ) by @stevhliu in [ #46606 ] Fix AttributeError in auto_factory when model_class lacks config_class ( #46669 ) by @atharv1945 in [ #46669 ] [CB] Slice logits inside the model ( #46660 ) by @remi-or in [ #46660 ] ci: add NO_COLOR=1 to suppress ANSI color codes in CI output ( #46659 ) by @ydshieh in [ #46659 ] Fix dynamic RoPE not resetting inv_freq when layer_type is None ( #46624 ) by @Incheonkirin in [ #46624 ] Better processing tests ( #46374 ) by @zucchini-nlp in [ #46374 ] ci: add merge_group trigger to pr-ci-caller.yml ( #46668 ) by @ydshieh in [ #46668 ] skip invalid quant_cache test for nemotron_h ( #46368 ) by @kaixuanliu in [ #46368 ] Revert "Disable PR CI workflow for PRs from forked repo. during the weekend" ( #46652 ) by @ydshieh in [ #46652 ] [CB] Fix seqlens and use TypedDict ( #46593 ) by @remi-or in [ #46593 ] Disable PR CI workflow for PRs from forked repo. during the weekend ( #46609 ) by @ydshieh in [ #46609 ] Update post release ( #46608 ) by @vasqu in [ #46608 ] Fix peft lower bound ( #46605 ) by @hmellor in [ #46605 ] Fix docstring formatting issues causing Sphinx autodoc warnings ( #46596 ) by @kurtmckee in [ #46596 ] Significant community contributions The following contributors have made significant changes to the library over the last release: @ydshieh Update workflow callers to use transformers-ci ( #47040 ) Add tiny_model_id support to ProcessorTesterMixin for memory-sensitive tests ( #47005 ) Install in docker ( #46910 ) [CI] Use pre-computed _OLD_MODELS in test_new_models_require_torchvision_backend ( #46882 ) Fix secondary rate limit when downloading artifacts in slack report ( #46796 ) [CI] Fix artifact download path in self-comment-ci workflow ( #46769 ) ci: add comment explaining why secrets are not inherited in security gate ( #46750 ) ci: trigger PR CI on ci-* branches ( #46746 ) ci: disable CircleCI by replacing config with no-op ( #46721 ) ci: grant pull-requests:write to the security gate caller ( #46715 ) Reduce per_page from 100 to 50 in GitHub API calls to avoid server errors ( #46678 ) ci: add NO_COLOR=1 to suppress ANSI color codes in CI output ( #46659 ) ci: add merge_group trigger to pr-ci-caller.yml ( #46668 ) Revert "Disable PR CI workflow for PRs from forked repo. during the weekend" ( #46652 ) Disable PR CI workflow for PRs from forked repo. during the weekend ( #46609 ) @Mi-Jiazhi Add HunYuan VL model ( #46417 ) @tarekziade chore(linter): add TRF018 modeling rule ( #46259 ) only in the original repo ( #46982 ) the CI status should be a comment ( #46976 ) Insert a Grafana badge in the PR ( #46774 ) call transformers-ci in a nightly run ( #46811 ) @casinca Add Xiaomi MiMo-V2 ( #45144 ) @JJJYmmm [new model] Add Zyphra/ZAYA1-8B ( #45862 ) @ebezzam Fix typo in Qwen3 ASR no_split_module ( #47002 ) Fix Xcodec2 attention to be non-causal. ( #46963 ) Use common floats_list method for feature extractor tests. ( #46956 ) Add xcodec2 model ( #44178 ) @meatybobby Add support for RADIO models ( #46425 ) @douglas-reid 🚨 [gemma 3/4] Fix bidirectional attention masking crossing sliding window boundaries ( #46850 ) @Sunt-ing Fix Mamba2 chunked-prefill / speculative decoding for Zamba2, Nemotron-H, Bamba, FalconH1 and GraniteMoeHybrid ( #46741 ) Reject assisted generation for LFM2 and LFM2-MoE (set _is_stateful) ( #46937 ) Don't pin the gated delta net norm to cuda:0 with a hardcoded device ( #46817 ) Fix prompt lookup decoding crash when no EOS token is configured ( #46790 ) Fix left-padding token selection in BioGptForSequenceClassification ( #46782 ) Fix offloaded cache device mismatch on hybrid models ( #46748 ) Fall back to the for-loop grouped_mm on CPU ( #46743 ) @eustlb Add Nemotron 3.5 ASR Streaming ( #46565 ) [NemotronAsrStreaming] fix pipeline ( #46870 ) [NemotronAsrStreaming] processor without modular ( #46865 ) Add Nemotron ASR Streaming ( #46332 ) [fix] enable base64 str audio in load_audio ( #46694 ) @vasqu [ Dia ] Fix docs ( #46923 ) [ CB ] Add FA2 to the fast path ( #46729 ) [ Kernels ] Trigger proper kernelization on use_kernels=True ( #46755 ) [ CI ] Fix some failures introduced by myself 😬 ( #46751 ) 🚨 [ Kernels ] Sync to latest version ( #46039 ) [ Templates ] Update members ( #46720 ) [ Blip2 ] Fix cross attention reshape ( #46695 ) Update post release ( #46608 ) @mbtariq82 Qwen3 ASR and Forced Aligner ( #43838 ) @remi-or [CB] Changes to increase max_batch_tokens ( #46712 ) [CB] Fix issues with FA read / writes ( #46765 ) [CB] Fix offloading ( #46587 ) [Fix] Serve drops some attributes at serialization ( #46680 ) [CB] Slice logits inside the model ( #46660 ) [CB] Fix seqlens and use TypedDict ( #46593 ) @jiqing-feng Fix BitNet packed-weight unpacking dtype ( F.linear dtype mismatch) ( #46808 ) Fix Evolla eager attention for the GQA text decoder ( #46860 ) Fix flex_attention block mask creation when get_seq_length returns a tensor ( #46802 ) Lazily build the default kernel mapping to decouple kernels from normal transformers usage ( #46681 ) @bzantium Add MiniCPM3 ( #41116 ) @MHRDYN7 Add Videoprism ( #39895 ) @YangKai0616 [RecurrentGemma] Support attn_implementation dispatch ( #46320 ) [blip_2] Support attn_implementation dispatch ( #46401 ) [CTRL] Support attn_implementation dispatch ( #46073 )