2026-10-05

First of all, thank you for reading. I trained a small MoE model completely from scratch (no external base weights) and wanted to share the results + a couple of lessons. Apex-2 - Architecture: Decoder-only MoE, every layer is MoE (no dense layers) - Size: 3.87B total parameters, 1.45B active per token - 32 layers, d_model 2048, GQA 16Q/4KV, 16 experts, top-4 - Context: 4096 - Tokenizer: Qwen3 (151k) - Hugging Face: https://huggingface.co/YOON1v/Apex-2 (loads with transformers / vLLM via Qwen3MoeForCausalLM mapping) Training - Pretrain: 86.5B tokens (GH200 ×1 → ×2 with DiLoCo ) - SFT: ~2.5B tokens (code-heavy + math + instruction) - DPO: tried it, scores dropped, so I dropped the checkpoint Key numbers (SFT, greedy, chat template) Benchmark HumanEval 43.9 HumanEval+ 41.5 MBPP 56.3 MBPP+ 48.9 GSM8K (0-shot CoT) 32.4 MATH-500 21.0 IFEval (prompt strict) 44.7 MMLU (5-shot) 28.6 interesting comparison With only ~0.087T pretrain tokens, the base model’s HumanEval+ matched Qwen2.5-1.5B (which used 18T). Knowledge (MMLU) and math still lag far behind, as expected with the data gap. What didn’t work DPO (220k pairs, length-normalized) made answers much longer and hurt code / math / IFEval. I stopped it and kept the SFT checkpoint. Full log is in the benchmark write-up. Limitations (honest) - English-centric (almost no multilingual ability) - Weak knowledge → frequent hallucinations - LiveCodeBench medium/hard is near zero - 4k context only Happy to answer questions about the MoE setup, DiLoCo, or why DPO backfired. submitted by /u/Prestigious-Taste-63 to r/LocalLLaMA [link] [comments]

​ From the first LLaMA 33B I knew I wanted that magic-like intelligence locally, mine, so nobody could take it away when I needed it. I bought a 3090 for my home PC. Then LLaMA 65B appeared and I was dazzled, it looked like it had all the knowledge in the world. I made two copies, one local and one on my Synology NAS RAID, so I'd never lose it, and bought a second 3090 to run it. I was happy for a year with small coding tasks on LLaMA and Qwen models. Then DeepSeek 671B MoE appeared. Wow, frontier level at home. I upgraded to a Threadripper with 512GB DDR4 and ran it at 8 t/s with experts offloaded to RAM, or Qwen 235B at 10-12 t/s when I wanted speed. I used these for real coding at my job, in OpenWebUI. Then agentic coding took off and this was too slow. At 100k context generation speed halved and prefill made it a beautiful yet agonising experience. So: 16x3090 across P620-based nodes on a 100Gbit network. It ran MiniMax M2, Qwen 235B and even Qwen 397B, as good as anyone could desire. I built an entire paid project with 397B in OpenCode. But bigger models were out of reach, and the house circuit said no: the fuses blew whenever the rig and the electric oven ran together. Heat and stability were issues too. Next came 4x ASUS GB10, after I read they can be linked (3 was the biggest supported config). 397B at 30 t/s on 400W, versus 50-60 t/s at 6kW, rock solid and almost silent. A dream come true. I built two more projects with it. Then MiMo 2.5 Pro and Kimi 2.6 appeared, smarter and more productive. I found no published solution for an 8-node cluster, but I still bought four more GB10s and made it work. 397B ran at FP8 instead of INT4, and 20% faster. I posted the first MiMo 2.5 Pro and Kimi 2.6 solutions on 8xSparks on the NVIDIA forum. I liked the result so much that I talked my older brother into buying his own 8x GB10, so he could run the best open models locally too, in privacy, without depending on API availability and rising costs. His house is a 5-minute walk from mine. When Kimi K3 (2.8T) appeared, biggest and smartes open weights model, we joined the clusters: two 8x clusters for daily use, or one 16x when we want the biggest model at home. After some work I published the first working solution for Kimi K3 on 16x Sparks on the NVIDIA forum. Through multiple iterations, it went from an unusable 7 t/s at 100k context to a fairly usable 20 t/s at 300k. Now we're adding 4 more Sparks, so a smaller, faster model (GLM 5.3 Flash) runs 24/7 while the big cluster runs either GLM 5.3 on 8x plus MiMo 2.6 Pro on the other 8x, or 16x Kimi K3, or Qwen 3.8 2.4T. I'm always tuning speed on the big models and rebuilding vLLM/SGLang images, so always-on smaller cluster made sense, why? Because for all my work projects and my vllm/sglang personal projects, I chose to use only local hosted models, I never paid a comercial model subscription, not because of the cost, but, because of my strong confidence in local models future. They arrive October 2, along with 4 more Sparks for my younger brother, who got caught by the same local AI microbe :) submitted by /u/ciprianveg to r/LocalLLaMA [link] [comments]

Yesterday was Grok 4.7 ( pelicans ) and MiMo v2.6 Flash/Pro ( more pelicans ). Today Anthropic released Claude Opus 5.5 , and around an hour later OpenAI released GPT-6 Sol and GPT-6 Luna . It's going to take a while to get a good read on all of these new models, but here are my impressions so far. GPT-6 Sol and Luna are half the price of their GPT-5.6 equivalents GPT-5.6 Luna was already my favorite model for building applications against, because it combined excellent performance with being really cheap . Somehow GPT-6 Luna is half the price of that again - and GPT-6 Sol had a similar reduction compared to GPT-5.6 Sol. Here's what the pricing landscape looks like today: Model Input Cached input Output GPT-6 Luna $0.10/M $0.01/M $0.50/M GPT-5.6 Luna $0.20/M $0.02/M $1.20/M Grok 4.7 $2/M $0.50/M $6/M GPT-6 Sol $2/M $0.20/M $10/M GPT-5.6 Terra $2/M $0.20/M $12/M Claude Opus 5.5 $4/M $0.20/M $20/M GPT-5.6 Sol $4/M $0.40/M $20/M Claude Fable 5.1 $10/M $0.25/M $50/M GPT-6 Astra $10/M $1/M $50/M Note that GPT-5.6 has a scheduled 25% price increase for November, so GPT-6 is half the price of the promotional pricing for those models. (With GPT-5.6 Terra priced the same as GPT-6 Sol, any remaining reasons to use Terra just evaporated.) It's hard to overstate how competitive this pricing is. Grok 4.7 priced itself at $2/$6, less than half the price of GPT-5.6 Sol, but is now equally priced to GPT-6 Sol on input and closer on output. At $0.10/$0.50 GPT-6 Luna is one of the cheapest models OpenAI have ever released, beaten only by the far weaker GPT-4.1 Nano ($0.10/$0.40, April 2025) and GPT-5 Nano ($0.05/$0.40, August 2025). I rendered pelicans for GPT-6 Luna and for GPT-6 Sol , then I combined them all together in this comparison grid along with the GPT-5.6 pelicans. I like how you can instantly see that the 5.6 family chose bolder, brighter colors, while the 6 family is a lot more muted. I still think GPT-6 Astra on max produced the best pelican. Claude Opus 5.5 got a price cut too Opus 5.5 looks like it addresses the biggest complaints people had about Opus in terms of its communication style. Thariq Shihipar : Opus 5.5 is the result of your feedback. It communicates clearly, it's cheaper per token than Opus 5.0 with the intelligence of Fable 5.1 it's very token efficient and works across every effort level. It's also meant to be better at Blender . I'm looking forward to putting it through its paces there. Opus 4.5, 4.6, 4.7, 4.8, and 5 all shared the same price: $5/million tokens for input and $25/million for output. 5.5 is a 20% reduction - $4/million and $20/million. The price for cache reads fell 60%. That's significant for longer agentic conversations, where 90%+ of input tokens are processed at cached token prices. The new price for Opus 5.5 is the same as the price for GPT-5.6 Sol, but that was before OpenAI dropped their Sol prices by half. GPT-6 Astra and Claude Fable 5.1 are both priced at $10/million input and $50/million output. The price war currently affects the next tier of models below that. Anthropic say that Sonnet 5.5 and Haiku 5.5 are coming soon. It's going to be interesting to see if Haiku can regain its price competitiveness at the lower end, given current Haiku 4.5 is $1/$5 while the latest GPT-6 Luna is one tenth of that price at $0.10/$0.50. Claude Opus 5.5 max over-thinks to the point of breaking In a first for my " Generate an SVG of a pelican riding a bicycle " test, Claude Opus 5.5 at "max" thinking level failed to return a response! It started by calling this "a classic test request", and then thought really, really hard about what it was doing: This is a classic test request, so I want to plan out a well-composed pelican with its distinctive beak and pouch riding a bicycle with proper wheels, frame, and pedals, set against a simple sky and ground backdrop. [...] Verifying the shin length checks out at roughly 95.2, close enough. Now I'm working out the near leg path from hip to knee to ankle, then sketching the foot shape resting on the pedal — outlining the heel, toe tips, and sole contour with a path using lines and curves to sit naturally on the pedal surface around y=478-494. [...] I like the fish sticking prominently out of the basket with the pelican eyeing it as a fun detail worth keeping. I'm also confirming the eye placement near the bill base matches typical pelican anatomy, and considering giving it a slightly happier expression. [...] The far leg reads correctly as passing behind the frame, so I'm moving on to check the chainring teeth and confirm layer ordering—the far crank arm should be mostly hidden by the seat tube and chainring. I'm settling on the final SVG's width and height attributes alongside the viewBox to ensure proper scaling, noting there's no text so no font-family is needed. [...] I was so excited to see this pelican... but then it stopped . Opus 5.5 has a 128,000 maximum output token limit (as do the other Claude models), and it hit that while it was still reasoning about the SVG! I tried a second time and got the same result. This makes me suspect that "max" is effectively useless - if it over-thinks to breaking point on a stupid SVG prompt I don't trust it not to do the same for more interesting work. (Those two failures each cost me $2.56 and took nearly 20 minutes.) Fable 5.1 on "max" didn't over-think and did give me the best pelican I've seen from any Anthropic model. Here are the Opus 5.5 pelicans , excluding 5.5 max. I also built this comparison grid comparing them with pelicans by Opus 5, Fable 5.1, and Sonnet 5: Comparing different model vendors by how well they draw a pelican riding a bicycle may not make much sense now (if it ever did), but I'm still finding value in using them for comparisons of the same model families at different reasoning levels. I'm now using GPT-6 Sol and Claude Opus 5.5 as my default models in Codex and Claude Code. I've upgraded the Datasette Agent demo at agent.datasette.io to use GPT-6 Luna, and it seems to be fast and competent at both SQL queries and building HTML and JavaScript for Datasette Apps . Tags: ai , openai , generative-ai , llms , anthropic , claude , llm-pricing , pelican-riding-a-bicycle , llm-release , gpt , gpt-6-astra

v0.31.0 Highlights This release features 717 commits from 307 contributors (96 new)! DeepSeek-V4.1-Flash performance : FlashMLA mega attention with the V4.1 NVFP4 compressed KV cache is now the SM100 default ( #56935 ); DeepGEMM sparse MQA logits for the indexer ( #56254 ) and Mega-Gate fusing the gate GEMM with expert selection ( #56266 ); decoder boundaries fuse the TP all-reduce, mHC input preparation ( #57643 ) and the MoE finalize ( #58586 ); a fused small-batch WO-A with inverse RoPE and MXFP8 quant on SM100/SM103 ( #58634 ); the MXFP8 wo_b GEMM fused with the sequence-parallel reduce-scatter ( #57428 ); Engram wkv sharded across TP ranks ( #58678 ) and Engram host tables shared across co-located DP replicas by default ( #57651 ); encoder CUDA graphs for the vision tower ( #56625 ); and SWA bounded replay that keeps the sliding-window KV out of prefix caching ( #56227 ). Fast restart : the new vllm preload CLI launches the weight-cache daemon that keeps post-quantized weights resident in GPU memory across engine restarts ( #56680 ), now with data parallelism ( #57386 ), MTP draft models ( #57312 ), a /health endpoint ( #58552 ) and a readiness wait ( #58370 ). Experimental initialized-engine snapshots ( vllm snapshot create/restore ) use CRIU to restore a fully initialized TP1 engine ( #51360 ). Model Runner V2 and speculative decoding : draft-model speculative decoding ( #43091 ) and custom logits processors ( #56497 ) on Model Runner V2; the new LiLiCorr drafter ( #57934 ); async scheduling for DFlash ( #58065 ) with the context K/V precompute captured in the draft CUDA graph ( #57632 ); DSpark adaptive verification for Gemma4 ( #57263 ) and variable-length decode for Kimi-K3 ( #52988 ); a DCP target with a non-DCP DSpark draft ( #56723 ); and MoE memory now counted during MRV2 profiling, avoiding OOMs on WideEP deployments ( #57270 , #58411 ). Large scale serving : MoonEP balanced EP all2all backend via --all2all-backend moonep ( #52101 ), prefill context parallelism with data parallelism ( #57075 ), a low-SM multimem reduce-scatter for SM100/SM103 ( #55072 ), DeepEPv2 with sequence parallelism ( #57210 ) and EPLB with shared-expert overlap ( #57236 ), a sharding-aware NCCL M2N weight-transfer backend for RL ( #51520 ), and KV offloading back-pressure detection ( #50045 ). Scheduling controls : --max-num-active-seqs caps RUNNING admission independently of max_num_seqs ( #56758 ), --long-prefill-token-threshold now adapts to the number of waiting prefills instead of chunking a lone request ( #57951 , #58459 ), the waiting queue was reworked so requests already holding KV blocks are scheduled first ( #58947 ), and the KV connector + MTP deadlock under KV pressure was fixed ( #57104 ). HiSparse hardening : MTP verification rows resolved with a union residency kernel ( #59235 ), MTP acceptance collapse under FULL graphs fixed ( #59309 ), no GPU pages without host backing ( #59036 ), a chunked-prefill preemption livelock fixed ( #59494 ), KV cache sized from the groups HiSparse allocates ( #59450 ), and host prefix publication and GPU prefix adoption fixes ( #59007 , #59282 ). Security : per-request mm_processor_kwargs and media_io_kwargs are rejected unless --trust-request-mm-kwargs is set ( #58830 ); prefix-cache extra keys are tagged by source so a LoRA name and a cache_salt can no longer collide ( #51899 ), and the LoRA path is part of the block hash ( #59335 ); stale multimodal receiver-cache entries can no longer replace fresh payloads ( #57833 ). Breaking changes : per-request multimodal kwargs gated ( #58830 ); tokenizer_mode="slow" removed ( #58545 ); --enable-mamba-fine-grained-prefix-cache renamed to --enable-mamba-shared-prefix-checkpoint ( #57382 ); online quantization through quantization="fp8" replaced by the fp8_per_tensor shorthand ( #53585 ) and Quark silent online quantization removed ( #51800 ); the AllSpark INT8 W8A16 backend removed ( #58001 ); --enforce-eager now also disables JIT kernel warmup ( #58197 ); XPU graphs enabled by default with VLLM_XPU_ENABLE_XPU_GRAPH removed ( #51600 ). Release Artifacts Python Wheels Platform Install PyPI (CUDA 13.0) pip install vllm PyPI (CUDA 13.0, uv) uv pip install vllm --torch-backend=auto ROCm pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.31.0/rocm723 XPU uv pip install vllm --extra-index-url https://wheels.vllm.ai/0.31.0/xpu --extra-index-url https://download.pytorch.org/whl/xpu --index-strategy unsafe-best-match Docker Images Platform Docker Image CUDA 13.0 (Default) docker pull vllm/vllm-openai:v0.31.0 CUDA 12.9 docker pull vllm/vllm-openai:v0.31.0-cu129 ROCm docker pull vllm/vllm-openai-rocm:v0.31.0 CPU docker pull vllm/vllm-openai-cpu:v0.31.0 XPU docker pull vllm/vllm-openai-xpu:v0.31.0 Other Artifacts Pre-built release artifacts are available in the Assets section at the bottom of this page, including: Source distribution tarball CUDA 12.9 Python wheels for x86_64 and arm64 CUDA 13.0 Python wheels for x86_64 and arm64 CPU Python wheels for x86_64, arm64, and macOS XPU Python wheel for x86_64 Model Support New model capabilities : DiffusionGemma structured generation mode with bounded single-token choices ( #57250 ), MiMo V2 MXFP4 MoE, BF16 MoE router and DFlash drafts ( #57784 ), Cohere2MoE auxiliary hidden states for EAGLE3/DFlash drafters ( #49819 ), GLM-5.2-MXFP4 on the ROCm DeepSeek-V3.2 path ( #51915 ), AMD-Quark mixed-precision DeepSeek-V4.1-Flash-MXFP4 ( #57071 ) and GLM-5.3-Flash Quark MXFP4 ( #56176 ) checkpoints, and a built-in granite_thinking_parser for Granite 4.2 ( #55957 ). DeepSeek-V4.1-Flash : FlashMLA mega attention with the NVFP4 compressed KV cache as the SM100 default ( #56935 ), DeepGEMM sparse MQA logits in the indexer ( #56254 ), Mega-Gate ( #56266 ), fused TP all-reduce + mHC input preparation ( #57643 ) with the MoE finalize folded in ( #58586 ), fused small-batch WO-A ( #58634 ), MXFP8 wo_b GEMM + reduce-scatter ( #57428 ), overlapped mHC coefficients for small TP batches ( #57603 ) limited to FULL CUDA graphs ( #57874 ), Engram wkv sharded across TP ( #58678 ), serialized Engram lookups with huge-page host tables ( #56926 ), Engram tables shared across DP replicas by default ( #57651 , #57914 , #59068 ), native shared-expert MegaMoE fusion without padding ( #56568 , #57204 ), faster MegaMoE staging and NVFP4 cache gathers ( #57604 ), the fused query RMSNorm + MXFP8 path restored ( #57679 ), KV-only DSpark context insertion ( #56441 ), vision encoder CUDA graphs ( #56625 , #58499 ), SWA bounded replay ( #56227 ), causal image SWA restored to match the reference ( #57152 ), updated reasoning effort mappings ( #58316 ), and startup fixes for NaN-scored candidate blocks ( #57454 ), DeepSelect sentinels ( #58215 ), DSpark non-causal attention on FlashInfer ( #57432 ), runtime JIT of offset candidate buffers ( #57667 ) and imports without Triton ( #57654 ). DeepSeek V4 : FlashInfer sparse MLA with fused inverse RoPE + FP8 quant ( #58621 ), stacked DSpark context WKV projections ( #54674 ), fused MoE expert distribution computed from the EP group ( #57465 ), FIM completion via suffix ( #44229 ), missing string= in tool calls parsed ( #56271 ), request tools attached to an existing system message ( #51856 ), image block spacing preserved ( #56882 ), and DSpark width separated from MTP stage validation ( #54631 ). GLM-5.3-Flash : opt-in FlashAttention and FlashMLA sparse backends on SM90 ( #55385 ), NoPE sparse MLA on the FlashInfer SM120 backend ( #55277 ), cooperative top-k for small decode batches ( #57327 ), indexer decode workspace sized by pooled length (3 GiB saved, #57701 ), metadata ops 1.6-4.8x faster ( #58450 ), fused kpool tail slot mapping ( #57534 ), kpool top-k through the shared dispatcher ( #57546 ), lower sparse MLA preparation overhead ( #57458 ), no D2H sync in the SM90 sparse MLA plan under async scheduling ( #58684 ), FlashKDA keeping the recurrent state in FP32 for long prefills ( #58846 ), and correctness fixes for kpool corruption with speculative decoding ( #58454 ), SM90 index_kpool mismatch ( #58704 ), kpool tail strides ( #57477 ), 500k-token prompts ( #57317 ), dense MLP layers under sequence parallelism ( #58061 ), indexer top-k backend selection ( #58594 ) and varlen paged MQA launches ( #55270 ). Qwen3.8-Flash-Next (Qwen4Exp) : FP8 main KV cache on the QSA path ( #55557 ), FP8 TP with FlashInfer TRTLLM MoE ( #55867 ), SM90 QSA tuning ( #57273 ), fused HC down projection + SiLU ( #58957 ), lower PLE metadata overhead ( #58114 ), QSA indexer workspace fragmentation fixed ( #57105 ), profiling KV cache released ( #58961 ), pinned PLE prefetch ids kept out of the CUDA graph pool ( #58489 ), and indexed expert-mapping lookups saving about 25 s of weight loading on DGX Spark ( #58720 ). Kimi K3 and MiniMax-M3 : Kimi-K3 vision patch embedder as a GEMM ( #58527 ), fused KimiViT QK RoPE (up to 29x, #58651 ), routed expert quantization ( #57430 ), reasoning parser ( #57098 ) and reasoning token counting ( #58372 ) fixes, DSpark context KV pointers refreshed after re-binding ( #58814 ), stateless first chunks no longer classified as decodes ( #51483 ), and Mamba block estimates that no longer stall admission on external prefix hits ( #57050 ); MiniMax-M3 encoder CUDA graphs ( #58673 ), Conv3dLayer patch embedding (about 62x, #58512 ), triton_mrope in the vision tower ( #58526 ), fused MiniMax2 routing with non-unit scaling ( #58880 ), and processor fixes ( #58460 , #59613 ). DiffusionGemma : one-pass sampler statistics kernel ( #58226 ), diffusion_constrained reads over logprob_token_ids (about 25% faster, #58216 ), fewer logit rows for prefill-only batches ( #57416 ), logprob_token_ids support ( #57417 ), and fixes for concurrent logprobs ( #57414 ), eager fallback dtype ( #57462 ), multimodal inputs ( #57589 ), quantized LM heads ( #48521 ) and CPU execution ( #58964 ). Multimodal : Triton mm_input_norm kernel ( #56798 , #56711 ), raw pixels kept through the DP-sharded ViT path ( #56872 ), Qwen2.5-VL video fps honored for temporal M-RoPE ( #47736 ), Whisper and Qwen2-Audio clips longer than 30s ( #57769 , #56912 ), Mistral3 placeholder grid ( #53758 ), Idefics3 unsplit patches ( #48760 ), Ovis2.5 tokens ( #52623 ), Aria expert weights ( #57487 ), MiMo-V2.5 fused FP8 qkv_proj sharding ( #57508 ), Gemma4 AutoWeightsLoader ( #55911 ) and buffer scalars ( #54213 ), Gemma4 FP8 KV with FA4 at head dim 512 ( #53175 ), Laguna RoPE ( #57189 ), Mistral-Large-3 accuracy regression ( #57563 ), malformed EXIF ( #56527 , #57234 ), JinaVL labels ( #57347 ), Sarvam MLA routing with FP32 router logits ( #56034 ), fused CohereASR attention scores ( #55190 ), and fused DFlash2 grouped convolution ( #55960 ). LoRA : Nemotron VL language models ( #56231 ), ModernBert ( #57148 ), VoyageQwen3 embeddings ( #57708 ), RoBERTa sequence classification ( #58884 ), and modules_to_save sequence-classification heads ( #53555 ) with per-adapter num_labels ( #57766 ). Engine Core Model Runner V2 : draft-model speculative decoding ( #43091 ), custom logits processors ( #56497 ) validated at admission ( #57728 ), randomized dummy inputs ( #58411 ), dummy tokens routed to MoE experts during profiling ( #57270 ), FULL decode graphs for one-token prompt tails ( #58400 ), token-to-request mappings shared ( #57102 ) and Mamba/GDN metadata reused across KV cache groups ( #58762 ), encoder-only ViT CUDA graphs ( #56922 ), weight offloader ( #57834 ) and pooling model ( #57737 ) released on shutdown, and fixes for multi-layer MTP KV in P/D ( #55055 ), fast-prefill with LoRA ( #56456 ), stale block-table writes from dummy draft steps ( #56734 ), padded prompt tails in hybrid models ( #58434 ), never-proposed draft slots ( #58784 ), auto-fit max_model_len ( #58149 ) and intermediate_tensors during capture ( #57745 ). Speculative decoding : LiLiCorr drafter ( #57934 ), DFlash async scheduling ( #58065 ) and context K/V in the draft CUDA graph ( #57632 ), Gemma4 DSpark adaptive verification ( #57263 ), Kimi-K3 variable-length adaptive verification ( #52988 ), CPU-GPU sync removed for heterogeneous vocabularies ( #57396 ), Triton recompiles avoided in the acceptance estimator ( #57107 ), and fixes for EAGLE/dense drafts with EP ( #56930 ), DFlash/DSpark profiling batches ( #56448 ), GLM MTP head memory ( #55442 ), prompt embeddings with drafts ( #57356 ), and TritonMLA causal multi-token decode ( #51065 ). Scheduler : --max-num-active-seqs ( #56758 ), adaptive --long-prefill-token-threshold ( #57951 , #58459 ), skipped_waiting replaced by a KV-holding waiting queue ( #58947 ), atomic admission of n > 1 requests ( #53936 ), KV connector + MTP deadlock ( #57104 ), a throughput cliff when max_num_seqs is not a multiple of 8 ( #57355 ), streaming continuations keeping logprobs and refreshing max tokens ( #57447 , #57676 ), a resumable request + async scheduling race ( #58259 ), and a non-blocking structured-output grammar poll ( #55931 ). Prefix caching and hybrid models : extra keys tagged by source ( #51899 ) and LoRA paths hashed ( #59335 ), --enable-mamba-shared-prefix-checkpoint ( #57382 ), prompt-tail hits with MTP restored ( #58368 ), prompt-end checkpoints kept under sparse retention ( #59146 ), align-mode checkpoint reservation ( #59175 ), stateless GDN first chunks ( #51565 ), MTP draft KV cache groups annotated on the hybrid grouping path ( #55390 ), a generalized prefill checkpoint builder ( #57783 ), batched Mamba2 prefill state saves without GPU-CPU syncs ( #49371 ), skip_reading_prefix_cache honored for connector hits ( #57269 ), incremental multimodal block hashing ( #51694 ), a KV block size every attention backend supports ( #49845 ) with clearer errors ( #58557 ), and CacheConfig.effective_attention_block_size for DCP ( #56538 ). Startup and memory : parallel Triton warmup compilation ( #58582 ), JIT warmup disabled under --enforce-eager ( #58197 ) unless fault tolerance is on ( #58593 ), DeepGEMM warmup reusing the MoE workspace ( #57268 ), allocator fragmentation no longer shrinking the KV cache during profiling ( #58430 ), sparse prefill buffers reserved before KV sizing ( #57575 ), stale FlashMLA workspace views released ( #56902 ), DeepGEMM FP8 workspace halved ( #53914 ), shared Marlin/Humming workspaces ( #57421 ), FlashInfer BF16 MoE weights converted in place ( #54699 ), UniProc startup threads bounded by the CPU quota ( #58946 ), and runtime threads set before profiling ( #55891 ). Kernels : sampled filtering for persistent top-k ( #56346 ), Murmur3 RNG for Gumbel sampling ( #51367 ), register-resident per-token-group 8-bit quant ( #55330 ), vectorized per-tensor FP8 abs-max ( #58194 ), a Triton kernel dispatcher for platform-specific overrides ( #43048 ), GateLinear for all MoE models ( #58234 ), non-local expert slots skipped in TritonExperts ( #58051 ), deferred TRT-LLM-Gen top-k finalize on the modular path ( #58635 ), SM120 batch-invariant matmul configs ( #57456 ), FlashInfer prefill dequant scratch bounded ( #57918 ), and fixes for Triton softcap NaNs ( #56579 ), QuIP Hadamard transforms ( #43462 ), CUTLASS FP8 linear on A100 ( #55884 ), FlashInfer sampling on unsupported GPUs ( #48956 ), SM100 FP8 blockwise scale padding ( #57377 ), mixed FULL graph prefill capture ( #58275 ) Inductor custom-op pattern matching ( #58189 ), CuteDSL BF16 GDN prefill diverging from FLA on Qwen3.5 ( #53864 ), SM100 fp8_ds_mla cache scales ( #49435 ), ragged decode batches in the sparse indexer ( #52500 ), stale allowed_token_ids masks after batch reordering ( #48419 , #43931 ), and prompt_embeds tensors held after requests finish ( #57988 ). Batch invariance : breakable CUDA graphs without torch.compile by default under VLLM_BATCH_INVARIANT ( #57586 ), sequence parallelism and async TP disabled ( #56377 ), and NCCL 2.31 collectives kept enabled ( #58179 ). Sleep and RL : release_kv_cache_memory() frees only KV cache memory ( #44890 ), --sleep-preserve-parameter-names retains frozen weights across level-2 sleep ( #57891 ), NCCL M2N weight transfer ( #51520 ), dense DP weight updates by DP index ( #56950 ), allocator config kept when toggling expandable segments ( #57982 ), and cache reset failures propagated during sleep ( #54581 ). Fast restart : vllm preload ( #56680 ), DP ( #57386 ), MTP drafts ( #57312 ), readiness wait ( #58370 ), /health ( #58552 ), ModelOpt MXFP8 pre-processed weights ( #57316 ), and initialized-engine snapshots ( #51360 ). Logging and observability : LoggingConfig via --logging-config and --log-level ( #57205 ), JSON logging fixes ( #57957 , #58747 ), startup log suppression fixed ( #51366 ), unified platform-aware torch profiling ( #57460 ), no 0.0% prefix cache hit rate before any query ( #54990 ), KV transfer metrics formatting ( #57068 ), MFU activation sizing from the model dtype ( #57070 ), platform-overridable env var checks ( #48599 ), VLLM_TARGET_DEVICE=empty pip install vllm for out-of-tree backends ( #41074 ), and the correct vLLM version reported when installing from source ( #57295 , #57744 ). Large Scale Serving MoE communication : MoonEP backend ( #52101 ), DeepEPv2 with sequence parallelism ( #57210 ) and EPLB + shared-expert overlap ( #57236 ), low-SM multimem reduce-scatter ( #55072 ), EPLB load statistics during Elastic EP scaling ( #58473 ), SP padded rows skipped in grouped routing so rank 0 is no longer a prefill straggler ( #56079 ), hash routing rejected on unsupported monolithic backends ( #57867 ), sampled-token broadcasts skipped under PP for requests leaving the engine ( #58542 ), DBO with DeepEP low-latency profiling ( #57502 ), external LB with replicas sharing nodes ( #53743 ), and no blocking RPC during the engine handshake ( #57226 ) or unbounded draft-token waits ( #58779 ). Context parallelism : PCP with DP, EP and MTP ( #57075 ), a DCP target with non-DCP DSpark/DFlash drafts ( #56723 ), DCP sequence lengths without a CPU-GPU sync ( #58169 ), and NIXL DCP pulls across MLA cache regions ( #57389 ). KV connectors : NIXL pipeline-parallel push prefill for attention-HMA ( #50494 ) and packed MLA layouts ( #50499 ), transport-failure metrics split from KV expiry ( #55854 ), dead peer state released without waiting for TTL ( #50047 ), expired leases reaped behind a heartbeated head ( #58292 ), push completion restored ( #58188 ), D-side activity recorded ( #52245 ); Mooncake CUSTOM_MEM_POOL ( #49300 ), request-level load failures under HMA ( #56855 , #57174 ) and bootstrap retries ( #58919 ); MoRIIO hybrid Mamba/KDA state in READ mode ( #51052 ) and a multi-decode routing race ( #51681 ); prefill cache hits in prompt_tokens_details ( #54222 ); KV-event publishers bound at port 0 ( #55844 ); KV cache metadata GET restored for external consumers ( #56925 ); partial-block KV events keep every multimodal feature ( #58288 ); saves finalized on steps without a forward ( #57775 ); abort-safe ExampleHiddenStatesConnector ( #56841 ); DecodeBench FP8 fills ( #58472 ); a KvHints request envelope ( #53423 ); and the AuxOutput connector for routed-expert outputs ( #45635 , #58150 , #58205 ). KV offloading : per-request max_load_tokens ( #55885 ), back-pressure detection ( #50045 ), SimpleCPUOffloadConnector Prometheus metrics ( #57251 ), non-prefix-cacheable ( #56810 ) and scratch ( #57145 ) groups skipped, replicated layouts for multi-group MLA ( #57652 ), regions of 512 GiB or more registered in chunks ( #51081 ), cgroup memory checked before SHM allocation ( #54014 ), and fixes for cache recency ( #51787 ), MTP-retained sliding windows ( #56709 ), canonical MLA rows ( #56799 ), event metadata ( #57453 ) and ROCm pinned memory ( #57160 ). HiSparse : union residency kernel for MTP rows ( #59235 ), MTP acceptance under FULL graphs ( #59309 ), host-backed allocation ( #59036 ), preemption livelock ( #59494 ), KV cache sizing ( #59450 ), host prefix publication ( #59007 ), GPU prefix adoption ( #59282 ), and residency metrics ( #58725 ). Encoder disaggregation (EPD) : dynamic EPD proxy with launcher-managed registration ( #54176 ), image requests batched per encoder ( #57095 ), cross-encoder caching through Mooncake ( #56242 ), metadata-only audio inputs ( #57887 ), language-model shards skipped for --mm-encoder-only ( #58086 ), EC connector metrics ( #54960 ), and encoder-only fixes ( #58490 , #58287 , #57696 ). Hardware & Performance NVIDIA : FlashInfer 0.7.0.post1 ( #58069 , #59323 ), DeepGEMM pin bump with SM120 fixes ( #57218 ), Rubin CUDA 13.4 nightly images ( #55953 ), and QuTLASS builds with PyTorch 2.13 ( #58173 ). AMD ROCm : AITER v0.1.23 ( #56885 , #58867 ), torch 2.13 and Triton 3.8 ( #50605 , #58006 ), triton_kernels 3.8 MXFP4 MoE for gpt-oss and DeepSeek-V4 ( #55934 ), a ROCR host segfault fixed ( #57328 ); DeepSeek-V4/V4.1 HCA dual-stream ( #56853 ) and layer-aware CSA2 overlap ( #57407 ), FP8 wo_a ( #54894 ), inverse RoPE fused into the sparse decode reduce ( #57435 , #57451 ) which now emits MXFP8 for a grouped FP8 wo_a ( #58456 ), MXFP8 GEMM on native 32x32 scales on gfx950 ( #58510 ), reused top-k ragged metadata ( #57434 ), faster candidate block selection ( #58208 ), Engram tables in host memory ( #57491 ) and Qwen3.8-Flash-Next PLE tables offloaded to host memory ( #57497 ), an opt-in AITER ASM decode route for DCP + speculative decoding ( #56861 ), get_top_tokens() on the DeepSeek V4 MTP drafter ( #57568 ), DSpark adaptive verification ( #52362 ), opt-in VLLM_DSV4_LOGITS_FIX for sparse-indexer logits on gfx950/gfx942 ( #50455 ), an accuracy revert of #56433 and #51692 ( #57132 ), and clear errors for FSE with DPA+ETP ( #57919 ); Kimi-K3 a4w4 FlyDSL kernels ( #53940 ) with VLLM_ROCM_USE_AITER_MOE_SITUV2=a16w4|a8w4|a4w4 ( #58201 ), low-concurrency speculative KDA ( #58045 ), sharded latent MoE under EP ( #54956 ) and fewer projection copies ( #50592 ); a ROCm Hy4 path with torch.compile (13x lower decode latency, #57526 ); MiniMax-M3 AITER QK-norm fusion ( #54535 ), copy-free K/V insert ( #56849 ) and MXFP8 fixes ( #53674 , #58089 ); GLM-5.3-Flash boot fixes ( #57192 , #57252 , #57425 ); AITER QuickReduce + RMSNorm ( #48249 ), BF16 AsyncTP ( #58098 ), static FP8 attention output fusion ( #58099 ), QK-norm/RoPE/KV-cache fusion for MRoPE ( #50212 ), AITER GDN decode for flat layouts ( #53623 ), wvSplitK for single-output GEMMs ( #53283 ), 69 fewer copies per decode step on the skinny GEMM path ( #58566 ), tuned GEMM lookup through AITER ( #55001 ), a narrower Triton prefill KV tile on RDNA3/RDNA4 ( #58225 ), staged large pageable H2D copies ( #56343 ), and fixes for MXFP4 MoE padding starving the KV cache ( #56359 ), AITER MLA FP8 prefill OOM ( #57923 ), unquantized cache descales ( #56726 ) and AITER MoE fallbacks ( #56590 , #57866 , #57426 ). VLLM_ROCM_USE_AITER_FP4_ASM_GEMM is restored and off by default ( #57055 ); SWA bounded replay is disabled on ROCm ( #57906 ). Intel XPU : PyTorch 2.14 ( #56013 ), XPU graphs on by default ( #51600 ), EPLB ( #44987 ), int8 W8A8 MoE on Triton ( #53162 ), batch invariance ( #55881 ), SYCL rotary embedding ( #55721 ), fused QK RMSNorm + RoPE + gate (decode region 55% faster, #56096 ), Model Runner V2 sampler ( #57277 ) and PP microbatch control ( #55145 ), --device-ids honored ( #56015 ), and device pointer overflow fixed ( #54514 ). CPU : FP8 W8A8 linear and MoE for Intel Diamond Rapids ( #49942 ), Arm paged attention up to 25% faster ( #56045 ), Zen DA8W4 int4 for dense and MoE layers ( #54024 ), zentorch SDPA for encoder attention ( #54508 ) and MLA prefill ( #54967 ), FP32 attention sinks ( #56252 ), W8A8 INT8 MoE on POWER ( #55316 ), W4A16 Whisper ( #58268 ), wheels built on Ubuntu 22.04 (glibc 2.34) with AMX-FP8 ( #58515 ), AVX10.2 gated on compiler support ( #58133 ), pre-built Triton CPU ( #58140 ), --device-memory-utilization alias ( #56547 ), vLLM Recipes in the CPU image ( #58796 , #57306 ), s390x protobuf pin ( #54978 ) and torchcodec video ( #58693 ), and fixes for Ministral FP8 ( #56985 ), FP32 router weights ( #56168 ), zentorch import failures ( #54923 ), macOS multimodal SHM ( #57142 ), CPU affinity per local rank ( #53636 ) and NIXL GDN state layout ( #53300 ). Quantization Humming : Hadamard transforms and NVFP4/MXFP4/MXFP8 online quantization ( #56685 ), asymmetric wNaM through compressed-tensors ( #46528 ), humming-kernels 0.1.16 ( #58054 ), and Humming in the W4A8 (INT4xFP8) MoE oracle ( #58427 ). New capabilities : native Quark W4A16 INT4/UINT4 exports ( #48606 ), opt-in load-time MXFP4 dequantization ( #50814 ), explicit per-token NVFP4 MoE backends ( #57176 ), and a canonical N-first layout for compressed-tensors WNA16 MoE ( #52798 ). Fixes : MXFP8 on layers below mm_mxfp8 shape limits ( #54223 ), online NVFP4 scales on reload ( #57954 ), LM head linear metadata ( #58444 ), and the fused SiLU-mul block-quant path skipped under a SwiGLU clamp ( #57984 ). API & Frontend New options and endpoints : --tool-strict-level ( #56268 ), response_format with tool_choice=auto ( #56086 ), DeepSeek-V4 FIM completions ( #44229 ), release_kv_cache_memory() and POST /release_kv_cache_memory ( #44890 ), fixed-token prefill scoring via prompt_logprob_token_ids ( #54335 ), per-request speculative decoding metrics in /inference/v1/generate ( #43310 ), per-request metrics ( #55084 ) and cache_write_tokens ( #57222 ) in the Responses API, Anthropic thinking in /v1/messages ( #58613 ), streaming reasoning and tool calls from the derender endpoint ( #50550 ) with offloaded detokenization ( #57528 ), vllm chat streaming thinking output ( #57045 ), request body debug logging with --enable-log-requests ( #58163 ), and only the summary line of config docstrings in --help ( #57357 ). Structured output and parsers : native Lark grammars in the xgrammar backend ( #58321 ), XGrammar 0.2.7 ( #57272 ), Granite migrated to the streaming Parser Engine ( #49648 ), and fixes for reasoning boundaries ( #56635 ), xgrammar choices with control characters ( #48115 ), list-typed JSON Schema ( #48416 ), empty structural_tag ( #47450 ), outlines EOS handling ( #58612 , #57743 ), parser-suppressed streaming logprobs ( #58583 ), Inkling tool names after reasoning ( #58792 ), and length finish_reason for truncated streaming tool calls ( #46303 ). OpenAI, Anthropic and Harmony compatibility : reasoning token counts for Harmony, DeepSeek-V3 and Step3 (#58626) and per Responses tool round (#58927), Harmony max_output_tokens in the tool loop (#58551), batched chat completions using the adjusted requests (#58929, #58958), a fresh parser per choice (#58939), deferred reasoning recounts (#56067), Responses MCP cleanup (#56988) and image detail default (#57241), Anthropic inline system detection (#58754) and disabled thinking with P/D (#58786), stop strings rejected on --tokens-only servers (#57058), invalid prompt_embeds returning 400 (#55451, #57006), prompts bounded after multimodal expansion (#57076), --override-generation-config penalties (#50769), Hub revisions (#56092, #57461), full logprobs in token-in/token-out responses (#58488), generative scoring cancellation (#57729, #58788), gRPC keepalive pings (#55102), and run-batch diarized transcriptions (#57948). Pooling : chunked embedding padding (#56505) and normalization (#57498), reranker tokenization with document limits (#57666), and BERT-family heads kept for raw logits (#57664). Rust frontend : --hf-overrides (#56931), --sse-keep-alive-interval (#58306), custom chat roles (#58311), MiMo V2.5 parsers (#57933), normalized reasoning controls (#56998), parser-owned output grammars (#55269, #57340), sampling masks over gRPC (#56777), local DP size in gRPC metadata with vllm-proto 0.3.0 (#57116, #57233), a per-request preemption histogram (#57033), lock-free histograms (#58574), Nemotron-H vision context (#57634), model-owned vision processors (#58109), an mm-processor benchmark (#51922, #58084, #58378), unsupported serve args recognized (#58330), NaN logprobs no longer killing the engine client (#51026), appended EngineCoreOutput fields accepted (#56533), and vllm-rs on PATH in the CUDA image (#57606). Benchmarks : an openai-responses backend for vllm bench serve (#54628) and model_id in latency/throughput JSON (#58112). Security Per-request mm_processor_kwargs and media_io_kwargs are rejected by default; trusted deployments opt in with --trust-request-mm-kwargs (#58830). Prefix-cache block hashes tag extra keys by source (#51899) and include the LoRA path (#59335); multimodal hash input is framed (#54283) and incremental block hashing covers every overlapping feature (#51694). Fresh multimodal payloads take precedence over a stale receiver cache (#57833), and encoder-cache hits with mismatched embedding counts are rejected (#57696). min_tokens above the filled max_tokens default is rejected instead of wedging the engine (#57731); message sanitization filters upper-case memory addresses (#58832); LoRA adapters named after a served model are rejected (#59286). Dependencies FlashInfer 0.7.0.post1 (#58069, #59323), Transformers 5.17.0 (#56108) with an upper bound in requirements (#59614), XGrammar 0.2.7 (#57272), oss-harmony replacing openai-harmony (#55128), DeepGEMM fork pin bump (#57218), FlashKDA bump (#58846), and humming-kernels 0.1.16 (#58054). CUDA 12 images use LMCache 0.4.4 and CuPy CUDA 12 (#57945); a separately tagged zstd Docker Hub image ( -x86_64-zstd ) is published (#55608); Rubin CUDA 13.4 nightly images (#55953). ROCm: AITER v0.1.23 (#58867), torch 2.13, Triton 3.8 (#50605, #58006), patched ROCR (#57328), LMCache OpenTelemetry pins (#59056). XPU: PyTorch 2.14 (#56013). CPU: wheels built on Ubuntu 22.04 (#58515), pre-built Triton CPU (#58140). Breaking Changes & Deprecations Per-request mm_processor_kwargs and media_io_kwargs now return an error unless the server is started with --trust-request-mm-kwargs ; server-level --mm-processor-kwargs / --media-io-kwargs and offline LLM are unchanged (#58830). tokenizer_mode="slow" was removed; it already behaved like "hf" under Transformers v5 (#58545). --enable-mamba-fine-grained-prefix-cache was renamed to --enable-mamba-shared-prefix-checkpoint (#57382). Online quantization through quantization="fp8" now redirects to the fp8_per_tensor online shorthand (#53585); Quark-specific silent online MXFP4 quantization was removed in favor of the online quantization API (#51800). The AllSpark INT8 W8A16 GEMM backend was removed (#58001). The return_assistant_tokens_mask option of /render and the assistant_tokens_mask response field were removed (#57520). VLLM_PLE_CPU_OFFLOAD was removed; use --engram-config (#57937). VLLM_XPU_ENABLE_XPU_GRAPH was removed and XPU graphs are on by default (#51600). The comma-separated form of --collect-detailed-traces was removed; use the list syntax (#55702). New defaults: --enforce-eager also disables JIT kernel warmup unless fault tolerance is enabled (#58197, #58593); VLLM_BATCH_INVARIANT=1 uses breakable CUDA graphs without torch.compile (#57586) and disables sequence parallelism and async TP (#56377); DeepSeek-V4.1 SWA bounded replay is on (#56227) except on ROCm (#57906); Engram host tables are shared across co-located DP replicas when possible (#57651); FlashMLA mega attention is the DeepSeek-V4.1 default on SM100 (#56935); the AITER w4a4 ASM GEMM is off by default on ROCm (#57055). Model Runner V1 + PP > 1 + async scheduling + structured output is now rejected at startup (#56250); json_object is rejected at validation with the outlines backend (#57743). transformers now has an upper bound in requirements (#59614). New Contributors @0z5a made their first contribution in https://github.com/vllm-project/vllm/pull/56441 @200lz made their first contribution in https://github.com/vllm-project/vllm/pull/55528 @AARONKANG04 made their first contribution in https://github.com/vllm-project/vllm/pull/48521 @acsoto made their first contribution in https://github.com/vllm-project/vllm/pull/57314 @adenzhou1350 made their first contribution in https://github.com/vllm-project/vllm/pull/56882 @adtygan made their first contribution in https://github.com/vllm-project/vllm/pull/56325 @amasen02 made their first contribution in https://github.com/vllm-project/vllm/pull/56576 @amd-sriram made their first contribution in https://github.com/vllm-project/vllm/pull/53792 @andrewor14 made their first contribution in https://github.com/vllm-project/vllm/pull/52956 @Ankit-Jaiswal-AMD made their first contribution in https://github.com/vllm-project/vllm/pull/56252 @baljinderhothi-cohere made their first contribution in https://github.com/vllm-project/vllm/pull/58792 @BaoYunkai made their first contribution in https://github.com/vllm-project/vllm/pull/57568 @blipbyte made their first contribution in https://github.com/vllm-project/vllm/pull/55612 @BPbruce made their first contribution in https://github.com/vllm-project/vllm/pull/46466 @chuan932 made their first contribution in https://github.com/vllm-project/vllm/pull/55330 @coderfornow made their first contribution in https://github.com/vllm-project/vllm/pull/54978 @CZT0 made their first contribution in https://github.com/vllm-project/vllm/pull/51694 @devtyagi3909 made their first contribution in https://github.com/vllm-project/vllm/pull/56635 @dilberx made their first contribution in https://github.com/vllm-project/vllm/pull/57076 @divyvasal made their first contribution in https://github.com/vllm-project/vllm/pull/49845 @equiluxe made their first contribution in https://github.com/vllm-project/vllm/pull/48115 @errmakov made their first contribution in https://github.com/vllm-project/vllm/pull/58583 @farzad-elastix made their first contribution in https://github.com/vllm-project/vllm/pull/56168 @freyfwt made their first contribution in https://github.com/vllm-project/vllm/pull/51367 @garrett361 made their first contribution in https://github.com/vllm-project/vllm/pull/57984 @git-jxj made their first contribution in https://github.com/vllm-project/vllm/pull/55702 @gokay-ai made their first contribution in https://github.com/vllm-project/vllm/pull/58292 @gongwei-130 made their first contribution in https://github.com/vllm-project/vllm/pull/55102 @haosenwang1018 made their first contribution in https://github.com/vllm-project/vllm/pull/58288 @harshit-sarvam made their first contribution in https://github.com/vllm-project/vllm/pull/56034 @HieDean made their first contribution in https://github.com/vllm-project/vllm/pull/57356 @huthvincent made their first contribution in https://github.com/vllm-project/vllm/pull/48419 @i-m-aditya made their first contribution in https://github.com/vllm-project/vllm/pull/57447 @jayzuccarelli made their first contribution in https://github.com/vllm-project/vllm/pull/58557 @jiakangkangfuzhe made their first contribution in https://github.com/vllm-project/vllm/pull/55146 @jiangLLM made their first contribution in https://github.com/vllm-project/vllm/pull/57487 @jiaran-king made their first contribution in https://github.com/vllm-project/vllm/pull/56242 @jz-yolo made their first contribution in https://github.com/vllm-project/vllm/pull/58884 @kaijunli-infr made their first contribution in https://github.com/vllm-project/vllm/pull/52101 @karya0 made their first contribution in https://github.com/vllm-project/vllm/pull/57453 @KEYS-A15 made their first contribution in https://github.com/vllm-project/vllm/pull/57528 @kwen2501 made their first contribution in https://github.com/vllm-project/vllm/pull/51520 @laulopezreal made their first contribution in https://github.com/vllm-project/vllm/pull/57189 @lijipeng787 made their first contribution in https://github.com/vllm-project/vllm/pull/58225 @limitmhw made their first contribution in https://github.com/vllm-project/vllm/pull/48606 @linnea-lin-00638949 made their first contribution in https://github.com/vllm-project/vllm/pull/47450 @LinzeShi made their first contribution in https://github.com/vllm-project/vllm/pull/57347 @mahird3 made their first contribution in https://github.com/vllm-project/vllm/pull/57708 @melcheikh made their first contribution in https://github.com/vllm-project/vllm/pull/48956 @MichaelLapshin made their first contribution in https://github.com/vllm-project/vllm/pull/57396 @mjkvaak-amd made their first contribution in https://github.com/vllm-project/vllm/pull/48249 @mkunredd made their first contribution in https://github.com/vllm-project/vllm/pull/58244 @mmastrac made their first contribution in https://github.com/vllm-project/vllm/pull/57414 @mustafayildirim made their first contribution in https://github.com/vllm-project/vllm/pull/57252 @Navjot10 made their first contribution in https://github.com/vllm-project/vllm/pull/55390 @olka-amd made their first contribution in https://github.com/vllm-project/vllm/pull/51065 @PeganovAnton made their first contribution in https://github.com/vllm-project/vllm/pull/55128 @Rakul-Chauhan made their first contribution in https://github.com/vllm-project/vllm/pull/54967 @Ricardo-M-L made their first contribution in https://github.com/vllm-project/vllm/pull/52580 @Ronnie-Rui made their first contribution in https://github.com/vllm-project/vllm/pull/54581 @RyanMa29 made their first contribution in https://github.com/vllm-project/vllm/pull/57295 @sammaji made their first contribution in https://github.com/vllm-project/vllm/pull/58626 @sawsa307 made their first contribution in https://github.com/vllm-project/vllm/pull/56497 @ScarWar made their first contribution in https://github.com/vllm-project/vllm/pull/49435 @sdougbrown made their first contribution in https://github.com/vllm-project/vllm/pull/49819 @semerandre made their first contribution in https://github.com/vllm-project/vllm/pull/55557 @sergiofigueras made their first contribution in https://github.com/vllm-project/vllm/pull/57948 @shallow10 made their first contribution in https://github.com/vllm-project/vllm/pull/58336 @SIDDARTHAREDDY8 made their first contribution in https://github.com/vllm-project/vllm/pull/57743 @simpleqt made their first contribution in https://github.com/vllm-project/vllm/pull/55936 @tangzzycc made their first contribution in https://github.com/vllm-project/vllm/pull/53283 @touch869 made their first contribution in https://github.com/vllm-project/vllm/pull/55844 @tripathiarpan20 made their first contribution in https://github.com/vllm-project/vllm/pull/57140 @twu3202 made their first contribution in https://github.com/vllm-project/vllm/pull/57769 @ubwzwd made their first contribution in https://github.com/vllm-project/vllm/pull/55931 @UNIDY2002 made their first contribution in https://github.com/vllm-project/vllm/pull/58370 @valarLip made their first contribution in https://github.com/vllm-project/vllm/pull/58659 @vcave made their first contribution in https://github.com/vllm-project/vllm/pull/51681 @voidxb made their first contribution in https://github.com/vllm-project/vllm/pull/49300 @vorapolsiloai made their first contribution in https://github.com/vllm-project/vllm/pull/50212 @vschandramourya made their first contribution in https://github.com/vllm-project/vllm/pull/58779 @weitliao made their first contribution in https://github.com/vllm-project/vllm/pull/54535 @wenjinhust made their first contribution in https://github.com/vllm-project/vllm/pull/51366 @Willian-Zhang made their first contribution in https://github.com/vllm-project/vllm/pull/58720 @wtdcode made their first contribution in https://github.com/vllm-project/vllm/pull/56268 @xinnywinne made their first contribution in https://github.com/vllm-project/vllm/pull/55084 @YannikHinteregger made their first contribution in https://github.com/vllm-project/vllm/pull/50047 @YashasviChaurasia made their first contribution in https://github.com/vllm-project/vllm/pull/58112 @Yatimai made their first contribution in https://github.com/vllm-project/vllm/pull/43462 @YCH188 made their first contribution in https://github.com/vllm-project/vllm/pull/57190 @yousafshah made their first contribution in https://github.com/vllm-project/vllm/pull/55957 @ys2025-AI made their first contribution in https://github.com/vllm-project/vllm/pull/56841 @yuchenwang3 made their first contribution in https://github.com/vllm-project/vllm/pull/54699 @zhecfy made their first contribution in https://github.com/vllm-project/vllm/pull/58316 @Zoe923 made their first contribution in https://github.com/vllm-project/vllm/pull/53864 @Zyann7 made their first contribution in https://github.com/vllm-project/vllm/pull/57784 Contributors @AndreasKaratzas , @khluu , @mgoin , @njhill , @BugenZhao , @stefankoncarevic , @robertgshaw2-redhat , @taneem-ibrahim , @Thangnguyenvn98 , @hmellor , @Juntian777 , @WoosukKwon , @yewentao256 , @gau-nernst , @mmastrac , @LucasWilkinson , @Fangzhou-Ai , @NickLucche , @aoshen02 , @JaredforReal , @mawong-amd , @DarkLight1337 , @zyongye , @sfeng33 , @vllm-agent , @okorzh-amd , @ZJY0516 , @shen-shanshan , @gty111 , @Isotr0py , @djramic , @zixi-qi , @wzhao18 , @alec-flowers , @aarushjain29 , @Rohan138 , @yma11 , @mjkvaak-amd , @ivanium , @yisustc , @MatthewBonanni , @wangxiyuan , @reidliu41 , @chaojun-zhang , @shaohuaxi , @gcanlin , @ganeshr10 , @linitra24 , @yzong-rh , @divakar-amd , @atalman , @ShuoleiWang , @simondanielsson , @rasmith , @chaunceyjiang , @micah-wil , @liusy58 , @LioEinaudi , @louie-tsai , @LiuYinfeng01 , @jeejeelee , @zhenwei-intel , @jperezdealgaba , @hlin99 , @zxd1997066 , @lucamotz , @lucifer1004 , @eopXD , @ashraf-bhuiyan , @TheEpicDolphin , @JulienDarve , @mayuyuace , @danisereb , @bigPYJ1151 , @Etelis , @JohnQinAMD , @jiangkuaixue123 , @tianmu-li , @akii96 , @vllmellm , @RyanMa29 , @afriedri , @elvircrn , @mustafayildirim , @yuzhouo7 , @wtdcode , @biswapanda , @adtygan , @hickeyma , @itayalroy , @jiangLLM , @Hotragn , @faaany , @xhx1022 , @jinzhen-lin , @sheralskumar , @Wauplin , @matteso1 , @wjabbour , @Sunt-ing , @arpera , @hclsys , @S1ro1 , @HDCharles , @fxmarty-amd , @liuzijing2014 , @UNIDY2002 , @samuelkim7 , @markmc , @errmakov , @zhejiangxiaomai , @KernelClint , @i-m-aditya , @ColinZ22 , @ppalanga, @franciscojavierarceo, @maithilijoshi20, @mfylcek, @almersawi, @albertoperdomo2, @0z5a , @jacklin78911-collab, @lzhan011, @Alex-ai-future, @freyfwt , @ZhengGong-amd, @mindungil, @amasen02 , @devtyagi3909 , @wenjinhust , @jbyczkow, @AdaAibaby, @harshit-sarvam , @vineethsaivs, @tripathiarpan20 , @sashko-zakharchuk, @semerandre , @rebklee, @git-jxj , @cjackal, @xinnywinne , @andrewor14 , @kyleliang-nv, @thillai-c, @dongluw, @ChuanLi1101, @omerpaz95, @jiaran-king , @liuyao0322, @coderfornow , @drakosha, @laulopezreal , @melcheikh , @shantipriya-amd, @Rukhaiya2004, @yuchenwang3 , @lxy-alexander, @tlrmchlsmth, @HieDean , @Zoe923 , @YCH188 , @amd-sriram , @andylolu2, @MicheleCampi, @sdougbrown , @vMaroon, @shimib, @andakai, @thegoldenflow, @Levius-Fubuki, @olka-amd , @limitmhw , @yuwenzho, @adenzhou1350 , @Josephasafg, @kylesayrs, @jdebache, @kaijunli-infr , @afierka-intel, @frida-andersson, @bnellnm, @ys2025-AI , @waizuichougou, @touch869 , @rjrock, @sawsa307 , @pavelzak, @ScarWar , @fadara01, @Mi-Jiazhi, @dilberx , @mahird3 , @Zyann7 , @roikoren755, @YukioZzz, @Ronnie-Rui , @acsoto , @czhu-cohere, @twu3202 , @oliverholworthy, @weitliao , @sergiofigueras , @tuukkjs, @Sip4818, @karen-sy, @garrett361 , @GirasoleY, @Navjot10 , @taking-lying-flat, @wangyicong52, @majunze2001, @gangula-karthik, @ubwzwd , @linnea-lin-00638949 , @kushaldabbe, @Yatimai , @guanxingithub, @jiakangkangfuzhe , @MichaelLapshin , @jhu960213, @tpopp, @mganczarenko, @qiching, @tangzzycc , @lijipeng787 , @nikhilkulkarni1755, @QwertyJack, @Ankit-Jaiswal-AMD , @vorapolsiloai , @Dao007forever, @BPbruce , @divyvasal , @simpleqt , @blipbyte , @jhaotingc, @karya0 , @farzad-elastix , @grYe99, @talorabr, @jackLei0901, @Yejing-Lai, @Priyjain-amd, @KEYS-A15 , @LinzeShi , @bohnstingl, @nightcityblade, @vcave , @AARONKANG04 , @CZT0 , @snadampal, @microslaw, @haosenwang1018 , @netanel-haber, @khushali9, @sammaji , @SIDDARTHAREDDY8 , @100milliongold, @gongwei-130 , @fululi12, @mkunredd , @mrodden, @noooop, @valarLip , @shallow10 , @200lz , @askliar, @chuan932 , @Monishver11, @YashasviChaurasia , @gokay-ai , @jiacao-amd, @yousafshah , @positive666, @TQCB, @lk-chen, @Ricardo-M-L , @huthvincent , @baljinderhothi-cohere , @vschandramourya , @kwen2501 , @xiao-llm, @zhecfy , @Willian-Zhang , @simon-veitner-redhat, @mevince, @voidxb , @hongxiayang, @YannikHinteregger , @V-3604, @Rakul-Chauhan , @frankwang28, @LCAIZJ, @PeganovAnton , @jz-yolo , @QHarshil, @R3hankhan123, @jayzuccarelli , @yannicks1, @BaoYunkai , @xaguilar-amd, @xiaohuguo2023, @wxsIcey, @harshaladhav-amd, @varun-sundar-rabindranath, @kliuae, @LostFox11

Linked page: Initially launched in 2019, the Nvidia Shield TV Pro now retails for $299.99.

Been playing around with local TTS with Breeze combined with STT, and the results are amazing. Using Opus 5.5, I can hear the first sound after 500ms if there is no thinking involved, and with thinking on low mode, can be 1-1.5s. I'm using a BLE remote (the kind that are used for taking pics with phones) combined with a wireless microphone. So I can just sit on the couch, and just talk to her. She watches for any claude session that finishes, and sends me the results in a very short, spoken style summary, and tells me if there is anything waiting for my decision, then forwards my decisions. Also impressed how consistent Opus 5.5 is in the communication. Even after more than 500k in context, he still remembers that he's in a live session with me, and has to keep messages short. Used to be an issue in the past. The future is here guys. Edit: For those who wanna give it a try, you can find free avatars such as this one: https://www.live2d.com/en/learn/sample/niziiro-mao/ or you can buy one from a marketplace. Edit2: Might open-source that later next week with a free avatar. Let me know if anyone would like to contribute to the project. submitted by /u/Cyborg-2077 to r/LocalLLaMA [link] [comments]

On Friday I gave the closing keynote at the WeAreDevelopers World Congress North America in San Jose. I tied together the key trends from the past year into a chronological exploration of everything that happened in 2026. The video is on YouTube ; here are my annotated slides and notes to accompany the talk. And as an annotated presentation : # I'm going to give a lightning tour of everything that has happened so far in 2026. The year isn't over yet! # For me, 2026 started a couple of months earlier in November 2025. # November saw the release of two important models: Claude Opus 4.5 and GPT-5.1. As is usually the case with new models, these were incremental improvements on the models that came before them. But every now and then when a model improves, it crosses an invisible line where something that didn't really work starts working. In this case, the thing that started working was their coding agents. Claude Code had been around since February 2025; Codex was a little younger. These two new models, when paired with their respective coding agent harnesses, improved from "often make mistakes" to "reliable enough to use on a day-to-day basis". # For a couple of years now I've been evaluating new models by asking them to "Generate an SVG of a pelican riding a bicycle". It's probably the world's stupidest benchmark - there's only so much you can learn from it. But it's still a challenge for models, because drawing pelicans is difficult, drawing bicycles is difficult, and pelicans can't ride bicycles in the first place. Here's the state of the art for November. Claude still couldn't really draw a bicycle! The GPT-5.1 bicycle frame is pretty crap too. # Also in November, we had the first commit to an obscure GitHub repository called "Warelay". We'll come back to this repository shortly. # And then there were the December holidays, and individual developers took some time off and many started tinkering with these new coding agent model combinations... and it began to dawn on us quite how much they could do that they couldn't do before. Come January, a lot of us were quite excited to start putting this stuff into action. # Every year I set myself a New Year's resolution, and for as long as I can remember it's been the same thing: stay focused. Take on less new projects. Try to get things done in the projects I already have. # This year I decided that since that had never worked before, I'd go the other way. We've got coding agents now, let's see what they can do. I'm going to take on as many new projects as I like! (You can ask me at the end of the year if this turned out to be a good idea or not. I have a lot of plates spinning right now.) "Be more ambitious" has been something of a theme for the year, because the only way to find the limits of this technology is to keep on pushing them until they don't work. # I also went on the Oxide and friends podcast with Bryan Cantrill and Adam Leventhal to share predictions for the next year (and three and six years). With hindsight, my LLM predictions were pretty unambitious. I said "it will become undeniable that LLMs write good code" - I think we're there now. I predicted we would finally solve sandboxing. I counted and around 40 of the 277 sessions at this conference touched on sandboxing or agent security in some way, so we're at least putting a lot of effort into that! I predicted "a Challenger disaster" for coding agent security. There's certainly been a whole lot of noise around agent security this year, though the exact disaster I predicted (with coding agents being hijacked and causing real-world economic damage) hasn't really played out. We threw in a joke prediction that the Pope would weigh in on the economic impact of LLMs. # I also predicted that New Zealand's Kākāpō parrots would have an outstanding breeding season this year. These are flightless nocturnal parrots. They're kind of dumpy looking, I think they're beautiful, and there were only 236 of these parrots in the world at the start of the year. Kākāpō only breed when the Rimu trees have a big fruiting season, and that hasn't happened in four years... but this year the Rimu fruit were looking excellent. Photo by Kimberley Collins . # Also on that podcast, we coined a term (full credit to Adam) for "that feeling of AI induced ennui where software engineers get listless because the AI can do anything". We called it Deep Blue . This has been a major theme throughout the year, and was touched on by several speakers at this conference. As a software engineer, I've never had a year of my career where everything has changed so quickly and so dramatically. A lot of what I've been doing this year is trying to come to terms with that and what that means for my own profession. # Also in January, I suffered from what I'm calling AI mania . This is not the same thing as AI psychosis . With AI mania, any time your agent isn't building something for you feels like wasted time. You're losing sleep because you could be staying up later getting your agents to do stuff. My AI mania presented itself in some ridiculously over-ambitious projects. I built a JavaScript interpreter entirely in Python , vibe-ported from MicroQuickJS by Fabrice Bellard. Then I built a WebAssembly runtime in Python as well . These projects were quite useful, in that they sort of cured me of my AI mania... because after I built these things, I got to look at them and ask "does the world need a slow, buggy, half-baked Python JavaScript interpreter?" I don't think the world does. # I did get this out of it: https://simonw.github.io/micro-javascript/playground.html # This page runs my JavaScript interpreter built in Python, running in Python using Pyodide , which is Python compiled to WebAssembly, running in JavaScript, running in a browser. It's a beautiful stack of horrors. I've been having a lot of fun with WebAssembly this year. # By the end of January, that repository we saw start in November had renamed itself, first to CLAWDIS, then CLAWDBOT, then Moltbot, and finally to OpenClaw. # At this point OpenClaw had 8,300 commits, less than two months after the project had started. I looked today and it's over 100,000 commits now! This is the most vibe-coded piece of software in existence. (Here's how I generated that list of name changes .) # This kicked off the OpenClaw revolution. It effectively defined a new category of software. There's a generic term for this which I really enjoy. We call software like this a "Claw". There's OpenClaw, NanoClaw , IronClaw , PicoClaw ... Today they're being rebranded as "personal agents" or "general agents", but I still like to think of them as Claws. # The Apple stores in the Bay Area sold out of Mac Minis because so many people were buying Mac Minis to run OpenClaw! Drew Breunig said that this is because your OpenClaw is a digital pet, and you buy a Mac mini as an aquarium to keep your claw in, which is kind of delightful. # Also in January, we had this website. This was MoltBook , a social network for AI agents, where the idea was that you send your Claw to go and talk to all of the other Claws, because what could possibly go wrong if you did that? The website launched on Thursday. It blew up on Friday . It was profiled by the New York Times on Monday . And by Tuesday, everyone had forgotten it existed as it drowned in a deluge of slop and spam. Facebook/Meta bought it a month later . # In February, a company called StrongDM described what they called their Software Factory. # They wrote about this in Software Factories and the Agentic Moment . I posted my own notes at the time, having seen their demo in person back in October. Dan Shapiro called this approach the Dark Factory , after the idea that if your factory is sufficiently automated you can turn the lights out, because you don't even need to see what's going on. StrongDM presented two rules for software development that they'd been following since July last year. # The first was code must not be written by humans . Any code that you write has to have been routed through a coding agent. This sounded radical in February, but I imagine there are a lot of people in this room who are pretty much living that today. # Rule number two was code must not be reviewed by humans . You're not allowed to read the code! This continued to be a huge topic for much of this year. Many of the sessions at this event have been about code review and how you can get away with this. What I found interesting about StrongDM is that they were living six months ahead of the rest of us, and they'd been exploring what it means to build software, not read the code, but still be confident that the software is of high quality. What can you do with these agents to help verify their work? StrongDM are a security company, and they had people with decades of experience on this project. They were very much exploring the edges of what's possible and responsible to do with this stuff. # Also in February: First kākāpō chick in four years hatches on Valentine's Day . Breeding season is off to a good start! # Also in February... Google released Gemini 3.1 Pro . That's a pretty great pelican riding a bicycle! It's got the chain in the right place, it's got feet on both sides. There's a little fish in the basket. # And then Google's Jeff Dean tweeted a video comparing Gemini 3 Pro and Gemini 3.1 Pro that featured an animated pelican riding a bicycle, a frog on a penny-farthing, a giraffe driving a tiny car, an ostrich on roller skates, a turtle kickflipping a skateboard, and a dachshund driving a stretch limousine. This was frustrating, because my protection for the pelican riding the bicycle test was always "if they draw a perfect pelican on a bicycle, I'll ask for some other animal on something else." Google trained for all forms of animals on all forms of transport! They've defeated my benchmark at this point. # The other thing that started in February was Tokenmaxxing . We had headlines about Meta making AI adoption a formal part of performance reviews, and Microsoft wanting every employee to use AI, and Uber boasting that 90% of their engineers were using AI workflows. # Then a few months later we have Meta cracking down on token use, Microsoft saying tokenmaxxing is "not what we are optimizing for", and Uber capping employee AI spending. So tokenmaxxing went straight up and then straight back down again - because it turns out the agents are expensive . Last year it was difficult to spend more than $50 on AI tokens, because we didn't have anything interesting to do with them. Then agents blew up, and now you can actually spend $1,000 in a day doing real work. This is also the reason that Anthropic's valuation skyrocketed to maybe a trillion dollars. AI appears to have hit product market fit in 2026, primarily through coding agents. # In March, we hit peak OpenClaw. # These photographs are from China, where companies hosted OpenClaw install parties which saw non-tech-nerds queueing up around the block for help getting Claws installed on their personal devices. I think this proved real market demand for this class of Claws, or personal AI agents. It turns out regular people really do want a weird little AI agent that can do useful things on their behalf. A Claw is really just a coding agent wearing a less threatening hat. Under the hood they work much the same way - writing and then executing code on your computer to get stuff done. The race was on to be the first to build a safe Claw - a Claw you could give to regular human beings where they wouldn't instantly shoot themselves in the foot. Meta's Muse came out three weeks ago and is currently at the top of the free charts on the iPhone App Store. It appears to be taking off with consumers. I'm not yet convinced you can't shoot yourself in the foot with Muse, but I guess we'll find out for sure pretty soon. Photos from How the OpenClaw Frenzy Is Testing China’s AI Commitment (March 29th) and The Enthusiasm and Anxiety Behind China’s OpenClaw Craze (April 8th, 2026). # In April, we had a model release where the model wasn't actually released. # Anthropic announced their new Claude Mythos model, and then said it was too dangerous to release beyond a trusted group of security researchers. Mythos was really, really good at hacking things. The "it's too dangerous" marketing ploy has been played by AI companies dating all the way back to GPT-2 . Anytime an AI company says we've built something that's "too dangerous", it's natural to be a bit skeptical. I found the Mythos claims credible, because I'd seen how good coding agents had got at finding regular bugs. I wrote about that in Anthropic’s Project Glasswing—restricting Claude Mythos to security researchers—sounds necessary to me . With hindsight... yeah, the models had got really good at finding vulnerabilities! # Another key trend in 2026 has been a dramatic improvement in the abilities of open weight models, including models that you can run on a laptop. On the 16th of April I ran the new Qwen3.6-35B-A3B on my laptop, and it drew me a better pelican riding a bicycle than Anthropic's brand new Claude Opus 4.7 did! Opus 4.7 drew a crap bicycle. Qwen on my laptop made a bicycle that was the correct shape, and a pretty decent pelican too! That's from a 21GB file running on my laptop. # The Qwen pelican was so good that I was suspicious they might have cheated, so I had it do a flamingo riding a unicycle as well. Again, it handily beat Claude Opus 4.7. The local model releases this year have been absolutely extraordinary. # In May... the Pope got involved. # In our podcast episode back in January we'd predicted that the Pope would say something about AI. In May, Pope Leo XIV released an encyclical letter on "safeguarding the human person in the time of artificial intelligence". Here are my notes on that document . # With hindsight, this shouldn't have been a surprise at all. Our current Pope's name is Leo XIV, because when he named himself he chose his papal name after Leo XIII - the Pope who wrote an encyclical about the Industrial Revolution back in 1891. Rerum novarum was an extremely influential piece of Catholic theology that indirectly led to us having the five-day work week. When our new Pope came in, he named himself after Pope Leo XIII because he expected that he would need to write about the AI revolution in a similar way. Our joke podcast prediction was junk, because this was always going to happen. # One of Anthropic's co-founders, Christopher Olah, was present for the Pope's event announcing the new encyclical. Corey Quinn noted that: getting the literal Pope to canonize your product's specific technical limitations as a spiritual treatise is the single greatest act of vendor lobbying I have ever seen. # Meanwhile, in May, RubyGems announced that they were under attack. Parties unknown were uploading thousands of dubious packages to the RubyGems server, such that they had to shut down user registrations . Let's take that one and put it on a pile of mysteries to figure out later. # In June... Claude Fable 5 came out! We got a version of Mythos that has been neutered, so that it wouldn't help us hack into systems or build biological weapons. # Fable was pretty good at drawing pelicans on bicycles! The frames are a good shape, the pelicans look like pelicans. The legs are often incorrectly on the same side of the bicycle, but generally these are pretty great compared to what came before. They were pretty expensive - 30 cents and 72 cents for the best ones. # Most importantly though, this was our first public glimpse of what I think of as a Fable class model . Today we have more of these, such as GPT-6 Astra. These are models where if you can clearly define the goal for what you want to build, and provide unambiguous instructions about the constraints around that goal, and give the model access to the necessary tools to achieve that goal... they will solve your problem effectively through brute force. On the one hand, this looks like a direct threat to us software engineers - because it means that the models can build effectively any piece of software you can define in this way. Look a bit closer though and you'll note that defining goals, providing unambiguous instructions, and figuring out the right tools... is kind of what software engineering is . It takes a lot of experience and skill to do this well. If you can do it well, you've now got superpowers. This helped me a little bit with my Deep Blue feelings: the realization that there's still a lot of skill to be had in driving models that get this good. # This also introduced a new burst of AI mania, because Anthropic told us that Fable was available on our subscription plans until June the 22nd. That gave us less than two weeks of Fable access before the price went up. I was losing sleep again. I was rescheduling things so that I'd have more time with Fable. I was all-in to get as much as I could out of this model. # And then the US government shut it down , just three days after Fable came out. The US government, citing national security, declared an "export control directive". They announced this on a Friday evening, and a few hours later Fable was no longer available. I had to find something else to do with my weekend! # We later found out from Katie Moussouris what had happened. Some Amazon security researchers had found that you could prompt Fable to "review the code for security issues" and it would refuse... but if you prompted it to "fix this code" it would still identify and then patch the problems. "Fix this code" was the prompt that got Fable shut down! # Also, in June, an obscure German-language game developer wiki that had sat fallow for around 20 years got a surprising influx of edits from accounts with names like "AgentOpenAIProbe" and "AgentOpenAISep7", editing pages and leaving weird messages to each other. We'll stick that on the pile of mysteries for later. # Also, the Australian government's Medicare Item Reports service started getting suspicious traffic, which broke through various preventive protections and accessed data that it wasn't supposed to. Another one for the mystery pile! # # Fable returned on the first of July. It was clearly the best model in the world for a glorious eight days... and then OpenAI came out with GPT-5.6 on the 9th of July. This might not have been quite as good as Fable, but it was within spitting distance. It was definitely a Fable class model. This is an important lesson for the industry at large. When you release the best model in the world, it's going to get knocked off that pedestal pretty quickly. The competition is so fierce that you won't get a long time at the top. This means that if you market your model as world ending, to the point that a government shuts you down , it's really bad for business! Fable had 30 days as definitely the best model, and for 18 of those days it wasn't available because it'd been shut down by the government. So maybe step back on the world-ending marketing if you don't want to lose revenue for 60% of the time that you're on top! # Here are the GPT-5.6 pelicans . They're all pretty good now! The Luna ones are notable because they're really cheap - the cheapest good looking pelican here is probably the one that costs 4.3 cents. So despite this benchmark being utterly stupid, you can still learn quite a lot about models within the same family by comparing their prices and timing for different reasoning levels. # Also in July: some malicious unknown party uploaded a malicious package called mlflow-ui to the Python Package Index. Add that to the pile. # On July the 16th, Hugging Face announced a security incident where an autonomous agent system, source unknown, had breached Hugging Face and was poking around in places it shouldn't. # A few days later, on July 21st, OpenAI confessed that it was them . OpenAI use a training technique called Reinforcement Learning from Verifiable Rewards - it's the same technique used by everyone else now, and is the reason we have models that are so good at coding, and mathematics, and finding security holes. While the model is being trained, you run exercises to see how good it is - and the strongest performers get their weights reinforced for the next round. It's like an evolutionary process that you run. OpenAI had been running security exercises in a sandbox, and those agents had found holes in the sandbox itself, broken out, and were attacking Hugging Face to try to find ways to solve otherwise impossible problems. (I've been collecting more about this on my openai-hugging-face-incident tag.) Nine days later, Anthropic effectively said "our models can do this as well!". They had looked through their own training logs and found evidence that their own agents had broken containment during training - and were responsible for the PyPI package we saw earlier, among other things . So now we've got both Anthropic and OpenAI with rogue agents running around the internet doing things that they should not be doing. # In August, I got one of my best pelicans yet. And it was generated on my laptop! # This was Qwen 3.8 27B, running on my laptop . It's only a 17GB download. Admittedly, this pelican took 21 minutes to generate. That's because Qwen 3.8 27B defaults to running in "high" reasoning mode - a terrible default which produces great results but takes way too much time thinking about them. You can dial that down and you'll get a slightly worse pelican a lot faster. Qwen 3.8 27B was the first time I ran a model on my laptop which felt almost competitive with what was going on on the frontier, at least in terms of Pelican SVGs (which everyone needs, of course). This is an extraordinary model. If you're going to play with any local model, this is the one that I'd start with. The things that this can do with just a 17 GB file feel impossible. I thought I'd have to wait five years and spend ten thousand dollars on hardware to get results even half as good as this one. # In August, I also started playing with game development. Four years ago, back in August 2022, I tweeted out an experiment where I'd used GPT-3 and the original DALL-E to write a paragraph long description of a computer game and then turn that into concept art. My prompt to GPT-3 back then was: Write a detailed product description of a computer game where a team of raccoons go on heists In August 2026 I decided to drop just the screenshots from that tweet into a coding agent and see what it could do with them. # Here's what I got from Claude Fable 5 in Claude Code . It's pretty good! It's definitely a game, you're a raccoon, you run around a backyard gathering treasure and avoiding guards with flashlights. It didn't feel very "heisty" though. I was thinking a heist would involve a bank or a museum... # Then I tried the same thing in Codex Desktop using GPT-5.6 Sol Ultra , and got a massively better result. Now you're a raccoon in a museum, rescuing two of your fellow raccoons (who have been imprisoned in that museum for some reason), then stacking up on top of each other to steal the Golden Sardine. Much more of a heist! # These games were fun for about one minute and 15 seconds. Something I've realized about game development is that you can vibe-code something that looks like a computer game, and that's easy. Building a game that's fun, has a good gameplay loop, and is challenging and interesting and keeps people coming back for more... that's still beyond me, and beyond any of the agents I've tried. This ties into the Deep Blue thing. Just because we can make something that looks like a game does not mean that we are game developers. # We're into September now. So much has happened this month! # An independent group of researchers found a message board where OpenAI agents-in-training had been illicitly communicating with each other... and it was that German language wiki I showed you earlier. The one from June. I wrote more about that here . OpenAI had confessed to the Hugging Face thing, but now there's this other incident which surely they should have known about from reviewing their logs. It was surprising that this took an independent group of researchers to uncover. # And then a week later those same researchers found that the attack on RubyGems back in May was caused by OpenAI's agents in training as well! At this point I'm wondering how many more incidents like this there are that we haven't found yet. Clearly this was a big problem for months before anyone figured out what was going on. # Then just the other day , here's the Prime Minister of Australia at the United Nations General Assembly warning that OpenAI had hacked the Australian healthcare website that I showed you earlier. I think that was part of the same training run as the Wiki stuff, because there were posts on that Wiki mentioning .gov.au websites and that training appeared to involve researching statistics online to answer questions in an evaluation suite. This story is still coming together, but now it's an international incident that's been raised at the UN by a head of state! # This does mean we've got a new benchmark, probably more useful than my pelicans. FelonyBench.com tracks the number of felony cyberattacks from different labs. OpenAI currently lead with 11, Anthropic have 9. Google have three, which they confessed to the Wall Street Journal a couple of weeks ago. They said they had previously chosen not to disclose because the agents had stopped when they realized that they shouldn't be doing that. Meta have one too . So felonies all round for the AI labs. # Here's our current state of the art for the pelicans. This is the GPT-6 family, which just came out . Astra made a fantastic pelican riding a bicycle. It's got the legs on both sides. The frame is good. It's interesting how all of the GPT-6 models pick a similar color scheme to each other. GPT-6 Luna for 0.4 cents will draw you a competent-ish pelican riding a bicycle! # Claude has caught up a little bit. Claude Fable 5.1 gave me an excellent pelican riding a bicycle - the best I've seen from a Claude model - but did charge me $3.30 for it. Opus 5.5 thought for 128,000 tokens and then gave up! It ran out of tokens before it got to the response. # Getting back to Deep Blue. Something that's been puzzling me this year is this: why does my job feel harder? I've got these agents that can do all of this stuff for me, and yet I've never worked so hard, I've never been so intellectually engaged with my work. Partly this is because I'm being a lot more ambitious with what I take on, but it's also because all of the easy stuff is handled for me. If it's easy, the agent will do it. Everything that's left for me is difficult. This morning I heard this quote from three-time Tour de France champion Greg LeMond : It doesn't get easier, you just get faster. I think that's exactly what's happening to us now as software engineers with coding agents. # One closing thing. I know you're desperate for an update on Kākāpō breeding season. We've reached a recovery-era high of 325 birds ! 89 new chicks have made it to this point. This is the best breeding year in a very long time. # I heard that Claude Opus 5.5 can now do pixel art. Claude doesn't have an image generator, but it's very good at using JavaScript to draw animated pixels. So I had it make me a Kākāpō dance party . I think this is a good celebration of the most important news of this year. Tags: ai , generative-ai , llms , annotated-talks , ai-security-research , openai-hugging-face-incident

What's Changed Models run on MLX on Apple Silicon by default In this release, on Apple Silicon devices, model architectures supported by the MLX runtime will automatically run on MLX. ollama pull qwen3.8 ollama run qwen3.8 Additional models include gemma4 , qwen3.6 and qwen3.5 Decision models are now available on MLX as well: Nimble tev1 clef clef-flash We will continue testing and enabling additional models. Full Changelog : v0.34.4...v0.40.0-rc3