2026-10-06

hexagon: matmul and flash-atten scalability updates ( #29974 ) hexagon: head-parallel flash_attn partitioning for row-split multicore In row-split mode each core computes its output row shard of every MUL_MAT, but flash_attn was previously partitioning by Q tokens (flat qrow split) instead of by heads. This forced every core to read the full KV cache (all n_kv_heads), negating the memory bandwidth benefit of multicore on flash_attn. Change both HMX and HVX flash_attn kernels to partition by KV heads when n_kv_heads is divisible by n_cores: core i processes heads [i*n_kv_heads/N, (i+1)*n_kv_heads/N) exclusively, reading only its head shard of the KV cache. Falls back to the original token-block split when n_kv_heads % n_cores != 0 (e.g. Gemma-4 with 2 KV heads on 4 cores). Controlled by GGML_HEXAGON_FA_HEAD_SPLIT (default 1 = on). The flag is packed into bit 1 of the existing is_dst_fp32 kparams byte to stay within the 128-byte kernel_params blob limit. Measured gains at 4c row-split (PP t/s, ubatch=1024): Qwen3-0.6B: 6977 -> 11026 (+58%) llama-3.2-3B: 3717 -> 5522 (+49%) Qwen3.5-4B: 2739 -> 2855 (+4%) Gemma-4 MoE: no change (MoE FFN dominates, fallback path) TG is unchanged (flash_attn is a small fraction of decode time relative to the matmul+barrier cost per layer). hex-fa: cleanup kern_params and head-split selection hex-fa: add -fa-head-split option to run.py hex-mdev: update matmul solver to account for reduced work in row-split scenarios hex-mmid: better work splitting by expers in multi-dev scenarios hex-fa: update HMX gating based on the model/n-hvx/ctx-len sweep hex-fa: precompute softcap/scale on the host hexagon: flatten matmul into 2d to use HMX in multi-sequence hex-mm: cleanup kparams and use collapse to 3/4D -> 2D mapping hex-mm: fix typo in collapse fallback hex-mm: another pass at consistent naming for act tensors hex-mm: add support for colapsing dims in fused matmuls hex-build: fix WoS build errors hex-mm: make sure to enforce dst stride in can_collapse hex-fa: add a onliner commit for head-split check hex-fa: remove unused local head_split var hex-fa: tighten up can_split checks hex-mm: update unfused paths to use act instead src1 hex-mm: make sure to check all dsts for splitting hexagon: fix the second weight chunk address in the batched HMX matmul prologue hexagon: F16 activation and ragged N in the HMX matmul hex-mm: tighten the ragged/split checks in mdev cases hex-mm: enable MM fusion for F16 activations hex-mm: pass tiled sizes to the solver in fused paths hex-mmid: remove scalar divs from expert mapping loops hex-mmid: proper cacheline safety enforcement for mdev splits hex-mm: improve solver for mdev split scanarios and tail handling hex-mm: remove redundant checks hex-mm: fix fused HMX MUL_MAT_NX drops the final partial tile for quantized weights hex-mm: better handling of ragged shapes (removes scalar memset of vtcm) Co-authored-by: ebateni ebateni@qti.qualcomm.com Co-authored-by: Jhen-Jie Hong iainst0409@gmail.com Co-authored-by: Yiwei Shao yiwei@aizip.ai Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/52957661 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries Ubuntu arm64 (CUDA 13) - CUDA 13.4 libraries Ubuntu x64 (ROCm 10.0) Ubuntu x64 (OpenVINO) Ubuntu x64 (SYCL FP32) Ubuntu x64 (SYCL FP16) Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide Android: Android arm64 (CPU) Android arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide Windows: Windows x64 (CPU) Windows arm64 (CPU) Windows arm64 (OpenCL Adreno) Windows x64 (CUDA 12) - CUDA 12.4 DLLs Windows x64 (CUDA 13) - CUDA 13.4 DLLs Windows arm64 (CUDA 13) - CUDA 13.4 DLLs Windows x64 (Vulkan) Windows arm64 (Vulkan) Windows x64 (OpenVINO) Windows x64 (SYCL) Windows x64 (ROCm 10.0) openEuler: DISABLED openEuler x86 (310p) openEuler x86 (910b, ACL Graph) openEuler aarch64 (310p) openEuler aarch64 (910b, ACL Graph) UI: UI

Running oQ4e+MTP on oMLX 0.7.0, with still more to optimize. Prefill is 1,878 toks. I saw some other benchmarks below what id expect so i figured I would share. submitted by /u/cryotic to r/LocalLLaMA [link] [comments]

Hey all. We've spent the last weeks getting Qwen3.8-Flash-Next (125B MoE, 6B active) to run properly on one AMD Strix Halo box (Ryzen AI Max+ 395, 128 GB). Tonight we're releasing both the 95 GB EXL3 weights and a new version of Kyojin, our inference engine (built on ExLlamaV3, open). This is a first version, same as our GLM-5.3-Flash and MiMo-V2.6-Flash builds. We'd rather ship it and improve it in the open: speed and quality updates are coming for all three. Numbers, all from a fresh clone and build on the mini PC: Decode: 44 to 59 tok/s with speculative decoding depending on the task (chat ~47, code ~58, copy-heavy edits ~60). 32.7 tok/s without it. Prefill: 1,412 tok/s at 4K, 1,486 at 32K, 1,367 at 128K (server-reported). It stays nearly flat. Long context: 10/10 needles at 64K and at 128K, still 32 tok/s at 128K. Fidelity: 94.1 % top-1 agreement with the original FP8 model over 844 positions. One thing we're a bit stubborn about: speculative decoding here returns exactly the tokens plain decoding would. We check that on every release. For comparison, a llama.cpp user posted about 30 tok/s with speculation and about 500 tok/s prefill on this same mini PC (Vulkan, UD-IQ4_XS). Those are their numbers, not something we measured: https://github.com/ggml-org/llama.cpp/discussions/28512 Now the part where we're not first. Halogen 0.16.2 (v2 checkpoint) is faster than us: 39.8 vs 32.7 tok/s plain, 52 vs 47 on chat with speculation, and 10 to 20 % ahead on prefill when both are timed the same way from the client (1,306 vs about 1,460 at 4K, 1,394 vs about 1,720 at 16K). On code we're close (58.5 vs 51.2 on the median pass, they're ahead once warm). Where we do better is fidelity to the original model: 94.1 % top-1 agreement against 92.3 % for them, and a KL divergence 41 % lower on our side. Full table is on the model card. Closing the speed gap is what we do next: we're reworking the core of the engine, which will help every model it runs, not just this one. The hardware has room left. There's also an optional uncensor preset, off by default (4 refusals out of 100 harmful prompts instead of 99, benchmarks within noise). If your agents lean hard on tool calls, leave it off. Weights: https://huggingface.co/yamz-labs/Qwen3.8-Flash-Next-EXL3-Yamz Engine: https://github.com/Yamz-Labs/kyojin If you run it, we'd love your tok/s and hardware. And tell us what you want to see next. submitted by /u/Yaniss916 to r/LocalLLaMA [link] [comments]

Hello everyone. This is an update post on the model Nanosaur2. Again this is not my model. A complaint a lot of people had with the model was that despite it being very small (660m params), it didn't take a 660m param level of time to generate. That's been solved now, with a 4 step turbo. On a single 5090, you can generate 7 images a second. Again, this model is small enough you could easily run it on a phone, edge devices, wherever. It's also a great research model, so if you want to finetune on top of a small easy to tune model, or way to create adapters for the model, or whatever, go ahead, it's all there. This was created using bytedance's new DMAD method, and it works great. Quality is incredibly close to the original model at a 12.5x speedup. links: 4step model comfy workflow As per usual, if you have any questions, please message metal63 on discord. Do not message me . submitted by /u/FusionCow to r/StableDiffusion [link] [comments]

The era of generative AI has upended technology supply chains, and there is one ironclad rule in 2026: If it has memory or storage, it’s getting more expensive: arstechnica.visitlink.me/YomR85 Linked page: Initially launched in 2019, the Nvidia Shield TV Pro now retails for $299.99.

Research: Qwen3.8 27B addition in words Colin Frasier posted on Bluesky about an experiment he ran over two years ago using GPT-4o to see how well it could "compute the sum but return the answer in words" across increasingly large numbers. Here's the chart he shared of those results: I'm confident GPT-4o didn't cheat and use a calculator, especially since it got so many of the calculations wrong, but I was inspired to run the experiment again on local hardware (a DGX Spark) to explore the effect in a fully controlled environment. I pasted his image into a Codex Remote session (GPT-6 Astra) and had it run the same experiment using Qwen3.8-27B-Q4_K_M.gguf . Here's the result for a run of 30 attempts per combination with reasoning disabled: Then I ran it again with reasoning enabled. This took a lot longer per pair, so instead of running 30 samples per square I ran just one - which results in a much less visually appealing heatmap since each square is either 100% or 0%: It got the right answer in 167 out of 169 attempts, and since these were one-shot I'm confident a second run would produce different results here. Here's a version of the report that includes the reasoning traces from some of those larger calculations, which include text like this: Wait, let me redo this more carefully. 4,299,366,105,622 6,088,794,067,970 Let me align them: 4 2 9 9 3 6 6 1 0 5 6 2 2 6 0 8 8 7 9 4 0 6 7 9 7 0 Adding from right to left: Position 1 (units): 2 + 0 = 2 Position 2 (tens): 2 + 7 = 9 Position 3 (hundreds): 6 + 9 = 15, write 5, carry 1 Tags: mathematics , ai , generative-ai , local-llms , llms , qwen , llm-reasoning , dgx-spark

Overview llama.cpp v0.6.0 introduces the new llama_batch_ext extended batch API (with llama_process ) for mixed token/embedding inputs and MTP/deepstack state embeddings, adds support for the GLM-5.3-Flash (GLM5-Next) 320B hybrid model, the Clef decision model (text and vision) and MTP speculative decoding for Qwen4Exp, ships a new /v1/systemone server API for decision models (laya, julia-1, lev, openjev, kev, nimble), overhauls the Web UI with a Hugging Face Hub data layer and model download pipeline, adds a Metal tensor-API flash attention kernel for F16 KV, sparse flash attention for quantized K/V on Vulkan, and updates ggml to v0.26.0. Highlights New llama_batch_ext extended batch API with llama_process() , supporting mixed token/embedding batches and per-token "state" embeddings for MTP and deepstack models #24669 New models: GLM-5.3-Flash (GLM5-Next), a 320B text+vision hybrid model #27773 , and the Clef decision model, fully supported with both text and vision #29831 #29969 Qwen4Exp: high-quality support is now available, with MTP speculative decoding (~1.5x decode speedup on DGX Spark) and various correctness fixes #29761 #29751 llama and server: new /v1/systemone API supporting five decision models - laya, julia-1, lev, openjev (+vision), kev #29818 Metal: new tensor API flash attention kernel for F16 KV #29570 Metal: new few-row MMA mat-mul kernels for speculative and batched decoding, up to ~3x faster mat-mul on Apple GPUs #29869 New llama_prefetch_rows() using MADVISE-based prefetching of PLE tensors in Qwen4Exp and Gemma4 #29599 API changes include/llama.h : new llama_batch_ext batch API with llama_embd , llama_process() and llama_process_type #24669 , new llama_get_causal_attn() #28876 , session formats bumped to LLAMA_SESSION_VERSION 11 and LLAMA_STATE_SEQ_VERSION 4 include/llama-cpp.h : added llama_batch_ext_ptr and deleter for the new extended batch API #24669 tools/mtmd/mtmd.h : mtmd_get_memory_usage() now returns an mtmd_memory_usage struct with image_max_tokens and use_non_causal #29773 tools/server : new /v1/systemone endpoint for decision models #29818 and /v1/embeddings now accepts typed vision/audio/video content #29556 New models GLM-5.3-Flash (GLM5-Next): 320B KDA/DSA hybrid text+vision model with mHC and MoE #27773 Clef decision model, fully supported with both text and vision #29831 #29969 Ling 3.0 VL, folded into the BailingMoeV3 architecture #29151 Nimble decision model #29844 Registered Lfm2BidirectionalForMaskedLM for LFM2.5-Encoder-230M/350M #29862 Added classifier_pooling support for rerankers #29627 Core changes Migrated examples, speculative decoding, mtmd and server to the new llama_batch_ext API #29385 #29601 ; batches now accept both embd and raw tokens #29622 Added llama_prec_policy and a model-driven W4A4 (NVFP4/MXFP4) mul_mat path #24364 Qwen4Exp: halved indexer score memory #29825 , optimized mask constructions #29824 , re-enabled the -sm tensor #28569 ; GLM5-Next: unique scatter rows for dead indexer slots #29745 KV cache: fixed restoring mismatched KV cache rotation #28498 , fixed K/V and recurrent state cleanup after failed restores #27530 and an invalid assert in recurrent memory #29799 Speculative decoding: probabilistic sampling for simple draft and MTP #27694 , fixed n-gram drafts rejected at temp > 0 after truncation #29924 , preserved original batch order for layer inputs #29019 , stop accepting draft tokens at EOG #29638 k-pool models: fixed unexpected graph reallocation #29958 and clamped kpool re-pool bound to existing pools #29805 Fixed tensor split for fused qkv with uneven K/V head sizes #29294 , gather recurrent states once so the reserve covers every split #29856 , and properly handle KV on training #28520 Context: do not re-reserve the scheduler when toggling causal_attn #28751 DFlash drafts: write Gemma embedding scale during conversion #29802 and add dflash support for MiMo #29650 Multi-modality changes Cap max_image to n_ubatch for non-causal models #29773 Fixed the mel preprocessor in LFM2 audio #29403 Migrated input processing to llama_batch_ext #29385 Server changes New /v1/systemone decision-model API with dedicated decision pipeline #29818 , extended to the nimble decision model #29844 Support vision input for Clef #29969 GET /v1/models and GET /models now report model input/output modalities in a new architecture object #29987 /v1/embeddings : accept typed content (vision/audio/video) input #29556 and return HTTP 400 for invalid embedding requests #29060 Allow RANK pooling batch splitting for causal LLM rerankers (Qwen3, Qwen3-VL) #28876 Reject partial media truncation #24076 Logging: self-contained colors and split child commands from logs in router mode #29895 , allow preset to set log file #29334 , fixed dead LLAMA_ARG_HF_REPO_FILE key in preset allow-list #29938 Fixed laya abort by limiting n_batch to n_ubatch #29903 Remove the built-in UI's service worker when the UI is not served #29565 UI changes New model download pipeline #27959 , Hugging Face Hub data layer #27947 , model memory-fit estimation #27957 and model id grammar for sidecars, quants and capability parsing #27946 Type-safe API types, fetch helpers and download-ready models store plumbing #29582 Shared model display primitives #29644 Fixed missing svg use and animation elements in preview and download #28962 Use toLocaleString() formatting consistently across chat message statistics #27990 ggml changes ggml updated to v0.26.0 : a new alloc_buffer_n / get_alloc_size_n buffer allocation API, sparse flash attention kernels on SYCL, Vulkan and Metal, and major lightning indexer improvements (halved score memory, tiling, MUSA support). The CPU backend gains BF16 ops and a tiled k-quant mul_mat, CUDA gains a model-driven W4A4 (NVFP4/MXFP4) mul_mat path plus MMVQ shared-expert fusion, and the Hexagon backend adds a sampler and more quant types. Model loading is faster with stricter GGUF size validation, Windows ARM64 MSVC builds are enabled, and WebGPU/OpenVINO/OpenCL/SYCL pick up numerous new kernels, ops and fixes. Assets Nightly build: b11429 More info Releases and versioning of ggml-org projects Changelog since v0.5.0 d812350 llama.cpp : bump version to 0.6.0 ( #29997 ) 4d60b4d common, server : report model input/output modalities in GET /models ( #29987 ) c06f841 sync : ggml f05c8b2 ggml : bump version to 0.26.0 (ggml/1652) e117148 CUDA: make the alloc_deps check batch independent ( #29986 ) 6c59c40 vulkan: fix Flash Attention shmem write out of bounds ( #29988 ) 3c9e747 vulkan: revert mul_mat_id tile selection PR #29182 ( #29936 ) b809b88 cuda: use the vector lightning indexer kernel on MUSA ( #29990 ) 994e8f2 ci : add "Require Docker" flag to make-release workflow ( #29989 ) 9d853bb webui: Use toLocaleString() format consistently across chat message statistics ( #27990 ) 8f9ae20 ci : disable failing test on virtual Metal device ( #29993 ) 9871df5 server: support vision input for Clef ( #29969 ) 8b2fbaf CUDA: Optimize accumulation in mmq for NVFP4 type ( #29857 ) 2ed93db ci : disable unused qemu in docker build ( #29984 ) 8e16421 server: reject partial media truncation ( #24076 ) 806eee9 vulkan: fix stale prealloc_y reuse across flash attention and soft_max ( #29591 ) b3daa07 vulkan: sparse flash attention for quantized K/V ( #29639 ) c173a53 llama : fix unexpected graph reallocation in the k-pool models ( #29958 ) 2107910 kv-cache: fix restoring mismatched KV cache rotation by saving exact rotation metadata ( #28498 ) e5983d6 ci : winget urls must be separate strings ( #29978 ) 4ca6b76 ci : fix docker workflow permissions ( #29979 ) 9f12cd4 ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60 ( #28479 ) ebe18be vulkan : Fix undeclared identifiers when -DGGML_VULKAN_RUN_TESTS=ON ( #29912 ) 8216c84 webgpu: add MMVQ support for Q1_0/Q5_0/Q5_1/Q3_K/Q5_K/Q6_K/MXFP4 ( #29483 ) 1b43d31 cuda: tile the lightning indexer over keys and tokens for 4 heads ( #29901 ) a3a1c47 metal : few-row MMA mat-mul ( #29869 ) 9d3aba6 CUDA: use MMVF for thin f16/bf16 mul_mat at small batch size ( #29633 ) d89651a CUDA: prefer whole-tile FlashAttention scheduling for efficient two-stage kernels ( #29435 ) a7fb71f log, server: self contained colors, split child commands from logs in router mode ( #29895 ) 0bb496d llama: support both embd + raw tokens in batch ( #29622 ) 2ca15f5 CUDA: refactor swizzling code ( #29612 ) a7b94df ggml-cpu: support BF16/FP16/FP32 K tails in tinyBLAS on x86 ( #29806 ) 0eb6d9a cuda : move neu_padded to where it is used ( #29940 ) 2e7c58c ci : windows llvm build requires ninja multi-config ( #29959 ) 7f2dd88 ci : add windows arm64 vulkan release ( #29954 ) bf79dbb AGENTS.md : revamp ( #29656 ) dbe4c3e chat-peg-parser : clear current_tool when pending_tool_call is reset ( #29942 ) 46847e6 ci : set default permissions ( #29945 ) 2bc5635 cuda : move blocks_per_col to where it is used ( #29939 ) dd26678 CUDA: fix MMQ memory fault if n_expert >> n_ubatch ( #29941 ) 16c163d vulkan: fix rdna4 mat_vec tuning ( #29934 ) 0504396 imatrix: calculate activation-based statistics for new format (GGUF) imatrices ( #14891 ) 8330e96 spec : fix n-gram drafts rejected at temp > 0 after truncation ( #29924 ) 6716df6 common : prepare load_from_models_dir() for path conversion ( #29674 ) bf9a0cc server : fix dead LLAMA_ARG_HF_REPO_FILE key in preset allow-list ( #29938 ) 0faee50 ci : pushing tag needs deploy key ( #29937 ) f98b31c ci : improve release flow ( #29913 ) 11fe021 webgpu: add f16 support to fill/set_rows ( #29897 ) 836d571 mtmd : fix deprecated strdup warning on Windows ( #29863 ) eec18f5 vendor : update cpp-httplib to 0.59.0 ( #29886 ) 1537a0a server : fix laya abort by limiting n_batch to n_ubatch ( #29903 ) edd6e2b common : add common_is_tty() helper and fix deprecated warnings on Windows ( #29860 ) 9bf55f4 chat : honor json_schema in Ling 3.0 parser ( #29813 ) a55e952 ci: fix flaky ADD_ADD f16 by using the fused ADD tolerance ( #29904 ) 436f6f8 graph: gather the recurrent states once so the reserve covers every split ( #29856 ) b92761a ggml-openvino: update to 2026.4.1, optimize performance, expand ops, improve device listing. ( #29852 ) cb7934c model : Add LFM2.5-Encoder-350M and LFM2.5-Encoder-230M ( #29862 ) 889edf4 qwen4exp : halve the indexer score memory ( #29825 ) 99b9548 model: add support for clef decision model (text-only) ( #29831 ) bed0a85 CUDA: fuse shared experts into MMVQ ( #29184 ) 4ebdf2c ci : use t4-medium for cuda jobs ( #29842 ) 1fb7ef3 spec : add probabilistic sampling for simple draft and MTP ( #27694 ) 134b2bb ggml-cuda : fix cpy transposed path corrupting non-contiguous dst ( #27663 ) 2923cf2 ggml-quants : avoid invalid rounding in qkx3 scale search ( #29817 ) dd4c286 ggml-cpu : fix soft_max_back wrong output when dst aliases src1 ( #27096 ) 46ca246 model: support nimble decision model ( #29844 ) d8fbd25 readme : add cmd install commands ( #29850 ) 926862e metal : add tensor API flash attention kernel for F16 KV ( #29570 ) a4cb4c6 llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev) ( #29818 ) 70849ee common : remove fs_open_ifstream() by using u8path() ( #29841 ) 8d81559 llama : silence unused-result warnings ( #29839 ) 6805ae3 llama : use GGML_ABORT instead of throw ( #29840 ) a8c9a4e opencl: use sigmoid f16 for bf16 ( #29787 ) 392ded6 SYCL: Q8_0 DMMV ESIMD and MMVQ wide load ( #29186 ) 9e258a6 vulkan: disable large matmul tile on Samsung GPUs with 32KB shared memory ( #28531 ) b933289 sycl: large register file for D=512 FA vec kernels ( #29062 ) c328acc sycl : do not use slow oneDNN reference matmul and fattn ( #28985 ) 4e2713c qwen4exp : optimize mask constructions ( #29824 ) 631109b ggml : add alloc_buffer_n to buffer type interface ( #23671 ) 254b177 ci : fix missing zdnn backend check ( #29837 ) fb4b273 vulkan: add logging to pipeline compile issues ( #29794 ) 207bdab pyproject : add linux platform marker to uv torch source ( #29177 ) 5fc4f3c hexagon: install rebuilt HTP skels ( #29828 ) 159c651 qwen4exp: fix tests ( #29819 ) a868c3e hexagon: add q2_k and q3_k quant type support ( #29717 ) ec7630a CUDA: fix 2 broken Volta FA cases ( #29803 ) 78e2964 llama: refer to segment documentation [no ci] ( #29074 ) f1cee99 common,rpc : fix cache dir creation through symlinks on buggy libstdc++ ( #29816 ) 68e79bd skill: note about model-specific CLI arguments + testings ( #29808 ) e358d59 ci: fix Fusion / metal by updating the qwen4exp baseline ( #29812 ) 81e39ad llama : clamp kpool re-pool bound to existing pools ( #29805 ) dcd387a hexagon: shared strided DMA copy for CPY and CONCAT, any-dim CONCAT via DMA ( #29685 ) d775ebf server: return HTTP 400 for invalid embedding requests ( #29060 ) 2b36825 convert : write Gemma embedding scale for DFlash drafts ( #29802 ) 42d9581 cuda : route sm70 to the Turing MMVQ nwarps table ( #29753 ) 13b4d71 metal : release temporary private transfer buffers ( #29777 ) 4b1622a webgpu: add bfloat16 support for MUL_MAT/MUL_MAT_ID/GET_ROWS- #29358 ( #29358 ) 869034b llama : fix invalid assert in recurrent memory ( #29799 ) b56f34a CUDA: Handle compute type for NVFP4 on cublass path ( #29173 ) c061df1 Qwen4Exp: add MTP ( #29761 ) 66e0c17 llama: fix qwen4exp ( #29751 ) 7677678 CUDA: Make CCCL configurable + pin it to 3.4.3 for CI jobs ( #29792 ) 552f18f mtmd: cap max_image to ubatch for non_causal models ( #29773 ) 5503b04 meta: clear inactive AllReduce shards with FILL, not SCALE ( #29793 ) def4d40 jinja : skip copying loop scope unless a loop filter needs it ( #29776 ) 32dd62e llama-mmap : avoid a second full-size copy of each tensor with direct-io ( #29749 ) f11d642 HIP: avoid treating CDNA as dgx spark for gqa_ratio 20 in fattn_mma dqk 576 ( #29572 ) 3aa0ce9 hex-workqueue: fix race condition in seqn getting out of sync with idx_read/write ( #29785 ) b0aca3c BLAS : Document AOCL-BLAS build and label the device AOCL-BLAS ( #29640 ) b8f96c3 common : add LLM-jp-4.1 Harmony dialect handler ( #29681 ) 3ec4df4 opencl: mark vec subgroup bcast as supproted for Adreno E17 compiler ( #29698 ) db33d3c vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3 ( #29734 ) 7dad6db llama-bench : fix verbosity filter to show GGML_LOG_ERROR ( #28229 ) 2232bc8 metal : use bf16 math for mxfp4 mul-mat ( #29770 ) 79625e0 llama-bench : fix docs ( #29464 ) 66bcc27 docs : refresh CPU ops support matrix ( #29666 ) 10f340d model : re-enable -sm tensor for qwen4exp ( #28569 ) 0c1e570 webgpu: fix SSM_SCAN binding aliasing ( #29750 ) f7b384c ggml-opencl : replace alloca() with std::vector ( #29765 ) f872b59 cuda: guard the iq4_nl dequantize row kernel against short rows ( #29683 ) a4d880f Hexagon: optimize ALLREDUCE with support for safe scatter mode ( #29757 ) feb9a3d args: fix cli download mmproj arg ( #28977 ) 4453b53 llama : preserve original batch order for speculative decoding layer inputs ( #29019 ) 4f31296 test-llama-archs : toggle causal_attn to catch graph shape changes ( #29724 ) b016f46 convert : fix LoRA conversion crash for Qwen3.5 V-head reorder ( #28324 ) 81ff93e llama: properly handle KV on training ( #28520 ) 60e9cf7 batch: migrate the rest of examples to llama_batch_ext ( #29601 ) 05af0d2 glm5-next: give dead indexer slots unique scatter rows ( #29745 ) 2149c00 ggml/gguf : fix integer overflow ( #29384 ) 876c75b codeowners : remove former ZenDNN owner ( #29747 ) b046420 cli: exit on stdin EOF and drop the console wide Ctrl+C broadcast ( #29722 ) 22bdcc4 mimo : support dflash (convert + feature extraction) ( #29650 ) ca2e203 jinja : support coerced array attributes ( #29574 ) bdeb855 ggml-et : remove useless alloca() ( #29663 ) 3b3d022 ci : fix Models Backend Check by shortening the hrm_text fixture ( #29744 ) 185103d llama: llama_prefetch_rows ( #29599 ) 2090f60 ggml : add BF16 unary, GLU, binary and scale ops (CPU, CUDA) ( #29675 ) 90c908d cpu: accept BF16 in src1 of mul_mat ( #28937 ) 8df332d model-conversion : add --add-bos to run org model script ( #29558 ) 4a096b8 ui : shared model display primitives ( #29644 ) 8664eae ui : model download pipeline ( #27959 ) 4cfb6d1 ui : model memory-fit estimation ( #27957 ) 9b43336 ui : Hugging Face Hub data layer ( #27947 ) f653250 ui : model id grammar for sidecars, quants and capability parsing ( #27946 ) fa2bde5 ui : type-safe API types, fetch helpers and download-ready models store plumbing ( #29582 ) 25747b0 openvino: serve GET_ROWS on a weight view from the base Constant ( #28381 ) db00347 ci : fix Fusion / metal by adding glm5-next to MTL.csv ( #29712 ) 272aad8 musa : define CUDA_ARCH for device passes ( #29508 ) 72db1e0 ci : add models backend check ( #29651 ) 2a53ace SYCL: reduce tensor allreduce sync with pinned host buffers ( #29604 ) 649dcb1 add GLM-5.3-Flash (GLM5-Next) support ( #27773 ) 931351e vendor: update BoringSSL to 0.20260929.0 ( #29669 ) eae11d2 ggml-zdnn: impl buffer reset, fix memory leaks ( #29637 ) 19e28a2 Hexagon f16 activation ops ( #29209 ) a6ea155 gguf : reject tensor size that wraps after padding ( #26979 ) d3954b9 ggml : check row bounds in get_rows_back ( #29575 ) 48de2a1 model : support classifier_pooling for rerankers ( #29627 ) 7fee178 hexagon: optimize concat op ( #29673 ) 6a2743f CUDA: bitonic argsort handles rows wider than one block ( #28957 ) 748d422 ggml-cuda: HIP: optimize packed byte subtraction ( __vsubss4 -> __vsub4 ) ( #29478 ) cee37ff ci: add zdnn backend build but not test ( #29541 ) 6dbbac4 opencl: fix get_tensor for q5_K adreno gemm_nonshuffle kernel ( #29555 ) 5c200e0 vulkan: Tune GDN kernel, fix Intel performance ( #29476 ) 83dd71f vulkan : Load F32 A matrix 2 at a time when its 2-aligned ( #29254 ) 94a0ae3 vulkan: MOE aware mat_mul_id tile selection ( #29182 ) da89bb3 ggml : fix c++ odr by properly using GGML_COMMON_DECL_CPP ( #29504 ) a3f84fa vocab : keep NORMAL in PLaMo-2 and PLaMo-3 ( #29580 ) 284153e ggml : accumulate f16 dot products in f32 on AVX512-FP16 ( #29545 ) b5cf8ce ggml : require input tensors to be GGML_OP_NONE ( #29647 ) e904318 hexagon: add FP32 GELU_ERF and GEGLU_ERF support ( #29631 ) d280808 common : stop accepting draft tokens at EOG ( #29638 ) ba0ba54 server : remove the built-in UI's service worker when the UI is not served ( #29565 ) 00af635 common : use fs::path for config dir ( #29649 ) c85b92c tests : adjust server string regex to also match m2 utlra results ( #29648 ) 31385c9 common : add fs_write_atomic() ( #29642 ) 8019dc5 ggml : collect all input tensors into graph_inputs ( #29634 ) 86ea01d ggml-zdnn: fix 0-row tensor crash ( #29636 ) 18b74ff musa: build the docker image and CI container from the MUSA SDK images ( #29624 ) c13e04e ggml : speed up model loading ( #29598 ) c8cda8b ci: remove gpu-rocm keyed directory logs ( #28940 ) 6d78fb0 llama : fix init in several tools/examples ( #29632 ) 18bbc46 metal: FWHT perf optimizations ( #29602 ) 0bc845d vulkan : reuse descriptor sets when bindings are constant ( #29280 ) 139997d chat : fix Muse Glimmer ignoring response_format json_schema with --jinja ( #29615 ) 76a5bc8 common : use fs::path for cache dirs ( #29595 ) 46e17a6 tests : skip pytest workers when PYTEST_WORKERS=1 ( #29610 ) fc07d78 ci : update the oneAPI toolkit to 2026.1 ( #29273 ) 526c43b mtmd: fix GCC 15 stringop-overflow in decode_embd_batch ( #29607 ) 1c47294 hex-scripts: show trace events smaller than 100nsec in perfetto ( #29614 ) 680a036 server : support typed content (vision/audio/video) input for /v1/embeddings endpoint ( #29556 ) 66e665c vulkan: include functional header ( #29597 ) 57b557c models: pad on the left with ggml_pad_ext ( #29567 ) 14ebbd5 ggml-openvino: mark unaligned batch-stride views unsupported ( #29603 ) f1ea206 batch: migrate speculative, mtmd and server to batch_ext ( #29385 ) 6c7a87f common : fix HF cache paths on Windows ( #29475 ) f00a64c webgpu: Handle unaligned writes in ggml_backend_webgpu_buffer_set_tensor ( #29471 ) d77dd08 tests : refactor test-recurrent-state-rollback ( #29426 ) 6f767fe ggml-cpu: enable tiled flash attention for non-vector-multiple head dims on x86 ( #29423 ) f916130 ci : ignore more vgpr spills in > 256 DQK fattn kernels ( #29571 ) 03a667a vulkan: fuse qwen4exp's SCALE -> SIGMOID -> SCALE -> hc_post chain ( #29520 ) c2a9e16 HIP: fix template skip for DKQ > 256 mfma kernels ( #29559 ) 4364bf7 metal: support left and circular padding in GGML_OP_PAD ( #29561 ) ed7ac35 context : do not re-reserve the scheduler when toggling causal_attn ( #28751 ) 0c6a6a7 Enables Windows ARM64 build with MSVC cl.exe ( #28362 ) 81ef10e tests : fix ggml init ( #29554 ) 5262471 vulkan: fix wrong results when a mul_mat reads a slice of a larger cache ( #28956 ) 4da6337 server : allow RANK pooling batch splitting for causal LLM rerankers (ie. Qwen3 and Qwen3-VL) ( #28876 ) a97cce8 common : avoid side effects around params parsing ( #29537 ) 136887b common : make string_split throw on invalid values ( #29518 ) 9adc7f4 convert : export YaRN scaling parameters for PLaMo-3 ( #29528 ) 6fd50a4 ci : bump ty to 0.0.84 ( #29529 ) 33c923d jinja : add support for dict builtin ( #29477 ) c9064dd opencl: refine bin kernel loading condition ( #29503 ) c829670 sycl: FWHT kernels for block widths above 512 ( #29243 ) 36d7b08 CUDA: tune fp16 tile FlashAttention configs for head sizes 40-112 ( #26289 ) 2ebd9ae HIP: Enable fattn-mma kernel on cdna for dkq > 256 for large batch sizes ( #28907 ) cea7462 vulkan: fix argsort kernel selection for Adreno ( #29469 ) da6c28e common : throw instead of abort on grammar without llguidance ( #29516 ) d7fb90e RPC: use RDMA completion channel to not spin ( #29440 ) 7fb2b08 ci : enable GGML_SCHED_DEBUG_REALLOC=1 for ctest workflows ( #29514 ) 187664b llama-bench : fix OOB access of hf_file ( #29515 ) 85ca3b5 hrm : fix layer placement of z_l_init weight ( #29512 ) 7ac59a6 hexagon: support tiled Q4_0 and Q8_0 GET_ROWS ( #29511 ) 2b129cc hexagon: support for backend sampler ( #29502 ) 9588757 cuda: support Nemotron 3 Puzzle state size 96 for ssm scan ( #28717 ) 694ec23 musa: build the docker images from the PH1 MUSA SDK image ( #29481 ) 6f856c7 cuda: add F16 input to the FWHT ( #29096 ) fcb3074 server : fix wake_fd warning on Windows ( #29479 ) 2145525 Revert "Change max context length for auto-fitting with unified KV ( #28849 )" ( #29437 ) 81bc6b8 jinja : implement sameas test ( #29448 ) 86a24a1 jinja : fix compile error ( #29468 ) 08618ff llama : fix K/V and recurrent state cleanup after failed restores ( #27530 ) a1de614 jinja : support noncall test statements with arg ( #29443 ) 965f897 polished Readme and llama-bench ( #28968 ) d834d44 ggml-cpu: tiled mul_mat for k-quants ( #27851 ) 9f70b2c opencl: add A8 Q8_0 non-MoE dp4a binary kernel ( #29439 ) 4e74811 hexagon: find software divide calls using binary inspection tool ( #29449 ) 171e884 vendor : update cpp-httplib to 0.58.0 ( #29407 ) 4b1a27f common,rpc : simplify fs_create_directory_with_parents() ( #29432 ) fcc8915 mtmd: fix mel preprocessor in LFM2 audio ( #29403 ) a25c986 opencl: add bin kernel kernel_gemm_noshuffle_q5_k_f32_32b_trans_ila_a8_bin , kernel_gemm_noshuffle_q5_k_q8_1_dp4a_ila_a8_bin ( #29401 ) e85e15c Fixing the vulkan build issue of legacy GLSLC version that has no cooperativeMatrix API support ( #29373 ) ( #29409 ) b248f4a gguf-py : ByteLevel processing defaults bos/eos to False ( #29422 ) d81aef1 gguf-py : TemplateProcessing has final word on add_special_token ( #29417 ) 27b20ba common : extract shared unicode path/string helpers ( #29415 ) e351231 metal: FWHT kernels for block widths above 512 ( #29095 ) 5a75f14 metal : split fa kernels into per-dtype libraries ( #29329 ) e9f824d llama : add llama_prec_policy + model-driven W4A4 path ( #24364 ) d028c69 HIP: bump HIP_VERSION requried for fp8 to avoid missing __hip_fp8_e4m3 support in 6.2 ( #29231 ) 66963a8 rpc: include nb in the get_alloc_size cache key and floor the result at ggml_nbytes ( #29283 ) cd74ef6 [SYCL] support sparse FA ( #28796 ) f9af9be musa: fix PH1 (MTT S5000) operator failures and build issues ( #29193 ) 1ab7e5a CUDA: fuse RMS_NORM + SCALE into one kernel ( #29393 ) f805c57 llama : fix tensor split for fused qkv with uneven K/V head sizes ( #29294 ) 4de0926 hexagon: add q5_k quant type support ( #29123 ) ed319fe hexagon: use DMA for contiguous dim1 CONCAT ( #29404 ) 84e76d8 metal : fix graph capture and handle empty graphs ( #29390 ) cdc0642 metal : optimize sparse FA + clean-up ( #29377 ) bced459 sync : ggml ( #29396 ) a02c7f5 hexagon: handle multi-sequence in concat_2d ( #29344 ) 5cf3a35 llama-grammar: fix numeric truncation for token_id parsing ( #29382 ) 07fc586 hexagon: dynamic quantizer improvements ( #29395 ) 97a418b hexagon: support I32 CPY and CONT ( #29379 ) a72e04a cuda : add F16 kernel support for CONV_2D_DW ( #29064 ) 8212c78 test: flush status ( #28352 ) 945064f ui : fix missing svg use and animation elements in preview and download ( #28962 ) fc343a8 llama: add llama_batch_ext ( #24669 ) 308883b server : change default pytest workers to 4 ( #29376 ) 70596c4 ci : use hf-jobs-cpu-performance, disable pytest workers ( #29369 ) 70c4e15 vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4 ( #27952 ) 6b790a9 vulkan: handle misalignment in conv_2d and conv_3d ( #29365 ) 3423f94 vulkan: tune KHR cooperative matrix support for Adreno GPUs ( #29328 ) 53ed051 cuda : add conv3d with implicit GEMM ( #29137 ) f830688 model : add Ling 3.0 VL support ( #29151 ) 2b70583 server,common : fix the GCC 12 stringop-overread false positive (again) ( #29325 ) 4c5957c test-save-load-state : print a per-model results table in --models mode ( #29316 ) 9710a32 hexagon: reject MUL_MAT_ID when src1 precision is F32 ( #29348 ) 013b31c scripts : make-release-desc - link previous release in changelog title ( #29336 ) bd4f514 convert : allow vision target for DFlash/Dspark ( #29339 ) b9ae43a server: allow preset to set log file ( #29334 ) d2e5458 tests: add -b/--backend option to test-llama-archs for testing a specific backend ( #27372 ) 6e60f35 ci : use hf-jobs-cpu-xl runner in server sanitize workflow ( #29297 ) fee39dd opencl: add A8 Q6_K non-MoE dp4a binary kernel ( #29057 )

The cool part is No training was needed. No hacking of the game state or algorithms needed Just simple instructions about what the snake can see, etc, and it can play in real time. 135ms is the turn limit of Google snake, so basically could be a human playing. Ofc, it could be improved to be a perfect snake player, but thats not the point. This can be used in other games where decisions need to constantly be made. Running Clef Flash (9B model at Q4 on an RTX 5080) submitted by /u/bigboyparpa to r/LocalLLaMA [link] [comments]

It’s now $20 cheaper than the M4 version available at Best Buy. www.theverge.com/gadgets/1004... Linked page: The first price reduction on the M6 model.

Article URL: https://reflection.ai/blog/introducing-beam Comments URL: https://news.ycombinator.com/item?id=49969183 Points: 433 # Comments: 136

Article URL: https://developers.cloudflare.com/changelog/post/2026-10-02-introducing-web-search-api/ Comments URL: https://news.ycombinator.com/item?id=49963171 Points: 537 # Comments: 244

Dominic-Madori Davis / TechCrunch : Ghost, which makes a $3,499 computer designed for AI agents and includes an RTX Pro 4000 SFF Blackwell GPU, emerges from stealth with an $11M seed led by a16z — For much of his life, Zain Javaid, 19, wanted to be a quant who uses math and data to guide his investment decisions. And for a while, he was one.

What's Changed Models run on MLX on Apple Silicon by default In this release, on Apple Silicon devices, model architectures supported by the MLX runtime will automatically run on MLX. ollama pull qwen3.8 ollama run qwen3.8 Additional models include gemma4 , qwen3.6 and qwen3.5 Decision models are now available on MLX as well: Nimble tev1 clef clef-flash We will continue testing and enabling additional models. Full Changelog : v0.34.4...v0.40.0-rc3

Release v5.14.0 New Model additions Inkling (fresh from Thinking Machines): 975B total, 41B active Add Inkling model #47347 by @molbap @Cyrilvallez @eustlb and @zucchini-nlp Inkling is a general-purpose multimodal model that accepts text, image and audio inputs and generates text outputs. It is intended for use in English and other languages, and across multiple coding languages. The model is designed to be used by developers building AI- powered applications, including agentic and tool-use systems, coding assistants, chatbots, and retrieval-augmented generation systems, and is suitable for general-purpose conversational use, instruction-following, and other natural language and multimodal tasks. It is released with open weights to support research, fine-tuning and integration into third-party products by downstream developers. TIPSv2 Links: Documentation Add TIPSv2 ( #46347 ) by @Ternura143 in #46347 TIPSv2 DPT Links: Documentation Add TIPSv2 ( #46347 ) by @Ternura143 in #46347 🚨 Breaking changes GPTNeoX now remaps embed_out to lm_head and GPTBigCode has _supports_attention_backend = True enabled for vLLM compatibility; users relying on the previous weight naming or attention backend behavior for these models should update their code accordingly. 🚨 Fix GPTBigCode and GPTNeoX for the Transformers modelling backend for vLLM ( #47198 ) by @hmellor Kernels Several kernel-related fixes and improvements were made, including pinning the kernels dependency to a compatible version in the benchmark workflow, removing a deprecated package_name argument from LocalLayerRepository , and making the DeepGEMM Triton fallback more robust when CUDA_HOME is unset or misconfigured. Additionally, SDPA prefill was updated to leverage the FlashAttention kernel with StaticCache , yielding significant performance gains (up to 260% faster for large input sizes). Pin kernels to compatible version in benchmark workflow ( #47339 ) by @tarekziade in [ #47339 ] [Fix] Remove deprecated argument from kernels call ( #47100 ) by @remi-or in [ #47100 ] [Fix] Make DeepGEMM triton fallback more robust ( #47126 ) by @remi-or in [ #47126 ] [sdpa] Allow prefill to use FA kernel with StaticCache ( #47094 ) by @Cyrilvallez in [ #47094 ] Generation Generation improvements include adding Multi-Token Prediction (MTP) decoding support, static ensemble verification for speculative decoding to improve draft token acceptance rates, and a fix for crashes in greedy assisted generation with different tokenizers. A misleading double-negative warning message for synced_gpus in continuous batching mode was also corrected. [generation] Fix misleading synced_gpus warning in continuous batching ( #47158 ) by @Partha-Shankar in [ #47158 ] [generate] Add proper MTP support ( #46229 ) by @Cyrilvallez in [ #46229 ] Fix crash in greedy assisted generation with different tokenizers ( #46936 ) by @Sunt-ing in [ #46936 ] [Generation] Add static ensemble verification for lossy speculative decoding ( #45979 ) by @kasakh in [ #45979 ] Performance Fixed a Flash Attention performance regression affecting models like Qwen3-VL and resolved a MoE decode optimization bug where the grouped-to-batched matrix multiplication switch was not applied to experts residing in submodels (e.g., VLMs with a nested text config). Fix FA performance regression ( #47134 ) by @andreasgoulas in [ #47134 ] Fix MoE decode optimization for experts living in a submodel ( #47107 ) by @IlyasMoutawwakil in [ #47107 ] Make doc builds faster ( #47099 ) by @mishig25 in [ #47099 ] Cache Cache dispatch logic was simplified by introducing explicit layer-type mappings for sliding and static layers, reducing complexity in cache routing. Additionally, fixes were made for read-only cache failures in CPU CI environments and for MPS graph cache growth during variable-length batch training on Apple Silicon. Fix CI read-only cache failures by patching cached_files in conftest ( #47043 ) by @ydshieh in [ #47043 ] trainer: clear MPS graph cache via torch_empty_cache_steps ( #45818 ) by @anagnorisis2peripeteia in [ #45818 ] [cache] Simplify cache dispatch based on layer_types ( #47118 ) by @Cyrilvallez in [ #47118 ] Bugfixes and improvements ci: cover xet as well (runtime error) ( #47338 ) by @tarekziade in [ #47338 ] [docs] TokenizersBackend fallback ( #47302 ) by @stevhliu in [ #47302 ] Resolve continuous batching XPU availability checks at runtime ( #47185 ) by @kaixuanliu in [ #47185 ] [Nit] Add kernels_fallback_ok kwarg to is_flash_attn_N_available ( #47318 ) by @remi-or in [ #47318 ] [Nit] Add expectations for gemma4 tests on H100 ( #47311 ) by @remi-or in [ #47311 ] [docs] DeepGEMM requirements ( #47324 ) by @stevhliu in [ #47324 ] DeepGEMM shouldn't pad on SM90 ( #47313 ) by @IlyasMoutawwakil in [ #47313 ] Fix half-precision torch.compile crash in DETR-family sine position embeddings ( #47238 ) by @David-Wu1119 in [ #47238 ] Fix hardcoded paths in siglip checkpoint/vocab loading ( #47178 ) by @XanxusCrypto in [ #47178 ] Update AMD CI runner groups to amd-mi300 ( #47307 ) by @Abdennacer-Badaoui in [ #47307 ] Point to Gemma 4 model in Gemma4ForCausalLM docstring example ( #47255 ) by @lefft in [ #47255 ] Fix Qwen Omni batched text postprocessing ( #47197 ) by @Sunt-ing in [ #47197 ] Fix AqlmConfig error messages to say "int" instead of "float" ( #47089 ) by @Sreekant13 in [ #47089 ] Fix check for interactive stdout in _style function ( #47283 ) by @smart8986 in [ #47283 ] Fix get_json_schema crash on non-string docstring choices ( #47072 ) by @Sreekant13 in [ #47072 ] Make MODEL_IDS_TO_TOKENIZERS_BACKEND capture all DeepSeek R1 distills ( #47296 ) by @hmellor in [ #47296 ] Update doc preprocessing regex to prevent ReDoS ( #47187 ) by @WilliamRoyNelson in [ #47187 ] Shard on read Dtensor aware ( #46717 ) by @3outeille in [ #46717 ] Switch AMD daily CI to mi300 runners ( #47259 ) by @Abdennacer-Badaoui in [ #47259 ] tests: reduce processor test memory usage by using tiny Hub checkpoints ( #47213 ) by @ydshieh in [ #47213 ] Torch compile backend defaults to "neuron" ( #47035 ) by @michaelbenayoun in [ #47035 ] Fix flash-attn Docker build broken by setuptools 83 removing pkg_resources ( #47251 ) by @ydshieh in [ #47251 ] Add heterogeneous config support (per-layer configuration) ( #45333 ) by @eladsegal in [ #45333 ] [fix] update integration test values ( #47146 ) by @eustlb in [ #47146 ] Fix DeepSpeed SP loss aggregation and LocalLayerRepository kwargs ( #47073 ) by @sshivampeta in [ #47073 ] tests only for the top 10 download models ( #47244 ) by @3outeille in [ #47244 ] Fix InputTokensDetails missing cache_write_tokens for openai>=2.34.0 ( #47248 ) by @ydshieh in [ #47248 ] Revert "Trigger a scheduled run" ( #47249 ) by @ydshieh in [ #47249 ] Remove Executorch from CI until latest version is supported and fully tested on CI env ( #47242 ) by @IlyasMoutawwakil in [ #47242 ] Be more defensive with remap_legacy_layer_types for custom models ( #47245 ) by @hmellor in [ #47245 ] Fix DistributedConfig docstring for unimplemented sp_plan ( #47237 ) by @3outeille in [ #47237 ] Switch mlinter to 0.1.2 ( #47172 ) by @tarekziade in [ #47172 ] Trigger a scheduled run ( #47209 ) by @ydshieh in [ #47209 ] Make executorch exporter tests always use xnnpack backend ( #47201 ) by @tarekziade in [ #47201 ] No agent PR descriptions ( #45790 ) by @Rocketknight1 in [ #45790 ] Clarify that max_steps is required for datasets without len ( #47155 ) by @albertvillanova in [ #47155 ] Cleanup pipelines, stop materializing generators ( #47142 ) by @Rocketknight1 in [ #47142 ] Fix device_map computation when the no_split_modules have different sizes ( #47203 ) by @Cyrilvallez in [ #47203 ] Add native FSDP2 module + migration ( #46707 ) by @3outeille in [ #46707 ] Fix experts implementation in two spots ( #47097 ) by @remi-or in [ #47097 ] [Fix] Remove old automatic cross attn pattern from output recorders ( #47117 ) by @remi-or in [ #47117 ] 🌐 [i18n-KO] Translate accelerator_selection.md to Korean ( #47157 ) by @kkwjk2718 in [ #47157 ] [i18n-KO] Translate optimum.md to Korean and fix Furiosa typo ( #47156 ) by @kkwjk2718 in [ #47156 ] [docs] fix curly quotes rendering to straight quotes ( #47135 ) by @clijo in [ #47135 ] Fix custom code which doesn't know about the new linear layer type names ( #47174 ) by @hmellor in [ #47174 ] Reject path traversal in the transformers_weights config field ( #46890 ) by @LinZiyuu in [ #46890 ] [docs] Custom code conversion mapping ( #47114 ) by @stevhliu in [ #47114 ] Add exporters min version requirements and test skip ( #47161 ) by @IlyasMoutawwakil in [ #47161 ] tests: reduce processor test memory usage and use tiny test assets ( #47168 ) by @ydshieh in [ #47168 ] Clarify input device placement in the Quicktour inference example ( #47136 ) by @samyuktahegde in [ #47136 ] Extend continuous batching memory prediction test to XPU ( #47159 ) by @sywangyi in [ #47159 ] Fix case where _LazyAutoMapping.register is passed a str key ( #47148 ) by @hmellor in [ #47148 ] [docs] MoE decode switching ( #47149 ) by @stevhliu in [ #47149 ] add XPU output expectations for minicpm3 tests ( #47092 ) by @kaixuanliu in [ #47092 ] Diffusion gemma: fix failed test cases ( #47025 ) by @kaixuanliu in [ #47025 ] add XPU Expectation for cosmos3_omni tests ( #46880 ) by @kaixuanliu in [ #46880 ] Fix IndexError Bug in XLMRoberta/Camembert ForMultipleChoice by restoring the pooler ( #47147 ) by @pariidanDKE in [ #47147 ] Skip caching_allocator_warmup on Neuron (no reuse pool to warm; currently OOMs) ( #47029 ) by @dacorvo in [ #47029 ] [docs] continuous batching (offloading behavior, max batch tokens, block size minimum) ( #46925 ) by @stevhliu in [ #46925 ] [docs] fix autolinks ( #46968 ) by @stevhliu in [ #46968 ] revert #47121 ( #47144 ) by @eustlb in [ #47144 ] Fix output labels for AudioFlamingo3 (and related) models ( #47112 ) by @ebezzam in [ #47112 ] Fix false len claims in Trainer docstrings ( #47131 ) by @albertvillanova in [ #47131 ] processor tests: use tiny Hub repos to reduce CI memory ( #47115 ) by @ydshieh in [ #47115 ] [serge] Fix 12 integration tests for model dac failing with output_mismatch (tensor values differ (6), other (6)) ( #47121 ) by @sergereview[bot] in [ #47121 ] Fix CLI compatibility with huggingface_hub 1.22 ( #47059 ) ( #47064 ) by @dhruv7477 in [ #47064 ] we want to run the CI in the release branches ( #47125 ) by @tarekziade in [ #47125 ] Small improvement ( #47128 ) by @Cyrilvallez in [ #47128 ] [Model] Support use_cache=False for DeepSeek V4 ( #46965 ) by @kylesayrs in [ #46965 ] docs-fix: IMDb dataset link in sequence classification guide ( #47062 ) by @abhishekkapoorx in [ #47062 ] Fix AltCLIP text embedding resize test ( #47079 ) by @IMvision12 in [ #47079 ] fix mask return-type contract regression and add correctness guard for ( #47019 ) by @kaixuanliu in [ #47019 ] Fix save_pretrained with offloading and weight conversions ( #47018 ) by @Cyrilvallez in [ #47018 ] Update dev ( #47044 ) by @vasqu in [ #47044 ] [ Gemma4 ] Update 1 integration test ( #47042 ) by @vasqu in [ #47042 ] Significant community contributions The following contributors have made significant changes to the library over the last release: @ArthurZucker v5.14.0 @tarekziade ci: cover xet as well (runtime error) ( #47338 ) Pin kernels to compatible version in benchmark workflow ( #47339 ) Switch mlinter to 0.1.2 ( #47172 ) Make executorch exporter tests always use xnnpack backend ( #47201 ) Remove executorch from all-latest-gpu image + add torch smoke test ( #47196 ) we want to run the CI in the release branches ( #47125 ) @remi-or [Nit] Add kernels_fallback_ok kwarg to is_flash_attn_N_available ( #47318 ) [Nit] Add expectations for gemma4 tests on H100 ( #47311 ) [Fix] Remove deprecated argument from kernels call ( #47100 ) [Fix] Make DeepGEMM triton fallback more robust ( #47126 ) Fix experts implementation in two spots ( #47097 ) [Fix] Remove old automatic cross attn pattern from output recorders ( #47117 ) @ydshieh tests: reduce processor test memory usage by using tiny Hub checkpoints ( #47213 ) Fix flash-attn Docker build broken by setuptools 83 removing pkg_resources ( #47251 ) Fix InputTokensDetails missing cache_write_tokens for openai>=2.34.0 ( #47248 ) Revert "Trigger a scheduled run" ( #47249 ) Fix CI read-only cache failures by patching cached_files in conftest ( #47043 ) Trigger a scheduled run ( #47209 ) tests: reduce processor test memory usage and use tiny test assets ( #47168 ) processor tests: use tiny Hub repos to reduce CI memory ( #47115 ) @eladsegal Add heterogeneous config support (per-layer configuration) ( #45333 ) @eustlb [fix] update integration test values ( #47146 ) revert #47121 ( #47144 ) @Ternura143 Add TIPSv2 ( #46347 )

https://x.com/AdithyaNLP/status/2107123924828049691?s=20 submitted by /u/Eliv_nurotic to r/accelerate [link] [comments]

TinyDecide is 10M Jev-like mode with 10M parameters and fits in just ~6MB. Smaller than every model on the Decision Index leaderboard and it punches way above its size . It runs almost anywhere: in the browser, Node.js, Python, Rust, and even on an ESP32. https://huggingface.co/TheREZOR/TinyDecide submitted by /u/TheRealREZOR to r/LocalLLaMA [link] [comments]

Google's Gemma 4 paired with BOTANIC-1 (trained on 320 plant species) can now pinpoint the exact DNA mutations behind drought resistance and higher yields — a task that once took years of physical crop breeding. In one test, it found the target mutation ranked #1 out of 2,494 possibilities in less than four minutes, potentially giving scientists the speed needed to future-proof our food supply against climate change. submitted by /u/EuphoricTomorrow2440 to r/accelerate [link] [comments]