2026-11-02

I'm sharing this here because this is working way better than it used to a few months ago with older models. I see people here posting slower speeds with much better hardware than mine and hopefully this information helps. I used gemini-flash to help me setup everything and I asked it to write up the sequence of steps to get me here - happy to provide more details if there's anything missing. Since I use gentoo some tools/commands might be unfamiliar to you. My 6 year old hardware: OS & CPU Gentoo Linux running on an AMD Ryzen Threadripper 3990X (64 cores, 128 threads). System Memory: 128 GB DDR4 RAM. GPU: 1 NVIDIA GeForce RTX 3090 Ti (24 GB GDDR6X VRAM, PCIe Gen 4 x16). Primary Storage: 4 TB SSD NVMe. Models Used, Download Sources, and Measured Speeds Model Architecture Quantization / Files Download Source Real Measured Speed DeepSeek-V4.1-Flash-1017 552B MoE (40 layers, 384 experts/layer, ~39B active) XQ / GSQ V2 + rank-1017-085.bin + FP4 KV + Engram NVMe tables IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored ~35 tok/s decode (with MTP/DSpark active) Qwen3.8-Flash-Next 125B MoE + 51B N-gram (GDN + QSA hybrid, ~3.5B active) pack-qfn45 (4-bit intermediate, DeltaNet linear KV) [IndrasMirror/Qwen3.8-Flash-Next-Swift-1.5-XQ-Uncensored]( https://huggingface.co/IndrasMirror/Qwen3.8-Flash-Next-Swift-1.5-XQ-Uncensored ) ~138–140 tok/s decode IMPORTANT: NVIDIA GPU Unlock Consumer Ampere cards (RTX 3000 series: GA102, GA104) feature artificial firmware throttles on FP16/BF16 matrix instruction issue rates compared to their enterprise workstation counterparts. The open-source gpu-unlock kernel module lifts this throttle prior to driver attachment. 1. Compile and Install the Kernel Module As root: git clone https://github.com/Zagitta/gpu-unlock.git /usr/src/gpu-unlock cd /usr/src/gpu-unlock make -j$(nproc) make install 2. Pass the Unlock Flag to the Kernel Command Line Add gpu_unlock.targets=all to your bootloader kernel parameters (e.g. in /etc/default/grub under GRUB_CMDLINE_LINUX_DEFAULT , or your direct EFISTUB / systemd-boot entry): gpu_unlock.targets=all Regenerate your boot configuration and rebuild your initramfs so the module is included and loads before the NVIDIA kernel modules: # If using GRUB: grub-mkconfig -o /boot/grub/grub.cfg # If using dracut: dracut -f --regenerate-all 3. NVIDIA Driver Options & Persistence Configure NVIDIA kernel driver parameters in /etc/modprobe.d/nvidia-perf.conf (as root): cat << 'EOF' > /etc/modprobe.d/nvidia-perf.conf options nvidia NVreg_PreserveVideoMemoryAllocations=1 options nvidia NVreg_EnableStreamMemOPs=1 options nvidia-uvm uvm_perf_enable_cpu_to_gpu_dma=1 EOF After rebooting, verify that the unlock executed: dmesg | grep -i gpu_unlock Enable persistence mode and set the GPU to its maximum factory power ceiling: nvidia-smi -pm 1 nvidia-smi -pl $(nvidia-smi -q -d POWER | grep "Max Power Limit" | head -n1 | awk '{print $5}' | cut -d. -f1) Step 2: System Dependencies & Tooling Install build dependencies and runtime libraries: emerge sys-devel/gcc:14 # If GCC 14 is not selected, set it to this version gcc-config -l # then replace <N> with the number from gcc-config -l): # gcc-config <N> && source /etc/profile # this won't compile with GCC > 14 emerge dev-util/nvidia-cuda-toolkit # for compleness, just in case emerge --ask --verbose \ dev-build/cmake \ dev-build/ninja \ dev-vcs/git \ net-misc/curl \ app-misc/jq \ sys-libs/liburing \ app-arch/snappy \ dev-lang/python Prepare the directory structure under your user environment: mkdir -p "$HOME/.local/bin" mkdir -p "$HOME/.local/llm-tools" mkdir -p "$HOME/.quetza-data/conductor" mkdir -p "$HOME/models/deepseek-v41" mkdir -p "$HOME/models/qwen-flash" # Dedicated Python environment for tool scripts & Hugging Face downloads python3 -m venv "$HOME/.local/llm-tools/venv_hf" source "$HOME/.local/llm-tools/venv_hf/bin/activate" pip install --upgrade pip pip install "huggingface_hub[cli]" torch safetensors numpy Step 3: Build the Xolotl Inference Engine Xolotl integrates the multi-tier memory management, NVMe Engram streaming, and DSpark speculative decoding logic needed to run large MoE models on a single GPU. cd "$HOME/.local/llm-tools" git clone https://github.com/Indras-Mirror/Xolotl.git cd Xolotl # Fetch pinned llama.cpp dependencies and apply patches bash tools/xolotl/fetch-llama.sh # Configure and compile for Ampere (sm_86) cmake -S . -B build -G Ninja \ -DCMAKE_BUILD_TYPE=Release \ -DSTRATA_ENABLE_CUDA=ON \ -DSTRATA_GGML_CUDA=ON \ -DSTRATA_MMQ_KQUANTS=ON \ -DCMAKE_CUDA_ARCHITECTURES=86 \ -DSTRATA_GGML_DIR="$PWD/third_party/llama.cpp" \ -DSTRATA_DSV41_FQ_OP=ON \ -DSTRATA_DSV41_HC_MIX=ON ninja -C build # Link the build directory so wrapper scripts resolve ds41_generate ln -sfn "$PWD/build" "$PWD/build-xolotl" Step 4: Model Downloads & Engram Extraction 1. DeepSeek-V4.1-Flash (552B MoE) Bash source "$HOME/.local/llm-tools/venv_hf/bin/activate" cd "$HOME/models/deepseek-v41" # Download model weights, draft sidecar, and expert rank index hf download IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored \ --include "v2/*" "tokenizer/*" "V41-MTP-Vision-1017*" \ --local-dir . # Download original FP8 Engram table shards hf download deepseek-ai/DeepSeek-V4.1-Flash-1017 \ model-00047-of-00048.safetensors \ model-00048-of-00048.safetensors \ model.safetensors.index.json config.json \ --local-dir ./FP8-engram # Extract Engram runtime binary tables onto NVMe cd "$HOME/.local/llm-tools/Xolotl" python3 tools/v41/extract_engram.py \ --input "$HOME/models/deepseek-v41/FP8-engram" \ --output "$HOME/models/deepseek-v41/engram_rt" 2. Qwen3.8-Flash-Next (125B MoE) cd "$HOME/models/qwen-flash" hf download Qwen/Qwen3.8-Flash-Next-GGUF \ --include "pack-qfn45/*" \ --local-dir . Step 5: MTP Speculative Decoding Setup MTP (Multi-Token Prediction) on DeepSeek-V4.1-Flash is driven by the DSpark sidecar engine: Draft Engine: Uses V41-MTP-Vision-1017.Q2_K.gguf as the proposal head. Expert Pruning: Restricts verification sweeps to the top 64 active draft experts ( dspark_expert_counts.txt:64 ), preventing PCIe bus saturation. On-the-fly Re-quantization ( STRATA_DENSE_REQUANT=q6_K ): Re-quantizes the dense attention and output heads from Q8_0 to Q6_K at launch. This reclaims 1.6 GiB of VRAM , fitting the speculative draft cache alongside active experts. PCIe Batching ( --tier-batch-pcie 0.25 ): Batches DMA transfers between the 80 GiB pinned host RAM arena and the GPU. 1. Expand Process Memory Boundary in the Engine Wrapper Allow the systemd user scope to hold the full 80 GiB arena plus runtime buffers: Bash sed -i 's/MemoryMax=84G/MemoryMax=110G/g' "$HOME/.local/llm-tools/Xolotl/tools/v41/serve/ds41-engine.sh" 2. Generate Base Configuration: $HOME/models/deepseek-v41/strata-v41.json cat << EOF > "$HOME/models/deepseek-v41/strata-v41.json" { "exe": "$HOME/.local/llm-tools/Xolotl/tools/v41/serve/ds41-engine.sh", "args": [ "-m", "$HOME/models/deepseek-v41/v2/V41-XQ-1017.gguf", "--backend", "cuda", "--experts", "gpu", "--engram", "$HOME/models/deepseek-v41/engram_rt", "--rank", "$HOME/models/deepseek-v41/v2/rank-1017-085.bin", "--ctx", "131072", "--stop", "1", "--arena-adapt", "--kv-fp4", "--route-bias", "0.5", "--route-bias-file", "0.3", "--arena-gib", "80", "--prompt-batch", "12288", "--tier-batch-pcie", "0.25" ], "cwd": "$HOME/.local/llm-tools/Xolotl", "tokenizer": "$HOME/models/deepseek-v41/tokenizer", "model_name": "deepseek-v4.1", "aliases": "deepseek-v41,deepseek/deepseek-v4.1-flash", "log": "$HOME/models/deepseek-v41/xolotl-v41.log", "port": 8000, "gpu": 0, "sampling": { "temperature": 0.1, "top_p": 0.95, "top_k": 64, "presence_penalty": 0.0, "frequency_penalty": 0.0 }, "api_monitor": true, "fit_max_tokens": true, "reasoning_budget_tokens": 4096, "env": { "STRATA_BUILD_DIR": "$HOME/.local/llm-tools/Xolotl/build", "STRATA_SERVE_DSPARK": "$HOME/models/deepseek-v41/V41-MTP-Vision-1017.Q2_K.gguf", "STRATA_ATTN_F16_GATHER": "1", "STRATA_NO_SCOPE": "1", "STRATA_SERVE_PASS_BIAS": "0", "STRATA_DENSE_REQUANT": "q6_K", "STRATA_SERVE_LOOKUP": "14", "STRATA_PRESTAGE_GIB": "1.5" } } EOF Step 6: Parameterized Launch Scripts 1. DeepSeek-V4.1 Launcher: $HOME/.local/bin/start-deepseek-custom.sh cat << 'EOF' > "$HOME/.local/bin/start-deepseek-custom.sh" #!/usr/bin/env bash set -euo pipefail MODEL_DIR="$HOME/models/deepseek-v41" XOLOTL_DIR="$HOME/.local/llm-tools/Xolotl" BASE_CONFIG="$MODEL_DIR/strata-v41.json" RUN_CONFIG="$MODEL_DIR/strata-v41-run.json" CTX="131072" PORT="8000" TEMP="0.1" THINK_BUDGET="4096" ARENA_GIB="80" PASS_BIAS="0" PROMPT_BATCH="12288" usage() { cat << USAGE Usage: $(basename "$0") [options] Options: -c, --ctx INT Context length in tokens (default: 131072; up to 1048576) -p, --port INT Port to bind (default: 8000) -t, --temp FLOAT Sampling temperature (default: 0.1) -b, --thinking INT Reasoning budget tokens (default: 4096) -a, --arena INT Pinned RAM expert arena in GiB (default: 80) --bias 0|1 Routing bias: 0=exact, 1=fast biased (default: 0) --batch INT Prompt batch size (default: 12288; auto-clamps to 8192 for >=512k) -h, --help Show help message USAGE exit 0 } while [[ $# -gt 0 ]]; do case "$1" in -c|--ctx) CTX="$2"; shift 2 ;; -p|--port) PORT="$2"; shift 2 ;; -t|--temp) TEMP="$2"; shift 2 ;; -b|--thinking) THINK_BUDGET="$2"; shift 2 ;; -a|--arena) ARENA_GIB="$2"; shift 2 ;; --bias) PASS_BIAS="$2"; shift 2 ;; --batch) PROMPT_BATCH="$2"; shift 2 ;; -h|--help) usage ;; *) echo "Unknown option: $1" >&2; exit 1 ;; esac done if [ "$CTX" -ge 524288 ] && [ "$PROMPT_BATCH" -gt 8192 ]; then echo "[-] Large context specified ($CTX tokens). Clamping --prompt-batch to 8192 to prevent graph OOM." PROMPT_BATCH=8192 fi python3 - << PY import json with open("$BASE_CONFIG", "r") as f: cfg = json.load(f) cfg["port"] = int("$PORT") cfg["max_context"] = int("$CTX") cfg["reasoning_budget_tokens"] = int("$THINK_BUDGET") cfg["sampling"]["temperature"] = float("$TEMP") args = cfg.get("args", []) def set_arg(flag, val): for i in range(len(args) - 1): if args[i] == flag: args[i+1] = str(val) return args.extend([flag, str(val)]) set_arg("--ctx", "$CTX") set_arg("--arena-gib", "$ARENA_GIB") set_arg("--prompt-batch", "$PROMPT_BATCH") cfg["args"] = args env = cfg.setdefault("env", {}) env["STRATA_SERVE_PASS_BIAS"] = "$PASS_BIAS" env["STRATA_PRESTAGE_GIB"] = "1.0" if int("$CTX") >= 524288 else "1.5" with open("$RUN_CONFIG", "w") as f: json.dump(cfg, f, indent=2) PY source "$HOME/.local/llm-tools/venv_hf/bin/activate" cd "$XOLOTL_DIR" echo "==> Launching DeepSeek-V4.1-Flash (552B MoE)..." echo " Context : $CTX tokens" echo " Port : $PORT" echo " Arena RAM : $ARENA_GIB GiB" echo " Prompt Batch : $PROMPT_BATCH" exec python3 tools/v41/serve/serve_ds41.py \ --engine strata \ --config "$RUN_CONFIG" \ --port "$PORT" \ --host 0.0.0.0 EOF chmod +x "$HOME/.local/bin/start-deepseek-custom.sh" 2. Qwen3.8-Flash-Next Launcher: $HOME/.local/bin/start-qwen.sh cat << 'EOF' > "$HOME/.local/bin/start-qwen.sh" #!/usr/bin/env bash set -euo pipefail XOLOTL_DIR="$HOME/.local/llm-tools/Xolotl" MODEL_DIR="$HOME/models/qwen-flash/pack-qfn45" PORT="8099" source "$HOME/.local/llm-tools/venv_hf/bin/activate" cd "$XOLOTL_DIR" echo "==> Launching Qwen3.8-Flash-Next (125B MoE)..." exec python3 -m serve.server \ --engine strata \ --config "$MODEL_DIR/strata-qwen.json" \ --port "$PORT" \ --host 0.0.0.0 EOF chmod +x "$HOME/.local/bin/start-qwen.sh" Step 7: Run # Start DeepSeek-V4.1-Flash on :8000 (default 128k context, 1M context work just fine too) $HOME/.local/bin/start-deepseek-custom.sh # Or start Qwen3.8-Flash-Next on :8099 $HOME/.local/bin/start-qwen.sh submitted by /u/uniVocity to r/LocalLLaMA [link] [comments]

I am open to the idea that I am misguided, but my findings feel worth sharing: Short video of my UI with 8 agents each explaining a codebase in parallel long prose: https://imgur.com/JSqlXIi A few days ago I read the Stanford/Berkeley TokenGate paper . The idea is to make long context cheap by pruning tokens up front. I tried it. It's fine for one question, but agents keep asking new ones, and by then the tokens they need are gone, so code recall drops hard. So I went the other way - don't throw anything away, just don't look at all of it at once. Keep the whole context in system RAM and at each step pull in only the ~1% that matters right now. A few days later, I have two 3090s run 8 coding agents at once, each holding their own 250K-token context : Every decode step, each attention layer reads only ~1% of the tokens, picked with a tiny rank-64 "key sketch". The full KV cache lives in system RAM ( int4 in my 80GB RAM VM, but int8 should fit in 128gb). The GPU keeps the sketch and a small hot set. Only the picked tokens cross PCIe. What it took: Triton kernels that build the sketch, score every token, pick the top ~2K and pack them for the stock int8 attention kernel. CUDA-graph safe. vLLM patches that switch dense/sparse per step, so short chats stay full speed. A RAM tier for the KV cache with a prefix cache and recurrent-state checkpoints, so follow-ups don't recompute. Follow-ups fetch only the picked rows. A scheduler fix so one agent reading a huge prompt doesn't freeze the rest. A frozen sketch basis (one file) plus the script to refit it. It's all in the repo as patches, not a fork. Apply them on top of HyperQwen @ ce9615c (a public vLLM fork) and run the install script. Tests and every bench script and result are included. Model: dbirks/Qwen3.8-27B-W4A16-AutoRound + HyperQwen's prepare/fetch_fast_variant.py . All numbers were measured with gpu-unlock loaded (a kernel module that roughly doubles 3090 BF16 tensor throughput, mostly faster prompt reading). Measured ( Qwen3.8-27B W4A16 , MTP-7 , 3090 Ti + 3090 at 250 W): 8 × 250K: ~545 tok/s total on explain-style output, ~790 on copy-heavy. 8 agents at 176–205K generating nonstop for 6 min: 657 tok/s box average, 69–90 per agent. Recall at 250K: 31/32. Follow-ups start in ~5 s. Single long chats are faster too : +15–41% at ~113K vs dense. Caveat: attention is approximate, so near-duplicate code lines occasionally get mixed up. Code: Serving (patches + kernels): https://github.com/bradj00/sparsekv-hyperqwen Rig: i9-12900K Z790 motherboard 96 GB DDR5 RAM 3090 Ti + 3090, power capped at 250 W submitted by /u/o2beast to r/LocalLLaMA [link] [comments]

SemiAnalysis : SemiAnalysis launches GDPx, an agentic capability benchmark measuring 11 domains, including chip design, cybersecurity, and RSI; Opus 5.5 tops with a 12% score — A comprehensive benchmark spanning RSI, hardware design, cyber, bio, finance, legal, quant trading, chip design, software engineering, materials science, and alignment

Good news, everyone! Having been inspired to revolutionize the field of computer science , I can now report that coding is solved! Sure, only if you care about x86 Linux binaries that print and exit, but "big journeys" and "small steps" and what not. CRAP (Compiler Replacement for Assembling Programs) is a Gemma 4 E2B fine tune that takes prompts like generate a Linux x86_64 binary that prints 'boop' and emits a working ELF binary (as hex, most of the time, every time). See, since programs end up as a binary anyway, why not just skip the middleman and write the binary itself? No compiler, no assembler, no tool calls. v1.1 is out with improved multi turn and out of distribution performance . Trust me, I know, this is super important work. https://preview.redd.it/gngsbvezc1zh1.png?width=1076&format=png&auto=webp&s=481f6ea0897d2f324272b59b6bf42c337daf3b2d v1.1 eval numbers for the nerds (eval is scored on behavior: the binary has to run, print the right string and exit with the right code): Held out legacy set: 20/23, same as v1 Novel strings: 26/40 ( v1 : 8/40) Multi turn (~3 turns): 10/12 ( v1 : 0/12) Doubled character strings ( boom , etc.): 18/24 ( v1 : 2/24) Random jank exit codes it never trained on: 23/25 An interesting tidbit: one training run hit 0.038 loss while still failing 11 of its 12 trained exit codes, so gradient descent != task completion ascent. Aaannndddnd it runs in your browser. https://preview.redd.it/xkz0lvr1d1zh1.png?width=760&format=png&auto=webp&s=5c5be9a9c3f4cee8663f20fea79097480a857438 The demo loads the Q6_K GGUF (~3.6 GiB) through wllama on WebGPU, streams the hex, parses it and hands you the ELF to run yourself (naturally, you need to download and run the actual binary, the browser won't do that). Q6_K scores 109/144 against the full f16's 114/144 (the five additional failed for Q6 are wrong values, not totally broken binaries). Q4_K_M scored 3/144, so the demo doesn't use it (that quant is crap, not CRAP) ... concerning . Also, tested in Chrome on an M4 Pro Mac; Firefox loads it but generates slowly and nothing else has been tested. Demo: https://huggingface.co/spaces/matthewhaynesonline/crap-demo Weights (v1.1): https://huggingface.co/matthewhaynesonline/gemma-4-E2B-it-crap-1.1 "Paper": https://github.com/matthewhaynesonline/crap-public/blob/main/paper/paper.pdf Code: https://github.com/matthewhaynesonline/crap-public Blog post: https://blog.studiohaynes.com/2026/07/19/compilers-overrated.html Video: https://youtu.be/55IBMzFQNYU Fine print for the buzzkills: only programs that print a string and exit can be generated. Obviously, a compiler does that deterministically, correctly and in milliseconds; CRAP does it with several gigabytes of weights, slower and occasionally incorrectly. submitted by /u/matthewhaynesonline to r/LocalLLaMA [link] [comments]

So, I was asleep on Strata and its specialized forks. Tried basalt and i'm very impressed with the speed/intelligence QFN gives you. Here is my benchmark: https://preview.redd.it/8qhp92ql73zh1.png?width=2946&format=png&auto=webp&s=2abf6d03db5124c8491baf980b8cd3b5367fdb9f I cannot get it to work fast enough with windows, because W11 eats 1.5GB of VRAM and 6GB of RAM. At that point, experts spill to RAM and CPU gets hit hard... decode/prefill drops by 70% because there is not enough of available RAM anyway. Switched experimentally to cachy just to test... and It's amazing that my AI box can now reach 9K~ prefill. And regarding quality, its just better than qwen 27b at many tasks. Yet to find something it genuinely cannot do. The prefill boost is around 2-3x coming from ninfer and dense 27b model. Cherry on top is, that I can even fit vision adapter to it, and that helps a lot for certain scenarios regarding coding. I usually run some small model on another machine for OCR and VISION as an adapter (sidecar) my harness can use for being to able look at things. Vivid user of omo stack, configured now for locally hosted qwen: https://github.com/code-yeongyu/oh-my-openagent It's a glorified pi harness, but it does so much, and im quite happy with its robustness, anyway, QFN 3.8 rocks, and now im really thrilled if qwen 4 will be any good, because I suspect i may be able to run the new flash model this fast basically. So I hope it will be good :D Im genuinely interested how 2x 5060/5070/5070ti fares. Because anything above 100~ decode and 3k prefill is genuinely usable for work. And it is my experience that 27b dense -> 125BA6 is a very good quality and speed jump. Absolutely love it now. Also, finetunes can be easily converted into basalt format if you need to do it yourself, especially those swift/reverse engineering variants are good. Anyway: CachyOS, 64gb RAM, 32GB VRAM, 5090, 200tps decode, 9k tps prefill, nvfp4 QFN 3.8, 256k context! If you have similar hardware, and are not using at least strata... but more specially basalt, you sleep on performance hard! See ya! PS: Would absolutely love to see your perf benchmarks if you have similar setups edit-- Running config (without vision,.. but it fits nicely): ~/basalt-linux/basalt_config.json ```json { "server": { "model": "/mnt/g/../basalt/models/Qwen3.8-FN-NVFP4.basalt", "gpu": "0", "host": "127.0.0.1", "port": 8080, "parallel": 1 }, "engine": { "max-context": 262144, "kv": "int8", "prompt-cache": 6, "expert-cache": "auto", "prefill": "auto", "pool-workers": 4, "vram-reserve-mib": 500, "resident-budget-gib": 44 } } ``` bin/basalt --serve --model /mnt/g/../basalt/models/Qwen3.8-FN-NVFP4.basalt \ --max-context 262144 --kv int8 --prompt-cache 6 --expert-cache auto --prefill auto \ --pool-workers 4 --vram-reserve-mib 500 --resident-budget-gib 44 --grammar on submitted by /u/Rozzemak to r/LocalLLaMA [link] [comments]

Sam Altman recently said that "from a global allocation perspective, actually, it's quite terrible to have everybody have a bunch of the world's limited supply of DRAM sitting in their own computer, and being used maybe 5% of the time." Challenge accepted. Here's my "quite terrible" home setup: an RX 9070 XT 16 GB, a 5800X3D, 64 GB of DDR4 that's now worth more than the rest of the PC, and a PCIe 3.0 NVMe. It runs DeepSeek V4.1 Flash at 1M context (IST-DAS 1.75 bpw; the lab reports 98.7% of the full model's task average) through Strata , fully offline, with the experts spread across VRAM, RAM and the SSD. Generation at ~7 tok/s was fine. The problem was prompt reading : every new OpenCode session re-read the full ~37K-token system prompt plus tools, about 100 s before the first token. That's datacenter-grade waiting without the datacenter. So I patched the DeepSeek engine. Two changes: 1. KV splice: stop re-reading what you already read Agents put per-session text (working directory, date, project CLAUDE.md, skill paths with random IDs) inside the system prompt, so the prefix cache misses on every new session. The engine now aligns a new prompt against every saved conversation, cuts the stale lines out of the cached state (moving the cached rows down, rotating the window rings, recording a RoPE gap), and reads only the changed lines plus your message. Changed lines move as whole lines, so the model never sees fragments. 2. Resident-only routing: make the RAM earn its keep Even a short prompt touches nearly every expert of every layer, so prompt reads streamed most of the model off the SSD: 58 GB for a 1.2K-token prompt . During prompt reading only, tokens now pick from experts already sitting in VRAM or RAM. Generation routes normally. Turns out home DRAM is pretty useful when you actually let it do things. Results (same machine) as seen above on the body image Caveats: HIP (gfx1201) only. I haven't built or tested CUDA, SYCL or Metal. The splice is approximate: tokens after a cut were computed with the removed text in context. I haven't seen it matter for short guidance lines, but there's no formal eval. Resident routing changes answers slightly (substitute experts during prompt reads). My hard coding suite didn't move (12/13 both ways), but that's one run each. Both are on by default in the branch ; STRATA_DS4_REORDER=0 and STRATA_DS4_PROMPT_RESIDENT=0 turn them off. It also includes a one-line fix for a GPU page fault I hit on a 37K-token prompt (#4912). Branch: https://github.com/PhialsBasement/Strata/tree/dsv41-kv-splice Build (HIP example): git clone -b dsv41-kv-splice https://github.com/PhialsBasement/Strata.git cd Strata cmake -S . -B build-hip -G Ninja -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_HIP=ON \ -DSTRATA_ENABLE_CUDA=OFF -DCMAKE_HIP_ARCHITECTURES=gfx1201 -DSTRATA_BUILD_DSV41=ON cmake --build build-hip --target strata-dsv41 DS4_CHUNK_MIN=16 STRATA_DSV41_EXE=$PWD/build-hip/strata-dsv41 strata launch --preset <your preset> If you use OpenCode, also add "interleaved": { "field": "reasoning_content" } to the model entry, otherwise OpenCode drops earlier reasoning and the cache misses more often. Upstream issues with details: #4921 (splice), #4936 (resident routing). If you try it on other hardware, especially CUDA, numbers in those threads would really help get it merged. Utilization report for Sam pissman, VRAM holding experts, RAM holding experts, SSD streaming the rest at full bandwidth. The only idle component in this system is my wallet, thanks to DRAM prices. submitted by /u/HearMeOut-13 to r/LocalLLaMA [link] [comments]

I have been researching what the ultimate diff tool would look like, based on actual diffs in open source repositories. OmniDiff can diff: code (25 languages), config, images, PDFs, archives, cursors, fonts... It's fast. It can diff C++ files of tens of thousands of lines in under 100ms, on a 2023 ThinkPad T14s. It's open source. AGPLv3. It's based on real diffs. I looked at the 7400 repositories that are listed in Gentoo packages, sampled 2000 diffs and solved them manually to get a good understanding of what is needed, before putting together the diff pipelines for each content type. Incidentally, this also forms the basis of the CI pipeline, so the robustness of the codebase is quite high. I'm still developing it. It's v0.3.0 at this time. But I have been using it daily as my git difftool for months now, and it works well. AI Disclosure: Yes, I do use AI. No, I do not vibecode. Comments URL: https://news.ycombinator.com/item?id=50325253 Points: 228 # Comments: 58

Dear fellow Llamas, after almost 5 months of development, I finally have the pleasure of announcing to you the immediate availability of the biggest Heretic release ever, Heretic 2.0: https://github.com/p-e-w/heretic Heretic is the leading software for removing censorship from language models. It now boasts 37k GitHub stars and more than 8,500 models on Hugging Face, with 88 million(!!!) total model downloads. With Heretic 2.0, the project is entering a new era. Heretic is expanding from a purpose-built tool for censorship removal to a general-purpose model modification tool that can do amazing things. Besides removing censorship, it can suppress slop, reduce sycophancy, alter emotional alignment, and pretty much anything else that you can think of. Of course, censorship removal remains the primary and default function, and in fact works much, much better than ever before. Heretic 2.0 is the most advanced refusal suppression system ever built. Your time is precious, so here's the TL;DR for this release: Directional abliteration is replaced by Arbitrary-Rank Ablation (ARA) , a novel steering method that I developed from scratch for Heretic. It's vastly superior to abliteration, and has been used with great success by enterprising testers for several months. Now it is the default (classic abliteration remains available as an option). Heretic has been completely rebuilt around plugins . Scorer plugins define what Heretic optimizes for. Modifier plugins implement the actual steering method. Plugins are easy to write, can be loaded directly from a simple Python file, and have a well-defined API. Heretic can now run non-interactively, which among other things has enabled us to implement full end-to-end tests for much tighter quality control. Model support has been expanded to include models with hyper-connections, and support for thinking models has been substantially improved. Heretic 2.0 is not backwards compatible with Heretic 1.x configurations. See the migration guide for what you need to change in order to make your old configurations work. What hasn't changed is that removing censorship from a model remains as simple as running heretic Qwen/Qwen3.5-4B See the default configuration file for what Heretic has to offer beyond that. Now for some details... ARA: Heretic's new steering method Directional ablation, or "abliteration", is based on the assumption that refusals (or whatever else you are trying to suppress) are represented by a single direction (that is, a rank-1 subspace) in residual space. That this is the full picture has been doubted in research for a while, and there are now several papers exploring alternatives such as multi-directional ablation or "representation cones". At the beginning of this year, I started experimenting with a method that completely abandons the idea that the ablation transformation must be restricted to a fixed-rank subspace. In March, I published the initial version of Arbitrary-Rank Ablation in this pull request (which was later closed as I re-implemented ARA on top of the plugin system in this PR ). ARA steers model behavior by modifying matrices based on an objective function that encodes the basic properties of the change we want: Harmful prompts should be represented in residual space in a way that's similar to harmless prompts, while the representations of harmless prompts should change as little as possible. This objective is used with the L-BFGS optimizer to modify matrices one-by-one, a kind of micro-training that steers representations without any rank restriction. ARA has been extensively tested and refined with the help of the community. Notably, kabachuha made it compatible with Heretic's LoRA system, and varunbln found a substantial performance improvement. I intend to post an article explaining ARA in more detail soon. Using scorer plugins to make Heretic do what you want Heretic operates by finding modification parameters that optimize scores , values that measure the quality of the resulting model. Ever since the 1.0 release, Heretic has used two hardcoded scores: Refusal count to measure decensoring strength, and KL divergence to measure quality retention. With Heretic 2.0, you can freely define your own scores by writing and loading scorer plugins. Arbitrarily many scorer plugins can be loaded and optimized for simultaneously, and each score can be either minimized or maximized during optimization. See the default configuration file for how to do this. The scorer plugin system is the result of months of effort by red40maxxer, with whom I exchanged hundreds of messages over dozens of review cycles to figure out the best possible design. This is a monumental contribution that has changed how Heretic works from the ground up. Besides the keyword (refusal) count and KLD scorer plugins, which remain the default, Heretic ships with two other scorers: A benchmark scorer , which can compute scores for many standard benchmarks, and a BERT classifier . The BERT classifier, developed by long-time Heretic contributor Vinay Umrethe, can use an arbitrary ModernBERT-type model to classify LLM responses, and score them based on which labels the classifier produces. Vinay also trained a refusal classifier model (used by default with this plugin), which experiments have shown to be substantially more accurate at identifying model refusals than the naive keyword matching method that Heretic employs by default. While the plugin is slower than keyword matching, the greater accuracy can steer Heretic towards producing a higher-quality modified model. An alternative configuration, which uses an emotion classifier, is also provided in the Heretic repository. The plugin system is well-documented in both the README and the code. If you are interested in experimenting with custom plugins, an agent should have no difficulty cooking up a plugin and a matching configuration file that together do what you want, after ingesting the Heretic repository. Well, I hope you like it! Cheers :) submitted by /u/-p-e-w- to r/LocalLLaMA [link] [comments]

I was trying to sell my old AM4 system, X470 / Ryzen 5 5600 / 64GB / RX580 / 1TB NVME and it didn't sell. Granted it was a bit pricey at 750€, because I took into account the current local RAM and NVME pricing. (Buying the same RAM and SSD new costs 775€ if you go to the cheapest sellers in my country) I decided to try and sell it as an AI system: removed the RX580, put in 2 of my 4 R9700. I set the system up with Fedora to run Qwen3.6 (latest release at the time) and ComfyUI; to show it working. It sold almost instantly! I priced it based on new prices for all of the DDR4 memory, PCI-E 4.0 ssd, both R9700s. I added another 150 euro for the other parts. Total price 5400€! I have since bought back two brand new R9700 from abroad for 250€ less per card than I had sold my old ones for! Effectively I ended up selling the original system for 1300€ instead of the 750€ I originally asked for. Except now I still have the RX580 and fresh warranty on the R9700s. tl;dr know the market for (used) parts before you buy a complete PC for local AI! submitted by /u/sernamenotdefined to r/LocalLLaMA [link] [comments]

Release v5.20.0 New Model additions Muse Spark Open Muse Spark Open is a multimodal mixture-of-experts model from Meta that takes text, images, audio, and video as input and generates text. It has 816B total parameters (42B active per token) and a context window of up to 1,048,576 tokens shared across modalities. It is trained for agentic coding tasks such as multi-file editing, debugging, and tool use. Checkpoints are available in BF16 and NVFP4 formats. Links: Documentation | Blog Post Add Muse Spark Open ( #49644 ) by @guarin in #49644 Ling 3.0 Ling 3.0 Flash is a hybrid Mixture-of-Experts language model from InclusionAI. Its decoder groups five Kimi Delta Attention (KDA) layers with one Multi-head Latent Attention (MLA) layer. The MLA blocks use interleaved rotary embeddings and a head-wise output gate, and the feed-forward blocks combine routed and shared experts. It is natively supported in Transformers as BailingHybridForCausalLM , so it loads without trust_remote_code=True . Links: Documentation [Model] Add native Ling 3.0 Flash support ( #49446 ) by @icavan in #49446 Kolibri1 Kolibri 1 is Aleph Alpha's mixture-of-experts reasoning model, focused on German and English. Each of its 50 layers routes every token to 6 of 384 experts using sigmoid routing, and adds one shared expert. Four sliding-window attention layers alternate with one full-attention layer, and RoPE is applied only in the sliding-window layers. Links: Documentation Add Kolibri 1 ( #49463 ) by @lucamotz in #49463 MiniCPM-V 4.7 MiniCPM-V is a series of efficient multimodal large language models developed by OpenBMB . Like MiniCPM-V 4.6, the MiniCPM-V 4.7 architecture pairs a SigLIP vision encoder that has a window-attention merger with a Qwen3.5 language model backbone, and supports both 4x and 16x visual downsampling modes. The main addition over 4.6 is canvas M-RoPE : instead of numbering visual tokens along a single 1-D sequence, the model lays every image out on a 2-D canvas and assigns each visual token a (temporal, height, width) position, so slices of the same image keep their spatial relationship and video frames keep their temporal order. Fix more stuff ( #48979 ) by @zucchini-nlp in #48979 UnlimitedOcr UnlimitedOcr is a single 3B-parameter OCR model from Baidu with a standard 32,768-token context length. It builds on DeepSeek OCR and replaces all decoder attention layers with Reference Sliding Window Attention (R-SWA). This keeps the KV cache constant during decoding, so the model can transcribe dozens of pages in a single forward pass. It supports "gundam" mode (640x640 tiles with dynamic cropping) and "base" mode (a single 1024x1024 view), and it can output region detections with bounding boxes. Links: Documentation | Paper Add Unlimited OCR ( #46836 ) by @guarin in #46836 UltraBERT UltraBERT is a family of bidirectional text encoders trained with a combination of masking ratios, pruning, and distillation for classification and retrieval tasks. The released encoders support sequences of up to 32,768 tokens. They use rotary position embeddings, RMSNorm, SwiGLU feed-forward layers, and alternating local and global attention (every third layer is global). Checkpoints include UltraBERT-150M and UltraBERT-380M for masked language modeling and fine-tuning, and UltraBERT-Embed-150M for text embeddings. Links: Documentation | Paper Integrate the UltraBERT architecture ( #49596 ) by @tomaarsen in #49596 MOSS-Transcribe-Diarize MOSS-Transcribe-Diarize 0.9B is an end-to-end audio understanding model for long-form multi-speaker transcription, speaker diarization, and timestamps. It combines a Whisper-style encoder, a 4× frame merge step, a multi-modal projector, and a Qwen3 language model. It produces joint transcription and diarization in a single pass, with segments formatted as [start][S01]text[end] . It supports long-form audio through Whisper-window chunking in the processor, promptable transcription with hotword biasing, and 50+ languages. Links: Documentation Add MOSS-Transcribe-Diarize model ( #48386 ) by @itazap in #48386 Breaking changes 🚨 General breaking notice that inference with EP has regressions in speed; we are working on fixing this ASAP! 🚨 Deprecated checks and methods have been removed from cache_utils.py (affecting models such as minimax_m3_vl and pi0), so any code still calling those deprecated cache methods must switch to the current cache API. 🚨 Remove deprecated cache utils methods ( #49605 ) by @guarin Accidental imports from other models were removed from several model files (e.g. embedding_gemma2, ijepa, lightglue), and the modular converter now avoids adding such cross-model imports. If you imported these names through those model modules, import them from their original modules instead. 🚨 Remove imports from other models ( #49473 ) by @guarin The minimum kernels version is now 0.17.2, which allows use_fallback=KernelizeFallback.ALL so that a failed kernel load falls back to the original implementation instead of raising an error. Upgrade kernels to 0.17.2 or later. 🚨 Bump min kernels to 0.17.2 for better fallback ( #49235 ) by @SunMarc A new SubConfigSpec centralizes the initialization of sub-configs, so default sub-configs are collected from dataclass fields instead of by instantiating the class, and this touches the configuration utilities and many model configs. Custom or existing nested configs may need to be updated to declare their sub-configs through this mechanism. 🚨 SubConfigSpec for centralized init of subconfigs ( #48282 ) by @zucchini-nlp The LUKE and mLUKE tokenizers now treat entity special tokens such as <ent> as special tokens (not split, honored by skip_special_tokens , normalized) while keeping the same spacing as regular vocabulary tokens when decoding. Tokenization and decoding output involving these tokens may therefore differ from before, so check any code or saved outputs that depend on the old behavior. 🚨🚨🚨 luke fixe ( #48408 ) by @itazap Exporters Transformers exporters now support quantization directly during export across Dynamo, ONNX, OpenVINO, and ExecuTorch. The new unified quantization API integrates PT2E, ONNX Runtime, and NNCF quantizers, enabling static and dynamic quantization as well as weight-only compression (INT8/INT4), with configurable calibration workflows. This makes it easier to export optimized models for efficient inference across different backends. For more information see the docs: https://huggingface.co/docs/transformers/exporters_quantization Parallelization This release improves distributed training and inference. FSDP × TP model and optimizer checkpoints can now be saved and loaded across different parallel configurations, and expert-parallel dispatch, tensor-parallel plans, static-cache generate() , and the caching allocator's byte count were fixed. Tied LM heads in composite models now get their own TP plan entry, enable_expert_parallel applies the EP plan alone again, TPU loading uses the `tor FSDP x TP Saving / Loading, Optimizer ( #49256 ) by @michaelbenayoun in [ #49256 ] FSDP x TP Saving / Loading ( #48802 ) by @michaelbenayoun in [ #48802 ] Don't set a device per rank for gloo tensor-parallel tests ( #49636 ) by @IlyasMoutawwakil in [ #49636 ] fix(distributed): keep enable_expert_parallel as EP alone ( #49633 ) by @dacorvo in [ #49633 ] Fix a port allocation race in FSDP tests ( #49455 ) by @icavan in [ #49455 ] Expert-parallel dispatch hands the experts the received rows, not the all-to-all's pending result ( #49421 ) by @IlyasMoutawwakil in [ #49421 ] Fix static-cache generate() with host-side inputs and under TP ( #49317 ) by @dacorvo in [ #49317 ] Composite lm head tp plan ( #49080 ) by @IlyasMoutawwakil in [ #49080 ] Fix: use torch.accelerator device APIs so TP loading works again on TPU ( #49502 ) by @tengomucho in [ #49502 ] Count ep and fsdp in caching_allocator_warmup byte calculation ( #49384 ) by @3outeille in [ #49384 ] Do not wrap the model in FSDP for expert-parallel token dispatch ( #49383 ) by @3outeille in [ #49383 ] Generation This release fixes static-cache generate() bugs with host-side inputs and under tensor parallelism. It also speeds up Whisper generation on TPU by cutting device syncs and recompilation, and fixes Whisper batched timestamp decoding for tensor and NumPy inputs. Pipelines get a cleanup of generation-argument handling that removes spurious warnings. Add job to automatically generate Intel CPU Docker image ( #49560 ) by @regisss in [ #49560 ] Image decoding moves to torchcodec ( #47963 ) by @zucchini-nlp in [ #47963 ] Optimize Whisper generation for TPU and device sync ( #49426 ) by @tengomucho in [ #49426 ] Generation cleanup in pipelines ( #49269 ) by @Rocketknight1 in [ #49269 ] Fix Whisper batch timestamp decoding for tensors ( #49419 ) by @Kingofbooks in [ #49419 ] Vision Vision processing was updated in several places. Image decoding now uses torchcodec, Molmo2's image and video kwargs follow the library's standard conventions, and Pixtral Flash Attention no longer detects false image boundaries after the axial RoPE refactor, which had caused CUDA illegal memory access errors and incorrect output. Several CI and Docker changes also shipped: GPU images moved to Python 3.11, a numpy/pyarrow conflict Fix numpy/pyarrow conflict in the XPU Docker image ( #49582 ) by @jiqing-feng in [ #49582 ] CI: Move GPU images to python 3.11 ( #49352 ) by @tarekziade in [ #49352 ] Update molmo2 processing images/videos kwargs ( #49563 ) by @guarin in [ #49563 ] Fix Pixtral Flash Attention image boundaries after axial RoPE refactor ( #49373 ) by @oliverholworthy in [ #49373 ] Kernels This release improves kernel support across hardware. flash_attention_2 now falls back to the AITER kernel on ROCm when flash-attn isn't installed, and T5-style models get a contiguous SDPA position-bias mask so they use fused kernels again. The Hub rotary kernel is now limited to CUDA inference because its backward pass gave incorrect gradients. The Atlas Inference Gated DeltaNet kernels for Strix Halo and Fall back to the AITER kernel for flash_attention_2 on ROCm ( #49608 ) by @Abdennacer-Badaoui in [ #49608 ] Make the SDPA position_bias mask contiguous so T5-style models use fused kernels ( #49517 ) by @IvenHsu01 in [ #49517 ] Use the Hub rotary kernel for inference only on CUDA ( #49439 ) by @qgallouedec in [ #49439 ] [Atlas Inference] Enable torch.compile for the Strix Halo Gated DeltaNet hub kernel ( #49453 ) by @AzeezIsh in [ #49453 ] [Atlas Inference] Fix GB10 Gated DeltaNet hub kernel for transformers 5.17+ and enable torch.compile ( #49452 ) by @AzeezIsh in [ #49452 ] Cache This release fixes several cache-related bugs: static-cache generate() now handles host-side nested inputs and tensor parallelism correctly, the offloaded cache prefetch no longer races with the offloading copy, and Phi3 LongRoPE cache rebuilds at the context boundary now receive the full prefix. It also counts the query token in MiMo-V2-Flash's sliding window and corrects the quantized cache (hqq/quanto) documentation exam Fix offloaded cache prefetch racing with the offloading copy ( #49363 ) by @qgallouedec in [ #49363 ] docs: update quantized cache hqq/quanto examples and documentation ( #47212 ) by @DavidJohnQuinlan in [ #47212 ] [MiMo-V2-Flash] Count the query token in the sliding window ( #49170 ) by @guapisolo in [ #49170 ] Fix Phi3 LongRoPE cache rebuild at the context boundary ( #49361 ) by @LE0-Lin in [ #49361 ] Bugfixes and improvements [docs] Hardware docs template ( #49575 ) by @stevhliu in [ #49575 ] Add MIC contract lock files for PR CI integration ( #49631 ) by @ydshieh in [ #49631 ] Use aiter-rope version 3 on ROCm ( #49595 ) by @Abdennacer-Badaoui in [ #49595 ] Remove missing methods from docs ( #49637 ) by @guarin in [ #49637 ] Translate torch.mm(..., out_dtype=...) to ONNX ( #49635 ) by @IlyasMoutawwakil in [ #49635 ] [docs] Hardware section ( #49541 ) by @michaelbenayoun in [ #49541 ] Raise when a torch fallback would drop packed-sequence boundaries ( #49474 ) by @qgallouedec in [ #49474 ] Fa packed sequence decode ( #49610 ) by @SunMarc in [ #49610 ] fix nested set_attn_implementation ( #49138 ) by @eustlb in [ #49138 ] feat: Add MTP support to Mellum2/2.1 ( #49531 ) by @shadeMe in [ #49531 ] extend onnx exporters tests to non-cuda devices ( #49627 ) by @kaixuanliu in [ #49627 ] Document video processor backends ( #49470 ) by @DimensionSTP in [ #49470 ] Make sinusoidal position embeddings compileable ( #49482 ) by @jiqing-feng in [ #49482 ] Save the PEFT adapter of a load-time sharded model from its own parameters ( #49500 ) by @qgallouedec in [ #49500 ] fix: Pass the MTP layer type to per-layer-type rotary embeddings ( #49468 ) by @shadeMe in [ #49468 ] Add LFM2 to the GGUF loading path ( #49400 ) by @francip in [ #49400 ] Add MtpModel helpers for frameworks that own the drafting loop ( #49544 ) by @hmellor in [ #49544 ] Build docs with the Python version from .python-version ( #49603 ) by @tarekziade in [ #49603 ] Use mlp_layer_types for AXK1 so the MoE layer layout honours moe_layer_freq ( #49518 ) by @hmellor in [ #49518 ] Combine Nvidia+AMD run-slow CI comments into one ( #49598 ) by @ydshieh in [ #49598 ] Fix flaky streamer timeout tests ( #49583 ) by @tarekziade in [ #49583 ] Fix TPU related tests issues ( #49568 ) by @tengomucho in [ #49568 ] fix(distributed): bind TPU ranks to device index 0, not LOCAL_RANK ( #49543 ) by @tengomucho in [ #49543 ] Use transformers init everywhere ( #49573 ) by @guarin in [ #49573 ] [docs] Context parallelism during eval ( #48663 ) by @stevhliu in [ #48663 ] Remove examples ( #49357 ) by @SunMarc in [ #49357 ] Accept whole-number floats for integer tool arguments in parse_response ( #49547 ) by @yonigozlan in [ #49547 ] update Intel CPU Dockerfile ( #49408 ) by @jiqing-feng in [ #49408 ] Add missing video processor auto docstrings ( #49569 ) by @guarin in [ #49569 ] Add chat input sanitization ( #47386 ) by @Rocketknight1 in [ #47386 ] Fix create_chunked_causal_mask() crashing with block_sequence_ids ( #49457 ) by @kaixuanliu in [ #49457 ] Fix SinglePositionMultiTokenCandidateGenerator signature ( #49413 ) by @jiqing-feng in [ #49413 ] Minicpm 4.7 update integration tests ( #49565 ) by @zucchini-nlp in [ #49565 ] Stop treating HunYuan's DynamicNTKAlpha RoPE as dynamic ( #49445 ) by @jiqing-feng in [ #49445 ] Fix Gemma3nVision2TextModelTest::test_model_parallelism split ( #49562 ) by @tarekziade in [ #49562 ] Fix UnboundLocalError in SuperPoint forward under torch.compile ( #49409 ) by @kaixuanliu in [ #49409 ] Preserve resolved remote backend references in AutoVideoProcessor ( #49511 ) by @DimensionSTP in [ #49511 ] Fix PR slow CI suggestion: install transformers in a venv ( #49558 ) by @tarekziade in [ #49558 ] Fix PR slow CI suggestion: install transformers, fail on script errors ( #49549 ) by @ArthurZucker in [ #49549 ] Remove deprecated data helpers ( #49360 ) by @SunMarc in [ #49360 ] Fix test_all_params_have_gradient for intentionally frozen parameters ( #49264 ) by @thillai-c in [ #49264 ] Flex attn fix for molmo2 test ( #49530 ) by @molbap in [ #49530 ] fix bridgetower cross-attn mask shape bug ( #49447 ) by @kaixuanliu in [ #49447 ] Use Siglip2Tokenizer when the Hub tokenizer_class is GemmaTokenizer ( #49245 ) by @jayzuccarelli in [ #49245 ] Fix mixed BOS prefix validation in WatermarkDetector ( #49496 ) by @dangruiyuan in [ #49496 ] Deprecate unused get_uniform_frame_indices ( #49464 ) by @archiep867-sys in [ #49464 ] [tests] Some fixes for test suite failures on TPU ( #49425 ) by @tengomucho in [ #49425 ] Second phase of torch 2.6 cleanups ( #49396 ) by @Rocketknight1 in [ #49396 ] Guard the checks for current_accelerator() and bump Torch version ( #49395 ) by @Rocketknight1 in [ #49395 ] Use one static Transformers CI badge in PR descriptions ( #49471 ) by @tarekziade in [ #49471 ] fix: treat every declared field of the config class as a default fixes #49456 ( #49466 ) by @tarekziade in [ #49466 ] Keep AutoVideoProcessor class lookup lazy ( #49458 ) by @DimensionSTP in [ #49458 ] Add Molmo2 ( #43451 ) by @SangbumChoi in [ #43451 ] Ci fetches multimodal models recursively ( #49389 ) by @zucchini-nlp in [ #49389 ] Recurse for all subconfigs ( #49461 ) by @zucchini-nlp in [ #49461 ] Aria, Ernie4.5-VL-MoE, Gemma4, Inkling, MiniMax-M3-VL, Qwen3.5-MoE: run the model tests on the VLM/ALM testers ( #49404 ) by @IlyasMoutawwakil in [ #49404 ] Fix CI slack reporting: surface crashed/missing model jobs in model_results.json ( #49427 ) by @ydshieh in [ #49427 ] [Qwen3VLMoe] Update expected text in test_small_model_integration_test_expand ( #49301 ) by @ydshieh in [ #49301 ] Fix backend attribute in documentation and default backend in MistralCommonBackend ( #49149 ) by @samyuktahegde in [ #49149 ] More RoPE simplification, TF32 fixes ( #49328 ) by @Rocketknight1 in [ #49328 ] Complete PR #49228 : remove unused device=None from rope init in 11 model families ( #49285 ) by @ydshieh in [ #49285 ] Revert "[CI] Temporarily disable AMD scheduled CI trigger" ( #49433 ) by @ydshieh in [ #49433 ] Fix QuantizedCache residual flushing for multi-token updates ( #49418 ) by @LE0-Lin in [ #49418 ] Fix data-dependent branch in _ignore_causal_mask_sdpa under compile ( #49354 ) by @dacorvo in [ #49354 ] Fix Inkling moe_intermediate_size when dense_intermediate_size is present ( #49416 ) by @Talhax55z in [ #49416 ] fix bug for DebertaV2ForMaskedLM non-legacy load issue ( #49152 ) by @kaixuanliu in [ #49152 ] This PR fixes Gemma3n fp16 loading when missing weights need to be in… ( #49147 ) by @sywangyi in [ #49147 ] PR CI: grant actions: read to the reusable workflow ( #49417 ) by @tarekziade in [ #49417 ] [Bugfix] [Modeling] Fix DSV4 norm ( #47486 ) by @kylesayrs in [ #47486 ] [docs] Update Spaces ( #49242 ) by @stevhliu in [ #49242 ] handle empty precompiled charsmap in spm converter ( #49211 ) by @itazap in [ #49211 ] [CI] Temporarily disable AMD scheduled CI trigger ( #49394 ) by @ydshieh in [ #49394 ] [tests] Fix flaky CMVN tests in Speech2TextFeatureExtractionWithoutTorchaudioTest ( #49374 ) by @ydshieh in [ #49374 ] Let token dispatch run with ep_size != tp_size ( #49368 ) by @qgallouedec in [ #49368 ] [tests] Skip onnx.exactness for BitForImageClassification (ORT fp32 drift) ( #49366 ) by @ydshieh in [ #49366 ] Significant community contributions The following contributors have made significant changes to the library over the last release: @itazap Add MOSS-Transcribe-Diarize model ( #48386 ) handle empty precompiled charsmap in spm converter ( #49211 ) 🚨🚨🚨 luke fixe ( #48408 ) @ydshieh Add MIC contract lock files for PR CI integration ( #49631 ) Combine Nvidia+AMD run-slow CI comments into one ( #49598 ) Fix CI slack reporting: surface crashed/missing model jobs in model_results.json ( #49427 ) [Qwen3VLMoe] Update expected text in test_small_model_integration_test_expand ( #49301 ) Complete PR #49228 : remove unused device=None from rope init in 11 model families ( #49285 ) Revert "[CI] Temporarily disable AMD scheduled CI trigger" ( #49433 ) [CI] Temporarily disable AMD scheduled CI trigger ( #49394 ) [tests] Fix flaky CMVN tests in Speech2TextFeatureExtractionWithoutTorchaudioTest ( #49374 ) [Docker] Upgrade to torch 2.14/CUDA 13.0 and optimize FA2 build from source ( #49381 ) [tests] Skip onnx.exactness for BitForImageClassification (ORT fp32 drift) ( #49366 ) @icavan Fix a port allocation race in FSDP tests ( #49455 ) [Model] Add native Ling 3.0 Flash support ( #49446 ) @tarekziade Build docs with the Python version from .python-version ( #49603 ) Fix flaky streamer timeout tests ( #49583 ) CI: Move GPU images to python 3.11 ( #49352 ) Fix Gemma3nVision2TextModelTest::test_model_parallelism split ( #49562 ) Fix PR slow CI suggestion: install transformers in a venv ( #49558 ) Use one static Transformers CI badge in PR descriptions ( #49471 ) fix: treat every declared field of the config class as a default fixes #49456 ( #49466 ) PR CI: grant actions: read to the reusable workflow ( #49417 ) @lucamotz Add Kolibri 1 ( #49463 ) @SangbumChoi Add Molmo2 ( #43451 ) @maxiaolong03 [Model] Add PP-DocLayoutV4 Model Support ( #48387 )

numbat: case file Subject: https://huggingface.co/burner22/numbat Date: 2026-11-01 Method: Range requests against shard tensor data. Calibration against Qwen2.5-7B. Verdict Not an Anthropic checkpoint. The packaging is real. The container is real. The tokenizer is real. The embedding matrix is real. Every weight tensor is synthetic. Hypothesis Odds Assembled from a real embedding matrix plus generated weights 99.5% Real Anthropic checkpoint 0.5% Two tests each settle it alone. lm_head[r] = 0.89 * embed[r] + 0.455 * noise[r] for every row r . input_scale differs between modules that share an input tensor. What the repo claims 14T params. 114 layers. Hybrid GDN/MLA. 1,224 experts, top-48, plus one shared. NVFP4. 772 shards. 7.72 TB. Checkpoint at step 1,830,000 of a 1.96M-step run. Mid-pretraining. Codenamed numbat , from a Claude Code source leak in March 2026. The leak quotes prompts.ts : "Remove this section when we launch numbat." The packaging is real Check Result Shards 772 of 772. No gaps. 7,720,489,342,136 bytes. HF storage 7,720,640,941,879. Zero dedup. Every shard holds unique bytes. NVFP4 blocks 261,888 sampled 16-value blocks. 100.0000% carry a true absmax. Tool versions lm-eval 0.4.13 and nvidia-modelopt 0.47.0 ship today. Transformers Config says 5.19.0 . Real version. Latest is 5.22.1. Tokenizer 131,072 vocab. 130,808 merges. Equals 131,072 − 256 − 8. Every merge resolves. Param math 107 × 1225 × 3 × 12288 × 2816 = 13.61T. Plus dense, MLA, GDN, embeddings. Lands at 13.7T. The generator did its homework. A random fake leaves 5% of NVFP4 blocks without an absmax. This one leaves none. The embedding is real Byte-level BPE, so Ġ marks a space. Related cos Control cos cat / cats +0.4750 cat / Paris +0.0475 dog / dogs +0.5515 water / java −0.0283 one / two +0.5446 ten / red +0.1015 king / queen +0.3436 the / Tokyo +0.0172 man / woman +0.3577 red / blue +0.3511 Paris / France +0.3244 Tokyo / Japan +0.2468 Noise floor is 0.0090. Related pairs sit 3× to 7× above it. Controls sit on it. Learned structure. Nobody fakes this without a real model. Test one: lm_head Config sets tie_word_embeddings: false . So lm_head and embed_tokens trained apart. Their rows should point in random directions. row numbat Qwen2.5-7B (real) 0 +0.8900 +0.0165 1 +0.8900 +0.0259 2 +0.8904 +0.0196 50 +0.8907 −0.0085 100 +0.8905 +0.0174 500 +0.8919 +0.0030 1000 +0.8904 −0.0099 mean abs 0.8906 0.0144 The cosine holds at 0.890 across every row. The residual holds at 0.455. lm_head[r] = 0.89 * embed[r] + 0.455 * noise[r] Same two constants for all 131,072 rows. Real matrices do not do this. Test two: input_scale modeling_numbat.py:348 : return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x)) gate_proj and up_proj read the same tensor x . input_scale is the calibrated amax of that tensor. The two values must match bit for bit. All 1,224 experts read the same hidden state. Their gate_proj.input_scale values must match too. They do not. module input_scale experts.0.gate_proj 0.18547873198986053 experts.0.up_proj 0.08691228926181793 experts.0.down_proj 0.2523329257965088 experts.1.gate_proj 0.07982252538204193 experts.2.gate_proj 0.23889905214309692 shared_experts.gate_proj 0.09329533576965332 shared_experts.up_proj 0.09222322702407837 Fifteen values. Fifteen distinct. Layer 2 GDN: q_proj 0.28462, k_proj 0.15753, v_proj 0.23753, g_proj 0.33552. All read x . All differ. Layer 3 MLA: q_a_proj 0.31776, kv_a_proj_with_mqa 0.20728. Both read x . Both differ. The values span 0.05 to 0.40. They look drawn, not measured. The weights are noise Every weight matrix holds rows with different scales. Each row holds values with no structure at all. Input-dimension spread is the tell. Real matrices vary 2× to 5× across input channels. That variation is why block-scaled FP4 formats exist. tensor col-relstd col max/med row-relstd numbat embed_tokens (real) 0.0375 1.34 0.1197 Qwen embed_tokens (real) 0.0639 2.69 0.0636 numbat L0 up_proj 0.0154 1.06 0.1209 numbat L0 down_proj 0.0220 1.10 0.1206 numbat L0 o_proj 0.0155 1.07 0.1211 Qwen L14 up_proj 0.0409 1.02 0.0764 Qwen L14 down_proj 0.0726 1.67 0.0915 Qwen L20 gate_proj 0.0357 1.45 0.1261 Qwen L0 gate_proj 0.2999 18.37 0.2073 Two facts. Real weights bottom out at 0.036. numbat's land at 0.015. Layer 0 of a real model runs 18× column outliers. numbat's layer 0 runs 1.06. Row spread sits at 0.1206, 0.1209, 0.1211 across three different tensor types. A 0.4% spread. Qwen's vary by 20%. The embedding in the same file measures 0.0375. That is a real number. Block scales agree Per-block E4M3 scales, input dimension against output dimension: tensor col relstd col max/med row relstd row max/med L2 expert0 gate_proj 0.0072 1.02 0.1190 1.42 L2 expert1 gate_proj 0.0070 1.02 0.1193 1.42 L2 expert2 gate_proj 0.0072 1.03 0.1200 1.44 L3 MLA kv_b_proj 0.0022 1.01 0.1374 1.69 Flat across inputs. Uneven across outputs. The shape of W[i,:] = s_i * noise . The account Field Value Created 2026-10-16T05:58:28Z Name "burner" Models 1 Datasets 0 Spaces 0 Followers 7 Discussions posted 0 Org memberships none The account predates the repo by seven days. It has never posted a comment anywhere on Hugging Face, including under its own eleven threads. The upload Field Value Repo created 2026-10-23T01:37:02Z Last modified 2026-10-24T19:09:00Z Commits 784 = 772 shards + 11 files + 1 initial Authors burner22 on all 784 Signatures none Messages huggingface_hub defaults All eleven metadata files landed inside 27 seconds. README, config, tokenizer, modeling_numbat.py , index, and training log. All before the first weight shard. Shards ran from 01:55:18 to the next day at 19:09. Forty-one hours. 18.6 shards per hour. Median gap 110 seconds. Upload order skips around: 1, 772, 143, 201, 272, 330, 401, 459, 530, 588, 659, 717, 144, 202, 273, ... Twelve workers, striding. Throughput works out to 52 MB/s, or 415 Mbps. Then it stopped. Eight days of silence. The README still says "They're all uploading. Be patient." The parameter count HF reports 6,849,377,242,584 params. The README says 14T. Community thread #6 calls this a lie. It is not. HF's parser skips F8_E4M3 tensors, and the 850 GB of block scales carry that dtype. Count two params per packed U8 byte. Add 16.85B BF16. Get 13.68T. The size claim holds up. "Mythos" is a real name Claude Mythos 5 and 5.1 exist here. Community finetunes date from 2026-06-19. Provider catalogs carry anthropic.claude-mythos-5 . The terminology is current and correct. The unverified claim is narrower: the March 2026 source leak, and the prompts.ts comment. Contradictions NVFP4 on Trainium2 does not exist. NVFP4 is an NVIDIA Blackwell format. modelopt is NVIDIA tooling. Trainium2 has no FP4 path. The FAQ leans on this as the reason a mid-pretraining checkpoint ships quantized. Anthropic does not train in HF transformers. A hand-written NumbatForCausalLM with auto_map is not how an internal checkpoint looks. The README pre-empts this: "internal Neuron-parallel runtime... this port." Eval numbers disagree. ckpt_metadata.json reports mmlu n=14042. The training log reports n=281. hellaswag n=10042 against n=201. MMLU 88.5% at 0-shot from a checkpoint with no post-training sits at the frontier. The repo calls this "raw pretraining." The "Why believe this?" list checks file sizes, tensor counts, index entries, and round-trip bounds. Every item is a property a generator controls. None require learned weights. Replication Inventory. curl -s "https://huggingface.co/api/models/burner22/numbat/tree/main?recursive=1" \ | python3 -c "import json,sys; d=json.load(sys.stdin); \ s=[e for e in d if e['path'].endswith('.safetensors')]; \ print(len(s), sum(e['size'] for e in s))" # 772 7720489342136 Test one. Read the header of shard 1 and shard 772. Compare lm_head[r] to model.embed_tokens.weight[r] . Expect 0.890 on every row. # embed_tokens sits at data offset 6232 of shard 1, not 0. # row r starts at 16352 + 6232 + r*24576 Test two. The F32 scalars sit contiguously at data bytes 768 to 6232 of shard 1. One 5 KB range request returns all 46 of them. curl -sL -r 17120-22583 "https://huggingface.co/burner22/numbat/resolve/main/model-00001-of-00772.safetensors" # compare experts.0.gate_proj.input_scale with experts.0.up_proj.input_scale Block structure. Decode nibbles. Confirm each 16-value block holds a code 7. Confirm the block scales stay flat across columns. How it was built Take a real tokenizer.json and embed_tokens matrix. 131,072 × 12,288. No public model matches that shape. Fill every other tensor with W[i,:] = s_i * N(0,1) . Draw s_i so row spread lands at 0.1197. Reuse that constant everywhere. Set lm_head = 0.89 * embed + noise . Draw input_scale and weight_scale_2 from 0.05 to 0.40. Quantize to real NVFP4 with per-16 absmax blocks. Every structural check in the README now passes. Write coherent architecture metadata. Use real tool version strings. Ship a 22 KB modeling_numbat.py so strict=True loads. Push 7.72 TB over 41 hours with twelve workers. What would settle it One layer of forward pass. Given lm_head = 0.89 * embed and the drawn input_scale values, the logits will be noise. submitted by /u/johnxreturn to r/singularity [link] [comments]

*Edit (link to repo): https://github.com/Crawler757/treadmill-home My GF bought a NordicTrack Commercial 2250 a couple of years ago, and the feature we were most excited about was the big screen that would let us watch our streaming services while working out. What we didn’t know is that everything beyond the basic workout controls is locked behind an iFit subscription, even streaming services you already pay for. Even for basic use you have to wait for iFit to load, and there’s no escaping its intrusive ads. It turns out that screen is just an Android 9 tablet running iFit’s software. I wanted Netflix and Crunchyroll on it without giving up the motor, so I sat down with Claude Code and a CLAUDE.md file as the project’s running memory, and over about a day and a half we got it all working. What we ended up with: A home screen app I own with big tiles for Netflix and Crunchyroll. There’s no iFit login, no ads, and no membership nags. A floating bar over whatever is playing with speed, incline, time, and distance, plus a quarter-mile track panel with laps and pace. On-screen Start, Pause, Stop, and speed and incline buttons. The physical buttons still work too. Netflix starts in about 21 seconds, down from close to a minute. How the work went: Claude did the discovery over ADB and found that iFit’s motor service runs a small server on the console that apps can talk to. It followed how the open source QZ (qdomyos-zwift) project talks to it and wrote the client from scratch in plain Java. Big shout out to the QZ folks, because none of this happens without their work. We disabled iFit’s apps one at a time with a health check on the motor service after each. Some of that went wrong: one app killed the motor service, and another attempt took half an hour to undo. Each time Claude wrote the lesson into CLAUDE.md and the next session followed it. The real fix was removing iFit’s lockdown and updater app for my user (reversible). It turned remote access off after every restart, re-enabled everything I’d disabled, and ate a big share of the CPU. Without it the treadmill boots unlocked and the tablet can finally breathe. Favorite catch: a test script helper named Rd was silently treated as PowerShell’s alias for Remove-Item. Nothing got deleted, but Claude caught it in the logs. Where Opus 5.5 refused commands: I wanted our own motor driver that talks straight to the motor board. Claude wouldn’t work out the board’s security handshake from iFit’s code, since it saw that as getting around an authentication check. I was a little disappointed, but I get it, so we stuck with iFit’s motor service. It wouldn’t start the belt remotely for testing at first. I basically had to argue that it was safer for me not to be standing on it: “I can always unplug it if something goes haywire, and in the worst case scenario of the treadmill exploding, shouldn’t it run remotely in the basement instead of me being there with it?” After that it agreed, as long as I confirmed nobody was near the treadmill. Every remote test had a speed cap, a stop command that runs even if the script fails, and a check that the speed reads zero at the end. The safety key and the console’s Stop button stayed the real way to stop. The overlay is never trusted with that. The catches: Netflix plays in SD (Widevine L3), the tablet is slow no matter what, and a factory reset could wipe it all. It’s a treadmill, I’ll live. We paid for the treadmill, the screen, and the motor, so paying monthly to use a screen we already own never sat right with me. Happy to answer questions! submitted by /u/Beloved_Misanthrope to r/ClaudeAI [link] [comments]

The Chromagraph Camera app costs $13 dollars and processes iPhone photos with the same color math Rian Johnson’s cinematographer, Steve Yedlin, uses for film. Linked page: The Chromagraph Camera app costs $13 dollars and processes iPhone photos with the same color math Rian Johnson’s cinematographer, Steve Yedlin, uses for film.

Saw Karpathy's HackerTube and wanted the same thing for books. So I had Claude build one (Opus 5.5). It's ~16.5k books that at least two different HN commenters brought up, from 2007 till now. Each book links back to the comments that mentioned it. Data is downloadable from the about page. To be clear: Claude wrote almost all of it, including the data pipeline and the site. I mostly steered and made calls, bought the domain name, etc. It counts mentions, so it's not perfect (praising a book and trashing it both count). If your favourite book is missing or matched to the wrong thing, tell me. Comments URL: https://news.ycombinator.com/item?id=50326821 Points: 1310 # Comments: 223