2026-10-22

I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting. Instead of shrinking the model or skipping experts, they tackled the real bottleneck: getting expert weights from flash storage to computation fast enough. OpenBMB’s MiniCPM-V 4.7 uses a technique called Lookahead Expert Scheduling (LES) to run a 35B-A3B MoE model locally without changing its actual routing or dropping experts. The key idea: predict which experts the next layer might need, preload their weights from flash storage while current computations run, and execute ready experts first. Combined with optimized weight layouts, mixed-precision quantization, and DSpark speculative decoding, LES achieves: 20.51 tok/s on iPhone 18 Pro Max 6.07× faster than the 3.38 tok/s baseline 26.7% smaller expert weights with only a 0.16-point drop across five benchmarks Around 3 GB peak memory in a low-memory configuration The implementation, including expert scheduling and DSpark integration, is open source in llama.cpp-omni . The interesting part is that LES hides memory bottlenecks without changing which experts the model actually executes. This approach could potentially work with other MoE models, too. submitted by /u/GasSmooth7439 to r/LocalLLaMA [link] [comments]

Reposting again since original post got removed by Reddit for some reason (I have no idea why) A few of you pointed out in my last post that comparing Flash-Next and Opus with different harnesses and effort levels wasn't fair. So here is a more controlled comparison. Same task for every run: a real feature from the LlamaStash roadmap (save llama-server's KV cache before an unload and restore it when the model comes back). Six combinations, each run twice, so 12 PRs. Flash-Next ran locally on my Strix Halo laptop (Flow Z13) via Halogen (~Q4 quant). Opus 5.5, Flash-Next, and Kimi K3 scored every PR on seven review questions plus CI. Combination Score (of 8) Time API cost Tokens (in / cached / out) Claude Code + Opus 5.5, xhigh 6.94 (87%) 43.5 min $17.78 507k / 48.3M / 203k Claude Code + Opus 5.5, medium 6.25 (78%) 23.4 min $6.52 242k / 13.4M / 95k Claude Code + Flash-Next, medium 5.5 (69%) 45.3 min $0.32 445k / 13.5M / 82k Pi + Flash-Next, xhigh 5.44 (68%) 63.8 min $0.67 316k / 35.0M / 134k Claude Code + Flash-Next, xhigh 5.38 (67%) 49.5 min $0.39 269k / 18.9M / 99k Pi + Flash-Next, medium 5.06 (63%) 48.1 min $0.56 242k / 29.1M / 117k Scores are the average of both runs. Time, cost and tokens are from the second run, where the 100k-token benchmark numbers were given in the prompt. Flash-Next cost is at the QwenCloud price for Qwen3.8-Flash; I ran it locally, so it cost me only electricity. TL;DR All 12 runs finished and all 12 PRs pass the end-to-end test. Opus wrote better code, 82% on average against 68% for Claude Code + Flash-Next and 66% for Pi + Flash-Next. The difference is in edge cases, tests and docs. The core feature works fine. The gap between Opus and Qwen is not that wide. Flash-Next took 1.1x to 2.1x as long as Opus at the same effort. With Flash-Next, Claude Code and Pi scored the same on the review questions. CI passed on 4 of 4 Claude Code PRs and 1 of 4 Pi PRs. xhigh helped Opus a bit at 2.7x the cost. For Flash-Next, medium and xhigh scored about the same, and medium was faster in all 4 pairs. Opus is still ahead, but Flash-Next was right behind, on my laptop, for the cost of electricity. Also note that the models are not on the same league. Opus is a frontier model run at FP16 or atleast FP8. I ran Flash-Next on my laptop at 4.16 bpw . Flash-Next at FP8 should show a bit more quality. The machine running Flash-Next ran at 70W TDP ceiling for the entire duration. I could have run it at 90W for slightly better speed but the fan noise and thermal stress was not worth it. Full write-up with the detailed methodology and the per-run results and what each model missed: https://deepu.tech/qwen3.8-flash-next-vs-claude-opus-showdown All 12 PRs are public if you want to read the code: https://github.com/deepu105/llamastash-b4r2/pulls submitted by /u/deepu105 to r/LocalLLaMA [link] [comments]

I saw this today on X and decided to try it, I havent seen it posted here yet. I ran it on a single 3090 benchmarked it with qwen 3.8 27B before and after the optimization and saw these results: Observed prompt-processing throughput improved 22–30% ; generation speed improved 1–5% . All tests passed without GPU errors. Edit: Keep in mind performance gains can be higher, my 3090 is power limited to 230 watts including during my tests submitted by /u/Striking_Luck_886 to r/LocalLLaMA [link] [comments]

Recursive Flint Flint brings Recursive's coding model to Ollama. It is designed for the work coding agents do across a repository: understanding unfamiliar code, tracing bugs, implementing features, and adding tests. Reasoning, multi-turn tool use, and a 262K-token context window support longer tasks across files and tool interactions. Flint is an Apache 2.0-licensed, 40B-parameter mixture-of-experts model with approximately 2.5B parameters active per token. In its release evaluations , Recursive reports 54.9% on Terminal-Bench 2.1 and 74.2% on SWE-bench Verified. ollama run flint Note: Flint is supported on MacOS. Other GPU support will be enabled in a subsequent release. Full Changelog : v0.40.7...v0.40.8

Bria released Fibo 2.0 yesterday and I wanted to see how it does with a character LoRA and create something that I can actually post (all my other datasets are not royalty-free). The dragon is a plush mascot I use for my LoRA portfolio. 15 training images. What I used: Bria's train_lora.py from the Fibo-2.0-dev repo 1024px, rank 64, alpha 64, lr 1e-4 constant, AdamW, batch 1, grad accum 2 400 steps, the likeness was already there at 200, and 300 was the best (this might hugely differ for human characters but I did not want to do a synthesized human) Rented an A100 40GB on Vast. It peaked at about 23 GB VRAM, ran at 6.2 s/step, and training took about 42 min. Each edit took about 80-90 seconds About $2 in GPU time in total The lora works really well with a plain edit instruction and doesn't change much else If you add a reference image from the dataset, it can give better results (It drew the outfits like the space suit.) but it often changes the whole picture On Abbey Road I asked it to add the dragon as a fifth person but it just replaced one of the Beatles (I don't know which one, sorry.) I kept it because I found it funny. The full settings and VRAM numbers are on my tracker, link in the comments. Fully free, no ads, just my referral links. submitted by /u/SirToki to r/StableDiffusion [link] [comments]

Memory-native evolution requires making memory an intrinsic capability of foundation models, enabling them to continuously accumulate and utilize information. To this end, we introduce MemT-1.0, a new architecture of memory foundation models that internalize memory as part of model computation. Building on Metis, MemT-1.0 employs Gated DeltaNet-2 with a stable token-level memory update mechanism within each turn. We further introduce Mixture-of-Storage to scale native memory capacity. Following a deliberate backbone preparation, we develop a two-stage training pipeline consisting of continued pre-training and post-training, supported by 34B tokens of training data and efficient training and inference infrastructure. We release a model family comprising MemT-1.0-Nano and MemT-1.0-Flash, and conduct a comprehensive evaluation of memory and general capabilities. The results show that they outperform the parametric memory baselines on average and remain competitive with retrieval-augmented methods, while largely preserving general capabilities. At a history length of 128K tokens, MemT-1.0-Nano achieves approximately 1.86× end-to-end inference speedup and a 66× reduction in persistent storage relative to full context KV cache . By internalizing retrieved procedural experience, MemT-1.0 improves task success on agentic tasks, providing a practical step toward memory-native evolution. arXiv : https://arxiv.org/abs/2610.26825 PDF : https://arxiv.org/pdf/2610.26825 GitHub : https://github.com/MemTensor/MemT-1.0 HuggingFace : https://huggingface.co/collections/MemTensor/memt-1 submitted by /u/pmttyji to r/LocalLLaMA [link] [comments]

KittenML has just released code and weights of Kitten-TTS 2, which is a super-efficient frontier TTS model, and the world's first 1.58-bit TTS model . It runs in real time on just a CPU, with zero-shot voice cloning, multilingual support, and expressive / emphasis control. Code and weights are open-source. Ternary(1.58-bit) Weights means it can run locally or be self-hosted for super-cheap. Github: https://github.com/KittenML/KittenTTS Hugging Face: https://huggingface.co/KittenML/kitten-tts-2 KittenML's Discord: https://discord.com/invite/VJ86W4SURW Key features and Advantages Ternary TTS: Every linear weight in the body is one of three values (-1, 0, +1) with group size of 128 for scales. Tiny footprint and fast CPU inference. The model has 1.7B parameters. Real-time on CPU and consumer GPU's too : No GPU needed for real-time generation. Optimized ternary kernels for CPU inference provided. Expressive voice cloning : zero-shot cloning from less than 5 seconds of reference audio, and it keeps the expressivity, accent information etc. No fine-tuning needed. Multilingual TTS: Model supports voice cloning in 20+ languages. Cross-language voice cloning can be used to create accented speech too. Expressive / emphasis control: you can steer the delivery and stress specific words, not just pick a voice. Details in the Readme. Open source (hell yeah!) : code and weights are open-source and free to use. It's an oss alternative to elevenlabs for gpu-poor folks. easy to self-deploy or run locally. 7. What changed from V0.8 to 2: higher quality. Completely new architecture, natural sounding speech, ternary weights, zero-shot voice cloning, multilingual, and emphasis/emotion control. submitted by /u/ElectricalBar7464 to r/LocalLLaMA [link] [comments]

Article URL: https://duckdb.org/2026/10/21/announcing-duckdb-200 Comments URL: https://news.ycombinator.com/item?id=50181136 Points: 395 # Comments: 52

Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading. Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns. This provides accurate multi-step lookahead over future expert usage, allowing the runtime to organize expert movement and execution ahead of verification and improve both communication and execution efficiency. Under single-request inference, Specter achieves an average 1.50x (up to 1.88x) speedup in time per output token (TPOT) over the state-of-the-art system. Specter makes memory-constrained, single-GPU MoE inference more practical on commodity hardware . Our code is available at this https URL . arXiv : https://arxiv.org/abs/2511.14102 PDF : https://arxiv.org/pdf/2511.14102 GitHub : https://github.com/wwwwwf/specter_asplos submitted by /u/pmttyji to r/LocalLLaMA [link] [comments]

I run a small exam-prep site for pilot students in India and have no CS background. Over the last few months I’ve been building this with Claude Code: a 3D model of Delhi airport that shows the real traffic live. What it does: Real flights landing, taxiing and taking off, pulled from from licensed , self hosted and public sources Tap any aircraft to follow it, or sit in the cockpit Aircraft are drawn in their real airline colours Mumbai, Bengaluru and Hyderabad are live too, and it now tracks traffic about 50 km out How Claude helped: I’d describe what I wanted in plain English (“the plane should slow down on the runway like a real one”, “the gear should fold into the wing properly”) and Claude wrote the code, set up the server, and built checks so a new version can’t go live if it breaks something. The hardest parts were making taxiing look natural and keeping it fast on a normal phone. It’s free, no sign-up: airport.readyaviator.com Happy to answer questions about how it was built. submitted by /u/ReadyAviatorOfficial to r/ClaudeAI [link] [comments]

Jacob Cooper / Railway Blog : Cloud provider Railway says an unauthorized party may have accessed customer encrypted environment variables and, for some, obtained the decrypted contents — Earlier this morning, Railway detected anomalous activity involving systems that store customer environment variables.

Hey r/SaaS , For the past 7 months, I’ve been building RPGMapEditor.com - a web-based map editor for tabletop RPGs and D&D players. Up until august, the project was generating $0. I had built free generators (fantasy maps, dungeons, cities) which started bringing in organic search traffic (around 1,000+ monthly clicks from Google), but nobody was converting to the paid tier. 2 weeks ago, I redesigned the conversion flow between the free tools and the premium editor ($3/mo tier & €14.99 lifetime license). Here is what happened in October: - 📈 MRR: €1.000+ (First recurring subscribers!) - 🔍 100% Organic traffic (0 spent on ads) all comes from Bing, and Google Search! Key takeaways that moved the needle: Free tools are great for SEO, but you MUST give users a clear reason to save/export in the editor. Micro-pricing ($3/mo) works insanely well for hobbyist niches (D&D/gaming) where people hesitate to spend $20+/mo. Organic search traffic compound interest is real – it took 3 months of low clicks before Search Console shot up. It’s small money, but going from absolute zero to actual validated revenue feels incredible. Happy to answer any questions about the tech stack, SEO strategy, or conversion tweaks! submitted by /u/ExistentialHugger to r/SaaS [link] [comments]

Article URL: https://docs.gitlab.com/releases/patches/patch-release-gitlab-19-5-1-released/ Comments URL: https://news.ycombinator.com/item?id=50185869 Points: 305 # Comments: 73

Article URL: https://www.php.net/index.php Comments URL: https://news.ycombinator.com/item?id=50180943 Points: 276 # Comments: 2

Release v5.19.0 New Model additions EmbeddingGemma2 EmbeddingGemma 2 is a multimodal embedding model from Google built on the Gemma 4 architecture. It encodes text, images, audio, and video, individually or combined in one input, into a shared 768-dimensional vector space for cross-modal retrieval, semantic similarity, clustering, and classification. It uses Matryoshka Representation Learning, so embeddings can be truncated to 512, 256, or 128 dimensions. It also offers configurable visual and video token budgets, and unused vision or audio towers can be disabled at load time to save memory. Links: Documentation Smthn smthn ( #49364 ) by @vasqu in #49364 Breaking changes All MoE models whose routers compute logits now return them when output_router_logits=True , following the Qwen3-MoE pattern (a router_logits recorder on the base model, MoeModelOutputWithPast from the backbone, and a MoE causal LM output from the head), so code that relied on the previous outputs or their absence should read the router logits from these output classes. 🚨 Return router logits from every MoE model that computes them ( #48920 ) by @qgallouedec Owlv2ForObjectDetection.embed_image_query now selects the query box with the highest objectness score, as in the original OWLv2 notebook, instead of the OWL-ViT heuristic, so image-guided query embeddings and detections may differ from earlier releases. 🚨 Select the OWLv2 image query by objectness ( #49200 ) by @qgallouedec The "paged|" prefix for SDPA and flash attention implementations is deprecated, so users should set the regular attention implementation (e.g. sdpa or flash_attention_2 ) for continuous batching instead of paged|sdpa or paged|flash_attention_2 . 🚨 Attention 🚨 Deprecate "paged|" prefix for SDPA and flash ( #49112 ) by @remi-or The regular flash and SDPA attention functions ( flash_attention.py , sdpa_attention.py ) now support continuous batching directly, and "paged|..." implementations for these are redirected to them, while eager still requires the "paged|eager" prefix. 🚨 Attention 🚨 Make regular attention support CB ( #49101 ) by @remi-or In continuous batching, the cache update for the index-based and block-table paths is now fused into a single call, which slightly changes the cache update function's behavior and affects any custom code that calls the separate update paths. 🚨 [CB] 🚨 Fuse update for index and block table path ( #49088 ) by @remi-or Continuous batching internals changed in preparation for removing "paged" : `max 🚨 [CB] 🚨 Little fixes before removing "paged" ( #49069 ) by @remi-or Parallelization Expert parallelism gains a token-dispatch implementation, selected via the new ep_dispatch_experts plan rule and now the default for Qwen3 MoE and Mellum, which removes the requirement that EP size equal TP size. The Trainer was also adapted to work with expert parallelism, and the docs now note that PEFT adapters support tensor parallelism. A CI-related fix for pipeline-parallel chart2table inference was also included. [ distributed ]: adapt Trainer to work Expert Parallel ( #48873 ) by @3outeille in [ #48873 ] [ distributed ] Add expert-parallel token dispatch, default for Qwen3 MoE ( #48865 ) by @3outeille in [ #48865 ] Fix pp chart2table inference ( #49225 ) by @zucchini-nlp in [ #49225 ] [docs] TP for PEFT adapters ( #49060 ) by @stevhliu in [ #49060 ] Cache This release fixes quantized cache handling: generate no longer mutates the user's cache_config , and QuantizedLayer.reorder_cache is repaired. It also adds per-layer cache configuration, so DynamicCache and StaticCache initialize each layer from its own config (sliding window, attention chunk size, conv states, and attention head counts) to better support heterogeneous models. Separately, the deprecation cycle on mask and cache Fix quantized cache ( #48700 ) by @jiqing-feng in [ #48700 ] [CI] check_bad_commit: use EFS cache to avoid Xet FUSE OOM (exit 137) ( #49273 ) by @ydshieh in [ #49273 ] Support per-layer cache configuration ( #48178 ) by @eladsegal in [ #48178 ] Remove deprecation cycle on mask and cache ( #49231 ) by @Cyrilvallez in [ #49231 ] Bugfixes and improvements Revert "[CI] Temporarily disable AMD scheduled CI trigger" ( #49271 ) ( #49362 ) by @ydshieh in [ #49362 ] ALM testx fIxing... ( #49355 ) by @zucchini-nlp in [ #49355 ] Fix DataCollatorForLanguageModeling ignoring seed=0 ( #49349 ) by @akanyaani in [ #49349 ] Fix explicit kernelization modes for evaluation models ( #49339 ) by @DimensionSTP in [ #49339 ] fix(cohere_compass): add video to input_modalities ( #49320 ) ( #49323 ) by @destopianpirate in [ #49323 ] GPT-OSS: read the clamped SwiGLU's alpha and limit from the config ( #49346 ) by @IlyasMoutawwakil in [ #49346 ] [ distributed ] Default MoE ep_plans to token dispatch ( #49160 ) by @3outeille in [ #49160 ] Move CI to Python 3.11, with the version set in .python-version ( #49311 ) by @tarekziade in [ #49311 ] Bump doc-builder main docs workflow pin ( #49350 ) by @paulinebm in [ #49350 ] [ distributed ] decoupled tp_plan and ep_plan ( #48859 ) by @3outeille in [ #48859 ] [ distributed ] Add dense and expert device mesh ( #48857 ) by @3outeille in [ #48857 ] Bump doc-builder workflow pin ( #49342 ) by @paulinebm in [ #49342 ] Allow registering new processing backends ( #48984 ) by @zucchini-nlp in [ #48984 ] Fix WatermarkDetector repeated-ngram counting ( #49315 ) by @LE0-Lin in [ #49315 ] Fix minicpm 4.6V video batching and CI ( #49259 ) by @zucchini-nlp in [ #49259 ] remove colwise_gather_output for lm_head for DeepseekV4 ( #49314 ) by @3outeille in [ #49314 ] Fix the labels docstring of CLIPSegForImageSegmentation ( #49001 ) by @qgallouedec in [ #49001 ] [CI] Temporarily disable AMD scheduled CI trigger ( #49271 ) by @ydshieh in [ #49271 ] Remove the unused pandas entry from setup.py deps ( #49329 ) by @tarekziade in [ #49329 ] Fix run_glue.py label ids becoming strings with pandas>=3 ( #49326 ) by @tarekziade in [ #49326 ] [Fix] Remove old test file with two ancient tests ( #49280 ) by @remi-or in [ #49280 ] Fix chunked prefill with multi-axis position ids (Qwen3-VL) ( #49319 ) by @dacorvo in [ #49319 ] Why override when you can fix in Mixin ( #49105 ) by @zucchini-nlp in [ #49105 ] Do RoPE with mul instead of matmul ( #49265 ) by @Rocketknight1 in [ #49265 ] Deprecated pipelines should redirect; removed pipelines should raise ( #49308 ) by @LysandreJik in [ #49308 ] Enable model test on upcoming torch_tpu backend ( #49207 ) by @tengomucho in [ #49207 ] Fix encoder repetition penalty source rows after batch expansion ( #49262 ) by @gitedmond in [ #49262 ] Preserve masked EOS scores in exponential decay length penalty ( #49260 ) by @gitedmond in [ #49260 ] [CI] Avoid fetching all artifacts when only specific ones are needed ( #49282 ) by @ydshieh in [ #49282 ] Revert "[CI] ssh-runner: add optional cache_type input to switch between bucket and EFS runners ( #49159 )" ( #49270 ) by @ydshieh in [ #49270 ] Add Trainer.loss_is_scaled_for_ga to declare whether compute_loss already scales for gradient accumulation ( #49240 ) by @qgallouedec in [ #49240 ] fix precomputed flash kwargs clashing ( #47928 ) by @IlyasMoutawwakil in [ #47928 ] Reapply modular examples ( #49230 ) by @Cyrilvallez in [ #49230 ] Remove some stuff that does not exist ( #49232 ) by @Cyrilvallez in [ #49232 ] Remove rotaries deprecation cycle ( #49228 ) by @Cyrilvallez in [ #49228 ] [ GTE ] Add memory mixin ( #49234 ) by @vasqu in [ #49234 ] [CB] Make CB more device agnostic and add support for XPU ( #49156 ) by @remi-or in [ #49156 ] correct final metrics due to resuming from checkpoint ( #49208 ) by @SunMarc in [ #49208 ] [ESM/Persimmon/NllbMoe/ESMFold] use safetensors repos to fix Xet FUSE SIGBUS/IOError on bucket CI runners ( #49194 ) by @ydshieh in [ #49194 ] Dev update ( #49212 ) by @vasqu in [ #49212 ] Significant community contributions The following contributors have made significant changes to the library over the last release: @vasqu Smthn smthn ( #49364 ) [ GTE ] Add memory mixin ( #49234 ) Dev update ( #49212 ) @remi-or [Fix] Remove old test file with two ancient tests ( #49280 ) [CB] Make CB more device agnostic and add support for XPU ( #49156 ) 🚨 Attention 🚨 Deprecate "paged|" prefix for SDPA and flash ( #49112 ) 🚨 Attention 🚨 Make regular attention support CB ( #49101 ) 🚨 [CB] 🚨 Fuse update for index and block table path ( #49088 ) [Refactor] Make some flash-attention utils more readable ( #49071 ) 🚨 [CB] 🚨 Little fixes before removing "paged" ( #49069 )

Panache — RTS Belle Époque pour mobile Well this is delightful: I released Battle Drones yesterday under an Apache 2.0 License and there's already a French language fork which reskins the game to take place in steampunk 1900s France and adds an animated intro! Panache est un jeu de stratégie en temps réel pour mobile et navigateur, dans la France de la Belle Époque (vers 1900) : dirigeables, Fardiers à vapeur, Creusot cuirassés, automates en képi, Gargantua la forteresse marchante — et beaucoup de panache. You can play Panache here - it's worth clicking that link just to experience the new intro video. So far it's mostly just aesthetic updates as opposed to gameplay changes. The music has had a French twist added to it as well. The new steampunk models are so cool! Here's a page with the original Battle Drones models and that same page with the new Panache alternatives . This is a comparison of the original and the new design for the Goliath: Via @Meffysto69 Tags: game-design , open-source