Every new paper was read against your interests. These are the ones worth your time.
Offline demo. These papers are fictional samples and the scores come from a keyword heuristic, not Jev. Add a TypeSafe or OpenRouter key to run the real model.
We review methods for uncertainty estimation in large language models, covering verbalized confidence, sampling consistency, and probe-based approaches. We organize the literature by the kind of prediction being scored and discuss open problems in calibration for long-form generation.
Retrieval-augmented generation pipelines typically rerank a fixed number of passages. We show that retrieval quality for question answering depends strongly on query difficulty, and train a lightweight controller that chooses retrieval depth per query. The controller cuts reranker calls by 58% while improving answer accuracy on four question answering benchmarks.
Mira Okafor, Tomás Lindqvist, Hae-won Seo et al. · cs.CL, cs.AI
Agent evaluation 95%
Abstract
Benchmarks for LLM agents usually score only the final answer of a multi-step episode. We introduce TraceGrade, a benchmark of 1,240 multi-step tool-use tasks across web, shell, and spreadsheet environments in which every intermediate action is verified against an executable checker. Evaluating 14 LLM agents, we find that final-answer accuracy overstates step-level correctness by up to 31 points. Code and tasks are publicly released.
Jonas Brandt, Aiko Tanaka, Lucía Ferrer · cs.CL, cs.LG
LLM calibration 95%RAG & retrieval 35%
Abstract
Language model predictions are often overconfident, which makes abstention unreliable. We propose a conformal procedure that turns token-level signals into calibrated uncertainty estimation for open-domain question answering. Across five datasets, our method reduces expected calibration error by 43% and lets the language model abstain with a guaranteed error rate.
We argue that evaluating LLM agents by success rate alone hides the cost of multi-step tool-use. We propose reporting success per dollar and per minute and show how rankings on popular agent benchmarks change under cost-aware metrics.
We study whether LLM agents can evaluate their own multi-step tool-use trajectories. Agents' self-reported confidence is poorly calibrated: across three agent benchmarks, trajectories rated 90% likely to succeed succeed only 61% of the time. A simple uncertainty estimation probe on hidden states recovers much of the gap.
We release SciRetrieve, a benchmark of 9,800 expert-written scientific questions with passage-level relevance labels, to measure retrieval quality for retrieval-augmented question answering. Dense retrievers and rerankers still miss a supporting passage for 27% of questions.
Long-context LLM inference is dominated by KV cache memory. We introduce a 4-bit quantization scheme for the KV cache with per-channel outlier handling that keeps perplexity within 0.1 of full precision and enables 2.6x cheaper inference at 128k context.
Wei Zhang, Olivia Grant, Rafael Moreno et al. · cs.CL, cs.LG
Efficient inference 89%LLM calibration 35%
Abstract
Speculative decoding accelerates LLM inference by verifying draft tokens in parallel. We propose SpecTree, which drafts a tree of continuations and prunes it with a learned acceptance model. SpecTree delivers 3.1x faster decoding than autoregressive sampling with identical outputs, and composes with quantization for cheaper inference. Code is available.
Speech language models generate discrete units slowly. We adapt speculative decoding to unit sequences and obtain 2.4x faster inference with no loss in speech quality.
arxiv:2609.99044
Near misses (8): just under your maybe bar, useful for tuning thresholds