01Case study
Paper Radar
Reads every new arXiv paper each morning and surfaces the few you should actually read.
- $0.06a day to read all of arXiv (~1,500 papers)
- 96.9%recall on 4 unseen Cochrane reviews
- 78%less screening work
- 7public radars running every weekday
The problem
arXiv now receives more than 30,000 papers a month (32,040 in June 2026), about 1,500 per weekday announcement. Nobody reads the listing any more, and the usual fixes each have a blind spot:
- Keyword alerts miss papers that use different words.
- Embedding recommenders quietly drop whatever doesn't look like your past reading.
- Running a chat LLM over the whole firehose every day is slow and expensive, so most tools only look at a pre-filtered sample.
The idea: judgements, not summaries
Most AI tools read a paper and write you a summary. Paper Radar writes nothing. It asks the model one yes/no question per interest and gets a calibrated probability back, such as introduces an agent benchmark: 0.97, and your code decides what that means.
A judgement costs a fraction of a generated sentence and leaves nothing to parse. That makes it cheap enough to read all 1,500 papers instead of a pre-filtered handful, which is exactly where keyword alerts and recommenders lose the paper you actually wanted.
Architecture
Key design decisions
Atomic questions, composed in code
Each interest is its own yes/no question. Relevance is max(weight × p) by default, or noisy-OR so papers matching several interests rank higher.
One request per paper
Every question for a paper goes into a single call, so more interests add tokens but almost no latency.
Retrieve, then judge
The model sees only the title, abstract and categories. Authors and affiliations don't bear on the topic, so they are deliberately left out.
A cost cascade
The judging model filters thousands of papers; an optional generative model writes one sentence for only the ten that made the cut.
Thresholds you can verify
calibrate computes the Brier score, ECE and a precision/recall table from your own labels and fits thresholds to them, instead of trusting vendor numbers.
One-click feedback
Every paper carries 👍 / 👎 links that open a pre-filled GitHub issue; the next run folds them into labels.jsonl and closes the issues.
Extension: systematic-review screening
A review's title-and-abstract stage has the same shape as a daily radar but the opposite logic: a record is eligible only if every inclusion criterion holds, and one exclusion criterion disqualifies it. So screening mode scores with the geometric mean of the inclusion criteria, keeps every record it ever screened, and prints the PRISMA 2020 counts and WSS (work saved over sampling) the field uses.
A counter-intuitive finding
Splitting compound criteria into more of them took one benchmark topic from 10.0% work saved to 0.1%. Screening is a conjunction, so each extra criterion is one more chance to veto a record. Write the fewest criteria an abstract can actually answer.
Results and validation
Validated on Cochrane reviews from CLEF eHealth 2019, against reviewers' real title-and-abstract decisions, with criteria quoted from each review's published selection criteria. These four were never looked at during development:
| Held-out review | Records | Eligible | Recall | Work saved | Cost |
|---|---|---|---|---|---|
| Antihypertensives: RAS inhibitors vs other classes | 12,319 | 88 | 95.5% | 68.1% | $0.37 |
| Thromboprophylaxis implementation | 3,574 | 11 | 100% | 95.1% | $0.11 |
| Cyclodestructive procedures for glaucoma | 2,456 | 12 | 100% | 95.8% | $0.08 |
| Psychological therapies for depression in COPD | 1,098 | 16 | 100% | 92.5% | $0.03 |
| Total | 19,447 | 127 | 96.9% | 78.0% | $0.60 |
The eight reviews used during development did far worse (38–48% pooled). Performance is dominated by the review, not the tool: the held-out set is large and low in prevalence (0.3–1.5% eligible), which is what a real search looks like. All three rounds of pre-registered predictions, wrong ones included, are published in the repo.
Day to day: 50 new papers judged in 5 seconds for $0.002 (about 932 input tokens per paper); 501 papers in 33 seconds for $0.02.
Honest limits
- Abstracts only, not full papers.
- Not a replacement for a human screener: use it as a second screener or to set the order a human reads in.
- Jev launched in September 2026 and its benchmark numbers are the vendor's own, so pin the model version and calibrate.