AgentBench: Evaluating LLMs as Agents
Direct match: evaluates language models as agents in interactive environments.
A research habit, with less sorting
Describe your research interest or share paper links. Let the skill retrieve public titles and abstracts, then get a ranked page, reasons and files you can keep.
MIT licensed · No separate API key for conversation mode · Normal Claude plan usage applies
RESEARCH INTEREST
LLM agent benchmarks and tool use
Direct match: evaluates language models as agents in interactive environments.
Related method: teaches models to use tools, rather than introducing an agent benchmark.
Saved test with two real papers. Claude used labelled web summaries after the metadata API was blocked. Scores are self-reported, uncalibrated and may vary; this page does not search or rank live.
Start where you are
See the saved two-paper real test, including its sources and retrieval limitation. The 44-paper fictional demo is also available. Viewing needs no installation or key.
Browse the offline demo →View the real-paper test →Upload and enable skill 0.2.0, then give a research interest or arXiv/PubMed/DOI links. Public retrieval needs network access or existing Claude web tools. The conversation ranks and exports the results.
Install and get a prompt →Subscribe to the existing public Jev feeds, or configure your own daily API workflow. Scheduling and external API judging require separate setup.
Read the public Jev feeds →Fork a Claude API workflow →From download to first result
This page only prepares a prompt; it sends nothing. In Claude, public retrieval sends the query or identifiers to metadata sources or permitted web tools. Missing abstracts and failures are disclosed.
Keep the result, not just the chat
Scores, reasons and abstracts in one place.
CSVA working tableKeep, annotate and compare in your spreadsheet.
JSONA traceable recordPreserved IDs, scores and input provenance.
RSSA feed to publishShortlisted papers, ready for hosting.
Links above open the saved real-paper test, using labelled web summaries. HTML/CSV/JSON retain all scored papers; RSS includes must-read and maybe. The original fictional report remains available in the repository.
Before your first run
No separate model API key for conversational retrieval, ranking or exports. Normal Claude plan usage applies. Retrieval makes public metadata requests and needs permitted network or existing web tools. Scheduled model API judging is separately billed.
No. You can request a single search in the current conversation. Daily schedules, long-term storage and publishing still need separate setup.
No. Claude's scores are self-reported and uncalibrated. Label selected and unselected papers yourself, fit thresholds on one batch and evaluate on a separate later batch.
Claude web Opus 5.5 loaded the bundle, checked its file hashes and exported reports. The 0.2.0 link test retrieved two real papers through existing Claude web tools after the metadata API was blocked; summaries were labelled. Local tests: 250; Python 3.11–3.13 CI passed. Functional checks do not establish accuracy or every-client compatibility.
Your environment, package version, the step that failed and a redacted error. Do not post keys, private abstracts or unpublished research. A first-run report is more useful than a star.
Help shape the next version
Report a successful trial or a blocker: your field, the route you tried and whether the output helped. You choose what to share; the page sends no analytics.