LLM agent 評測基準與工具使用:連結排序

Interest: LLM agent 的評測基準(benchmarks)與工具使用能力(tool use)

must_read · 0.93

AgentBench: Evaluating LLMs as Agents

直接命中「LLM agent 評測基準」:提出 8 個環境的 agent benchmark,比較商用與開源模型並分析失敗原因;工具使用僅隱含於環境互動,未專門評測 API 呼叫。

Source: claude-web · Retrieved: 2026-11-13T19:11:30+00:00 · web-summary

Abstract

[Web summary, paraphrased by the fetch tool; not the original abstract] Introduces AgentBench, a benchmark of eight environments testing how well LLMs act as agents in reasoning and decision-making tasks. Top commercial models perform strongly while many open-source models up to 70B lag well behind. Failures are attributed mainly to weak long-term reasoning, decision-making and instruction following; better instruction following and high-quality multi-round alignment data may help, and training on code has mixed effects across agent tasks. Datasets, environments and the evaluation package are publicly released.

maybe · 0.74

Toolformer: Language Models Can Teach Themselves to Use Tools

高度相關於「工具使用能力」:自監督學習何時、如何呼叫計算機、搜尋、翻譯等 API;但它是訓練方法而非 agent 評測基準,也非多步 agent 設定,故列為 maybe。

Source: claude-web · Retrieved: 2026-11-13T19:11:30+00:00 · web-summary

Abstract

[Web summary, paraphrased by the fetch tool; not the original abstract] Language models solve new tasks from few examples but struggle with arithmetic and factual lookup. Toolformer teaches itself, self-supervised with a handful of demonstrations per API, to decide which tools to call, when, with what arguments, and how to use results. Tools include a calculator, a Q&A system, two search engines, a translation system and a calendar. It improves zero-shot performance on many downstream tasks, often matching much larger models, while keeping language modeling ability.