appbench
Give it an app description and it one-shots that app with several agent frameworks — raw Anthropic API, Claude Agent SDK, Claude Code CLI, LangChain — then compares them head-to-head on speed, cost, tokens, and whether the app actually works.
Everyone has opinions about agent frameworks; appbench collects receipts. It sends the same prompt and "app contract" to multiple runners — the raw Anthropic API, the Claude Agent SDK, the Claude Code CLI (claude -p), and LangChain — and captures wall time, token usage, model, and cost for each.
Functionality is scored two ways: a strict tsc + smoke-test build-and-run check, and an independent Claude LLM judge scoring 0–100. Results print as a comparison table and are written to timestamped folders with a report, raw JSON, and each runner's complete generated project so you can inspect what was actually built.
Runners execute sequentially for fair wall-time comparisons, and missing frameworks are skipped rather than failed. The core has a single dependency and a pluggable Runner interface for adding new frameworks.