Skip to content

Compare the guidance you give coding agents.

PluginBench runs the same coding tasks with and without your instructions, skills, or plugins, keeping the harness, model, and reasoning effort fixed. Compare correctness, resource use, and the code each attempt adds to decide which guidance is worth using.

Install

pipx install pluginbench
Runs on
Linux or macOS
Needs
Python 3.12+, Git, Docker with Compose and Buildx
Agent
Codex CLI; verified traces: 0.162.1, 0.167.0, 0.167.1. Other versions warn; incompatible or unverified inherited history leaves tokens unavailable.

Full reference