Compare the guidance you give coding agents.
PluginBench runs the same coding tasks with and without your instructions, skills, or plugins, keeping the harness, model, and reasoning effort fixed. Compare correctness, resource use, and the code each attempt adds to decide which guidance is worth using.
pipx install pluginbench- Runs on
- Linux or macOS
- Needs
- Python 3.12+, Git, Docker with Compose and Buildx
- Agent
- Codex CLI; verified traces: 0.162.1, 0.167.0, 0.167.1. Other versions warn; incompatible or unverified inherited history leaves tokens unavailable.
- Source
- filipgutica/pluginbench