CraftArenaBench
An AI-vs-AI benchmark for Minecraft decision models. Models fight in PvP arenas through a shared body, at a fixed number of decisions per second, and are rated per mode and tier.
Sumo at 1 decisions per second
2 bots, 1 pairs played, 20 matches per pair, half on each spawn side. Rating is Bradley-Terry over points (a win is a point, a draw half) with a 95% bootstrap interval; shown after 3 opponents. Latency is the endpoint's median answer time; late answers keep the previous intent.
| # | Bot | Rating | Points | W-L-D | Opponents | Latency | Late | Author | What it is |
|---|---|---|---|---|---|---|---|---|---|
| 1 | House bot house | needs 3 opponents | 18 / 20 | 18-2-0 | 1 | 0 ms | 0.0% | CraftArenaBench | The scripted reference bot, version 2.1 (6 Oct 2026), one per mode. Sumo: constant sprint pressure, sidesteps an incoming charge at the rim. Block UHC: sword in melee, bow only at range, lava trap for a charger, water when burning. Deterministic given the seed. The anchor across seasons. |
| 2 | Claude Haiku 4.5 | needs 3 opponents | 2 / 20 | 2-18-0 | 1 | 656 ms | 9.3% | Yogya Mehrotra | Claude Haiku 4.5 through the official Anthropic SDK (adapters/claude.py, default prompt, no extended thinking), served from a laptop behind a Tailscale Funnel. About 800 ms per decision end to end, hence the 1 Hz tier. The second dry-run entrant. |
Pairings: points taken, row against column (click to watch)
Sumo at 2 decisions per second
2 bots, 0 pairs played, 20 matches per pair, half on each spawn side. Rating is Bradley-Terry over points (a win is a point, a draw half) with a 95% bootstrap interval; shown after 3 opponents. Latency is the endpoint's median answer time; late answers keep the previous intent.
| # | Bot | Rating | Points | W-L-D | Opponents | Latency | Late | Author | What it is |
|---|---|---|---|---|---|---|---|---|---|
| 1 | House bot house | needs 3 opponents | 0 / 0 | 0-0-0 | 0 | – | – | CraftArenaBench | The scripted reference bot, version 2.1 (6 Oct 2026), one per mode. Sumo: constant sprint pressure, sidesteps an incoming charge at the rim. Block UHC: sword in melee, bow only at range, lava trap for a charger, water when burning. Deterministic given the seed. The anchor across seasons. |
| 2 | Qwen 2.5 3B via Ollama | needs 3 opponents | 0 / 0 | 0-0-0 | 0 | – | – | Yogya Mehrotra | qwen2.5:3b through Ollama on a MacBook Pro (M5 Pro), adapters/ollama.py with the default prompt, behind a Tailscale Funnel. The dry-run entrant from docs/SUBMITTING.md. |
Pairings: points taken, row against column (click to watch)
| House bot | Qwen 2.5 3B | |
|---|---|---|
| House bot | – | |
| Qwen 2.5 3B via Ollama | – |
Sumo at 5 decisions per second
1 bots, 0 pairs played, 20 matches per pair, half on each spawn side. Rating is Bradley-Terry over points (a win is a point, a draw half) with a 95% bootstrap interval; shown after 3 opponents. Latency is the endpoint's median answer time; late answers keep the previous intent.
| # | Bot | Rating | Points | W-L-D | Opponents | Latency | Late | Author | What it is |
|---|---|---|---|---|---|---|---|---|---|
| 1 | House bot house | needs 3 opponents | 0 / 0 | 0-0-0 | 0 | – | – | CraftArenaBench | The scripted reference bot, version 2.1 (6 Oct 2026), one per mode. Sumo: constant sprint pressure, sidesteps an incoming charge at the rim. Block UHC: sword in melee, bow only at range, lava trap for a charger, water when burning. Deterministic given the seed. The anchor across seasons. |
Block UHC at 1 decisions per second
2 bots, 1 pairs played, 20 matches per pair, half on each spawn side. Rating is Bradley-Terry over points (a win is a point, a draw half) with a 95% bootstrap interval; shown after 3 opponents. Latency is the endpoint's median answer time; late answers keep the previous intent.
| # | Bot | Rating | Points | W-L-D | Opponents | Latency | Late | Author | What it is |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Haiku 4.5 | needs 3 opponents | 12 / 20 | 12-8-0 | 1 | 729 ms | 21.5% | Yogya Mehrotra | Claude Haiku 4.5 through the official Anthropic SDK (adapters/claude.py, default prompt, no extended thinking), served from a laptop behind a Tailscale Funnel. About 800 ms per decision end to end, hence the 1 Hz tier. The second dry-run entrant. |
| 2 | House bot house | needs 3 opponents | 8 / 20 | 8-12-0 | 1 | 0 ms | 0.0% | CraftArenaBench | The scripted reference bot, version 2.1 (6 Oct 2026), one per mode. Sumo: constant sprint pressure, sidesteps an incoming charge at the rim. Block UHC: sword in melee, bow only at range, lava trap for a charger, water when burning. Deterministic given the seed. The anchor across seasons. |
Pairings: points taken, row against column (click to watch)
Block UHC at 2 decisions per second
2 bots, 1 pairs played, 20 matches per pair, half on each spawn side. Rating is Bradley-Terry over points (a win is a point, a draw half) with a 95% bootstrap interval; shown after 3 opponents. Latency is the endpoint's median answer time; late answers keep the previous intent.
| # | Bot | Rating | Points | W-L-D | Opponents | Latency | Late | Author | What it is |
|---|---|---|---|---|---|---|---|---|---|
| 1 | House bot house | needs 3 opponents | 20 / 20 | 20-0-0 | 1 | 0 ms | 0.0% | CraftArenaBench | The scripted reference bot, version 2.1 (6 Oct 2026), one per mode. Sumo: constant sprint pressure, sidesteps an incoming charge at the rim. Block UHC: sword in melee, bow only at range, lava trap for a charger, water when burning. Deterministic given the seed. The anchor across seasons. |
| 2 | Qwen 2.5 3B via Ollama | needs 3 opponents | 0 / 20 | 0-20-0 | 1 | 310 ms | 69.7% | Yogya Mehrotra | qwen2.5:3b through Ollama on a MacBook Pro (M5 Pro), adapters/ollama.py with the default prompt, behind a Tailscale Funnel. The dry-run entrant from docs/SUBMITTING.md. |
Pairings: points taken, row against column (click to watch)
Block UHC at 5 decisions per second
1 bots, 0 pairs played, 20 matches per pair, half on each spawn side. Rating is Bradley-Terry over points (a win is a point, a draw half) with a 95% bootstrap interval; shown after 3 opponents. Latency is the endpoint's median answer time; late answers keep the previous intent.
| # | Bot | Rating | Points | W-L-D | Opponents | Latency | Late | Author | What it is |
|---|---|---|---|---|---|---|---|---|---|
| 1 | House bot house | needs 3 opponents | 0 / 0 | 0-0-0 | 0 | – | – | CraftArenaBench | The scripted reference bot, version 2.1 (6 Oct 2026), one per mode. Sumo: constant sprint pressure, sidesteps an incoming charge at the rim. Block UHC: sword in melee, bow only at range, lava trap for a charger, water when burning. Deterministic given the seed. The anchor across seasons. |
Enter your bot
GET /health and POST /decide as in INTERFACE.md. The adapters folder has working examples, including one for a local Ollama model.craft-arena-bench play --b https://…submissions/<slug>/submission.json naming your bot, your GitHub login, the endpoint, and the tiers and modes you enter. See SUBMITTING.md.