D20bench - Benchmarking LLMs with Dungeons & Dragons Combat

A few months ago I wrote about building a D&D combat simulator . Once BattleCast could run whole battles on its own, a question started nagging me: if I handed the controls to a language model, would it actually play well? Not “can it talk about tactics” - can it stand on a bridge, look at eight combatants, and make the right call?

That question turned into D20bench, an open-source benchmark where LLMs fight each other (and a scripted AI) under real SRD 5e combat rules. This post is the story of the idea, the mistakes I made along the way, the results, and how you can get involved.

The idea

Most LLM benchmarks ask a model to produce text and then grade the text. I wanted something different: a benchmark where the model has to make decisions, the environment is deterministic, and the outcome is impossible to argue with. Either your team won the fight or it did not.

Tactical D&D combat is quite a nice fit for this. It has positioning, resource management, action economy, risk assessment and teamwork - all in one turn. And because I already had a rules engine from BattleCast, the hard part (adjudicating the rules) was already done.

The core contract of D20bench: the engine generates every legal action for the current game state, the model picks exactly one by its id, and the engine resolves it with seeded dice. The model supplies intent, nothing else.

This matters more than it might seem. The model cannot invent actions, cannot fudge dice, and cannot delegate to a smarter system. If it wins, it won by making better choices. Matches run in a ladder and every agent gets an Elo rating - including battlecast.smart, my scripted AI, which acts as the calibration anchor in the middle of the table.

The models are not stupid, your harness is

When I first ran the benchmark, the models looked terrible. They ended turns without attacking, wandered around aimlessly and lost to a few hundred lines of scripted heuristics. It was tempting to conclude that LLMs simply cannot play D&D.

To be fair, some of them can’t. But when I dug into the decision traces, the real story was different: most of the “stupidity” was my harness design. A deep review found problems like these:

• Matches were capped at 3-4 rounds and scored by remaining HP - and I never told the models. They were playing a game whose win condition was hidden from them. Good D&D play (buffs, positioning, healing) was actively penalised.
• Movement was offered as a curated menu of 16 squares, biased towards extremes. The traces showed models asking for perfectly sensible squares - “move to the dying Bard at (7,7) to finish them off” - and getting rejected because that square was not on the menu.
• Opportunity attacks on the LLM side were silently declined, while the scripted AI always took its own. A free damage tax, invisible in the results.
• Nobody explained the grid. Positions were in squares, ranges in feet, and the model was left to guess that one square is 5 ft and diagonals are free.

Each fix was small: state the objective and the round cap in the prompt, explain the geometry, provide distances, widen the movement menu, make reactions real decisions, raise the round cap so fights actually finish. None of it made the models smarter. All of it let their existing intelligence through.

After the fixes, wasted turns (ending a turn with an attack still available) dropped from common to literally zero. In fact, the very first validation run produced something I did not expect: DeepSeek beat my scripted AI in a 16-round fight by repeatedly healing and stabilising downed allies while my “smart” AI let its own team bleed out. It lost the damage race and won the war. That is real tactics, and my benchmark had been hiding it.

If you are building any LLM evaluation, this is the lesson I would take away: before you conclude the model is weak, audit what you are actually asking it to do. Read the raw traces. The bug is usually on your side of the API.

The results

The proper calibration run was a 48-match round robin: DeepSeek V4 Flash and Qwen 3.5 Flash (the cheap tier), GLM 5.2 (a stronger reasoning model), and battlecast.smart, across two maps, both side assignments and two seeds. This is calibration-grade rather than paper-grade: enough to see whether the benchmark discriminates, not enough to publish definitive model rankings.

It cost $54 to run on OpenRouter, and $44 of that was GLM alone. A smaller 12-match smoke run with two cheap models cost about $3. I have not put frontier models through the ladder yet because the harness is still the thing under test. I would rather spend the first dollars finding harness bugs than burn expensive-model money on results I might have to throw away.

A language model sits at the top of the table, above the scripted AI. GLM 5.2 beat it 5-3 head to head, and it won on the chokepoint map - the positioning-heavy board where models used to lose every single game. The rankings were identical across both seeds, so this is not seed luck.

I find the shape of this table more exciting than the winner. A stronger model calibrates higher, the scripted baseline sits in the middle, and the weak model falls to the bottom with a 200 Elo gap. In other words: the benchmark discriminates. Before the fixes, an 8B model was statistically indistinguishable from the baseline. Now capability shows up in the standings.

The traces are full of small moments that make it feel real. GLM keeping Hunter’s Mark on a wild-shaped druid “to pop its wild shape quickly”. A warlock declining an opportunity attack because “the target is 55 ft away, too far for a dagger”. A ranger spreading out from a wounded ally “to avoid AoE clustering”. You do not get this from a multiple-choice benchmark.

Watching the models think

Numbers are nice, but the thing that convinced me this project is worth continuing is the replay viewer. Every match records a full replay: every action, every dice roll, and - my favourite part - the model’s rationale for every single decision.

The viewer plays matches back on the actual battle map, borrowing the token art and visual style from BattleCast. There is a combat log synced to the animation, and a Decision panel that shows what the model was thinking as it acted. The screenshot above is GLM 5.2 closing out a win: the only enemy left is an unconscious Bard with one failed death save, and the model casts Shatter specifically to stack a second failure while noting its own allies are safe from the blast. Cold-blooded, rules-accurate, and entirely its own idea. You do not get that from a leaderboard number.

Running it yourself

Do not drive this by hand. This whole project was built with a coding agent, and running it with one is by far the nicest way to use it. If you have Claude Code , Codex, or a similar agent, you need exactly three things:

• Node.js 20 or newer
• an OpenRouter API key (this is what pays for the models’ decisions)
• the repo: git clone https://github.com/bjedrzejewski/d20bench.git

Then open the repo with your agent, put your key in .env.local (there is a .env.example showing the format - just OPENROUTER_API_KEY=), and simply ask for what you want. Some prompts that work well:

“Set this project up, run the tests, and then run one free match between the scripted agents so I can see it working.”

“Run a match between claude-haiku and battlecast.smart on the chokepoint map and tell me what happened.”

“Create a small season with openrouter:vendor/model and DeepSeek against battlecast.smart, two seeds, cap the cost at $5, run it and show me the standings.”

“Build the replay viewer data and serve it so I can watch the matches.”

The agent will find the CLI, the season definitions and the docs on its own - the repo is documented specifically so that agents can navigate it. Asking for a cost cap in your prompt is the one thing I would always do: the runner enforces it and resumes cleanly, so a typo in a season definition cannot surprise you with a bill.

If you prefer to drive it manually, the short version is:

npm install && npm run build && npm test

# free match between scripted agents
npm run d20bench -- match run \
  --scenario public.hero-mirror-chokepoint-l5.v1 \
  --red baseline.focus-fire --blue battlecast.smart --seed 1

# a full Elo season with a hard cost cap
npm run d20bench -- llm ladder run \
  --season llm-actual-fairfix-tournament-v1 --resume --max-cost 10

# replay viewer at http://localhost:8734
npm run replays:data && npm run replays:serve

Any OpenRouter model id works as an agent (openrouter:vendor/model), seasons are defined in llm-season.ts, and npm run d20bench -- llm audit --season <id> verifies the fairness properties of a finished run.

Where this goes next

There is a clear path from “working benchmark” to something the community can rely on, and I would love help with all of it:

• A proper leaderboard. More seeds, confidence intervals, and a published site where every rating links to watchable replays. The current runs are calibration-grade, not final science.
• Filling out the ladder. There is a 55 Elo gap between DeepSeek and GLM 5.2 with nothing in between, and nothing above GLM yet. Mid-tier and frontier models would tell us how far the benchmark can discriminate before it saturates.
• A misplay detector. The replays contain everything needed to annotate wasted turns, bad focus fire and missed heals automatically. That would turn “model A beat model B” into “here is exactly why”.
• Harder tactical problems. Both current maps are symmetric mirrors. Asymmetric objectives (defend the door, escort the wizard, hold out for five rounds) would test planning in a way symmetric deathmatches cannot.

How you can contribute

D20bench is open source at github.com/bjedrzejewski/d20bench . It is TypeScript, the engine is deterministic and well tested, and pretty much every part of it can be improved. Some concrete ideas, roughly in order of difficulty:

• Run more models. Any OpenRouter model id works out of the box - an agent is just openrouter:vendor/model. The ladder needs more data points, especially in the gap between the cheap tier and the strong reasoning models. Season definitions live in llm-season.ts and a small season costs a few dollars.
• Design scenarios. There are currently two main battle maps. New scenarios (different party compositions, terrain, objectives) are self-contained and the scripted baselines make it easy to sanity-check balance before spending API money.
• Extend the action space. Every D&D mechanic needs a concrete legal-action representation before models can use it. Each new one (grappling, readied actions, more spells) comes with a clear pattern to follow and existing tests to copy.
• Compress the observation. Models currently receive about 15k tokens of game state per decision. Smarter serialisation would cut costs for everyone running seasons.

If any of this sounds interesting, open an issue or have a look at the harness audit docs in the repo.

Summary

• D20bench makes LLMs play real D&D combat: the engine generates legal actions, the model picks one, seeded dice decide the rest.
• The models initially looked stupid, and it was almost entirely my fault - hidden win conditions, curated menus and silent rule asymmetries. Audit your harness before judging your models.
• After the fairness fixes, the benchmark discriminates cleanly: GLM 5.2 now tops the table above my scripted AI, with identical rankings across seeds.
• Every match is replayable, including the model’s reasoning for every decision.
• You can run it yourself with Node and an OpenRouter key - a scripted match is free and a small LLM season costs a few dollars.
• It is open source and there is plenty to do - models to run, scenarios to design, mechanics to add.

I started this to find out whether language models can play D&D. It turns out some of them quite genuinely can - once you stop getting in their way.

Till next time!