D20bench - Benchmarking LLMs with Dungeons & Dragons Combat
A few months ago I wrote about building a D&D combat simulator . Once BattleCast could run whole battles on its own, a question started nagging me: if I handed the controls to a language model, would it actually play well? Not “can it talk about tactics” - can it stand on a bridge, look at eight combatants, and make the right call? That question turned into D20bench, an open-source benchmark where LLMs fight each other (and a scripted AI) under real SRD 5e combat rules. This post is the story of the idea, the mistakes I made along the way, the results, and how you can get involved. ...