Rogue
Bench – LLMs play the game Rogue
- Free
- Web
- Other
- Always free
- No credit card
What is Rogue?
Rogue Bench is an open source benchmark that evaluates large language models by placing them in the classic Rogue dungeon crawler game. Rather than solving isolated tasks, LLMs must navigate a complete game environment, make tactical decisions, manage resources, and adapt to procedurally generated challenges. This tests reasoning, planning, and real-time decision-making in a complex, partially observable world. The project provides structured evaluation metrics and supports multiple LLM models, allowing researchers and practitioners to compare model performance on practical problem-solving tasks that require sustained reasoning over many steps.
Key features
Game-based LLM evaluation using Rogue gameplay as a testing environment
Multi-model support for testing and comparing different LLMs
Structured metrics to measure reasoning, planning, and decision-making
Procedurally generated game instances for varied evaluation conditions
Open source codebase available on GitHub for transparency and extension
Pros & cons
Advantages
- Tests LLM abilities in real, complex environments rather than isolated prompts
- Neutral ground for model comparison with clear success metrics
- Open source design allows verification and customisation
- Reveals practical problem-solving strengths and weaknesses
- Applicable to AI research, model development, and capability assessment
Limitations
- Results specific to roguelike game mechanics; performance may not transfer to other domains
- Requires computational resources to run game simulations at scale
- Limited to evaluating LLMs; not applicable to other AI systems
- Game performance depends on prompt engineering and model-specific factors
- Smaller benchmark than some general LLM evaluation suites
Use cases
Comparing performance across different LLM models
Assessing reasoning and planning capabilities in complex environments
Research on AI game-playing and decision-making under uncertainty
Selecting models for tasks requiring sustained multi-step reasoning
Evaluating improvements to LLM architectures and training approaches
Ready to try Rogue?
Pricing
Get started with Rogue
Click through to Rogue and start using it now.
- Always free
- No credit card