bettermdptools

Bettermdptools is a lightweight toolkit for working with Gymnasium environments using classic planning and tabular reinforcement learning methods.

It is designed to help users get up and running quickly, explore standard RL algorithms, and experiment with environments like FrozenLake, Taxi, Blackjack, and CartPole without heavy framework overhead.

---

Getting started

Install

Install from PyPI:

pip install bettermdptools

pygame installation issues (Python 3.11+)

If you encounter errors installing pygame on Python 3.11 or newer, try:

pip install pygame --pre

• https://stackoverflow.com/questions/74188013/python-pygame-not-installing

Google Colab notes (NumPy + Gymnasium)

Some Gymnasium-compatible environments may require a NumPy downgrade.

When using Google Colab - after installing numpy<2 you must restart the Colab session for the change to take effect.

Typical workflow:

pip install "numpy<2"

Then:

Runtime → Restart session

---

Quick example (FrozenLake)

Below is a minimal example using value iteration on FrozenLake:

import gymnasium as gym
from bettermdptools.algorithms.planner import Planner
from bettermdptools.utils.plots import Plots

env = gym.make("FrozenLake8x8-v1", render_mode=None)

V, V_track, pi = Planner(env.P).value_iteration()

Plots.values_heat_map(
    V,
    title="FrozenLake Value Iteration - State Values",
    size=(8, 8),
)

Example notebooks

The fastest way to explore the library is through the notebooks in /examples .

---

Entrypoint APIs (optional)

Bettermdptools provides optional high-level entrypoints for quick experimentation and hyperparameter search.

See:

• docs/api/experiments_api.md
• docs/api/optuna_search_api.md

---

Development and tooling

Dependency management

The project supports standard pip workflows. For development, using Poetry is recommended to ensure reproducible environments.

Example:

poetry install
poetry shell

Code quality

The codebase uses the following tools during development:

• ruff for fast linting
• black for code formatting

Typical usage:

ruff check .
black .

---

Documentation

API documentation is generated using pdoc (not pdoc3) and lives in the docs/ directory.

To regenerate documentation locally:

pdoc --include-undocumented -d numpy -t docs-templates --output-dir docs bettermdptools

---

Contributing

Pull requests are welcome.

Guidelines:

• Use numpy-style docstrings
• Add or update tests when introducing new functionality
• Prefer explicit, readable code over clever or showy abstractions
• Format code with black and check linting with ruff before submitting
• Keep public APIs stable and avoid breaking changes unless clearly justified
• Prefer small, focused pull requests with a limited number of file changes

Basic workflow:

1. Fork the repository
2. Create a feature branch
3. Commit changes
4. Open a pull request

---