The Alignment League Benchmark (ALB) is a suite of multi-agent games, designed to measure how well AI agents align, coordinate, and collaborate with others (both AIs and humans).
CoGames is the games environment for ALB. You can use it to:
- create new games
- train agents to play existing ALB games
- submit those agents to the ALB leaderboard
There's one ALB game right now: Cogs vs Clips.
Install cogames as a Python package.
pip install cogamesUsing uv
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# Create a virtual environment
uv venv .venv
source .venv/bin/activate
# Install cogames
uv pip install cogamesUsing Docker
# Ensure Python 3.12 is available
FROM python:3.12-slim
# Ensure C/C++ compiler is available
RUN apt-get update && \
apt-get install -y --no-install-recommends build-essential && \
rm -rf /var/lib/apt/lists/*
# Install cogames
RUN pip install --no-cache-dir cogamesUsing Colab
Play an easy mission in Cogs vs Clips using:
cogames tutorial playThe game will open in a new window, and the terminal will give you instructions to complete training mission.
-
Log into the ALB leaderboard with your GitHub account.
cogames login
-
Upload one of our starter policies under your name.
cogames upload --policy "class=cogames.policy.scripted_agent.starter_agent.StarterPolicy" --name "$USER.README-quickstart-starter-policy"
-
List the seasons currently running.
cogames seasons
-
Submit the policy to one of the running seasons.
cogames submit "$USER.README-quickstart-starter-policy:v1" --season beta-cogsguard
To learn more, see:
- Creating a policy: Creating a custom policy and evaluating it
- Training: Training a custom policy and evaluating it
- Submitting: Submitting to the leaderboard and understanding the results
If you want help, or to share your experience, join the Discord.
CogsGuard is a cooperative territory-control game. Teams of AI agents ("Cogs") work together to capture and defend junctions against automated opponents ("Clips") by:
- gathering resources and depositing them at controlled junctions
- acquiring specialized roles (Miner, Aligner, Scrambler, Scout) at gear stations
- capturing neutral junctions using Aligners (costs 1 heart + 1 influence)
- disrupting enemy junctions using Scramblers (costs 1 heart)
- defending territory from Clips expansion
Read MISSION.md for a thorough description of the game mechanics.
There are many mission configurations available, with different map sizes, junction layouts, and game rules.
Overall, CogsGuard aims to present rich environments with:
- Territory control: Capture and defend junctions to score points each tick
- Role specialization: Four roles (Miner, Aligner, Scrambler, Scout) with distinct capabilities and dependencies
- Dense rewards: Agents receive reward every tick proportional to territory controlled
- Partial observability: Agents have limited visibility of the environment
- Required multi-agent cooperation: No single role can succeed alone; Miners need Aligners to capture junctions, Aligners need Miners for resources, Scramblers must clear enemy territory for Aligners to advance
The ALB leaderboard runs in seasons. Each season has two pools:
- Qualifying pool: Where new submissions start. Your policy plays matches against other policies in the pool.
- Competition pool: Policies that score above a threshold in qualifying get promoted here.
To see active seasons and their pools:
cogames seasonsWhen you submit a policy, it gets queued for matches against other policies in its pool. Our focal metric is VORP (Value Over Replacement Policy), which estimates how much your agent improves team performance compared to a baseline.
VORP is calculated by comparing:
- Replacement mean: The average score when only other pool policies play (no candidate)
- Candidate score: The score when your policy plays
The difference tells us how much value your policy adds to a team. A positive VORP means your policy makes teams better; a negative VORP means teams perform worse with your policy than without it.
You can evaluate VOR locally before submitting:
cogames pickup --policy <YOUR_POLICY> --pool <POOL_POLICY>To check your submission status and match results:
cogames submissions
cogames leaderboard --season beta-cogsguardMost commands are of the form cogames <command> --mission [MISSION] --policy [POLICY] [OPTIONS]
To specify a MISSION, you can:
- Use a mission name from the registry given by
cogames missions(e.g.training_facility_1). - Use a path to a mission configuration file (e.g.
path/to/mission.yaml). - Alternatively, specify a set of missions with
--mission-set.
To specify a POLICY, use one of two formats:
-
URI format (for checkpoint bundles):
- Point directly at a checkpoint bundle (directory or
.zipcontainingpolicy_spec.json) - Examples:
./train_dir/my_run:v5,./train_dir/my_run:v5.zip,s3://bucket/path/run:v5.zip - Use
:latestsuffix to auto-resolve the highest version:./train_dir/checkpoints:latest
- Point directly at a checkpoint bundle (directory or
-
Key-value format (for explicit class + weights):
class=: Policy shorthand or full class path fromcogames policies, e.g.class=lstmorclass=cogames.policy.random.RandomPolicy.data=: Optional path to a weights file (e.g.,weights.safetensors). Must be a file, not a directory.proportion=: Optional positive float specifying the relative share of agents that use this policy (default: 1.0).kw.<arg>=: Optional policy__init__keyword arguments (all values parsed as strings).
You can view all the commands with
cogames --helpand you can view help for a given command with:
cogames [COMMAND] --helpUsage: cogames missions [OPTIONS] SITE
List available missions. This command has three modes: 1. List sites: Run with no arguments to see all available sites. 2. List missions at a site: Pass a site name (e.g., 'cogsguard_machina_1') to see its missions. 3. Describe a mission: Use -m to describe a specific mission. Only in this mode do --cogs, --variant, --format, and --save have any effect.
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ site TEXT Filter by site (e.g., cogsguard_machina_1) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Describe ──────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission -m MISSION Mission to describe │ │ --cogs -c INTEGER Override agent count (requires -m) │ │ --variant -v VARIANT Apply variant (requires -m, repeatable) │ │ --format [yaml|json] Output format (requires -m) │ │ --save -s PATH Save config to file (requires -m) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames missions List all sites cogames missions cogsguard_machina_1 List missions at site cogames missions -m cogsguard_machina_1.basic Describe a mission cogames missions -m arena --format json Output as JSON
Usage: cogames evals [OPTIONS]
List all eval missions
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames variants [OPTIONS]
List all available mission variants
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames describe [OPTIONS] MISSION
Describe a mission and its configuration
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * mission TEXT Mission name (e.g., hello_world.open_world) [required] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Configuration ─────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --cogs -c INTEGER Number of cogs (agents) │ │ --variant -v VARIANT Apply variant (repeatable) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames describe hello_world.open_world Describe mission cogames describe arena -c 4 -v dark_side With 4 cogs and variant
Usage: cogames make-mission [OPTIONS]
Create a custom mission from a base template
╭─ Mission ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission -m MISSION Base mission to start from │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Customization ─────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --cogs -c INTEGER RANGE [x>=1] Number of cogs (agents) │ │ --width INTEGER RANGE [x>=1] Map width │ │ --height INTEGER RANGE [x>=1] Map height │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --output -o PATH Output file path (.yml or .json) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames make-mission -m hello_world -c 8 -o my_mission.yml 8 cogs cogames make-mission -m arena --width 64 --height 64 -o big.yml 64x64 map cogames play -m my_mission.yml Use custom mission
Usage: cogames play [OPTIONS]
Play a game interactively. This runs a single episode of the game using the specified policy. By default, the policy is 'noop', so agents won't move unless manually controlled. To see agents move by themselves, use `--policy class=random` or `--policy class=baseline`. You can manually control the actions of a specific cog by clicking on a cog in GUI mode or pressing M in unicode mode and using your arrow or WASD keys. Log mode is non-interactive and doesn't support manual control.
╭─ Game Setup ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission -m MISSION Mission to play (run cogames missions to list) │ │ --variant -v VARIANT Apply variant modifier (repeatable) │ │ --cogs -c N Number of cogs/agents [default: (from mission)] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --policy -p POLICY Policy controlling cogs (noop, random, lstm, or path) [default: class=noop] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Simulation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --steps -s N Max steps per episode [default: 1000] │ │ --render -r [gui|vibescope|unicode|log|none] gui=MettaScope, vibescope=VibeScope, unicode=terminal, │ │ log=metrics only │ │ [default: gui] │ │ --seed INTEGER RNG seed for reproducibility [default: 42] │ │ --map-seed SEED Separate seed for procedural map generation │ │ [default: (same as --seed)] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --save-replay-dir DIR Save replay file for later viewing with cogames replay │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames play -m cogsguard_machina_1.basic Interactive cogames play -m cogsguard_machina_1.basic -p class=random Random policy cogames play -m cogsguard_machina_1.basic -c 4 -p class=baseline Baseline, 4 cogs cogames play -m cogsguard_machina_1 -r unicode Terminal mode
Usage: cogames replay [OPTIONS] FILE
Replay a saved game episode from a file in the GUI
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * replay_path FILE Path to the replay file (.replay or .bin) [required] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames replay ./replays/game.replay Replay a saved game cogames replay ./train_dir/my_run/replay.bin Replay from training run
Usage: cogames scrimmage [OPTIONS]
Evaluate a single policy controlling all agents.
This command is equivalent to running `cogames run` with a single policy.
╭─ Mission ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission -m MISSION Missions to evaluate (supports wildcards) │ │ --mission-set -S SET Predefined set: integrated_evals, spanning_evals, diagnostic_evals, all │ │ --cogs -c N Number of cogs (agents) │ │ --variant -v VARIANT Mission variant (repeatable) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --policy -p POLICY Policies to evaluate: (URI[,proportion=N] or NAME or │ │ class=NAME[,data=FILE][,proportion=N]...) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Simulation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --episodes -e N [x>=1] Number of evaluation episodes [default: 10] │ │ --steps -s N [x>=1] Max steps per episode [default: 1000] │ │ --seed N [x>=0] Seed for evaluation RNG [default: 42] │ │ --map-seed N [x>=0] MapGen seed for procedural maps [default: (same as --seed)] │ │ --action-timeout-ms MS [x>=1] Max ms per action before noop [default: 250] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --format FMT Output format: yaml or json │ │ --save-replay-dir DIR Directory to save replays │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames scrimmage -m arena.battle -p lstm Single policy eval
Usage: cogames run [OPTIONS]
Evaluate one or more policies on missions. With multiple policies (e.g., 2 policies, 4 agents), each policy always controls 2 agents, but which agents swap between policies each episode. With one policy, this command is equivalent to `cogames scrimmage`.
╭─ Mission ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission -m MISSION Missions to evaluate (supports wildcards) │ │ --mission-set -S SET Predefined set: integrated_evals, spanning_evals, diagnostic_evals, all │ │ --cogs -c N Number of cogs (agents) │ │ --variant -v VARIANT Mission variant (repeatable) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --policy -p POLICY Policies to evaluate: (URI[,proportion=N] or NAME or │ │ class=NAME[,data=FILE][,proportion=N]...) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Simulation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --episodes -e N [x>=1] Number of evaluation episodes [default: 10] │ │ --steps -s N [x>=1] Max steps per episode [default: 1000] │ │ --seed N [x>=0] Seed for evaluation RNG [default: 42] │ │ --map-seed N [x>=0] MapGen seed for procedural maps [default: (same as --seed)] │ │ --action-timeout-ms MS [x>=1] Max ms per action before noop [default: 250] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --format FMT Output format: yaml or json │ │ --save-replay-dir DIR Directory to save replays │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames run -m cogsguard_machina_1.basic -p lstm Evaluate single policy cogames run -m cogsguard_machina_1 -p ./train_dir/my_run:v5 Evaluate a checkpoint bundle cogames run -S integrated_evals -p ./train_dir/my_run:v5 Evaluate on mission set cogames run -m 'arena.*' -p lstm -p random -e 20 Evaluate multiple policies together cogames run -m cogsguard_machina_1 -p ./train_dir/my_run:v5,proportion=3 -p class=random,proportion=5 Evaluate policies in 3:5 mix
Usage: cogames pickup [OPTIONS]
Evaluate a policy against a pool of other policies and compute VOR
╭─ Mission ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission -m MISSION Mission to evaluate on [default: cogsguard_machina_1.basic] │ │ --cogs -c N [x>=1] Number of cogs (agents) [default: 4] │ │ --variant -v VARIANT Mission variant (repeatable) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --policy -p POLICY Candidate policy to evaluate │ │ --pool POLICY Pool policy (repeatable) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Simulation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --episodes -e N [x>=1] Episodes per scenario [default: 1] │ │ --steps -s N [x>=1] Max steps per episode [default: 1000] │ │ --seed N [x>=0] Base random seed [default: 50] │ │ --map-seed N [x>=0] MapGen seed for procedural maps [default: (same as --seed)] │ │ --action-timeout-ms MS [x>=1] Max ms per action before noop [default: 250] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --save-replay-dir DIR Directory to save replays │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames pickup -p greedy --pool random Test greedy against pool of random
Usage: cogames diagnose [OPTIONS] POLICY
Run diagnostic evals for a policy checkpoint
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * policy TEXT Policy specification: URI (bundle dir or .zip) or NAME or │ │ class=NAME[,data=FILE][,kw.x=val] │ │ [required] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Evaluation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission-set -S SET Eval suite to run [default: diagnostic_evals] │ │ --experiments NAME Specific experiments (subset of mission set) │ │ --cogs -c N Agent counts to test (repeatable) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Simulation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --steps -s N Max steps per episode [default: 1000] │ │ --episodes -e N Episodes per case [default: 3] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames diagnose ./train_dir/my_run Default diagnostics cogames diagnose lstm -S tournament Tournament suite cogames diagnose lstm -c 4 -c 8 -e 5 Custom cog counts
Usage: cogames version [OPTIONS]
Show version information for cogames and dependencies
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames docs [OPTIONS] DOC
Print documentation (run without arguments to see available docs)
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ doc_name DOC Document name (readme, mission, technical_manual, scripted_agent, evals, mapgen) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames docs List available documents cogames docs readme Print README cogames docs mission Print mission briefing
Usage: cogames policies [OPTIONS]
Show available policy shorthand names
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: Use these shorthand names with --policy or -p: cogames play -m arena -p class=random Use random policy cogames play -m arena -p class=baseline Use baseline policy
Usage: cogames validate-policy [OPTIONS]
Validate the policy loads and runs for at least a single step
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * --policy -p POLICY Policy specification: URI (bundle dir or .zip) or NAME or │ │ class=NAME[,data=FILE][,kw.x=val] │ │ [required] │ │ --setup-script TEXT Path to a Python setup script to run before loading the policy │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Tournament ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --season SEASON Tournament season (determines which game to validate against) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --server URL Tournament server URL (used to resolve default season) │ │ [default: https://api.observatory.softmax-research.net] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames login [OPTIONS]
Authenticate with CoGames server
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --force -f Re-authenticate even if already logged in │ │ --timeout -t SECS Authentication timeout in seconds [default: 300] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --login-server URL Authentication server URL [default: https://softmax.com/api] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames login Authenticate with default server cogames login --force Re-authenticate even if already logged in
Usage: cogames submissions [OPTIONS]
Show your uploads and tournament submissions
╭─ Filter ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --policy -p POLICY Filter by policy name (e.g., 'my-policy' or 'my-policy:v3') │ │ --season SEASON Filter by tournament season │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --login-server URL Authentication server URL [default: https://softmax.com/api] │ │ --server -s URL Tournament server URL [default: https://api.observatory.softmax-research.net] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --json Print raw JSON instead of table │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames submissions All your uploads cogames submissions --season beta-cogsguard Submissions in a season cogames submissions -p my-policy Info on a specific policy
Usage: cogames seasons [OPTIONS]
List currently running tournament seasons
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --login-server URL Authentication server URL [default: https://softmax.com/api] │ │ --server -s URL Tournament server URL [default: https://api.observatory.softmax-research.net] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --json Print raw JSON instead of table │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames leaderboard [OPTIONS]
Show tournament leaderboard for a season
╭─ Tournament ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --season SEASON Tournament season name [default: beta-cogsguard] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --login-server URL Authentication server URL [default: https://softmax.com/api] │ │ --server -s URL Tournament server URL [default: https://api.observatory.softmax-research.net] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --json Print raw JSON instead of table │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames leaderboard --season beta-cogsguard View rankings
Usage: cogames upload [OPTIONS]
Upload a policy to CoGames
╭─ Upload ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * --name -n NAME Name for your uploaded policy [required] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * --policy -p POLICY Policy specification: URI (bundle dir or .zip) or NAME or │ │ class=NAME[,data=FILE][,kw.x=val] │ │ [required] │ │ --init-kwarg -k KEY=VAL Policy init kwargs (can be repeated) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Files ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --include-files -f PATH Files or directories to include (can be repeated) │ │ --setup-script PATH Python setup script to run before loading the policy │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Tournament ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --season SEASON Tournament season (default: server's default season) │ │ --no-submit Upload without submitting to a season │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Validation ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --dry-run Run validation only without uploading │ │ --skip-validation Skip policy validation in isolated environment │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --login-server URL Authentication server URL [default: https://softmax.com/api] │ │ --server URL Tournament server URL [default: https://api.observatory.softmax-research.net] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames upload -p ./train_dir/my_run -n my-policy Upload and submit to default season cogames upload -p ./run -n my-policy --season beta-cvc Upload and submit to specific season cogames upload -p ./run -n my-policy --no-submit Upload without submitting cogames upload -p lstm -n my-lstm --dry-run Validate only
Usage: cogames submit [OPTIONS] POLICY
Submit a policy to a tournament season
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ * policy_name POLICY Policy name (e.g., 'my-policy' or 'my-policy:v3' for specific version) [required] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Tournament ────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --season SEASON Tournament season name │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Server ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --login-server URL Authentication server URL [default: https://softmax.com/api] │ │ --server -s URL Tournament server URL [default: https://api.observatory.softmax-research.net] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames submit my-policy Submit to default season cogames submit my-policy:v3 --season beta-cvc Submit specific version to specific season
Usage: cogames tutorial play [OPTIONS]
Interactive tutorial - learn to play Cogs vs Clips
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames tutorial cogsguard [OPTIONS]
Interactive CogsGuard tutorial - learn roles and territory control
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help Show this message and exit. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Usage: cogames tutorial make-policy [OPTIONS]
Create a new policy from a template. Requires --trainable or --scripted.
╭─ Policy Type ───────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --trainable Create a trainable (neural network) policy │ │ --scripted Create a scripted (rule-based) policy │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --output -o FILE Output file path [default: my_policy.py] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames tutorial make-policy -t -o my_nn_policy.py Trainable (neural network) cogames tutorial make-policy -s -o my_scripted_policy.py Scripted (rule-based)
Usage: cogames tutorial train [OPTIONS]
Train a policy on one or more missions. By default, our 'lstm' policy architecture is used. You can select a different architecture (like 'stateless' or 'baseline'), or define your own implementing the MultiAgentPolicy interface with a trainable network() method (see mettagrid/policy/policy.py). Continue training from a checkpoint using URI format, or load weights into an explicit class with class=...,data=... syntax. Supply repeated -m flags to create a training curriculum that rotates through missions. Use wildcards (*) in mission names to match multiple missions at once.
╭─ Mission Setup ─────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --mission -m MISSION Missions to train on (wildcards supported, repeatable for curriculum) │ │ --cogs -c N Number of cogs (agents) [default: (from mission)] │ │ --variant -v VARIANT Mission variant (repeatable) │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Policy ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --policy -p POLICY Policy to train (URI (bundle dir or .zip) or NAME or │ │ class=NAME[,data=FILE][,kw.x=val]) │ │ [default: class=lstm] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Training ──────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --steps N [x>=1] Number of training steps [default: 10000000000] │ │ --batch-size N [x>=1] Batch size for training [default: 4096] │ │ --minibatch-size N [x>=1] Minibatch size for training [default: 4096] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Hardware ──────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --device DEVICE Device to train on (auto, cpu, cuda, mps) [default: cpu] │ │ --num-workers N [x>=1] Number of worker processes [default: (CPU cores)] │ │ --parallel-envs N [x>=1] Number of parallel environments │ │ --vector-batch-size N [x>=1] Vectorized environment batch size │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Reproducibility ───────────────────────────────────────────────────────────────────────────────────────────────╮ │ --seed N [x>=0] Seed for training RNG [default: 42] │ │ --map-seed N [x>=0] MapGen seed for procedural map layout [default: (same as --seed)] │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Output ────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --checkpoints DIR Path to save training checkpoints [default: ./train_dir] │ │ --log-outputs Log training outputs │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Other ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ --help -h Show this message and exit │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Examples: cogames tutorial train -m cogsguard_machina_1.basic Basic training cogames tutorial train -m cogsguard_machina_1.basic -p class=baseline Train baseline policy cogames tutorial train -p ./train_dir/my_run:v5 Continue from checkpoint cogames tutorial train -p class=lstm,data=./weights.safetensors Load weights into class cogames tutorial train -m mission_1 -m mission_2 Curriculum (rotates) Wildcard patterns: cogames tutorial train -m 'machina_2_bigger:*' All missions on machina_2_bigger cogames tutorial train -m '*:shaped' All "shaped" missions cogames tutorial train -m 'machina*:shaped' All "shaped" on machina maps
If you use CoGames in your research, please cite:
@software{cogames2025,
title={CoGames: Multi-Agent Cooperative Game Environments},
author={Softmax},
year={2025},
url={https://github.com/metta-ai/metta}
}