Illustris (Modernized Fork)

Modern Python package for reading and analyzing Illustris simulation data.

This is a modernized and extended fork of the original illustris_python package, featuring:

• 🚀 Modern CLI tools with async data downloading
• 🧪 Comprehensive pytest test suite with fixtures
• 📚 Complete documentation with examples and guides
• ⚡ Performance optimizations and better error handling
• 🔧 Developer-friendly setup with uv and modern Python practices

Note: This fork maintains full API compatibility with the original package while adding significant new functionality.

Features

• Data Loading: Load snapshots, group catalogs, and merger trees
• CLI Tools: Command-line interface for data management and documentation
• Modern Testing: Comprehensive test suite with pytest fixtures
• Documentation: Auto-generated Sphinx documentation

Installation

# Install with uv (recommended)
uv add illustris

Development

# Clone the repository
git clone https://github.com/bjoernbethge/illustris.git
cd illustris

# Install with uv (recommended)
uv sync --dev

Quick Start

CLI Usage

The illustris command provides tools for data management and documentation:

# Documentation
illustris -docs -generate          # Build documentation
illustris -docs -serve             # Serve documentation (port 8000)
illustris -docs -serve -p 8080     # Serve on custom port

# Data Management
illustris -data -test              # Download complete test data (TNG50-4, all files)
illustris -data -load TNG50-4      # Download latest snapshot
illustris -data -load TNG50-4 -snap 99  # Download specific snapshot
illustris -data -list-sims         # List available simulations
illustris -data -list-snaps TNG50-4 # List snapshots for simulation

Python API

import illustris

# Load snapshot data
gas_data = illustris.snapshot.loadSubset(
    basePath="data/TNG50-4/output", 
    snapNum=99, 
    partType='gas',
    fields=['Coordinates', 'Masses']
)

# Load group catalog
halos = illustris.groupcat.loadHalos(
    basePath="data/TNG50-4/output",
    snapNum=99,
    fields=['GroupMass', 'GroupPos']
)

# Load subhalos
subhalos = illustris.groupcat.loadSubhalos(
    basePath="data/TNG50-4/output", 
    snapNum=99,
    fields=['SubhaloMass', 'SubhaloPos']
)

Configuration

Create a .env file for TNG API access:

# Copy example configuration
cp env.example .env

# Edit with your API key
ILLUSTRIS_API_KEY=your_api_key_here
ILLUSTRIS_DATA_DIR=./data  # Optional: custom data directory

Get your API key from: https://www.tng-project.org/users/register/

Testing

The project includes comprehensive tests using pytest:

# Run all tests
uv run pytest

# Run specific test modules
uv run pytest tests/test_snapshot.py
uv run pytest tests/test_groupcat.py

# Run with coverage (automatically included)
uv run pytest

# Run with detailed coverage report
uv run pytest --cov-report=term-missing

# Run only tests that don't require data
uv run pytest -m "not requires_data"

Test Data

Complete Test Data:

illustris -data -test

• Downloads ~5.3 GB of complete TNG50-4 dataset
• Enables all functionality tests including:
  • All 11 snapshot chunks for loadSubset testing
  • Group catalog files for halo/subhalo analysis
  • Offsets files for loadHalo testing
  • SubLink trees for merger analysis
• Recommended for development and full validation

Test Results

With complete test data:

• ✅ 26 tests pass - Full functionality tested
• ⏭️ 0 tests skipped - All features available

Tests automatically detect available data and skip appropriately if files are missing.

Development

Project Structure

illustris/
├── src/illustris/          # Main package
│   ├── cli.py             # Command-line interface
│   ├── snapshot.py        # Snapshot data loading
│   ├── groupcat.py        # Group catalog loading
│   ├── sublink.py         # Merger tree loading
│   └── util.py            # Utility functions
├── tests/                 # Test suite
│   ├── conftest.py        # Pytest fixtures
│   ├── test_basic.py      # Basic functionality tests
│   ├── test_snapshot.py   # Snapshot loading tests
│   ├── test_groupcat.py   # Group catalog tests
│   └── test_sublink.py    # Merger tree tests
├── docs/                  # Documentation
└── data/                  # Downloaded simulation data

Adding Tests

Tests use pytest fixtures for consistent data access:

import pytest
import illustris

@pytest.mark.requires_data
def test_my_function(test_data_path, test_snapshot):
    """Test with real simulation data."""
    data = illustris.snapshot.loadSubset(
        test_data_path, test_snapshot, 'gas'
    )
    assert data is not None

def test_basic_function():
    """Test without data dependencies."""
    result = illustris.snapshot.partTypeNum('gas')
    assert result == 0

Building Documentation

# Generate API documentation
illustris -docs -generate

# Serve locally with auto-reload
illustris -docs -serve

# Documentation will be available at http://localhost:8000

API Reference

Snapshot Module

• loadSubset(basePath, snapNum, partType, fields=None) - Load particle data
• loadHalo(basePath, snapNum, haloID, partType, fields=None) - Load halo particles
• partTypeNum(partType) - Get particle type number
• snapPath(basePath, snapNum, chunkNum=0) - Get snapshot file path

GroupCat Module

• loadHalos(basePath, snapNum, fields=None) - Load halo catalog
• loadSubhalos(basePath, snapNum, fields=None) - Load subhalo catalog
• loadSingle(basePath, snapNum, haloID=None, subhaloID=None) - Load single object
• gcPath(basePath, snapNum, chunkNum=0) - Get group catalog path

SubLink Module

• loadTree(basePath, snapNum, id, fields=None, onlyMPB=False) - Load merger tree
• numMergers(tree, minMassRatio=1e-10, massPartType='stars') - Count mergers
• treePath(basePath, chunkNum=0) - Get tree file path

License

This project is licensed under the MIT License - see the LICENSE.md file for details.

Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Add tests for new functionality
5. Run the test suite (uv run pytest)
6. Commit your changes (git commit -m 'Add amazing feature')
7. Push to the branch (git push origin feature/amazing-feature)
8. Open a Pull Request

Changelog

Latest Changes

• Simplified CLI: Streamlined data management commands
• Complete Test Data: Single -test option for full dataset downloads
• Modern Testing: Migrated from nose to pytest with fixtures
• API Integration: Direct TNG API access for data downloads
• Improved Documentation: Auto-generated Sphinx docs with CLI integration
• Better Error Handling: Robust error handling for missing data
• Smart Test Skipping: Tests automatically adapt to available data

Data Download Features

• Complete Downloads: Full dataset for comprehensive testing (~5.3 GB)
• Intelligent Chunk Detection: Automatically detects all available file chunks
• Automatic TNG API authentication
• Progress indicators for large downloads
• Smart file organization (snapshots, group catalogs, offsets, trees)
• Support for all TNG simulations (TNG50, TNG100, TNG300, Illustris)
• Robust timeout handling for large files

Testing Improvements

• Session-scoped fixtures for efficient testing
• Automatic data directory detection
• Separate tests for data-dependent and basic functionality
• Comprehensive validation of file structures and data integrity
• Smart skip mechanisms for unavailable data
• Detailed test output with progress indicators

Support

• Documentation: https://bjoernbethge.github.io/illustris
• TNG Project: https://www.tng-project.org/
• API Documentation: https://www.tng-project.org/data/docs/api/

Troubleshooting

Common Issues

Permission Denied Errors:

# Make sure you have write permissions to the data directory
chmod 755 data/

Incomplete Downloads:

# Remove corrupted files and re-download
rm -rf data/TNG50-4/output/snapdir_099/
illustris -data -test

Missing API Key:

# Set up your environment file
cp env.example .env
# Edit .env and add your TNG API key

Test Failures:

# Download complete test data for full functionality
illustris -data -test
uv run pytest tests/ -v