tap-openproject

A Singer tap for extracting data from OpenProject API. Built with the Meltano Singer SDK.

Features

• ✅ Meltano SDK-based - Modern Singer tap implementation
• ✅ Incremental sync - Efficient data extraction using updatedAt timestamp
• ✅ Stream Maps - Built-in inline transformation support
• ✅ Automatic pagination - Handles large datasets seamlessly
• ✅ Schema validation - Full JSON Schema with type checking
• ✅ Rate limiting & retries - Built-in resilience for API calls
• ✅ Cloud & self-hosted - Works with any OpenProject instance

Installation

Prerequisites

• Python 3.8 or higher
• pip or Poetry

Install from Source

git clone https://github.com/surveilr/tap-openproject.git
cd tap-openproject

# Option 1: Using pip
pip install -e .

# Option 2: Using Poetry
pip install poetry
poetry install

Quick Start

1. Create Configuration

Create a config.json file:

{
  "api_key": "your-openproject-api-key",
  "base_url": "https://your-instance.openproject.com/api/v3",
  "start_date": "2024-01-01T00:00:00Z"
}

2. Discover Available Streams

tap-openproject --config config.json --discover > catalog.json

3. Run Extraction

tap-openproject --config config.json --catalog catalog.json

4. Incremental Sync

# State is automatically managed
tap-openproject --config config.json --catalog catalog.json --state state.json > output.singer

See QUICKSTART.md for more details.

Getting Your API Key

1. Log into your OpenProject instance
2. Click your avatar → My Account
3. Navigate to Access tokens in the sidebar
4. Click + API to generate a new token
5. Copy the token immediately (you won't see it again)
6. Use it as api_key in your configuration

Configuration Options

Configuration Options

Parameter	Type	Required	Default	Description
api_key	string	Yes	-	OpenProject API key
base_url	string	Yes	-	Base URL of OpenProject instance API (include /api/v3)
timeout	integer	No	30	HTTP request timeout in seconds
max_retries	integer	No	3	Maximum retry attempts for failed requests
start_date	datetime	No	-	ISO 8601 date for incremental sync starting point
user_agent	string	No	tap-openproject/0.3.0	User-Agent header value

Available Streams

This tap extracts data from 12 OpenProject API endpoints, organized into three categories:

Core Streams

Stream	Endpoint	Replication	Description
projects	/projects	Incremental	Project metadata with parent relationships
work_packages	/work_packages	Incremental	Tasks, bugs, features with all relationships

Reference Data Streams

Stream	Endpoint	Replication	Description
statuses	/statuses	Full	Work package status definitions
types	/types	Incremental	Work package types (Bug, Task, Feature, etc.)
priorities	/priorities	Full	Priority level definitions
roles	/roles	Full	Role definitions for memberships
users	/users	Incremental	User accounts (may require admin access)

Transactional Streams

Stream	Endpoint	Replication	Description
versions	/versions	Incremental	Project milestones/releases
time_entries	/time_entries	Incremental	Time tracking data
relations	/relations	Full	Work package dependencies
memberships	/memberships	Incremental	Project membership assignments
attachments	/work_packages/{id}/attachments	Full	File attachments (child of work_packages)

Field Flattening

All streams that reference other entities include flattened fields for easier querying. For example, work_packages includes:

• type_id, type_title - Work package type
• status_id, status_title - Current status
• priority_id, priority_title - Priority level
• project_id, project_title - Parent project
• assignee_id, assignee_title - Assigned user
• author_id, author_title - Creator
• version_id, version_title - Target version
• parent_id - Parent work package ID

The attachments stream includes:

• author_id, author_title - User who uploaded the file
• container_id, container_type, container_title - Parent object (WorkPackage, Meeting, etc.)
• download_url - Direct download link

This enables direct JOINs between streams without parsing HAL _links objects

Usage with Meltano

This tap is designed to work seamlessly with Meltano:

# Add the tap to your Meltano project
meltano add extractor tap-openproject --custom

# Configure
meltano config tap-openproject set api_key YOUR_API_KEY
meltano config tap-openproject set base_url https://your-instance.openproject.com/api/v3

# Run extraction
meltano run tap-openproject target-jsonl

Configuration in meltano.yml

plugins:
  extractors:
    - name: tap-openproject
      pip_url: -e /path/to/tap-openproject
      config:
        api_key: ${OPENPROJECT_API_KEY}
        base_url: https://your-instance.openproject.com/api/v3
        start_date: '2024-01-01T00:00:00Z'

Development

Setup Development Environment

git clone https://github.com/surveilr/tap-openproject.git
cd tap-openproject

# Using Poetry (recommended)
poetry install
poetry shell

# Or using pip
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'

Run Tests

poetry run pytest tests/ -v

# With coverage
poetry run pytest tests/ --cov=tap_openproject

Code Quality

# Linting
poetry run ruff check .

# Formatting
poetry run ruff format .

Project Structure

tap-openproject/
├── pyproject.toml           # Poetry dependencies & metadata
├── README.md
├── LICENSE
├── CHANGELOG.md
├── CONTRIBUTING.md
├── QUICKSTART.md
├── meltano-hub.yml          # Meltano Hub plugin definition
├── tap_openproject/
│   ├── __init__.py
│   ├── tap.py               # Main Tap class with SDK
│   ├── streams.py           # Stream definitions (SDK-based)
│   └── schemas/
│       └── projects.json    # JSON Schema for projects stream
├── tests/
│   └── test_projects_stream.py
└── examples/
    ├── config.example.json
    └── openproject.surveilr[singer].py

SDK Capabilities

Built with Meltano Singer SDK, this tap supports:

• ✅ catalog - Stream and property selection
• ✅ discover - Automatic schema discovery
• ✅ state - Incremental replication with bookmarks
• ✅ about - Plugin metadata output
• ✅ stream-maps - Inline data transformation
• ✅ schema-flattening - Automatic nested object flattening
• ✅ batch - Efficient batch processing

# Check capabilities
tap-openproject --about --format=json

Troubleshooting

Authentication Errors (401)

• Verify your API key is correct and hasn't expired
• Ensure the API key has sufficient permissions
• Check that you're using Basic Auth format (handled automatically)

Connection Errors

• Verify base_url includes /api/v3 at the end
• Check that your OpenProject instance is accessible
• Verify network/firewall settings allow outbound HTTPS

Rate Limiting (429)

• The SDK automatically retries with exponential backoff
• Adjust max_retries if needed
• Contact your OpenProject admin if limits persist

Resources

• OpenProject API Documentation
• Meltano Singer SDK
• Singer Specification
• Meltano Hub

Contributing

Contributions are welcome! See CONTRIBUTING.md for guidelines.

Changelog

See CHANGELOG.md for version history.

License

MIT License - see LICENSE file for details.

---

Built with the Meltano Singer SDK