From 1f7be5194123ae5123234f3428757e415370a5d9 Mon Sep 17 00:00:00 2001 From: Aaron Steven White Date: Tue, 10 Feb 2026 15:20:04 -0500 Subject: [PATCH] Updates docs. --- docs/api/behavioral.md | 24 ++++++ docs/api/data_collection.md | 24 ++++++ docs/api/dsl.md | 45 ++++++++++++ docs/api/evaluation.md | 17 +++++ docs/api/items.md | 47 ++++++++++++ docs/api/participants.md | 31 ++++++++ docs/api/resources.md | 27 +++++++ docs/api/simulation.md | 106 +++++++++++++++++++++++++++ docs/api/templates.md | 17 +++++ docs/developer-guide/architecture.md | 63 ++++++++++++---- docs/developer-guide/testing.md | 24 +++--- docs/index.md | 4 +- docs/user-guide/concepts.md | 3 +- mkdocs.yml | 6 ++ 14 files changed, 408 insertions(+), 30 deletions(-) create mode 100644 docs/api/behavioral.md create mode 100644 docs/api/data_collection.md create mode 100644 docs/api/dsl.md create mode 100644 docs/api/evaluation.md create mode 100644 docs/api/participants.md create mode 100644 docs/api/simulation.md diff --git a/docs/api/behavioral.md b/docs/api/behavioral.md new file mode 100644 index 0000000..77470b5 --- /dev/null +++ b/docs/api/behavioral.md @@ -0,0 +1,24 @@ +# bead.behavioral + +Behavioral data analytics and extraction utilities for judgment experiment responses. + +## Analytics + +::: bead.behavioral.analytics + options: + show_root_heading: true + show_source: false + +## Extraction + +::: bead.behavioral.extraction + options: + show_root_heading: true + show_source: false + +## Merging + +::: bead.behavioral.merging + options: + show_root_heading: true + show_source: false diff --git a/docs/api/data_collection.md b/docs/api/data_collection.md new file mode 100644 index 0000000..3890603 --- /dev/null +++ b/docs/api/data_collection.md @@ -0,0 +1,24 @@ +# bead.data_collection + +Data retrieval and integration from JATOS and Prolific platforms. + +## JATOS Integration + +::: bead.data_collection.jatos + options: + show_root_heading: true + show_source: false + +## Prolific Integration + +::: bead.data_collection.prolific + options: + show_root_heading: true + show_source: false + +## Data Merging + +::: bead.data_collection.merger + options: + show_root_heading: true + show_source: false diff --git a/docs/api/dsl.md b/docs/api/dsl.md new file mode 100644 index 0000000..457efe1 --- /dev/null +++ b/docs/api/dsl.md @@ -0,0 +1,45 @@ +# bead.dsl + +Domain-Specific Language for constraint expressions used in template slot filling and list partitioning. + +## Parser + +::: bead.dsl.parser + options: + show_root_heading: true + show_source: false + +## Evaluator + +::: bead.dsl.evaluator + options: + show_root_heading: true + show_source: false + +## Standard Library + +::: bead.dsl.stdlib + options: + show_root_heading: true + show_source: false + +## Abstract Syntax Tree + +::: bead.dsl.ast + options: + show_root_heading: true + show_source: false + +## Context + +::: bead.dsl.context + options: + show_root_heading: true + show_source: false + +## Errors + +::: bead.dsl.errors + options: + show_root_heading: true + show_source: false diff --git a/docs/api/evaluation.md b/docs/api/evaluation.md new file mode 100644 index 0000000..d5aaaad --- /dev/null +++ b/docs/api/evaluation.md @@ -0,0 +1,17 @@ +# bead.evaluation + +Metrics and evaluation utilities for convergence detection and inter-annotator agreement. + +## Convergence Detection + +::: bead.evaluation.convergence + options: + show_root_heading: true + show_source: false + +## Inter-Annotator Agreement + +::: bead.evaluation.interannotator + options: + show_root_heading: true + show_source: false diff --git a/docs/api/items.md b/docs/api/items.md index 182d784..e0f0b53 100644 --- a/docs/api/items.md +++ b/docs/api/items.md @@ -100,3 +100,50 @@ Stage 3 of the bead pipeline: experimental item construction with 9 task types. options: show_root_heading: true show_source: false + +## Model Adapters + +::: bead.items.adapters.base + options: + show_root_heading: true + show_source: false + +::: bead.items.adapters.huggingface + options: + show_root_heading: true + show_source: false + +::: bead.items.adapters.openai + options: + show_root_heading: true + show_source: false + +::: bead.items.adapters.anthropic + options: + show_root_heading: true + show_source: false + +::: bead.items.adapters.google + options: + show_root_heading: true + show_source: false + +::: bead.items.adapters.togetherai + options: + show_root_heading: true + show_source: false + +::: bead.items.adapters.sentence_transformers + options: + show_root_heading: true + show_source: false + +::: bead.items.adapters.registry + options: + show_root_heading: true + show_source: false + +::: bead.items.adapters.api_utils + options: + show_root_heading: true + show_source: false diff --git a/docs/api/participants.md b/docs/api/participants.md new file mode 100644 index 0000000..ac33ebb --- /dev/null +++ b/docs/api/participants.md @@ -0,0 +1,31 @@ +# bead.participants + +Participant metadata models and collection management. + +## Models + +::: bead.participants.models + options: + show_root_heading: true + show_source: false + +## Collection + +::: bead.participants.collection + options: + show_root_heading: true + show_source: false + +## Merging + +::: bead.participants.merging + options: + show_root_heading: true + show_source: false + +## Metadata Specification + +::: bead.participants.metadata_spec + options: + show_root_heading: true + show_source: false diff --git a/docs/api/resources.md b/docs/api/resources.md index 6792d46..3644535 100644 --- a/docs/api/resources.md +++ b/docs/api/resources.md @@ -56,3 +56,30 @@ Stage 1 of the bead pipeline: lexical items, templates, and constraints. options: show_root_heading: true show_source: false + +## Resource Adapters + +::: bead.resources.adapters.base + options: + show_root_heading: true + show_source: false + +::: bead.resources.adapters.glazing + options: + show_root_heading: true + show_source: false + +::: bead.resources.adapters.unimorph + options: + show_root_heading: true + show_source: false + +::: bead.resources.adapters.cache + options: + show_root_heading: true + show_source: false + +::: bead.resources.adapters.registry + options: + show_root_heading: true + show_source: false diff --git a/docs/api/simulation.md b/docs/api/simulation.md new file mode 100644 index 0000000..d1a0e08 --- /dev/null +++ b/docs/api/simulation.md @@ -0,0 +1,106 @@ +# bead.simulation + +Framework for simulating annotator responses with configurable noise models and task-specific strategies. + +## Runner + +::: bead.simulation.runner + options: + show_root_heading: true + show_source: false + +## Annotators + +::: bead.simulation.annotators.base + options: + show_root_heading: true + show_source: false + +::: bead.simulation.annotators.oracle + options: + show_root_heading: true + show_source: false + +::: bead.simulation.annotators.random + options: + show_root_heading: true + show_source: false + +::: bead.simulation.annotators.lm_based + options: + show_root_heading: true + show_source: false + +::: bead.simulation.annotators.distance_based + options: + show_root_heading: true + show_source: false + +## Noise Models + +::: bead.simulation.noise_models.base + options: + show_root_heading: true + show_source: false + +::: bead.simulation.noise_models.temperature + options: + show_root_heading: true + show_source: false + +::: bead.simulation.noise_models.random_noise + options: + show_root_heading: true + show_source: false + +::: bead.simulation.noise_models.systematic + options: + show_root_heading: true + show_source: false + +## Task-Specific Strategies + +::: bead.simulation.strategies.base + options: + show_root_heading: true + show_source: false + +::: bead.simulation.strategies.forced_choice + options: + show_root_heading: true + show_source: false + +::: bead.simulation.strategies.ordinal_scale + options: + show_root_heading: true + show_source: false + +::: bead.simulation.strategies.binary + options: + show_root_heading: true + show_source: false + +::: bead.simulation.strategies.categorical + options: + show_root_heading: true + show_source: false + +::: bead.simulation.strategies.multi_select + options: + show_root_heading: true + show_source: false + +::: bead.simulation.strategies.magnitude + options: + show_root_heading: true + show_source: false + +::: bead.simulation.strategies.free_text + options: + show_root_heading: true + show_source: false + +::: bead.simulation.strategies.cloze + options: + show_root_heading: true + show_source: false diff --git a/docs/api/templates.md b/docs/api/templates.md index a4cab4e..1dd96a0 100644 --- a/docs/api/templates.md +++ b/docs/api/templates.md @@ -32,3 +32,20 @@ Stage 2 of the bead pipeline: template filling strategies and constraint resolut options: show_root_heading: true show_source: false + +## Template Adapters + +::: bead.templates.adapters.base + options: + show_root_heading: true + show_source: false + +::: bead.templates.adapters.huggingface + options: + show_root_heading: true + show_source: false + +::: bead.templates.adapters.cache + options: + show_root_heading: true + show_source: false diff --git a/docs/developer-guide/architecture.md b/docs/developer-guide/architecture.md index 5edcb28..7f21800 100644 --- a/docs/developer-guide/architecture.md +++ b/docs/developer-guide/architecture.md @@ -12,7 +12,7 @@ bead implements a 6-stage pipeline for constructing, deploying, and analyzing la |-------|--------|---------| | 1 | `resources/` | Lexical items and templates with constraints | | 2 | `templates/` | Template filling strategies (exhaustive, stratified, random) | -| 3 | `items/` | Experimental item construction (8 task types) | +| 3 | `items/` | Experimental item construction (9 task types) | | 4 | `lists/` | List partitioning with constraint satisfaction | | 5 | `deployment/` | jsPsych experiment generation for JATOS | | 6 | `active_learning/` | Training with human-in-the-loop convergence | @@ -40,11 +40,12 @@ At every step, objects store only UUID references to their sources, never copyin ## Module Organization -bead consists of 14 top-level modules organized by function: +bead consists of 17 top-level modules organized by function: ### Core Pipeline Stages (6 modules) **bead/resources/** - Stage 1: Lexical items and templates + - `lexical_item.py`: LexicalItem, MultiWordExpression - `lexicon.py`: Lexicon collection - `template.py`: Template, Slot, TemplateSequence, TemplateTree @@ -56,6 +57,7 @@ bead consists of 14 top-level modules organized by function: - `template_generation.py`: Template generation utilities **bead/templates/** - Stage 2: Template filling + - `filler.py`: TemplateFiller (main engine) - `strategies.py`: Exhaustive, Random, Stratified strategies - `resolver.py`: ConstraintResolver @@ -63,19 +65,23 @@ bead consists of 14 top-level modules organized by function: - `combinatorics.py`: Combinatorial generation utilities **bead/items/** - Stage 3: Item construction + - `item.py`: Item, UnfilledSlot, ModelOutput, ItemCollection - `item_template.py`: ItemTemplate, TaskType, JudgmentType, ChunkingSpec - `forced_choice.py`: create_forced_choice_item() and batch utilities - `ordinal_scale.py`, `binary.py`, `categorical.py`, `multi_select.py`, - `magnitude.py`, `free_text.py`, `cloze.py`: Task-type utilities (8 types) + `magnitude.py`, `free_text.py`, `cloze.py`: Task-type utilities +- `spans.py`: Span annotation data models +- `span_labeling.py`: Span labeling utilities (9th task type) - `validation.py`: validate_item_for_task_type() - `constructor.py`: ItemConstructor for item creation - `generation.py`: Item generation utilities - `scoring.py`: Item scoring functions -- `adapters/`: Model integrations (HuggingFace, OpenAI, Anthropic, Google) +- `adapters/`: Model integrations (HuggingFace, OpenAI, Anthropic, Google, TogetherAI) - `cache.py`: Content-addressable cache for model outputs **bead/lists/** - Stage 4: List partitioning + - `experiment_list.py`: ExperimentList (stores item UUIDs) - `list_collection.py`: ListCollection - `constraints.py`: ListConstraint (8 types), BatchConstraint (4 types) @@ -84,6 +90,7 @@ bead consists of 14 top-level modules organized by function: - `stratification.py`: Stratification strategies **bead/deployment/** - Stage 5: Experiment generation + - `jspsych/generator.py`: JsPsychExperimentGenerator (batch mode only) - `jspsych/config.py`: ExperimentConfig, ListDistributionStrategy - `jspsych/trials.py`: Trial generation for jsPsych 8.x @@ -94,10 +101,11 @@ bead consists of 14 top-level modules organized by function: - `distribution.py`: 8 list distribution strategies **bead/active_learning/** - Stage 6: Training and convergence + - `loop.py`: ActiveLearningLoop orchestration - `selection.py`: UncertaintySampler for item selection - `strategies.py`: Active learning strategies -- `models/`: Task-specific models (8 types matching items/) +- `models/`: Task-specific models (9 types matching items/) - `forced_choice.py`: ForcedChoiceModel with GLMM support - `base.py`: ActiveLearningModel interface - `random_effects.py`: RandomEffectsManager for mixed effects @@ -110,9 +118,10 @@ bead consists of 14 top-level modules organized by function: - `registry.py`: Trainer registry - `config.py`: MixedEffectsConfig, RandomEffectsSpec -### Supporting Modules (8 modules) +### Supporting Modules (11 modules) **bead/data/** - Foundation layer + - `base.py`: BeadBaseModel (UUID, timestamps, metadata) - `identifiers.py`: generate_uuid() (UUIDv7) - `timestamps.py`: now_iso8601() (ISO 8601 timestamps) @@ -123,6 +132,7 @@ bead consists of 14 top-level modules organized by function: - `repository.py`: Data repository pattern **bead/dsl/** - Constraint DSL (7 files) + - `parser.py`: Lark-based parser for constraint expressions - `evaluator.py`: DSL evaluation engine - `stdlib.py`: Built-in functions (membership, comparison, arithmetic) @@ -132,6 +142,7 @@ bead consists of 14 top-level modules organized by function: - `__init__.py`: Module exports **bead/config/** - Configuration system (18 files) + - `config.py`: ProjectConfig (root configuration) - `paths.py`: PathsConfig - `resources.py`: ResourcesConfig @@ -144,12 +155,12 @@ bead consists of 14 top-level modules organized by function: - Plus 9 other modules: `defaults.py`, `env.py`, `loader.py`, `logging.py`, `model.py`, `profiles.py`, `serialization.py`, `validation.py`, `__init__.py` **bead/evaluation/** - Metrics and reporting + - `convergence.py`: ConvergenceDetector (Krippendorff's alpha) - `interannotator.py`: InterAnnotatorMetrics (Cohen, Fleiss, Krippendorff) -- `model_metrics.py`: ModelMetrics for classification -- `cross_validation.py`: CrossValidator for validation strategies **bead/simulation/** - Simulation framework + - `annotators/`: Simulated annotators - `base.py`: Annotator interface - `lm_based.py`: Language model annotators @@ -161,21 +172,24 @@ bead consists of 14 top-level modules organized by function: - `temperature.py`: Temperature-based noise - `random_noise.py`: Random noise injection - `systematic.py`: Systematic noise patterns -- `strategies/`: Task-specific simulation strategies (8 types) +- `strategies/`: Task-specific simulation strategies (9 types) - `base.py`: Strategy interface - One strategy per task type: `binary.py`, `categorical.py`, `cloze.py`, `forced_choice.py`, `free_text.py`, `magnitude.py`, `multi_select.py`, `ordinal_scale.py` - `dsl_extension/`: DSL extensions for simulation - `runner.py`: Simulation orchestration **bead/data_collection/** - Data retrieval + - `jatos.py`: JATOS API client for downloading results - `prolific.py`: Prolific metadata integration - `merger.py`: Merge JATOS results with Prolific metadata **bead/adapters/** - External resources + - `huggingface.py`: HuggingFace model integration **bead/cli/** - Command-line interface + - `main.py`: Click root command (entry point) - `resources.py`: Resource commands - `templates.py`: Template commands @@ -198,6 +212,25 @@ bead consists of 14 top-level modules organized by function: - `completion.py`: Shell completion - `utils.py`: CLI utilities +**bead/behavioral/** - Behavioral analytics + +- `analytics.py`: JudgmentAnalytics and aggregation +- `extraction.py`: Extract behavioral measures from experiment responses +- `merging.py`: Merge behavioral data across participants and sessions + +**bead/participants/** - Participant metadata + +- `models.py`: Participant, ParticipantIDMapping models +- `collection.py`: ParticipantCollection management +- `merging.py`: Merge participant data from multiple sources +- `metadata_spec.py`: ParticipantMetadataSpec and FieldSpec validation + +**bead/tokenization/** - Multilingual tokenization + +- `tokenizers.py`: WhitespaceTokenizer, SpacyTokenizer, StanzaTokenizer +- `config.py`: TokenizerConfig, TokenizerBackend +- `alignment.py`: Token-to-character alignment utilities + ## Design Principles bead follows five core design principles that guide all architectural decisions: @@ -468,11 +501,11 @@ partitioner.partition_with_batch_constraints( ### Task-Type Utilities Pattern -**Decision**: Provide 8 task-type-specific modules with consistent API for item creation. +**Decision**: Provide 9 task-type-specific modules with consistent API for item creation. -**Task types**: forced_choice, ordinal_scale, binary, categorical, multi_select, magnitude, free_text, cloze +**Task types**: forced_choice, ordinal_scale, binary, categorical, multi_select, magnitude, free_text, cloze, span_labeling -**API pattern** (consistent across all 8 types): +**API pattern** (consistent across all 9 types): ```python # Core creation function def create_forced_choice_item( @@ -519,7 +552,7 @@ validate_item_for_task_type(item, "forced_choice") # Raises ValueError if inval **Rationale**: 1. **Correctness**: Type-specific utilities enforce correct structure (e.g., forced_choice requires n_alternatives metadata) 2. **Discoverability**: IDE autocomplete shows create_forced_choice_item() in bead.items.forced_choice -3. **Consistency**: All 8 task types follow same API pattern +3. **Consistency**: All 9 task types follow same API pattern 4. **Future expansion**: Adding task type 9 follows established pattern **Comparison**: @@ -877,9 +910,9 @@ Items reference filled templates by UUID, not by importing FilledTemplate. Const bead's architecture prioritizes: 1. **Provenance**: UUID-based stand-off annotation creates unbroken provenance chains -2. **Modularity**: 15 modules organized by function, 6 pipeline stages +2. **Modularity**: 17 modules organized by function, 6 pipeline stages 3. **Type Safety**: Full Python 3.13 type hints with Pydantic v2 validation -4. **Flexibility**: Configuration-first design, 8 task types, 12 constraint types +4. **Flexibility**: Configuration-first design, 9 task types, 12 constraint types 5. **Research Validity**: GLMM support, batch deployment, convergence detection Key architectural decisions (no models.py, stand-off annotation, task-type utilities, GLMM modes, batch-only deployment, 12-type constraints, content-addressable caching) reflect lessons learned from production linguistic research workflows. diff --git a/docs/developer-guide/testing.md b/docs/developer-guide/testing.md index ca24e15..447fd78 100644 --- a/docs/developer-guide/testing.md +++ b/docs/developer-guide/testing.md @@ -101,33 +101,33 @@ uv run pytest tests/ **Specific module**: ```bash -uv run uv run pytest tests/resources/ -uv run uv run pytest tests/lists/ +uv run pytest tests/resources/ +uv run pytest tests/lists/ ``` **Specific file**: ```bash -uv run uv run uv run pytest tests/resources/test_lexical_item.py +uv run pytest tests/resources/test_lexical_item.py ``` **Specific test**: ```bash -uv run uv run uv run uv run pytest tests/resources/test_lexical_item.py::TestLexicalItemCreation::test_create_with_all_fields +uv run pytest tests/resources/test_lexical_item.py::TestLexicalItemCreation::test_create_with_all_fields ``` **With verbose output**: ```bash -uv run uv run pytest tests/ -v +uv run pytest tests/ -v ``` **Stop on first failure**: ```bash -uv run uv run pytest tests/ -x +uv run pytest tests/ -x ``` **Show print statements**: ```bash -uv run uv run pytest tests/ -s +uv run pytest tests/ -s ``` ## Fixtures @@ -289,7 +289,7 @@ bead uses pytest-cov to measure code coverage. Target >90% coverage for all modu **Run with coverage**: ```bash -uv run uv run pytest tests/ --cov=bead --cov-report=term-missing +uv run pytest tests/ --cov=bead --cov-report=term-missing ``` Output shows coverage per file with uncovered line numbers: @@ -316,7 +316,7 @@ The "Missing" column shows line numbers not covered by tests. Generate visual coverage report: ```bash -uv run uv run pytest tests/ --cov=bead --cov-report=html +uv run pytest tests/ --cov=bead --cov-report=html ``` Open htmlcov/index.html in a browser. This shows: @@ -740,7 +740,7 @@ def test_openai_integration(): Run tests excluding slow tests: ```bash -uv run uv run pytest tests/ -m "not slow" +uv run pytest tests/ -m "not slow" ``` ## Continuous Integration @@ -828,7 +828,7 @@ def sample_new_feature_data(): ### 4. Run Tests Locally ```bash -uv run uv run uv run pytest tests/lists/test_new_feature.py -v +uv run pytest tests/lists/test_new_feature.py -v ``` Verify all tests pass and coverage is >90%. @@ -850,6 +850,6 @@ Follow these testing practices: 7. **Use parametrize** for testing multiple inputs 8. **Include doctest examples** in docstrings 9. **Run tests before committing**: `uv run pytest tests/` -10. **Check coverage**: `uv run uv run pytest tests/ --cov=bead --cov-report=term-missing` +10. **Check coverage**: `uv run pytest tests/ --cov=bead --cov-report=term-missing` For architecture details, see [architecture.md](architecture.md). For contribution guidelines, see [contributing.md](contributing.md). For development setup, see [setup.md](setup.md). diff --git a/docs/index.md b/docs/index.md index 0df7531..1e655f2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,7 +8,7 @@ bead implements a 6-stage pipeline for linguistic experiment design: 1. **Resources**: lexical items and templates with constraints 2. **Templates**: template filling strategies (exhaustive, random, stratified, MLM, mixed) -3. **Items**: experimental item construction (8 task types) +3. **Items**: experimental item construction (9 task types) 4. **Lists**: list partitioning with constraint satisfaction 5. **Deployment**: jsPsych 8.x batch experiment generation for JATOS 6. **Training**: active learning with GLMM support and convergence detection @@ -16,7 +16,7 @@ bead implements a 6-stage pipeline for linguistic experiment design: ## Key Features - **Stand-off annotation** with UUID-based references for provenance tracking -- **8 task types**: forced-choice, ordinal scale, binary, categorical, multi-select, magnitude, free text, cloze +- **9 task types**: forced-choice, ordinal scale, binary, categorical, multi-select, magnitude, free text, cloze, span labeling - **GLMM support**: Generalized Linear Mixed Models with random effects - **Batch deployment**: server-side list distribution via JATOS batch sessions - **Language-agnostic**: works with any language supported by UniMorph diff --git a/docs/user-guide/concepts.md b/docs/user-guide/concepts.md index ebf2200..e0171c2 100644 --- a/docs/user-guide/concepts.md +++ b/docs/user-guide/concepts.md @@ -83,7 +83,7 @@ Output: `filled_templates/*.jsonl` Convert filled templates into experimental items with task-specific structure. -- **8 task types**: forced-choice, ordinal scale, binary, categorical, multi-select, magnitude, free text, cloze +- **9 task types**: forced-choice, ordinal scale, binary, categorical, multi-select, magnitude, free text, cloze, span labeling - **Factory functions**: task-specific constructors with validation - **Batch creation**: generate items from groups or cross-products @@ -135,6 +135,7 @@ Bead distinguishes between **task types** (UI presentation) and **judgment types - `magnitude`: enter numeric value (reading time, confidence) - `free_text`: open-ended text response - `cloze`: fill in blanks +- `span_labeling`: annotate text spans with labels **Judgment types** describe the measurement goal: diff --git a/mkdocs.yml b/mkdocs.yml index f31d346..3d8c9c6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -95,6 +95,12 @@ nav: - bead.deployment: api/deployment.md - bead.tokenization: api/tokenization.md - bead.active_learning: api/active_learning.md + - bead.simulation: api/simulation.md + - bead.evaluation: api/evaluation.md + - bead.dsl: api/dsl.md + - bead.behavioral: api/behavioral.md + - bead.participants: api/participants.md + - bead.data_collection: api/data_collection.md - bead.config: api/config.md - bead.data: api/data.md - Developer Guide: