Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 6 additions & 7 deletions docs/design/ltm--loops-that-matter.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,7 @@ For a link from `x` to `z` where `z = f(x, y, ...)`:
replacing `x` inside `x_rate`, or corrupting function names like `MAX`).
4. The link score is:
```
if (TIME = PREVIOUS(TIME)) then 0
if (TIME = INITIAL_TIME) then 0
else if ((z - PREVIOUS(z)) = 0) OR ((x - PREVIOUS(x)) = 0) then 0
else ABS(SAFEDIV((partial_eq - PREVIOUS(z)), (z - PREVIOUS(z)), 0))
* SIGN(SAFEDIV((partial_eq - PREVIOUS(z)), (x - PREVIOUS(x)), 0))
Expand All @@ -267,7 +267,7 @@ The ratio is wrapped in `ABS()` because flow-to-stock polarity is structural:
inflows always contribute positively (+1), outflows negatively (-1). The sign
is applied outside the absolute value. This equation returns 0 for the first
two timesteps (insufficient history for second-order differences), guarded by
`TIME = PREVIOUS(TIME)` and `PREVIOUS(TIME) = PREVIOUS(PREVIOUS(TIME))`.
`TIME = INITIAL_TIME` and `PREVIOUS(TIME, INITIAL_TIME) = INITIAL_TIME`.

### Stock-to-Flow Links

Expand Down Expand Up @@ -540,11 +540,10 @@ computational interval" strategy.
module-internal stocks to loop stock lists) is an implementation-specific
extension that enables correct cycle partitioning.

4. **PREVIOUS via stdlib module**: The `PREVIOUS()` function used in link score
equations is implemented as a standard library module (`stdlib/previous.stmx`)
using a stock-and-flow structure, not as a built-in function. This affects
initial-timestep behavior: `TIME = PREVIOUS(TIME)` is used to detect the first
timestep and return 0.
4. **PREVIOUS is intrinsic**: The `PREVIOUS()` function used in link score
equations is compiled as an intrinsic two-argument builtin. Unary syntax is
desugared to `PREVIOUS(x, 0)`. LTM first-timestep behavior is handled
explicitly with `TIME = INITIAL_TIME`.

5. **Relative loop score formula**: The implementation uses
`SAFEDIV(loop_score, sum_of_abs_scores, 0)` with explicit division-by-zero
Expand Down
9 changes: 0 additions & 9 deletions docs/tech-debt.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,15 +159,6 @@ Known debt items consolidated from CLAUDE.md files and codebase analysis. Each e
- **Owner**: unassigned
- **Last reviewed**: 2026-02-22

### 20. parse_source_variable Missing module_idents Context

- **Component**: simlin-engine (src/simlin-engine/src/db.rs)
- **Severity**: low
- **Description**: `parse_source_variable` calls `parse_var` (module_idents = None) rather than `parse_var_with_module_context`. Computing module_idents requires a model-level view (knowing which sibling variables are module-expanded), but salsa tracks individual variables. The consequence: `PREVIOUS(x)` where `x` is a user-written stdlib-call aux (e.g., `x = SMTH1(...)`) will compile to LoadPrev via the salsa-cached path instead of falling through to module expansion. The U+205A composite-identifier check in builtins_visitor.rs catches already-expanded names (the common case after a full compile), so this gap only affects the raw user-facing variable name at incremental-parse time. A fix would require computing a `module_idents` salsa input (set of user-written stdlib-call aux/flow names) at the model level and threading it into per-variable parse.
- **Measure**: Write a test: model with `x = SMTH1(input, 1)` and `y = PREVIOUS(x)`, verify y compiles to module expansion (not LoadPrev) via the salsa incremental path.
- **Owner**: unassigned
- **Last reviewed**: 2026-03-01

### 19. Flaky Hypothesis Tests in pysimlin Due to Slow Input Generation

- **Component**: pysimlin (src/pysimlin/tests/test_json_types.py)
Expand Down
6 changes: 3 additions & 3 deletions src/simlin-engine/CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Equation text flows through these stages in order:
1. **`src/lexer/`** - Tokenizer for equation syntax
2. **`src/parser/`** - Recursive descent parser producing `Expr0` AST
3. **`src/ast/`** - AST type system with progressive lowering: `Expr0` (parsed) -> `Expr1` (modules expanded) -> `Expr2` (dimensions resolved) -> `Expr3` (subscripts expanded). `array_view.rs` tracks array dimensions and sparsity.
4. **`src/builtins.rs`** - Builtin function definitions (e.g. `MIN`, `PULSE`, `LOOKUP`, `QUANTUM`, `SSHAPE`, `VECTOR SELECT`, `VECTOR ELM MAP`, `VECTOR SORT ORDER`, `ALLOCATE AVAILABLE`, `NPV`, `MODULO`, `PREVIOUS`, `INIT`). `is_stdlib_module_function()` is the authoritative predicate for deciding whether a function name expands to a stdlib module; shared by `equation_is_stdlib_call()` (pre-scan) and `contains_stdlib_call()` (walk-time). `builtins_visitor.rs` handles implicit module instantiation and routes `PREVIOUS`/`INIT` between opcode compilation and module expansion: 1-arg `PREVIOUS(var)` for simple scalar variables compiles to `LoadPrev`; 2-arg `PREVIOUS(x, init_val)`, nested, module-variable, and expression arguments fall through to module expansion. `INIT(x)` always compiles to `LoadInitial`. Tracks `module_idents` to prevent `PREVIOUS(module_var)` from using `LoadPrev` (modules occupy multiple slots).
4. **`src/builtins.rs`** - Builtin function definitions (e.g. `MIN`, `PULSE`, `LOOKUP`, `QUANTUM`, `SSHAPE`, `VECTOR SELECT`, `VECTOR ELM MAP`, `VECTOR SORT ORDER`, `ALLOCATE AVAILABLE`, `NPV`, `MODULO`, `PREVIOUS`, `INIT`). `is_stdlib_module_function()` is the authoritative predicate for deciding whether a function name expands to a stdlib module; shared by `equation_is_stdlib_call()` (pre-scan) and `contains_stdlib_call()` (walk-time). `builtins_visitor.rs` handles implicit module instantiation and PREVIOUS/INIT helper rewriting: unary `PREVIOUS(x)` desugars to `PREVIOUS(x, 0)`, direct scalar args compile to `LoadPrev`, and module-backed or expression args are first rewritten through synthesized scalar helper auxes. `INIT(x)` compiles to `LoadInitial`, using the same helper rewrite when needed. Tracks `module_idents` so `PREVIOUS(module_var)` never reads a multi-slot module directly.
5. **`src/compiler/`** - Multi-pass compilation to bytecode:
- `mod.rs` - Orchestration; includes A2A hoisting logic that detects array-producing builtins (VectorElmMap, VectorSortOrder, AllocateAvailable) during array expansion, hoists them into `AssignTemp` pre-computations, and emits per-element `TempArrayElement` reads
- `context.rs` - Symbol tables and variable metadata; `lower_preserving_dimensions()` skips Pass 1 dimension resolution to keep full array views for array-producing builtins. Handles `@N` position syntax resolution: in scalar context (no active A2A dimension, not inside an array-reducing builtin), `DimPosition(@N)` resolves to a concrete element offset; inside array-reducing builtins (`preserve_wildcards_for_iteration`), dimension views are preserved for iteration
Expand All @@ -29,7 +29,7 @@ Equation text flows through these stages in order:

- **`src/common.rs`** - Error types (`ErrorCode` with 100+ variants), `Result`, identifier types (`RawIdent`, `Ident<Canonical>`, dimension/element name types), canonicalization
- **`src/datamodel.rs`** - Core structures: `Project`, `Model`, `Variable`, `Equation` (including `Arrayed` variant with `default_equation` for EXCEPT semantics), `Dimension` (with `mappings: Vec<DimensionMapping>` replacing the old `maps_to` field), `DimensionMapping`, `DataSource`/`DataSourceKind`, `UnitMap`
- **`src/variable.rs`** - Variable variants (`Stock`, `Flow`, `Aux`, `Module`), `ModuleInput`, `Table` (graphical functions). `classify_dependencies()` is the primary API for extracting dependency categories from an AST in a single walk, returning a `DepClassification` with five sets: `all` (every referenced ident), `init_referenced`, `previous_referenced`, `previous_only` (idents only inside PREVIOUS), and `init_only` (idents only inside INIT/PREVIOUS). The older single-purpose functions (`identifier_set`, `init_referenced_idents`, etc.) remain as thin wrappers. `parse_var_with_module_context` accepts a `module_idents` set so `PREVIOUS(module_var)` falls through to module expansion instead of `LoadPrev`.
- **`src/variable.rs`** - Variable variants (`Stock`, `Flow`, `Aux`, `Module`), `ModuleInput`, `Table` (graphical functions). `classify_dependencies()` is the primary API for extracting dependency categories from an AST in a single walk, returning a `DepClassification` with five sets: `all` (every referenced ident), `init_referenced`, `previous_referenced`, `previous_only` (idents only inside PREVIOUS), and `init_only` (idents only inside INIT/PREVIOUS). The older single-purpose functions (`identifier_set`, `init_referenced_idents`, etc.) remain as thin wrappers. `parse_var_with_module_context` accepts a `module_idents` set so `PREVIOUS(module_var)` rewrites through a scalar helper aux instead of `LoadPrev`.
- **`src/dimensions.rs`** - Dimension context and dimension matching for arrays
- **`src/model.rs`** - Model compilation stages (`ModelStage0` -> `ModelStage1` -> `ModuleStage2`), dependency resolution, topological sort. `collect_module_idents` pre-scans datamodel variables to identify which names will expand to modules (preventing incorrect `LoadPrev` compilation). `init_referenced_vars` extends the Initials runlist to include variables referenced by `INIT()` calls, ensuring their values are captured in the `initial_values` snapshot. `check_units` is gated behind `cfg(any(test, feature = "testing"))` (production unit checking uses salsa tracked functions).
- **`src/project.rs`** - `Project` struct aggregating models. `from_salsa(datamodel, db, source_project, cb)` builds a Project from a pre-synced salsa database (all variable parsing comes from salsa-cached results). `from_datamodel(datamodel)` is a convenience wrapper that creates a local DB and syncs. `From<datamodel::Project>`, `with_ltm()`, and `with_ltm_all_links()` are all gated behind `cfg(any(test, feature = "testing"))` (retained only for tests and the AST interpreter cross-validation path); production code uses `db::compile_project_incremental` with `ltm_enabled`/`ltm_discovery_mode` on `SourceProject`.
Expand All @@ -42,7 +42,7 @@ The primary compilation path uses salsa tracked functions for fine-grained incre

- **`src/db.rs`** - `SimlinDb`, `SourceProject`/`SourceModel`/`SourceVariable` salsa inputs, `compile_project_incremental()` entry point, dependency graph computation, diagnostic accumulation via `CompilationDiagnostic` accumulator. `SourceProject` carries `ltm_enabled` and `ltm_discovery_mode` flags for LTM compilation. `Diagnostic` includes a `severity` field (`Error`/`Warning`) and `DiagnosticError` variants: `Equation`, `Model`, `Unit`, `Assembly`. `VariableDeps` includes `init_referenced_vars` to track variables referenced by `INIT()` calls. Dependency extraction uses two calls to `classify_dependencies()` (one for the dt AST, one for the init AST) instead of separate walker functions. `parse_source_variable_with_module_context` is the sole parse entry point (the non-module-context variant was removed). `variable_relevant_dimensions` provides dimension-granularity invalidation: scalar variables produce an empty dimension set so dimension changes never invalidate their parse results.
- **`src/db_analysis.rs`** - Salsa-tracked causal graph analysis: `model_causal_edges`, `model_loop_circuits`, `model_cycle_partitions`, `model_detected_loops`. Produces `DetectedLoop` structs with polarity.
- **`src/db_ltm.rs`** - LTM (Loops That Matter) equation parsing and compilation as salsa tracked functions. Handles link scores, loop scores, relative loop scores, and PREVIOUS module expansion for LTM synthetic variables.
- **`src/db_ltm.rs`** - LTM (Loops That Matter) equation parsing and compilation as salsa tracked functions. Handles link scores, loop scores, relative loop scores, and any implicit helper/module vars synthesized while parsing LTM equations.
- **`src/db_diagnostic_tests.rs`** - Verification tests for diagnostic accumulation paths.
- **`src/db_differential_tests.rs`** - Differential tests verifying `classify_dependencies()` produces identical results to the old per-category walker functions, plus fragment-phase agreement tests ensuring dt/init ASTs yield consistent dependency classifications.
- **`src/db_dimension_invalidation_tests.rs`** - Tests for dimension-granularity salsa invalidation: verifying that dimension changes only re-parse variables that reference those dimensions.
Expand Down
24 changes: 20 additions & 4 deletions src/simlin-engine/src/ast/expr1.rs
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,22 @@ impl Expr1 {
"vector_elm_map" => check_arity!(VectorElmMap, 2),
"vector_sort_order" => check_arity!(VectorSortOrder, 2),
"allocate_available" => check_arity!(AllocateAvailable, 3),
"previous" => check_arity!(Previous, 1),
// Unary PREVIOUS(x) desugars to PREVIOUS(x, 0).
// builtins_visitor may have already added the fallback
// at Expr0 level, so both 1-arg and 2-arg forms are valid.
"previous" => {
if args.len() == 1 {
let a = args.remove(0);
let zero = Expr1::Const("0".to_string(), 0.0, loc);
BuiltinFn::Previous(Box::new(a), Box::new(zero))
} else if args.len() == 2 {
let b = args.remove(1);
let a = args.remove(0);
BuiltinFn::Previous(Box::new(a), Box::new(b))
} else {
return eqn_err!(BadBuiltinArgs, loc.start, loc.end);
}
}
"init" => check_arity!(Init, 1),
_ => {
// TODO: this could be a table reference, array reference,
Expand Down Expand Up @@ -419,9 +434,10 @@ impl Expr1 {
Box::new(b.constify_dimensions(scope)),
Box::new(c.constify_dimensions(scope)),
),
BuiltinFn::Previous(a) => {
BuiltinFn::Previous(Box::new(a.constify_dimensions(scope)))
}
BuiltinFn::Previous(a, b) => BuiltinFn::Previous(
Box::new(a.constify_dimensions(scope)),
Box::new(b.constify_dimensions(scope)),
),
BuiltinFn::Init(a) => BuiltinFn::Init(Box::new(a.constify_dimensions(scope))),
};
Expr1::App(func, loc)
Expand Down
5 changes: 4 additions & 1 deletion src/simlin-engine/src/ast/expr2.rs
Original file line number Diff line number Diff line change
Expand Up @@ -805,7 +805,10 @@ impl Expr2 {
ctx.set_allow_dimension_union(prev);
result
}
Previous(e) => Previous(Box::new(Expr2::from(*e, ctx)?)),
Previous(a, b) => Previous(
Box::new(Expr2::from(*a, ctx)?),
Box::new(Expr2::from(*b, ctx)?),
),
Init(e) => Init(Box::new(Expr2::from(*e, ctx)?)),
};
// TODO: Handle array sources for builtin functions that return arrays
Expand Down
12 changes: 8 additions & 4 deletions src/simlin-engine/src/ast/expr3.rs
Original file line number Diff line number Diff line change
Expand Up @@ -189,8 +189,8 @@ impl Expr3 {
| Size(e)
| Stddev(e)
| Sum(e)
| Previous(e)
| Init(e) => e.references_a2a_dimension(),
Previous(a, b) => a.references_a2a_dimension() || b.references_a2a_dimension(),
Max(a, b) | Min(a, b) => {
a.references_a2a_dimension()
|| b.as_ref().is_some_and(|e| e.references_a2a_dimension())
Expand Down Expand Up @@ -965,9 +965,13 @@ impl<'a> Pass1Context<'a> {
req_a2a || pp_a2a || avail_a2a,
)
}
Previous(e) => {
let (new_e, has_a2a) = self.transform_inner(*e);
(Previous(Box::new(new_e)), has_a2a)
Previous(a, b) => {
let (new_a, a_has_a2a) = self.transform_inner(*a);
let (new_b, b_has_a2a) = self.transform_inner(*b);
(
Previous(Box::new(new_a), Box::new(new_b)),
a_has_a2a || b_has_a2a,
)
}
Init(e) => {
let (new_e, has_a2a) = self.transform_inner(*e);
Expand Down
17 changes: 12 additions & 5 deletions src/simlin-engine/src/builtins.rs
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ pub enum BuiltinFn<Expr> {
// ALLOCATE AVAILABLE(request, priority_profile, avail)
AllocateAvailable(Box<Expr>, Box<Expr>, Box<Expr>),
// builtins replacing stdlib modules
Previous(Box<Expr>),
Previous(Box<Expr>, Box<Expr>),
Init(Box<Expr>),
}

Expand Down Expand Up @@ -153,7 +153,7 @@ impl<Expr> BuiltinFn<Expr> {
VectorSortOrder(_, _) => "vector_sort_order",
AllocateAvailable(_, _, _) => "allocate_available",
// builtins replacing stdlib modules
Previous(_) => "previous",
Previous(_, _) => "previous",
Init(_) => "init",
}
}
Expand Down Expand Up @@ -251,7 +251,7 @@ impl<Expr> BuiltinFn<Expr> {
AllocateAvailable(a, b, c) => {
AllocateAvailable(Box::new(f(*a)?), Box::new(f(*b)?), Box::new(f(*c)?))
}
Previous(a) => Previous(Box::new(f(*a)?)),
Previous(a, b) => Previous(Box::new(f(*a)?), Box::new(f(*b)?)),
Init(a) => Init(Box::new(f(*a)?)),
})
}
Expand Down Expand Up @@ -280,7 +280,11 @@ impl<Expr> BuiltinFn<Expr> {
}
Abs(a) | Arccos(a) | Arcsin(a) | Arctan(a) | Cos(a) | Exp(a) | Int(a) | Ln(a)
| Log10(a) | Sign(a) | Sin(a) | Sqrt(a) | Tan(a) | Size(a) | Stddev(a) | Sum(a)
| Previous(a) | Init(a) => f(a),
| Init(a) => f(a),
Previous(a, b) => {
f(a);
f(b);
}
Inf | Pi | Time | TimeStep | StartTime | FinalTime | IsModuleInput(_, _) => {}
Max(a, b) | Min(a, b) => {
f(a);
Expand Down Expand Up @@ -455,8 +459,11 @@ where
| BuiltinFn::Size(a)
| BuiltinFn::Stddev(a)
| BuiltinFn::Sum(a)
| BuiltinFn::Previous(a)
| BuiltinFn::Init(a) => cb(BuiltinContents::Expr(a)),
BuiltinFn::Previous(a, b) => {
cb(BuiltinContents::Expr(a));
cb(BuiltinContents::Expr(b));
}
BuiltinFn::Mean(args) => {
args.iter().for_each(|a| cb(BuiltinContents::Expr(a)));
}
Expand Down
Loading