starting to revamp ZGraph

Author: James-Riordan

.dpignore

@@ -0,0 +1,3 @@
+.git
+.github
+.zig-cache

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,216 @@
+# ZGraph
+
+ZGraph is a high-performance, in-memory graph library with:
+
+- **Pluggable storages**: adjacency list, adjacency matrix, incidence matrix.
+- **Compile-time graph semantics**: `directed` / `undirected`, `weighted` / `unweighted`.
+- **Unified algorithms** that operate across storages via a common trait surface.
+- **Portable formats**:
+  - `.zgraph` – human-readable, human-writable
+  - `.zgraphb` – compressed, chunked binary for fast I/O / mmap
+- **Robust conversion strategies** (duplicate, streamed, chunked, copy-on-write) to switch layouts under tight memory budgets.
+
+ZGraphDB (paged container) builds on the same chunks and APIs; algorithms continue to live in ZGraph.
+
+## Status
+
+- ✅ Core storages compile and pass tests on Zig 0.16.
+- ✅ `AdjacencyList`, `AdjacencyMatrix`, `IncidenceMatrix` with CRUD + neighbor ops.
+- ✅ `.zgraph` / `.zgraphb` specifications documented; parsers/writers in progress.
+- 🚧 Conversion strategies module & CLI utilities.
+
+See **[ZGraph.Spec.md](./ZGraph.Spec.md)** for the canonical checklist and roadmap.
+
+## Install
+
+```bash
+zig build test
+zig build run
+```
+
+Zig: 0.16.x
+
+## Quick start
+
+```c++
+const GS = @import("src/lib/core/graph_storage.zig").GraphStorage;
+const Storage = GS(.AdjacencyList, true, true); // directed, weighted
+
+var g = Storage.init(std.heap.page_allocator);
+defer g.deinit();
+
+try g.addEdge(0, 1, 3.5);
+try g.addEdge(1, 1, 7.5);
+
+if (g.getNeighbors(0)) |edges| {
+    // ...
+}
+```
+
+### Convert storage (e.g., list → matrix) with a memory budget
+
+```zig
+const conv = @import("src/lib/conversion/strategies.zig");
+var dst = try conv.convertStorage(
+    std.heap.page_allocator,
+    Storage,                             // source type
+    .AdjacencyMatrix, true, true,        // dest type + semantics
+    &g,
+    .Streamed,                           // strategy
+    .{ .max_bytes_in_flight = 4 * 1024 * 1024 * 1024 }, // 4 GiB
+);
+defer dst.deinit();
+```
+
+### Import / Export
+
+```bash
+# text -> binary
+zig build run -- convert --in graph.zgraph --out graph.zgraphb
+
+# validate and print stats
+zig build run -- stats --in graph.zgraphb
+```
+
+## Design pillars
+
+- **Predictable performance**: zero-copy views, cache-friendly CSR, zstd block compression.
+- **Compatibility**: readers ignore unknown chunks; text & binary round-trip.
+- **Safety**: memory ownership is explicit; threads guarded by mutexes where needed.
+- **Tested**: unit, property, and fuzz tests; algorithm parity across storages.
+
+## Repo layout (idealized)
+
+```
+ZGraph/
+├─ README.md
+├─ ZGraph.Spec.md                  # Source of truth (feature checklist)
+├─ LICENSE
+├─ build.zig
+├─ zig.mod
+├─ docs/
+│  ├─ formats/
+│  │  ├─ zgraph-text-spec.md
+│  │  └─ zgraphb-binary-spec.md
+│  ├─ algorithms/
+│  │  ├─ traversal.md
+│  │  ├─ shortest_paths.md
+│  │  ├─ connectivity.md
+│  │  ├─ flow_cut.md
+│  │  ├─ centrality.md
+│  │  └─ spectral.md
+│  ├─ storage_design.md
+│  ├─ conversion_strategies.md
+│  ├─ perf_tuning.md
+│  ├─ safety_and_threading.md
+│  └─ roadmap.md
+├─ examples/
+│  ├─ tiny/
+│  │  ├─ triangle.zgraph
+│  │  ├─ weighted_digraph.zgraph
+│  │  └─ attributes.zgraph
+│  ├─ conversions/
+│  │  ├─ list_to_matrix.zig
+│  │  └─ streamed_live_convert.zig
+│  └─ cli/
+│     ├─ import_export.zig
+│     └─ metrics.zig
+├─ src/
+│  ├─ root.zig
+│  ├─ lib/
+│  │  ├─ core/
+│  │  │  ├─ node.zig
+│  │  │  ├─ edge.zig
+│  │  │  ├─ graph_storage.zig
+│  │  │  ├─ iterators.zig
+│  │  │  └─ attributes.zig
+│  │  ├─ data_structures/
+│  │  │  ├─ adjacency_list.zig
+│  │  │  ├─ adjacency_matrix.zig
+│  │  │  └─ incidence_matrix.zig
+│  │  ├─ formats/
+│  │  │  ├─ zgraph_text.zig
+│  │  │  ├─ zgraphb.zig
+│  │  │  └─ chunk_provider.zig
+│  │  ├─ conversion/
+│  │  │  ├─ strategies.zig
+│  │  │  ├─ duplicate_convert.zig
+│  │  │  ├─ streamed_convert.zig
+│  │  │  ├─ chunked_convert.zig
+│  │  │  └─ cow_convert.zig
+│  │  ├─ indices/
+│  │  │  ├─ csr.zig
+│  │  │  └─ degree_table.zig
+│  │  ├─ algorithms/
+│  │  │  ├─ traversal/
+│  │  │  │  ├─ bfs.zig
+│  │  │  │  └─ dfs.zig
+│  │  │  ├─ shortest_paths/
+│  │  │  │  ├─ dijkstra.zig
+│  │  │  │  ├─ bellman_ford.zig
+│  │  │  │  └─ floyd_warshall.zig
+│  │  │  ├─ connectivity/
+│  │  │  │  ├─ union_find.zig
+│  │  │  │  └─ strongly_connected.zig
+│  │  │  ├─ flow_cut/
+│  │  │  │  ├─ edmonds_karp.zig
+│  │  │  │  └─ dinic.zig
+│  │  │  ├─ centrality/
+│  │  │  │  ├─ pagerank.zig
+│  │  │  │  └─ betweenness.zig
+│  │  │  └─ spectral/
+│  │  │     ├─ laplacian.zig
+│  │  │     └─ power_iteration.zig
+│  │  ├─ io/
+│  │  │  ├─ file.zig
+│  │  │  ├─ csv.zig
+│  │  │  └─ zstd.zig
+│  │  ├─ mem/
+│  │  │  ├─ alloc.zig
+│  │  │  └─ pool.zig
+│  │  └─ util/
+│  │     ├─ bitset.zig
+│  │     ├─ hash.zig
+│  │     └─ time.zig
+│  └─ cli/
+│     ├─ zgraph.zig
+│     └─ subcommands/
+│        ├─ convert.zig
+│        ├─ validate.zig
+│        ├─ build_index.zig
+│        └─ stats.zig
+├─ tests/
+│  ├─ unit/
+│  │  ├─ adjacency_list_test.zig
+│  │  ├─ adjacency_matrix_test.zig
+│  │  ├─ incidence_matrix_test.zig
+│  │  ├─ formats_text_test.zig
+│  │  ├─ formats_binary_test.zig
+│  │  ├─ conversion_strategies_test.zig
+│  │  └─ algorithms_smoke_test.zig
+│  ├─ property/
+│  │  ├─ graph_idempotence_test.zig
+│  │  └─ random_graph_agreement_test.zig
+│  ├─ fuzz/
+│  │  └─ fuzzer.zig
+│  └─ data/
+│     ├─ tiny_graphs/*.zgraph
+│     └─ medium_graphs/*.zgraph
+├─ tools/
+│  ├─ scripts/
+│  │  ├─ gen_random_graph.zig
+│  │  └─ bench_matrix_vs_list.zig
+│  └─ perf/
+│     └─ flamegraph_instructions.md
+└─ benches/
+   ├─ micro/
+   │  ├─ csr_iter_bench.zig
+   │  └─ parse_zgraph_bench.zig
+   └─ macro/
+      ├─ bfs_vs_dijkstra_bench.zig
+      └─ convert_streamed_vs_duplicate.zig
+```
+
+## License
+
+MIT

.github/workflows/bench.yml

@@ -0,0 +1,141 @@
+# ZGraph — Unified Master Plan (Reconciled)
+
+This document merges the current repository state with the idealized goals and becomes the **single source of truth**. Items are marked as:
+- ✅ Done
+- 🚧 Partial / scaffolding present
+- ⭕ Not started
+
+---
+
+## A. Semantics & Types
+- ✅ A1. Compile-time graph semantics (`directed`, `weighted`) across storages
+- ✅ A2. Self-loops & multi-edges covered by tests
+- ⭕ A3. Attributes (node/edge KV with typed values)
+- ✅ A4. Numeric node IDs (u64/usize) in use
+- ⭕ A5. Storage-agnostic iterators (common trait iterators)
+
+## B. Storage Backends (in-memory)
+- ✅ B1. AdjacencyList: CRUD + neighbor ops + tests
+- ✅ B2. AdjacencyMatrix: CRUD + present bitset + tests
+- ✅ B3. IncidenceMatrix: rows, sign/weight semantics, edge_set, parallel builder, tests
+- ⭕ B4. CSR / ReverseCSR indices (standalone, optional overlays)
+- ⭕ B5. Attribute storage (typed, columnar, interning)
+
+## C. Common Trait Surface
+- ✅ C1. `GraphStorage(type, directed, weighted)` factory
+- ✅ C2. Core ops: init/deinit/add/remove/has/getNeighbors
+- ⭕ C3. Attribute API (node/edge getters/setters)
+- ⭕ C4. Iteration policy & attribute filters
+
+## D. Conversion Strategies
+- ⭕ D1–D7. Strategies + API + tests (`Duplicate`, `Streamed`, `Chunked`, `CopyOnWrite`)
+
+## E. File Formats
+- 🚧 E1. `.zgraph` text spec documented; runtime reader/writer pending
+- 🚧 E2. `.zgraphb` binary spec documented; runtime reader/writer pending
+
+## F. Algorithms
+- 🚧 F1. Traversal (BFS/DFS/CC) — source files exist; unify over trait; add tests
+- 🚧 F2. Shortest paths (Dijkstra/Bellman-Ford/Floyd-Warshall) — wire and test
+- 🚧 F3. Connectivity (SCC) — add Tarjan/Kosaraju; tests
+- 🚧 F4. Flow (Edmonds-Karp present; add Dinic); tests
+- 🚧 F5. Centrality (PageRank present; add Betweenness); tests
+- 🚧 F6. Spectral (Laplacian, eigen routines present); tests & trait integration
+
+## G. Indices & Acceleration
+- ⭕ G1. CSR build/use
+- ⭕ G2. Reverse CSR (directed)
+- ⭕ G3. Degree table
+- ⭕ G4. Index persistence in `.zgraphb` optional blocks
+- ⭕ G5. Rebuild-on-demand hooks
+
+## H. Attributes
+- ⭕ H1. Node & Edge KV (`int|float|bool|string`)
+- ⭕ H2. Storage-agnostic attribute map with typed accessors
+- ⭕ H3. Text↔Binary column mapping (string table)
+- ⭕ H4. Attribute filters in iterators
+- ⭕ H5. Tests for mixed types, missing values, interning
+
+## I. I/O & CLI
+- ⭕ I1. CLI subcommands: `convert`, `validate`, `build-index`, `stats`
+- ⭕ I2. Streaming I/O (chunked) for both formats
+- ⭕ I3. zstd dictionaries & auto-tuning
+
+## J. Concurrency & Memory
+- ✅ J0. IncidenceMatrix parallel builder + mutex guarding
+- ⭕ J1. Threading doc & invariants
+- ⭕ J2. Parallel builders for CSR & conversions where safe
+- ⭕ J3. Allocator strategy knobs (GPA/Arena/Page) + docs
+- ⭕ J4. Zero-copy `.zgraphb` readers (mmap)
+- ⭕ J5. Stress tests (OOM behavior)
+
+## K. Testing & Quality
+- ✅ K1. Unit tests for storages
+- ⭕ K2. Property tests (round-trips, cross-storage equivalence)
+- ⭕ K3. Fuzzers for text/binary parsers
+- ⭕ K4. Benchmarks (micro/macro) and tracked results
+- ⭕ K5. CI (Win/macOS/Linux; Zig 0.16.x)
+- ⭕ K6. `zig fmt` + static checks gates
+
+## L. Documentation
+- ✅ L1. README
+- 🚧 L2. Detailed specs for `.zgraph`/`.zgraphb` (needs sync with runtime)
+- ⭕ L3. Algorithm docs and complexity notes
+- ⭕ L4. Conversion strategy doc (memory math & decision table)
+- ⭕ L5. Perf tuning guide (allocators, cache, zstd params)
+- ⭕ L6. Roadmap milestones mapping to this checklist
+
+---
+
+# Execution Order (Do This Next, Exactly)
+
+## Phase 1 — Lock the Core Interfaces
+1. **Define common trait surface (`GraphLike`)** with adapters over existing storages:
+   - `neighbors(u)`, `hasEdge(u,v)`, `weight(u,v)?`, `nodeCount()`, `edgeCount()`
+2. **Iterator policy**: stable order + attribute filter hooks; basic `NodeIter`, `EdgeFromIter`
+
+## Phase 2 — Attributes (foundation for formats)
+3. **Typed attribute store** (node & edge): `int|float|bool|string` + string interning
+4. **Attribute API** on the trait surface + tests
+5. **Iterator filters** using attributes
+
+## Phase 3 — File Formats (runtime)
+6. **`.zgraph` reader/writer** (streaming CSV-ish with schemas; tolerant of unknown sections); round-trip tests
+7. **`.zgraphb` reader/writer** (chunked blocks + zstd + mmap-friendly); round-trip & parity tests
+8. **Text↔Binary parity**: load(text)->save(binary)->load(binary) == load(text)
+
+## Phase 4 — Conversion Strategies (memory-bounded)
+9. `convertStorage` API + **Duplicate** strategy
+10. **Streamed** strategy (destroy-as-you-go option; memory ceiling tests)
+11. **Chunked** strategy (node sharding; parallel)
+12. **Copy-On-Write** adapter (lazy migration) + background compaction
+13. **Invariants & ceilings** property tests
+
+## Phase 5 — Indices & Acceleration
+14. **CSR/ReverseCSR** build & use as optional overlays for any storage
+15. **Degree table** and **index persistence** blocks in `.zgraphb`
+16. **Rebuild-on-demand** hooks
+
+## Phase 6 — Algorithms (unify + verify)
+17. Traversal (BFS/DFS/CC) over trait + CSR; tests across storages
+18. Shortest paths (Dijkstra/BF/FW) with layout decision table; tests
+19. Connectivity/SCC (Tarjan/Kosaraju); tests
+20. Flow (finish Dinic); tests
+21. Centrality (PageRank + Betweenness); tests
+22. Spectral (laplacian, eigen); tests & perf
+
+## Phase 7 — CLI, Streaming I/O, Docs, Quality Gates
+23. CLI subcommands: `convert`, `validate`, `build-index`, `stats`
+24. Streaming I/O for both formats + zstd dictionaries
+25. Documentation: conversion strategies, perf tuning, thread model; keep specs synced
+26. Property tests, fuzzers, benches; CI on all platforms; formatting/static checks gates
+
+---
+
+## Acceptance Criteria (per phase, condensed)
+- **Formats:** Round-trips preserve counts, attributes, weights; unknown sections/blocks ignored.
+- **Conversion:** Peak memory measured below ceiling; equivalence of neighbor sets/weights post-convert.
+- **Indices:** CSR neighbors == storage neighbors; persisted indices reload correctly.
+- **Algorithms:** Identical results across storages (given same semantics); perf within expected bounds.
+- **CLI:** Non-zero exit on invalid files; stats match library queries; build-index regenerates CSR.
+- **Docs:** Examples compile; decision tables match implemented behavior.