Rewrite personal-graph in Rust with custom storage engine#56
Open
Rewrite personal-graph in Rust with custom storage engine#56
Conversation
Replace the entire Python codebase with a ground-up Rust implementation. No database wrappers — custom storage engine with page-oriented layout, buffer pool, WAL, B+tree indexes, HNSW vector index, and graph traversal. Architecture (5 crates): - pg-core: Node/Edge types, PropertyMap, GraphStorage/VectorIndex traits - pg-storage: 4KB slotted pages, LRU buffer pool, CRC32 WAL, B+tree indexes - pg-vector: HNSW approximate nearest neighbor search (L2/cosine/inner product) - pg-graph: PersonalGraph engine with BFS/DFS, shortest path, similarity search - pg-server: axum HTTP API with full CRUD, batch ops, traversal, vector search 62 tests, zero warnings, all passing. https://claude.ai/code/session_01BL34LfrRWc1ciedT3asAbu
…her engine - pg-storage: MVCC transaction manager (TxManager, Transaction RAII guard) - pg-storage: on-disk B+tree (disk_btree.rs) with WAL-backed writes - pg-storage: compaction module — reclaims dead MVCC cells from pages - pg-storage: GraphStore gains tx_manager field and compact() method - pg-vector: HNSW index binary persistence (save/load) with magic header - pg-cypher: full new crate — recursive-descent Cypher parser + executor - Supports MATCH, CREATE, MERGE, SET, DELETE, RETURN, WITH, UNWIND - Node/edge patterns, variable-length hops, property filters, ORDER BY - Executor evaluates against pg-graph's GraphBackend trait - pg-server: POST /query endpoint for OpenCypher queries - Fix lexer: 2..5 range token not misread as Float(2.0) - Fix parser: Token::Dash replaced with Token::Minus (lexer never emits Dash) - Fix parser: node variable detection for bare (n) patterns - All 101 tests passing across workspace https://claude.ai/code/session_01BL34LfrWc1ciedT3asAbu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is a major architectural rewrite of personal-graph from Python to Rust. The entire codebase has been restructured around a custom storage engine and vector index implementation, replacing the previous SQLite/TursoDB-based approach.
Key Changes
Complete language migration: Converted from Python to Rust for better performance and type safety
Custom storage engine: Implemented a from-scratch page-based storage engine (
pg-storage) with:Vector indexing: Built HNSW (Hierarchical Navigable Small World) vector index (
pg-vector) from scratch with distance metrics (Euclidean/L2)Core type system (
pg-core): Defined fundamental data structures:Graph engine (
pg-graph): High-level API combining storage and vector index with:HTTP API server (
pg-server): REST API handlers for graph operations using Axum web frameworkWorkspace structure: Organized as Rust workspace with modular crates for separation of concerns
Removed Python artifacts: Deleted all Python source files, FHIR ontology files, examples, tests, and documentation that were specific to the old implementation
Notable Implementation Details
https://claude.ai/code/session_01BL34LfrRWc1ciedT3asAbu