Skip to content

perf(chain-db): speed up chain sync batching and rollback handling#720

Merged
AndrewWestberg merged 1 commit intomasterfrom
newm_chain_db_improvement
Jan 31, 2026
Merged

perf(chain-db): speed up chain sync batching and rollback handling#720
AndrewWestberg merged 1 commit intomasterfrom
newm_chain_db_improvement

Conversation

@AndrewWestberg
Copy link
Member

Batch chain and ledger writes, cache per-block metadata lookups, and cap catch-up buffer growth to improve initial sync performance. Update gradle/kogmios versions and record perf PRD tasks.

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improve initial chain sync throughput by batching DB writes, reducing per-block query overhead, and preventing unbounded catch-up buffering while keeping rollback semantics correct.

Changes:

  • Cap sync batch growth during catch-up via configurable maxCatchupBlockBufferSize.
  • Gate rollback work to RollBackward events and add per-block metadata lookup caching.
  • Batch chain inserts / UTXO creation + spending updates; bump Gradle and Kogmios versions; add perf PRD/task docs.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
newm-chain/src/main/kotlin/io/newm/chain/daemon/MonitorAddressDaemon.kt Adds config-driven cap for catch-up buffer growth.
newm-chain/src/main/kotlin/io/newm/chain/daemon/BlockDaemon.kt Adds rollback gating, buffer-cap logic, and per-block metadata caching.
newm-chain-db/src/main/kotlin/io/newm/chain/database/repository/LedgerRepositoryImpl.kt Reworks spent/created UTXO writes into batch SQL/batchInsert to reduce N+1 and update overhead.
newm-chain-db/src/main/kotlin/io/newm/chain/database/repository/ChainRepositoryImpl.kt Switches chain writes to batchInsert and computes etaV in-memory per batch.
gradle/wrapper/gradle-wrapper.properties Updates Gradle wrapper version.
buildSrc/src/main/kotlin/Versions.kt Updates Kogmios version constant.
.opencode Adds agent config entry.
.agent/task/newm-chain-db-improvement/newm-chain-db-performance-tasks.json Adds tracked performance tasks list.
.agent/task/newm-chain-db-improvement/newm-chain-db-performance-prd.md Adds PRD describing performance plan and completion status.
.agent/skills/git-commit-formatter/SKILL.md Adds skill doc for conventional commit formatting.

@AndrewWestberg AndrewWestberg force-pushed the newm_chain_db_improvement branch 2 times, most recently from 6faa55c to 45b8a97 Compare January 30, 2026 22:00
Batch chain and ledger writes, cache per-block metadata lookups, and cap catch-up buffer growth to improve initial sync performance. Update gradle/kogmios versions and record perf PRD tasks.
@AndrewWestberg AndrewWestberg force-pushed the newm_chain_db_improvement branch from 45b8a97 to 427b7d9 Compare January 30, 2026 22:52
@AndrewWestberg
Copy link
Member Author

Dropping the prompt here for future dev's reference. I looped on this until the PRD was complete clearing the context window before each loop.

Implement the next highest priority task from @.agent/task/newm-chain-db-improvement/newm-chain-db-performance-tasks.json with additional context from @.agent/task/newm-chain-db-improvement/newm-chain-db-performance-prd.md . Plan your task carefully and ensure each update is precise and does not cause data integrity issues. Update any relevant tests if necessary. Execute the workflow @.agent/workflows/test.md . Once complete, update your status in the PRD and json tasks file. Then prompt the user for whether they want to make a git commit or not. If yes, use the git-commit-formatter skill and use your best judgment for wording the commit message automatically.

@AndrewWestberg AndrewWestberg merged commit cabff39 into master Jan 31, 2026
2 checks passed
@AndrewWestberg AndrewWestberg deleted the newm_chain_db_improvement branch January 31, 2026 00:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants