Skip to content

Add spike digest job with dedupe rules and report persistence#4

Open
artvandelay wants to merge 1 commit intomainfrom
codex/implement-scheduled-job-for-spike-events
Open

Add spike digest job with dedupe rules and report persistence#4
artvandelay wants to merge 1 commit intomainfrom
codex/implement-scheduled-job-for-spike-events

Conversation

@artvandelay
Copy link
Owner

Motivation

  • Provide a scheduled spike-reporting job that picks top spike events over a configurable window (e.g., 6h/24h) for downstream notifications.
  • Prevent repeated or noisy alerts by applying dedupe rules based on last reported time, score delta, and new page spread.
  • Produce a shareable digest in both Markdown and structured JSON containing term, score, direction, top pages, diff links, and snippets.
  • Persist sent reports and supporting evidence so downstream systems can query history and avoid duplicates.

Description

  • Added src/nlbt/spike_reports.py with dataclasses PageEvidence, SpikeEvent, and DedupeConfig and helper functions including select_top_spike_events, should_report_event, build_digest, ensure_reports_table, and run_scheduled_spike_digest to assemble and persist digests.
  • Digest assembly outputs both Markdown and a structured JSON payload that includes term, score, direction, top_pages, diff_links, and snippets.
  • Persisted reports into an SQLite reports table (created by ensure_reports_table) with stored payload_json and evidence_json for each reported item.
  • Exported the new helpers from the package __init__ and added tests/test_spike_reports.py covering dedupe rules and report persistence.

Testing

  • Ran pytest -q tests/test_spike_reports.py and it succeeded (2 passed).
  • Running pytest -q for the whole repository failed during collection due to unrelated test collection issues (ModuleNotFoundError: No module named 'nlbt') and is unrelated to the new spike report unit tests.
  • Unit tests exercise should_report_event, run_scheduled_spike_digest, and verify that data and snippets are persisted into the reports table.
  • The new tests import the package from src during test execution so they run in the repository layout used here.

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant