Feature Request: Subject Digest Feeds

[madpin]

Summary

Introduce "Subject Digest" feeds that let a user subscribe to a topic (e.g., "Genomics breakthroughs" or "AI policy") instead of a single RSS source. The platform should periodically gather relevant news for each subject, summarize the highlights, and expose the digest as a virtual feed that behaves like any other feed in the reader UI.

Why

• Reduce friction for users who know what they care about but do not know which feeds to import.
• Provide higher-level, LLM-powered digests that surface the most important updates per subject.
• Enable premium capabilities (future upsell) such as scheduled email digests, notifications, or collaborative subject sharing.

User Stories

• As a user, I can create a Subject Digest by naming it and providing seed keywords/topics, so the system knows what to monitor.
• As a user, I can optionally restrict a subject to specific sources (existing feeds, domains, or categories) or languages.
• As a user, I see each Subject Digest as a feed in my sidebar, with its own summary card, last refresh time, and unread counter.
• As a user, I can open a Subject Digest to read a generated synopsis (LLM summary) followed by the curated list of articles that informed it.
• As a user, I can edit, pause/resume, or delete a Subject Digest at any time.

Functional Requirements

1. Creation flow
   • Modal or dedicated page accessible from the feeds sidebar ("New Subject Digest").
   • Required inputs: Subject name, short description or intent, minimum one keyword or phrase.
   • Optional inputs: include/exclude sources (existing feeds, domains, RSS URLs), language/region, refresh cadence, summary tone (e.g., concise, detailed).
   • Validate keywords, throttle creation (rate limiting) to avoid abuse.
2. Storage
   • Persist Subject Digests either as a new table (e.g., SubjectDigest) or as a specialized Feed with settings.type = "subject_digest" containing keyword metadata.
   • Track last refresh timestamp, next scheduled refresh, status (active, paused, error), failure reason.
3. Ingestion / Retrieval
   • Background job that runs per subject: queries existing article corpus + external search APIs (News API, Google News RSS, etc.) filtered by keywords and optional constraints.
   • Deduplicate by URL/content hash. Respect user-level refresh ceilings.
4. Digest Generation
   • Summarize top N (configurable) articles via existing LLM infrastructure (see UserPreferences.llmDigestModel).
   • Store summary, bullet key points, and referenced article IDs for traceability.
   • Surface summary + links in UI with "Read more" anchors.
5. UX integration
   • Sidebar should display Subject Digest feeds with a unique badge/icon.
   • Article list view should display digest summary at the top, then associated articles (sorted by relevance or published date).
   • Provide quick actions (refresh now, edit subject, pause) from feed header.
6. Permissions & Quotas
   • Limit number of digests per plan (configurable constants or admin setting).
   • Enforce per-user job queueing to avoid overlapping LLM calls.
7. Observability
   • Log job duration, article count, summary token usage.
   • Surface failure states in UI with actionable messages.

Technical Notes

• Reuse UserFeed relationship so Subject Digests appear everywhere a normal feed does. Consider a Feed.settings.kind = "subject_digest" flag plus a subjectConfig JSON payload { keywords: string[], includeSources: string[], excludeSources: string[], language?: string, refreshIntervalMinutes?: number }.
• Extend ingestion worker (src/lib/jobs/) to branch on digest feeds: instead of fetching RSS, run search queries + scoring using embeddings (src/lib/embeddings).
• Summaries can use the existing LLM service (src/lib/llm) with the dedicated digest model preference (falls back to admin default if user doesn't override).
• Add REST endpoints under app/api/feeds/subject-digests for CRUD + manual refresh. Protect with session auth + Zod validation.
• Ensure caching layer (src/lib/cache) expires summaries when new refresh occurs.

Acceptance Criteria

• User can create at least one Subject Digest with required fields and see it listed instantly in the sidebar.
• Refresh job runs automatically (per configured interval) and populates the digest with >=1 relevant article when matching content exists.
• Digest view shows generated summary, key points, and the article list, with each article linking back to the original source.
• Editing a Subject Digest updates its configuration and affects the next refresh cycle.
• Paused digests are skipped by the scheduler; resuming re-enters them.
• Errors during refresh are surfaced in UI with an explanation and retry guidance.

Open Questions

• Do we need per-digest notification channels (email/push) in the first iteration? (likely no; future enhancement)
• Should we allow collaborative subjects shared across users or keep them private per user for now?
• Which external news/search providers are available within current infrastructure limits and legal constraints?

Out of Scope (v1)

• Email delivery of digests.
• Multi-user shared subjects.
• Fine-grained per-article feedback loops feeding back into subject weighting (can reuse existing feedback later).