Skip to content

Automatic detection and cleanup of removed topics from consumer groups #854

@karthikraina32

Description

@karthikraina32

Description:
Problem
Currently, when a consumer group removes topics from its subscription list, Burrow continues to track and report these removed topics as errors (stopped partitions) until either:

  • The Burrow service is manually restarted, or
  • The /v3/kafka/(cluster)/consumer/(group) DELETE endpoint is manually called

This creates operational overhead and can lead to false alerts in monitoring systems.

Current Workaround:
As mentioned in #14, the current solutions require manual intervention:

  1. Restart Burrow to refresh consumer group configurations
  2. Use the DELETE API endpoint to manually drop the consumer group

Proposed Enhancement:
Implement automatic detection of topic removal from consumer groups by:

  1. Periodic reconciliation: Burrow should periodically compare the list of topics currently consumed by a group (from Kafka metadata) against the topics it's tracking internally
  2. Topic removal detection: When a topic is no longer present in the consumer group's subscription but still exists in Burrow's internal state, automatically remove it from tracking
  3. Configurable interval: Add a configuration parameter (e.g., topic-reconciliation-interval) to control how frequently this check occurs (default: 5-10 minutes)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions