Discord Message Vector DB

A tool for retrieving Discord messages and storing them in a vector database for semantic search and analysis.

Overview

This project provides a secure, privacy-respecting way to retrieve messages from Discord channels using the Discord MCP Server, process them for privacy concerns, and store them in a vector database (ChromaDB) for semantic search and analysis.

Features

• Robust Message Retrieval: Fetch all messages from Discord channels with pagination support
• Privacy Protection: PII detection and redaction, opt-out registry
• Resilient Design: Checkpointing, error recovery, and circuit breaker patterns
• Vector Database Integration: Convert messages to embeddings for semantic search
• Ethical Considerations: Built-in privacy controls and data minimization

Installation

# Clone the repository
git clone https://github.com/yourusername/discord_vector_db.git
cd discord_vector_db

# Create a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Usage

Basic Usage

from discord_retriever.fetcher import DiscordMessageFetcher

# Initialize fetcher
fetcher = DiscordMessageFetcher(
    channel_id="731607577481314359",
    save_directory="messages"
)

# Retrieve all messages
fetcher.fetch_all()

Processing for Vector Database

from discord_retriever.processor import process_for_vector_db

# Process messages and add to vector database
collection = process_for_vector_db(
    messages_directory="messages",
    collection_name="discord_messages"
)

# Search for semantically similar messages
results = collection.query(
    query_texts=["tell me about security concerns"],
    n_results=5
)

Command Line Interface

# Fetch messages from a Discord channel
python -m discord_retriever.cli fetch --channel-id 731607577481314359 --save-dir messages

# Process messages for vector database
python -m discord_retriever.cli process --messages-dir messages --collection discord_messages

# Search for messages
python -m discord_retriever.cli search --collection discord_messages --query "security concerns"

Ethical Considerations

This tool is designed with privacy and ethics in mind:

• All personal identifiable information (PII) can be automatically redacted
• Users can opt-out of having their messages processed
• Data minimization principles are applied by default
• Secure storage options for sensitive data

License

MIT

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for details.