Add SMS transaction message parser with column value extraction

[Copilot]

Implements a pattern-based parser to extract structured transaction data from raw SMS messages (amounts, merchants, dates, transaction types, categories, references).

Implementation

• sms_extractor.py: Core SMSExtractor class with regex patterns for Indian/US currencies, merchant extraction, transaction type detection, and rule-based categorization (8 categories)
• predict_sms.py: CLI with text/JSON/CSV output formats, single message or batch file processing, optional merchant hints
• examples_sms.py: Demonstration of various extraction patterns
• Documentation: README_SMS.md (full docs), QUICK_START.md (reference), SOLUTION_SUMMARY.txt

Usage

# Extract structured data from raw SMS
python predict_sms.py --text "₹499 paid to Amazon" --format json

{
  "amount": 499.0,
  "transaction_type": "debit",
  "merchant": "Amazon",
  "category": "shopping",
  "date": null,
  "reference": null
}

# Programmatic access
from sms_extractor import SMSExtractor

extractor = SMSExtractor()
result = extractor.extract_all("Rs 1,250 debited from your account at Swiggy")
# Returns dict with amount, transaction_type, merchant, category, date, reference

Patterns Supported

• Currencies: ₹, Rs, INR, $, USD
• Transaction types: debit/credit/refund keywords
• Merchants: "paid to X", "at X", "from X" patterns
• Categories: keyword matching for food, shopping, transport, utilities, entertainment, healthcare, education, transfer
• Dates: DD/MM/YYYY, DD-Mon-YYYY formats
• References: UPI refs, transaction IDs

Handles ambiguous cases via optional merchant hints. Works with various bank SMS formats (tested: HDFC, SBI, ICICI, Paytm).

Original prompt

i want to input a raw sms messsage and get the extracted column values, how do i do it

[Chronological Review: The conversation began with the user requesting the extraction of features from SMS data, leading to the creation of various models and pipelines. The user iteratively refined their requests, focusing on specific tasks such as filtering data, training classifiers, and visualizing results. The conversation progressed through multiple requests for code implementations, error troubleshooting, and model evaluations, culminating in the user's desire to input raw SMS messages and receive extracted values.]

[Intent Mapping: The user explicitly requested the following:

1. Build a spending pattern clustering system from a CSV dataset.
2. Create a category classifier model for SMS messages.
3. Input raw SMS messages to extract specific column values.
4. Various data processing tasks including filtering, feature engineering, and model training.]

[Technical Inventory: Key technologies and tools mentioned include:

• Python for scripting and data manipulation.
• Libraries such as Pandas for data handling, Scikit-learn for machine learning, and Joblib for model serialization.
• Clustering algorithms like K-Means, DBSCAN, and Gaussian Mixture Models (GMM).
• Visualization techniques including PCA/t-SNE and radar charts.]

[Code Archaeology: The user worked on several files:

• nlp_model.ipynb: For training models.
• preprocess_pipeline.py: For data preprocessing.
• train_xgb.py: For training an XGBoost classifier.
• evaluate_xgb.py: For evaluating the trained model.
• predict_short_classifier.py: For making predictions based on input SMS messages.]

[Progress Assessment: The user has made significant progress in building models and pipelines but encountered issues with model predictions and data filtering. They are currently focused on extracting values from raw SMS messages.]

[Context Validation: All critical information for continuation is captured, including the user's goals for data processing, model training, and SMS message extraction.]

[Recent Commands Analysis: The last commands executed involved the user trying to run a prediction script (predict_short_classifier.py) which resulted in an AttributeError related to loading a model. The user also expressed frustration with the model's performance, indicating it was not providing useful predictions.]

1. Conversation Overview: - Primary Objectives: The user aimed to build a spending pattern clustering system and a category classifier for SMS messages, with specific requests for data processing, model training, and visualization. - Session Context: The conversation evolved through multiple iterations of requests, focusing on refining models, troubleshooting errors, and enhancing data processing techniques. - User Intent Evolution: The user shifted from general model building to specific tasks like filtering data, training classifiers, and extracting values from SMS messages.

2. Technical Foundation:

• Python: Used for scripting and data manipulation.
• Scikit-learn: For machine learning tasks including clustering and classification.
• Joblib: For saving and loading models.

3. Codebase Status:

• nlp_model.ipynb: Contains code for training models.
• preprocess_pipeline.py: Handles data preprocessing tasks.
• train_xgb.py: Implements training for an XGBoost classifier.
• evaluate_xgb.py: Evaluates the performance of the trained model.
• predict_short_classifier.py: Intended for making predictions based on SMS input.

4. Problem Resolution:

• Issues Encountered: The user faced an AttributeError when trying to load a model in predict_short_classifier.py, indicating a potential issue with the model's serialization.
• Solutions Implemented: The user has been refining their approach to model training and data processing but has not yet resolved the prediction issues.

5. Progress Tracking:

• Completed Tasks: Data loading and cleaning, initial model training, and feature engineering.
• Partially Complete Work: Ongoing efforts to improve model predictions and extract values from SMS messages.

6. Active Work State:

• Current Focus: The user is working on inputting raw SMS messages to extract specific column values.
• Recent Context: The user has been troubleshooting model predictions and refining their data processing pipeline.

7. Recent Operations:

• Last Agent Commands: The user executed python predict_short_classifier.py --merchant "Amazon" --text "₹499 paid to Amazon".
• Tool Results Summary: The command resulted in an AttributeError related to loading a model, indicating a problem with the model's serialization or the environment setup.
• Pre-Summary State: The user was actively trying to predict categories from SMS messages when the token budget was exceeded.
• Operation Context: These commands were part of the user's goal to create a functional SMS message classifier that could extract relevant ...

Created from VS Code via the GitHub Pull Request extension.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.