[FEATURE]: Integrating AWS CloudTrail ingestion source #228
Description
Why should this ingestion source be integrated into BuffaLogs?
-
AWS CloudTrail is a source of telemetry for cloud environments, logging all account activity related to AWS services. Integrating CloudTrail as an ingestion source into BuffaLogs will:
-
Give BuffaLogs' coverage in terms of cloud security monitoring by adding visibility into user actions, login attempts, privilege escalations, and API usage across AWS.
-
Enable analysis of cloud-specific login anomalies
-
Support compliance and forensic use cases, as CloudTrail logs are often mandated for auditing in regulated environments.
This integration ensures BuffaLogs becomes a more cloud-native and cloud-aware tool for anomaly detection in multi-cloud or hybrid infrastructures.
Visibility into a users AWS account activity is crucial for security and operational best practices and CloudTrail records events using Event history, CloudTrai lake and Trails (which capture a record of AWS activities into an S3 bucket). Ingesting this source would allow for access to manage all this data.
Additional Information
-
CloudTrail logs are delivered in JSON format and are usually stored in Amazon S3 buckets.
-
Each event contains structured metadata like eventTime, eventCategory, userIdentity, sourceIPAddress, awsRegion, and eventType, which can be mapped to the ECS schema.
-
AWS provides the Boto3 SDK in Python for interacting with S3 and listing/pulling CloudTrail logs.
-
Events can vary based on the AWS service, but normalization logic should account for dynamic structures.
Questions or Concerns for Development
- Should ingestion be batch-based or streaming ? Or should there be support for bot formats ?
Development Guidelines
-
Branch Name: feature/add_cloudtrail
-
Add configs for the ingestion source: into the config/buffalogs/ingestion.json file insert the settings and mapping fields for the ingestion source considered (e.g., AWS credentials, bucket name, region, prefix, ECS field mappings)
-
File to Create: Implement a new ingestion source file in the following directory, naming it cloudtrail_ingestion.py. This file must implement the three abstract classes defined in base_ingestion.py.
-
Update Enum and Factory: Add the new source to the SupportedIngestionSources Enum class and update the IngestionFactory.get_ingestion_class function to include the new source.
-
Testing: Create tests in the buffalogs/impossible_travel/tests/ directory, naming it test_cloudtrail_ingestion.py
-
Documentation: Create a documentation file in the docs/ingestion/ directory, naming it cloudtrail.md and include relevant screenshots showcasing the ingestion source in action.
Reference Implementation:
Check the existing implementation in elasticsearch_ingestion.py and its corresponding test test_elasticsearch_ingestion.py as a reference.