feat: Add AWS cloud infrastructure logs integration

[bhaskarvilles]

Description

Adds comprehensive cloud infrastructure log ingestion and analysis capabilities to Malcolm, enabling monitoring of hybrid cloud + on-prem environments.

Related Issue

Closes #232

What's New

• AWS VPC Flow Logs parser with security event detection
• AWS CloudTrail parser with threat detection
• Automated S3 log collector script
• Comprehensive documentation with setup guides
• Configuration templates for easy deployment

Features

✅ VPC Flow Logs parsing (v2 format)
✅ CloudTrail API activity parsing
✅ Automated S3 log collection
✅ ECS field mapping
✅ GeoIP/ASN enrichment integration
✅ Security tagging (unauthorized access, high-risk actions, etc.)

Security Event Detection

VPC Flow Logs

• rejected_traffic - Blocked connections
• high_volume_transfer - Large data transfers (>10MB)
• potential_port_scan - Multiple connection rejections

CloudTrail

• unauthorized_access_attempt - AccessDenied errors
• high_risk_action - Destructive operations (DeleteBucket, TerminateInstances, etc.)
• root_account_usage - Root user activity
• failed_authentication - Failed console logins

Testing

• Tested VPC Flow Logs parser with sample data
• Tested CloudTrail parser with sample events
• Verified ECS field mappings
• Confirmed integration with Malcolm's enrichment pipeline
• Tested with live AWS VPC Flow Logs (pending AWS account access)
• Tested with live CloudTrail logs (pending)
• Created dashboards (in progress)

Documentation

• ✅ User guide: docs/cloud-logs-integration.md
• ✅ Configuration: config/cloud-logs.env.example
• ✅ Contribution summary: CONTRIBUTION.md

Files Changed

File Path	Insertions
CONTRIBUTION.md	96
config/cloud-logs.env.example	25
docs/cloud-logs-integration.md	380
logstash/pipelines/enrichment/15_cloud_logs_vpc_flow.conf	130
logstash/pipelines/enrichment/16_cloud_logs_cloudtrail.conf	160
shared/bin/aws_log_collector.py	236
shared/bin/cloud-logs-requirements.txt	1

Summary

• Files changed: 7
• Total insertions: 961

Impact

This contribution enables Malcolm to:

• 🌐 Monitor hybrid cloud + on-prem environments
• 🔒 Detect cloud security threats
• 🔗 Correlate cloud API activity with network traffic
• 📊 Provide unified analysis platform for security teams

Checklist

• Code follows Malcolm's style guide
• Documentation updated
• Commit messages are descriptive
• ECS field mappings used consistently
• Unit tests added (next step)
• Dashboards created (next step)

Next Steps

1. Create OpenSearch dashboards for VPC Flow and CloudTrail
2. Add unit tests for parsers
3. Test with live AWS infrastructure logs
4. Address maintainer feedback

---

Author: @bhaskarvilles
Branch: Cloud-Infrastructure-Logs-Integration

[mmguero]

Thanks, I will review this in the next week or so and get it merged for probably a February release.

[mmguero]

Just wanted to let you know I saw your updates. It will probably be next week sometime before I'm able to carve out the time to review this, but know it's appreciated and we'll get them reviewed. I may end up moving the logstash filters to a separate (new) parse pipeline rather than putting them in "enrichment" but don't worry about it for now, that's easy to adjust. Cheers.

[bhaskarvilles]

@mmguero Thanks for your reply, Just adding more enhancements, i will wait for your review and for the next release, planning to integrate Azure, Akamai and other cloud provider logs as well.

[mmguero]

I'm on site at one of our funding sources this week, but I plan on looking at this and getting it merged next week. Thanks for your patience!

[mmguero]

I've got a number of concerns/questions about this PR.

• Configuration environment variables not used by Malcolm, or anywhere that I can see:
  • You've created this config .env file, but I don't see anywhere these variables are actually used or referenced other than in cloud-logs-integration.md. .env files in ./config/ are usually intended to be used internally by Malcolm containers.

$ cat config/cloud-logs.env.example |rg -o "\w+=" | tr -d '=' | tr '\n' '|' | sed "s/|$//" | x rg
config/cloud-logs.env.example
2:AWS_REGION=us-east-1
3:AWS_VPC_FLOW_LOGS_BUCKET=
4:AWS_VPC_FLOW_LOGS_PREFIX=AWSLogs/
5:AWS_CLOUDTRAIL_BUCKET=
6:AWS_CLOUDTRAIL_PREFIX=AWSLogs/
7:AWS_LOG_COLLECTION_ENABLED=false
8:AWS_LOG_COLLECTION_INTERVAL=300
11:AZURE_SUBSCRIPTION_ID=
12:AZURE_NSG_FLOW_LOGS_CONTAINER=
13:AZURE_LOG_COLLECTION_ENABLED=false
14:AZURE_LOG_COLLECTION_INTERVAL=300
17:CLOUD_LOGS_OUTPUT_DIR=/var/log/malcolm/cloud-logs
18:CLOUD_LOGS_RETENTION_DAYS=90
19:CLOUD_LOGS_CLEANUP_ENABLED=true
22:CLOUD_LOGS_MAX_FILES_PER_RUN=100
25:CLOUD_LOGS_INDEX_PREFIX=malcolm-cloud

docs/cloud-logs-integration.md
114:AWS_REGION=us-east-1
115:AWS_VPC_FLOW_LOGS_BUCKET=my-vpc-flow-logs
116:AWS_VPC_FLOW_LOGS_PREFIX=AWSLogs/123456789012/vpcflowlogs/
117:AWS_LOG_COLLECTION_ENABLED=true
118:AWS_LOG_COLLECTION_INTERVAL=300
121:CLOUD_LOGS_OUTPUT_DIR=/var/log/malcolm/cloud-logs
122:CLOUD_LOGS_RETENTION_DAYS=90

• Dashboards included are not compatible with how Malcolm initializes dashboards on startup:

  • The ./dashboards/dashboards/*.ndjson files you've added won't be automatically created/imported into Malcolm. For dashboards to be automatically created on startup and set up correctly with the index patterns, they need to be exported using the Malcolm dashboard export API. That endpoint ensures that the exported dashboard file has the replacers automatically put in it that, when re-imported, will be used to substitute in custom user index pattern settings that are found in ./config/opensearch.env.

• Your cloud-logs-integration.md talks about this data pipeline, but I don't see where any of it really plugs into Malcolm.

  • For example, it talks about this /var/log/malcolm/cloud-logs directory, which is stored in CLOUD_LOGS_OUTPUT_DIR, but that environment variable isn't used. When you pass it into aws_log_collector.py with --output-dir, then you talk about "Check Filebeat is monitoring the directory" but I don't see anywhere about filebeat actually being configured to it. And where would that even happen? This scripts is being run outside outside of the Malcolm containers, and so I assume you're talking about some user-managed external filebeat where those are watched?
  • We don't really have any existing code that does that sort of thing, like an external script that creates log files and then (if this is your intent) the log files are mounted into the Malcolm containers. I'd probably be more inclined to have your script send logs to Malcolm's JSON-over-TCP/TLS listener (configuration, "Expose Malcolm Service Ports", "Expose Filebeat TCP", etc.) and then route those to an entirely separate Logstash pipeline (pre-enrichment pipeline) rather than putting all of your logstash conf files in the enrichment pipeline.

Frankly, and no offense intended, my initial look at this makes me feel a lot of this is pretty heavily vibe-coded.