[santosr2]

What issues does your PR fix?

• resolves Support loading DAG definitions from S3 buckets #24

What does your PR do?

Summary

This PR introduces a new dags.bucketSync feature that enables automatic synchronization of DAG files from cloud storage buckets (S3, and future support for GCS/Azure) into Airflow pods. This provides an alternative to git-sync for teams storing DAGs in object storage.

Motivation

Many organizations store their Airflow DAGs in cloud storage buckets (S3, GCS, Azure Blob) rather than Git repositories. Previously, users had to:

• Manually implement custom init containers or sidecars
• Use PersistentVolumes with external sync mechanisms
• Build DAGs into container images

This feature provides native, first-class support for cloud storage bucket synchronization with the same user experience as git-sync.

Key Features

Supported Providers

• AWS S3 (including S3-compatible storage like MinIO, DigitalOcean Spaces, Ceph)
• Google Cloud Storage (GCS) - planned
• Azure Blob Storage - planned

Authentication Methods (S3)

1. IAM Roles for Service Accounts (IRSA) - Recommended
2. Kubernetes Secrets - Alternative
3. Inline Credentials - Not Recommended for production environments

Core Capabilities

• Automatic syncing - Continuous sync at configurable intervals
• SubPath support - Sync specific subdirectories within buckets
• Flexible sync flags - Pass custom AWS CLI flags for filtering
• Resource management - Configurable resource limits for sync containers
• Failure handling - Configurable max consecutive failures before aborting
• S3-compatible storage - Works with MinIO, DigitalOcean Spaces, etc.

Mutual exclusivity with Git-Sync

dags.bucketSync and dags.gitSync are mutually exclusive - only one can be enabled at a time. Both sync to /dags/repo/, which would cause conflicts if both were enabled. However, when/if we introduce the PVC approach to reduce the container count, as discussed in #25, we may enable the possibility of having both enabled to sync DAGs and/or plugins simultaneously.

New Values

• dags.bucketSync.enabled (default: false)
• dags.bucketSync.provider (default: s3)
• dags.bucketSync.subPath
• dags.bucketSync.syncInterval (default: 60)
• dags.bucketSync.resources
• dags.bucketSync.maxFailures (default: 0)
• dags.bucketSync.s3.bucket
• dags.bucketSync.s3.key (default: dags)
• dags.bucketSync.s3.region
• dags.bucketSync.s3.image.*
• dags.bucketSync.s3.syncFlags
• dags.bucketSync.s3.credentialsSecret
• dags.bucketSync.s3.credentialsSecretAccessKeyKey (default: aws_access_key_id)
• dags.bucketSync.s3.credentialsSecretSecretKeyKey (default: aws_secret_access_key)
• dags.bucketSync.s3.credentialsSecretSessionTokenKey (default: aws_session_token)
• dags.bucketSync.s3.accessKeyId
• dags.bucketSync.s3.secretAccessKey
• dags.bucketSync.s3.sessionToken
• dags.bucketSync.s3.endpoint

Note

This PR introduces AWS S3 support as the first cloud storage provider. The architecture is designed to easily extend to GCS and Azure in future PRs.

Checklist

For all Pull Requests

• Commits are signed off
• Commits have semantic messages
• Documentation updated
• Passes ct linting

For releasing ONLY

• Chart.yaml version bumped
• CHANGELOG.md updated