DoEKS v2 Development

[vara-bonthu]

We are actively restructuring the Data on EKS (DoEKS) repository to make it cleaner, modular, and easier to use and contribute to.
This effort is being tracked as DoEKS v2.

---

🚀 What’s Changing

1. New Repository Structure

• infra/ → Terraform Infrastructure

  • Centralized Terraform modules for EKS, networking, IRSA, Karpenter, and core services.
  • Single source of truth — blueprints will no longer duplicate infra code; instead, they will pull from this central location during installation.

• data-stacks/

  • Contains blueprint-specific configuration and examples (e.g., spark-on-eks, flink-on-eks).
  • Each blueprint follows a consistent structure:

    examples/
    terraform/
      data-stack.tfvars
    cleanup.sh
    deploy-data-stack.sh
    set-env.sh
    

  • Example: Spark on EKS data-stack.tfvars

• website/

  • Improved documentation site with updated guides, architecture diagrams, and benchmarks.

2. Add-ons Moving to ArgoCD Applications

• All platform add-ons are now managed as ArgoCD Applications for easier lifecycle management and GitOps workflows.
• See: ArgoCD Applications

---

General folder structure.

project/
├── infra/                          # Base infrastructure templates
│   ├── terraform/
│   │   ├── main.tf
│   │   └── s3.tf
│   ├── argocd-applications/
│   │   └── *.yaml
│   └── manifests/
│       └── deployment.yaml
│
└── data-stacks/
    └── spark-operator/             # Example blueprint
        ├── _local/                 # Working directory (auto-generated)
        ├── install.sh              # Installation script
        ├── extra.tf                # Blueprint-specific Terraform
        ├── s3.tf                   # Overrides base s3.tf
        ├── *.tfvars               # Blueprint variables
        └── manifests/
            └── deployment.yaml     # Overrides base deployment.yaml

---

1. Base + Blueprint Pattern

• infra/ contains common infrastructure code shared across all deployments
• blueprints/ contain specialized configurations that extend or customize the base
• Each blueprint is completely independent with its own state and working directory

2. Installation Process

When a user runs ./install.sh from a blueprint directory:

• • Copy base: Copy all files from infra/ to _local/
• • Apply data-stack: Copy blueprint files to _local/, overwriting any matching filenames
• • Deploy: Run terraform init/plan/apply from _local/

3. File Override Behavior

• • Files with matching names are completely replaced (not merged)
• • New files from data-stack are added
• • Folder structure is preserved - only files within folders are replaced
• This is meant to be used where you cannot use tfvars and should be avoided where we can.

4. User Experience

cd blueprints/spark-on-eks/
./install.sh

---

💡 Why This Matters

• Cleaner architecture — no more infra duplication across blueprints
• Faster updates — improve consistency and maintainability of modules
• Easier onboarding — new contributors can focus on blueprints, not boilerplate infra
• GitOps-native — standardizing on ArgoCD Applications improves day-2 operations

---

🔗 Follow & Contribute

• Preview the new structure: DoEKS v2 Branch
• Share feedback: Comment on this issue or open new ones tagged with v2.
• Feature requests: Create a new issue to request or propose enhancements for v2.

[vara-bonthu]

V2 PR has been merged now