[EPIC] Move to blue/green deployments for ROR API and Infrastructure

[adambuttrick]

Summary

Recent infrastructure changes have resulted in some temporary service degradations and short outages during rollouts. We need to move ROR's production deployment process to a proper blue/green model so that new versions can be brought up, validated, and cut over to without the live environment being affected.

Background

Our current setup runs production, staging, and dev as long-lived environments backed by AWS ECS Fargate behind ALBs and CloudFront, with Elasticsearch and MySQL in the private subnet. Deployments today update our services in place against the live prod environment. This means that:

• There is no fully provisioned, production-equivalent "next" environment to validate against real traffic before cutting anything over.
• Rollback requires redeployments rather than shifting traffic, lengthening the time to resolve when a bad change is shipped.
• Index and data changes are coupled to the single live environment.

Goal

Production deployments should:

1. Stand up a parallel green API stack alongside our live blue one.
2. Run health checks, tests, and synthetic traffic against the green setup before user traffic is shifted.
3. Cut over by shifting traffic at the ALB layer (or similar).
4. Keep blue warm long enough to allow instant rollback by shifting traffic back.

Questions and Decisions

• How do we shift traffic between colors? What signals trigger an automatic rollback?
• How does our WAF need to be configured so that swapping origin target groups doesn't break caching, redirect rules, or similar?
• What's the right Elasticsearch approach so that a full reindex doesn't touch the indx currently serving traffic
• What does the MySQL setup look like where we can run blue and green against the same database during a switch-over?
• How do we express blue/green in Terraform?
• What observability do we need per color so we can actually tell blue and green apart during a cutover window?
• Do staging and dev adopt blue/green too, or do they stay single-stack?

Acceptance criteria

• A production deployment of the ROR API and its infrastructure can be performed with zero user-visible degradation.
• Rollback from a bad release is just shifting traffic, not a redeployment, and ideally can be done in a few minutes.
• The full process is documented and reproducible by any developer working on ROR.