terraform-ghost-agent-platform

Terraform module that deploys the Ghost Agent Platform to a single EC2 VM in an AWS account. Provisions the VM, network, IAM, and AWS Secrets Manager entries, then bootstraps the stack via cloud-init - including cosign signature verification of the stack definition and images before any container starts.

What gets deployed

• 1 EC2 instance (t3.large by default, AL2023 AMI) running the full Ghost Agent Platform stack as docker-compose. Five service containers: the gateway, credential proxy, worker, UI, and an in-stack updater that applies UI-driven upgrades (see Updating the deployment).
• 1 EBS data volume (100 GB gp3 by default) mounted at /var/lib/exo - holds the database, TLS material, run artifacts, and signing-cert state. Configured with prevent_destroy = true to guard against accidental teardown.
• 1 Elastic IP providing a stable public address.
• 1 Security group - SSH from var.admin_cidr only, HTTP/HTTPS from var.public_ingress_cidrs (default: world).
• 1 IAM role + instance profile with scoped permissions: ECR pull on Ghost's image repos, Secrets Manager read on this module's own secrets, SSM core (for Session Manager).
• 2 Secrets Manager entries, both auto-generated by Terraform: the encryption key and the one-time setup-wizard claim token. Everything else (session-signing secret, admin account, connector tokens) is platform-generated or configured through the in-product setup wizard.
• (Optional) Route 53 A record when both var.domain_name and var.route53_zone_id are set.

Prerequisites

1. AWS account with admin or equivalent permissions to create EC2 / EBS / IAM / Secrets Manager resources.
2. A public subnet in a VPC. The subnet must allow inbound HTTP/HTTPS for Let's Encrypt cert issuance and public app access.
3. Cross-account ECR pull access from Ghost Security. Ghost publishes images to a private ECR registry; the AWS account running this module needs to be added to Ghost's repository policy before the EC2 can pull. See Onboarding below.
4. Terraform >= 1.5 with the AWS provider 6.x.

Onboarding

1. Share the target AWS account ID with Ghost Security. The account running this module needs to appear on Ghost's pull_allowed_accounts list - without it, terraform apply will succeed but cloud-init will fail to pull images.
2. Apply the module with a subnet and admin CIDR, then complete first-boot setup in the in-product wizard (see After apply). The registry defaults to Ghost's ECR, and the newest published release is resolved automatically at first boot; set image_tag only to pin a specific release.

Quick start

module "ghost_agent" {
  # Pin to a released tag for reproducibility (omit ?ref=… to track main).
  source = "github.com/ghostsecurity/terraform-ghost-agent-platform?ref=<latest release tag, e.g. v0.2.0>"

  # Network placement
  subnet_id  = "subnet-0abc123def456789"
  admin_cidr = "203.0.113.42/32"
}

output "url" {
  value = module.ghost_agent.bringup_url
}

# Access the VM. Uses AWS Systems Manager Session Manager - no SSH key
# pair needed (the module attaches the SSM role). For SSH instead, set
# `ssh_key_name` on the module and use `module.ghost_agent.ssh_command`.
output "ssm" {
  value = module.ghost_agent.ssm_session_command
}

This bring-up uses the default nip.io fallback for the public hostname (no DNS setup required) and pulls a real Let's Encrypt cert automatically. To use a custom FQDN instead, set domain_name and optionally route53_zone_id.

After apply

terraform apply returns as soon as the EC2 instance is created, but the on-VM bootstrap (mount EBS, install Docker + cosign, verify image signatures, pull images, start the stack) takes another ~3-5 minutes. Wait for cloud-init to finish before trying to log in - watch the bootstrap log and look for the final ===> Bootstrap complete at <timestamp> line.

Two ways to watch:

# From your laptop - no SSM/SSH needed. Console output is buffered
# by EC2 and typically lags 1-3 minutes, but works even before the
# SSM agent comes up:
aws ec2 get-console-output --region <region, e.g. us-east-1> \
  --instance-id "$(terraform output -raw instance_id)" \
  --latest --output text | tail -100

# Real-time, via SSM Session Manager (requires the SSM agent to be
# running on the VM - happens a minute or two into bootstrap):
$(terraform output -raw ssm_session_command)
sudo tail -f /var/log/ghost-agent-bootstrap.log

Once the ===> Bootstrap complete line appears the stack is up. Useful terraform outputs:

terraform output bringup_url                # e.g. https://3-14-15-92.nip.io
terraform output -raw claim_token           # one-time setup-wizard token
terraform output -raw ssm_session_command   # aws ssm start-session --region <region> --target i-01234...

Open the bringup_url in a browser - the bring-up hostname is served with a real Let's Encrypt certificate, so no certificate warning (browsing by bare IP instead falls back to a self-signed certificate with a one-time warning). Enter the claim token in the setup wizard, create the admin account, and set the domain and TLS mode. The stack applies the configuration - the affected services restart briefly - and comes up at the configured domain with a Let's Encrypt (or uploaded) cert.

For extra caution, the wizard can be claimed over an IAM-authenticated channel instead of the public listener: forward the port with aws ssm start-session --target <instance-id> --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["443"],"localPortNumber":["8443"]}' and browse https://localhost:8443 (self-signed there, but the transport is authenticated by SSM). Alternatively, provision with public_ingress_cidrs scoped to your own IP and widen it after setup.

The claim token is single-use and inert after setup. If it is lost or exposed before claiming, rotate it on the host with sudo exo-reissue-claim-token (installed by the bootstrap).

Domain, TLS, worker count, and connector tokens can be changed later under System → Settings in the UI.

Ghost support access (cross-account SSM)

By default this module publishes an IAM role, <name_prefix>-ssm-support (e.g. ghost-agent-ssm-support), that Ghost Security assumes to open an AWS Systems Manager Session Manager shell on the VM for support. Its trust is scoped to a single Ghost support role, so only that role can assume it. The role grants only ssm:StartSession on this one instance - an interactive shell, or a port-forward to reach the UI without opening any inbound ports - plus management of the operator's own sessions: no SSH key, no inbound ports, and no access to any other resource. The instance still dials out to the SSM service exactly as it does for your own ssm_session_command; nothing new is exposed to the internet.

terraform output ssm_support_role_arn   # the role Ghost assumes (empty if disabled)

To turn it off, set ghost_support_access_enabled = false. The role is removed and Ghost has no Session Manager path to the VM.

Updating the deployment

Three paths, in order of operator overhead.

In-app (preferred)

A workspace admin opens the UI's workspace dropdown → System → Version. The page shows the running tag and (after a check-against-ECR) the latest available release. Clicking Upgrade to vX.Y.Z dispatches the upgrade to the in-stack updater container, which:

1. Fetches and cosign verifys the release's stack bundle — the docker-compose for that tag — from ECR
2. Applies it to /opt/exo and rewrites TAG= in /opt/exo/.env
3. Pulls the release's images and converges the stack (docker compose up -d --remove-orphans)

Because the compose travels with the release, an upgrade can add, remove, or reconfigure containers — not only change image tags. The updater excludes itself from the converge so it doesn't kill the container running the upgrade. Brief ~10-30s window during the gateway swap where the API may 502 - acceptable for a single-host deployment.

Rollback works the same way: the page offers a one-click rollback to the prior tag (derived from the audit log of past upgrades), fetching that tag's bundle so the topology reverts too.

Out-of-band (operator SSH)

When the updater itself needs a version bump (it doesn't auto-update), or as break-glass if the in-app path is broken:

$(terraform output -raw ssm_session_command)   # or ssh_command
cd /opt/exo
sudo sed -i 's/^TAG=.*/TAG=vX.Y.Z/' .env
# The updater image is pinned separately; bump it to the same tag too:
sudo sed -i 's/^UPDATER_TAG=.*/UPDATER_TAG=vX.Y.Z/' .env
# Fetch the stack bundle (the docker-compose) for the target tag so the
# topology matches the release, then converge:
REGISTRY=$(grep '^REGISTRY=' .env | cut -d= -f2)
sudo oras pull "$REGISTRY/exo-stack:vX.Y.Z" -o /opt/exo
sudo docker compose -f docker-compose.prod.yml pull
sudo docker compose -f docker-compose.prod.yml up -d --remove-orphans

ECR repositories use immutable tags, so byte-identical reproduction is guaranteed.

Terraform replace (heavier)

Force a clean instance recycle by tainting the EC2 resource:

terraform apply -replace=module.ghost_agent.aws_instance.vm

The instance is replaced, cloud-init re-runs end-to-end (including cosign verification), but the data EBS volume and secrets persist. Useful when AMI / cloud-init changes need to land alongside a version change. The replacement boots the newest published release (or var.image_tag when pinned - in that case make sure the pin matches the version you want before triggering the replace, since it overrides whatever the running stack was on).

Signature verification

The stack bundle (the docker-compose for a release) and every service image are cosign verify'd against Ghost's published-workflow identity before they are applied or started — at first boot and on every in-app upgrade. The verify policy is encoded in var.image_signing_identity_regex + var.image_signing_oidc_issuer. Defaults match the Ghost Security publish workflow on a v* tag:

identity-regexp: ^https://github\.com/ghostsecurity/exo/\.github/workflows/publish-to-ecr\.yml@refs/tags/v[0-9]+\.[0-9]+\.[0-9]+(-.*)?$
oidc-issuer:     https://token.actions.githubusercontent.com

If a signature fails verification, cloud-init exits before docker compose up (and an in-app upgrade aborts before touching the stack). Cosign and the workflow signer are pinned in lockstep on both sides - Ghost's publish workflow signs with cosign-installer@v4.1.2 (cosign v3.0.6), and the cloud-init verify uses the same version.

Troubleshooting

Symptom	Where to look
terraform apply succeeds but the bringup URL doesn't respond	Cloud-init log: sudo tail -200 /var/log/ghost-agent-bootstrap.log
cosign verify failure in the bootstrap log	Image-tag mismatch, or Ghost's signing identity regex doesn't match. Verify with cosign verify <image>@<digest> locally to confirm.
RepositoryNotFoundException during image pull	Cross-account ECR access not yet granted - contact Ghost.
502 from Caddy for /api calls	Gateway container down. docker compose -f /opt/exo/docker-compose.prod.yml ps and check gateway logs.
Browser cert error / "not secure" warning	LE issuance failed (port 80 unreachable, DNS not resolving, rate limited). Check docker logs <caddy-container>.
Database not initializing	First-boot replica-set bootstrap timing. Verify /var/lib/exo is actually the EBS mount (mountpoint /var/lib/exo) - if not, the bootstrap script aborts before docker starts.
Credential-proxy / gateway fail at startup with cannot program address ... conflicts with existing route 0.0.0.0/0 (in the bootstrap log)	Docker libnetwork regression in AL2023 docker 25.0.16 - it can't attach a container to multiple bridge networks when one is internal: true, which the proxy and gateway both use. The module pins Docker to 25.0.14 and version-locks it (DOCKER_VERSION in files/user_data.sh.tpl); if you hit this, Docker was bumped past the pin - re-pin and terraform apply -replace the instance. Only bump the pin once a fixed docker build lands in the AL2023 repo (confirm the proxy comes up on its three networks).
terraform apply fails with secret with this name is already scheduled for deletion	A previous terraform destroy puts secrets into a 7-day soft-delete (recovery_window_in_days = 7 in secrets.tf). To re-deploy into the same account before the window expires, force-delete the leftovers: for s in encryption-key claim-token; do aws secretsmanager delete-secret --secret-id "ghost-agent/$s" --force-delete-without-recovery --region <region>; done

Persistent log locations on the VM:

• /var/log/ghost-agent-bootstrap.log - cloud-init bootstrap (single run, captured during first boot)
• /var/log/cloud-init-output.log - cloud-init's own log
• docker logs <container> - runtime container logs (gateway, credential-proxy, worker, caddy, database)

Data persistence and recovery

The EBS data volume at /var/lib/exo holds all survival-critical state:

• tls/ - MITM CA and service CA private keys, the gateway's mTLS identity, and the self-generated session-signing secret (jwt.secret). Losing the CAs breaks credential interception and enrolled runner identities; losing jwt.secret invalidates active sessions. Stored credentials are decrypted with ENCRYPTION_KEY, which lives in Secrets Manager on AWS, not on this volume.
• mongo-data/ - application database.
• caddy-data/ - Let's Encrypt account and issued certificates.
• runner-identity/ - per-runner client certs and slot locks.
• artifacts/ - run output (recoverable, but loss is a regression).

The module sets prevent_destroy = true on this volume to guard against terraform destroy. That protects against accidental teardown, but not against data corruption, an accidental delete inside the VM, or loss of the availability zone - so production deployments should add scheduled snapshots.

Scheduled snapshots

Set enable_data_volume_snapshots = true and the module creates a Data Lifecycle Manager (DLM) policy that takes a daily snapshot of the data volume, retaining data_volume_snapshot_retention_days (30 by default), plus the DLM service role it runs as. The policy targets this deployment's volume by a dedicated tag, so it does not touch any other volume in the account.

module "ghost_agent" {
  # ...
  enable_data_volume_snapshots        = true
  data_volume_snapshot_retention_days = 30
}

For a centralized backup policy across many resources, leave this off and target the data volume (tagged Name = "<deployment-name>-data") from your own AWS Backup plan instead.

EBS snapshots are crash-consistent, and MongoDB recovers a crash-consistent snapshot on startup via its journal. For an application-quiesced snapshot, stop the stack first (sudo docker compose -f /opt/exo/docker-compose.prod.yml down), snapshot, then bring it back up.

Restoring from a snapshot

Restore swaps the data volume for a fresh one created from a snapshot, then reconciles Terraform state so the module adopts it. The instance keeps its Elastic IP (the URL is unchanged), and the restored volume auto-mounts on boot - the /etc/fstab entry matches by filesystem UUID, which a snapshot preserves, so there's no manual remount.

Run from the directory where the module is invoked. Set the region and deployment name, and pick a snapshot:

REGION=us-east-1
NAME=ghost-agent            # var.name_prefix

# instance_id is a module output (surface it at your root, as the examples
# do); the data volume has no output, so look it up by its Name tag.
INSTANCE_ID=$(terraform output -raw instance_id)
OLD_VOL=$(aws ec2 describe-volumes --region "$REGION" \
  --filters "Name=tag:Name,Values=${NAME}-data" --query 'Volumes[0].VolumeId' --output text)
AZ=$(aws ec2 describe-volumes --region "$REGION" --volume-ids "$OLD_VOL" \
  --query 'Volumes[0].AvailabilityZone' --output text)

aws ec2 describe-snapshots --region "$REGION" --owner-ids self \
  --filters "Name=volume-id,Values=${OLD_VOL}" \
  --query 'reverse(sort_by(Snapshots,&StartTime))[].{Id:SnapshotId,Started:StartTime,State:State}' --output table
SNAP=snap-xxxx             # a State=completed snapshot from the list

Stop the instance (for a clean detach), swap the volume, and start it again:

aws ec2 stop-instances --region "$REGION" --instance-ids "$INSTANCE_ID"
aws ec2 wait instance-stopped --region "$REGION" --instance-ids "$INSTANCE_ID"

aws ec2 detach-volume --region "$REGION" --volume-id "$OLD_VOL"
aws ec2 wait volume-available --region "$REGION" --volume-ids "$OLD_VOL"

NEW_VOL=$(aws ec2 create-volume --region "$REGION" --snapshot-id "$SNAP" \
  --availability-zone "$AZ" --volume-type gp3 \
  --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=${NAME}-data},{Key=exo:snapshot-group,Value=${NAME}},{Key=ManagedBy,Value=terraform-ghost-agent-platform},{Key=Component,Value=ghost-agent-platform}]" \
  --query VolumeId --output text)
aws ec2 wait volume-available --region "$REGION" --volume-ids "$NEW_VOL"

aws ec2 attach-volume --region "$REGION" --volume-id "$NEW_VOL" --instance-id "$INSTANCE_ID" --device /dev/sdf
aws ec2 wait volume-in-use --region "$REGION" --volume-ids "$NEW_VOL"

aws ec2 start-instances --region "$REGION" --instance-ids "$INSTANCE_ID"
aws ec2 wait instance-running --region "$REGION" --instance-ids "$INSTANCE_ID"

Reconcile Terraform state so the module manages the new volume instead of the old one (adjust module.ghost_agent to your module instance name):

terraform state rm 'module.ghost_agent.aws_volume_attachment.data' 'module.ghost_agent.aws_ebs_volume.data'
terraform import 'module.ghost_agent.aws_ebs_volume.data' "$NEW_VOL"
terraform import 'module.ghost_agent.aws_volume_attachment.data' "/dev/sdf:${NEW_VOL}:${INSTANCE_ID}"
terraform plan   # expect no changes (benign tag drift at most)

Verify (via SSM: mountpoint /var/lib/exo is mounted and docker compose -f /opt/exo/docker-compose.prod.yml ps is healthy), then delete the old volume once you trust the restore:

aws ec2 delete-volume --region "$REGION" --volume-id "$OLD_VOL"

Two things to get right: the new volume must be in the instance's availability zone (handled above), and you must reconcile Terraform state before the next terraform apply - skip it and the apply re-attaches the old volume, reverting the restore. Keep the old volume until the restore is verified; it is your rollback.

Full recovery set

The volume holds MongoDB, the TLS CA keys, and the platform-generated session-signing secret, but stored credentials in MongoDB are ciphertext that only ENCRYPTION_KEY (in AWS Secrets Manager) can decrypt. Within the same account that secret persists independently of the volume, so a same-account restore needs nothing more. To rebuild in a different account, also copy ENCRYPTION_KEY from Secrets Manager: without it every stored credential is unrecoverable.

Other notes:

• Treat the volume ID as sensitive - anyone with ec2:DescribeVolumes can locate it.
• For dev/test instances: temporarily set prevent_destroy = false (one-line module edit) before terraform destroy.

Configuration knobs

See variables.tf for the full input list. Most useful overrides:

Variable	When to override
image_tag	Pin the first boot to a specific release instead of resolving the newest one
image_registry	Point at a mirror or staging registry instead of Ghost's ECR
domain_name + route53_zone_id	Real public domain instead of the nip.io fallback
instance_type	Higher throughput; in-place change supported (~1-2 min downtime, data persists)
data_volume_size_gb	Anticipate high workflow rate / artifact retention
public_ingress_cidrs	Lock down 80/443 to a CDN/WAF origin (note: scoping 80 too tightly will break Let's Encrypt cert renewal)
ssh_key_name	Use SSH instead of SSM Session Manager
enable_data_volume_snapshots	Turn on daily DR snapshots of the data volume (with data_volume_snapshot_retention_days)
ghost_support_access_enabled	Set false to remove the Ghost support role and disable cross-account SSM access
image_signing_identity_regex	Ghost rotates the publish workflow path or pre-release tag pattern

Production recommendations

A few things this module does not enforce, but which any production deploy should have in place:

• Tighten admin_cidr to an office IP or VPN egress range. The variable validates that the value is a CIDR, not that it's narrow.
• Use a remote Terraform backend (S3 with KMS encryption). The module generates secrets via random_* resources, so their values live in state - local state on a laptop is not appropriate.
• Enable scheduled snapshots of the data volume by setting enable_data_volume_snapshots = true (see Data persistence and recovery). The volume holds all survival-critical state; prevent_destroy guards against accidental teardown but not against corruption or accidental file deletion inside the VM.
• Add CloudWatch alarms on EC2 status checks and configure instance auto-recovery. A single-VM deployment has no built-in failover.

License

Apache License 2.0 - see LICENSE.

Copyright 2026 Ghost Security