Deployment

Overview

The backend runs as a Docker container on EC2 instances behind an AWS Network Load Balancer (NLB). Deployments are zero-downtime rolling updates — instances are taken out of the load balancer one at a time, upgraded, and put back.

Images are built once in CI and pushed to ECR. Instances pull the pre-built image at deploy time — nothing is built on the instances themselves.

Push to main
    → CI builds Docker image
    → pushes to ECR (tagged with commit SHA + latest)
    → rolling deploy to dev (automatic)

Manual trigger in GitHub Actions
    → rolling deploy to prod (choose which SHA to deploy)

Environments

Environment	Region	Triggered by
dev	us-west-1	Push to main
prod	us-west-2	Manual workflow_dispatch

ECR lives in us-west-1 (same region as dev). Prod pulls cross-region.

Deploying to Dev

Push to main. The deploy-dev-backend.yml workflow runs automatically if any files under growl-backend/ changed:

1. Builds the Docker image with GIT_COMMIT baked in as an env var
2. Pushes to ECR tagged with the full commit SHA and latest
3. Runs a rolling deploy across all dev EC2 instances

You can watch it in the Actions tab on GitHub.

Deploying to Prod

Go to GitHub → Actions → Deploy to Prod → Run workflow.

You’ll be prompted for an image_tag — paste the full commit SHA you want to deploy. This must be a SHA that was already built and pushed to ECR by a previous dev deploy.

Danger

Never deploy a SHA to prod that hasn’t gone through dev first.

Rolling Deploy (how it works)

For each instance, in order:

1. Deregister the instance from the NLB
2. Wait 60s for in-flight connections to drain
3. SCP the latest Makefile to the instance
4. SSH in and run make ecr-deploy (pulls image from ECR, stops old container, starts new one)
5. Re-register the instance with the NLB
6. Poll NLB health checks until the instance is healthy (up to 300s)
7. Pause 5s, then repeat for the next instance

If make ecr-deploy fails, the instance is re-registered with the old container still running before moving on.

Before the loop starts, the script verifies that Terraform state is in sync with AWS infrastructure. If there is drift, the deploy is aborted.

Running a Rolling Deploy Manually

You’ll need:

• AWS credentials configured locally (aws configure)
• The EC2 SSH private key (available in Bitwarden)
• The env file for the target environment (available in Bitwarden) — place it at growl-backend/terraform/.env.dev or growl-backend/terraform/.env.prod
• Terraform installed
• jq installed

cd growl-backend/terraform
terraform workspace select dev

./scripts/rolling-deploy.sh \
  --env dev \
  --ssh-key ~/.ssh/growl-dev-key \
  --image-tag <commit-sha>   # omit to use latest

For prod:

terraform workspace select prod

./scripts/rolling-deploy.sh \
  --env prod \
  --ssh-key ~/.ssh/growl-prod-key \
  --image-tag <commit-sha>

Infrastructure

Infrastructure is managed with Terraform. State is stored in S3 (growl-tf-state, us-west-1) with workspaces separating dev and prod.

cd growl-backend/terraform

# See what would change
terraform workspace select dev
terraform plan -var-file=terraform-dev.tfvars

# Apply changes
terraform apply -var-file=terraform-dev.tfvars

Danger

Never run terraform apply during a deploy. The rolling deploy script checks for drift and will abort if it detects any — apply infra changes first, then deploy.

Tfvars files (terraform-dev.tfvars, terraform-prod.tfvars) are committed to git. They must not contain sensitive values — secrets belong in .env.* files instead.

Environment Variables

App env vars live in growl-backend/terraform/.env.dev and .env.prod. These files are not committed to git — they are available in Bitwarden.

They are pushed to instances during provisioning (via cloud-init). If you change an env file, you need to re-provision the instances (via terraform apply) before the new values take effect — a rolling deploy alone will not pick them up.

The pre-commit hook automatically updates growl-backend/terraform/env-dev.sha256 and growl-backend/terraform/env-prod.sha256 when env files change.

Initial Instance Provisioning

When Terraform creates a new EC2 instance, cloud-init:

1. Installs Docker, AWS CLI, and other dependencies
2. Decodes the Makefile from the Terraform config
3. Writes the .env file
4. Runs make ecr-deploy to pull the latest image and start the container

Instances are added to the NLB target group automatically by Terraform. The container mounts /tmp/deployment-test from the host so deployment test logs persist across container restarts.