EKS and ECS services: let deploy steps set allowlisted environment variables

.github/workflows/module-definitions.yml

@@ -43,6 +43,16 @@ jobs:
         run: npm test
         working-directory: tools/ravion-modules
 
+      - name: Setup Go
+        uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5
+        with:
+          go-version: "1.23"
+          cache-dependency-path: test/go.sum
+
+      - name: Test deploy environment expression runtime
+        run: go test -run '^(TestDeployEnvironmentVariableAllowlist|TestECSDeployEnvironmentVariableAllowlist)$' -count=1 ./deploy_env_allowlist
+        working-directory: test
+
       - name: Run migration guardrails
         run: node tools/ravion-modules/dist/src/cli.js guardrails
 

README.md

@@ -81,9 +81,9 @@ sync by `node tools/ravion-modules/dist/src/cli.js readme` (enforced in CI, and
 | `rvn-cloudfront` | CloudFront CDN | v1.7.0 | `cdn/cloudfront/` |
 | `rvn-ec2-service` | EC2 Service | v2.5.0 | `compute/ec2_service/` |
 | `rvn-ecs-cluster` | ECS Cluster | v2.4.0 | `compute/ecs_cluster/` |
-| `rvn-ecs-nlb` | ECS Network Service | v2.5.0 | `compute/ecs_service/` |
-| `rvn-ecs-web` | ECS Web Service | v2.8.0 | `compute/ecs_service/` |
-| `rvn-ecs-worker` | ECS Worker | v2.4.0 | `compute/ecs_service/` |
+| `rvn-ecs-nlb` | ECS Network Service | v2.6.0 | `compute/ecs_service/` |
+| `rvn-ecs-web` | ECS Web Service | v2.9.0 | `compute/ecs_service/` |
+| `rvn-ecs-worker` | ECS Worker | v2.5.0 | `compute/ecs_service/` |
 | `rvn-efs` | EFS File System | v1.1.1 | `storage/efs/` |
 | `rvn-elasticache` | ElastiCache | v1.1.2 | `cache/elasticache/` |
 | `rvn-lambda` | Lambda Function | v2.4.0 | `compute/lambda/` |

compute/ecs_service/rvn-ecs-nlb-definition.yml

@@ -3,8 +3,8 @@ definition:
   name: ECS Network Service
   description: Network Load Balanced ECS service for exposing one to five TCP, UDP, or TLS ports through an ECS cluster Network Load Balancer.
 release:
-  version: 2.5.0
-  description: Automatically size sandbox build CPU, memory, and disk from previous builds, with an option to use fixed sizes.
+  version: 2.6.0
+  description: Let pipeline deploy steps set allowlisted environment variables, such as a release version computed earlier in the pipeline.
 module:
   inputs:
     - id: section_cluster
@@ -363,8 +363,14 @@ module:
         > 0 ? module.input.image_start_command : nil), "cpu": (module.input.capacity_provider == "ec2" ?
         int(float(module.input.task_cpu) * 1024) : int(float(module.input.fargate_size.vcpu) * 1024)),
         "depends_on": [{"container_name": "log_router", "condition": "START"}], "environment": ([{name:
-        "PORT", value: string(module.input.listeners[0].container_port)}] | concat(module.input.environment_variables !=
-        nil ? module.input.environment_variables : [])), "essential": true, "image":
+        "PORT", value: string(module.input.listeners[0].container_port)}] |
+        concat(module.input.environment_variables != nil ? module.input.environment_variables : []) |
+        concat(module.input.deploy_environment_variable_names != nil && deploy.input.environment_variables != nil ?
+        map(filter(deploy.input.environment_variables, #.name != "PORT" &&
+        #.name in module.input.deploy_environment_variable_names && #.value != nil &&
+        !(#.name in map(module.input.environment_variables ?? [], .name)) &&
+        !(#.name in map(module.input.secrets ?? [], .name))),
+        {"name": #.name, "value": string(#.value)}) : [])), "essential": true, "image":
         (module.input.build_source == "image_registry" ? (deploy.input.image_ref contains
         "sha256:" ? module.input.image_repository + "@" + deploy.input.image_ref :
         module.input.image_repository + ":" + deploy.input.image_ref) : (deploy.input.image_ref contains
@@ -401,7 +407,12 @@ module:
         (module.input.capacity_provider == "ec2" ? int(float(module.input.task_cpu) * 1024) :
         int(float(module.input.fargate_size.vcpu) * 1024)), "environment": ([{name: "PORT", value:
         string(module.input.listeners[0].container_port)}] | concat(module.input.environment_variables != nil ?
-        module.input.environment_variables : [])), "essential": true, "image": (module.input.build_source ==
+        module.input.environment_variables : []) | concat(module.input.deploy_environment_variable_names != nil &&
+        deploy.input.environment_variables != nil ? map(filter(deploy.input.environment_variables, #.name != "PORT" &&
+        #.name in module.input.deploy_environment_variable_names && #.value != nil &&
+        !(#.name in map(module.input.environment_variables ?? [], .name)) &&
+        !(#.name in map(module.input.secrets ?? [], .name))), {"name": #.name, "value": string(#.value)}) : [])),
+        "essential": true, "image": (module.input.build_source ==
         "image_registry" ? (deploy.input.image_ref contains "sha256:" ?
         module.input.image_repository + "@" + deploy.input.image_ref : module.input.image_repository +
         ":" + deploy.input.image_ref) : (deploy.input.image_ref contains "sha256:" ?
@@ -598,7 +609,9 @@ module:
 
     ## Application configuration
 
-    Use Runtime environment variables for plain container environment values. Ravion also sets PORT from the first listener's Container port.
+    Use **Runtime environment variables** for plain values, as an array of `{name, value}` objects. Ravion also sets `PORT` from the first listener's Container port.
+
+    Use **Deploy-provided environment variables** to allow a pipeline deploy step to set selected plain values. They cannot replace a Runtime environment variable or Runtime secret, and `PORT` is reserved for the network service.
 
     Use Runtime secrets for sensitive values. Secrets are injected into the ECS task from SSM Parameter Store or Secrets Manager using name and value_from objects.
 
@@ -664,6 +677,7 @@ module:
     | Listener port                    | Yes      | 80                       | Unique NLB port for each listener                |
     | Listener protocol                | Yes      | TCP                      | TCP, TLS, or UDP for each listener               |
     | Container port                   | Yes      | 3000                     | Container port targeted by each listener         |
+    | Deploy-provided environment variables | No  | -                        | Names a pipeline deploy step may set to plain values; `PORT` is reserved |
     | TLS certificate                  | Yes*     | -                        | Required for each TLS listener                   |
     | TLS SSL policy                   | No       | ELBSecurityPolicy-TLS13-1-2-2021-06 | SSL policy for each TLS listener        |
     | TLS ALPN policy                  | No       | None                     | Optional ALPN behavior for each TLS listener     |

compute/ecs_service/rvn-ecs-web-definition.yml

@@ -3,8 +3,8 @@ definition:
   name: ECS Web Service
   description: Web server ECS service for running an HTTP application behind an ECS cluster load balancer.
 release:
-  version: 2.8.0
-  description: Automatically size sandbox build CPU, memory, and disk from previous builds, with an option to use fixed sizes.
+  version: 2.9.0
+  description: Let pipeline deploy steps set allowlisted environment variables, such as a release version computed earlier in the pipeline.
 module:
   inputs:
     - id: section_cluster
@@ -586,8 +586,14 @@ module:
         > 0 ? module.input.image_start_command : nil), "cpu": (module.input.capacity_provider == "ec2" ?
         int(float(module.input.task_cpu) * 1024) : int(float(module.input.fargate_size.vcpu) * 1024)),
         "depends_on": [{"container_name": "log_router", "condition": "START"}], "environment": ([{name:
-        "PORT", value: string(module.input.container_port)}] | concat(module.input.environment_variables !=
-        nil ? module.input.environment_variables : [])), "essential": true, "image":
+        "PORT", value: string(module.input.container_port)}] |
+        concat(module.input.environment_variables != nil ? module.input.environment_variables : []) |
+        concat(module.input.deploy_environment_variable_names != nil && deploy.input.environment_variables != nil ?
+        map(filter(deploy.input.environment_variables, #.name != "PORT" &&
+        #.name in module.input.deploy_environment_variable_names && #.value != nil &&
+        !(#.name in map(module.input.environment_variables ?? [], .name)) &&
+        !(#.name in map(module.input.secrets ?? [], .name))),
+        {"name": #.name, "value": string(#.value)}) : [])), "essential": true, "image":
         (module.input.build_source == "image_registry" ? (deploy.input.image_ref contains
         "sha256:" ? module.input.image_repository + "@" + deploy.input.image_ref :
         module.input.image_repository + ":" + deploy.input.image_ref) : (deploy.input.image_ref contains
@@ -624,7 +630,12 @@ module:
         (module.input.capacity_provider == "ec2" ? int(float(module.input.task_cpu) * 1024) :
         int(float(module.input.fargate_size.vcpu) * 1024)), "environment": ([{name: "PORT", value:
         string(module.input.container_port)}] | concat(module.input.environment_variables != nil ?
-        module.input.environment_variables : [])), "essential": true, "image": (module.input.build_source ==
+        module.input.environment_variables : []) | concat(module.input.deploy_environment_variable_names != nil &&
+        deploy.input.environment_variables != nil ? map(filter(deploy.input.environment_variables, #.name != "PORT" &&
+        #.name in module.input.deploy_environment_variable_names && #.value != nil &&
+        !(#.name in map(module.input.environment_variables ?? [], .name)) &&
+        !(#.name in map(module.input.secrets ?? [], .name))), {"name": #.name, "value": string(#.value)}) : [])),
+        "essential": true, "image": (module.input.build_source ==
         "image_registry" ? (deploy.input.image_ref contains "sha256:" ?
         module.input.image_repository + "@" + deploy.input.image_ref : module.input.image_repository +
         ":" + deploy.input.image_ref) : (deploy.input.image_ref contains "sha256:" ?
@@ -870,7 +881,9 @@ module:
 
     ## Application configuration
 
-    Use Runtime environment variables for plain container environment values. Ravion also sets PORT from Container port.
+    Use **Runtime environment variables** for plain values, as an array of `{name, value}` objects. Ravion also sets `PORT` from Container port.
+
+    Use **Deploy-provided environment variables** to allow a pipeline deploy step to set selected plain values. They cannot replace a Runtime environment variable or Runtime secret, and `PORT` is reserved for the web service.
 
     Use Runtime secrets for sensitive values. Secrets are injected into the ECS task from SSM Parameter Store or Secrets Manager using name and value_from objects.
 
@@ -995,6 +1008,7 @@ module:
     | Image repository                | Yes*     | nginx                      | Required for image registry deployments            |
     | Start command                   | No       | []                         | Command arguments that override an image default CMD |
     | Container port                  | Yes      | 80                         | Port exposed by the app container                  |
+    | Deploy-provided environment variables | No  | -                          | Names a pipeline deploy step may set to plain values; `PORT` is reserved |
     | Health check path               | Yes      | /                          | HTTP path used by the target group health check    |
     | Deregistration delay (secs)     | No       | 300                        | Drain window for a task that is deregistering      |
     | Sticky sessions                 | No       | true                       | Keep clients on the same task when possible        |

compute/ecs_service/rvn-ecs-worker-definition.yml

@@ -3,8 +3,8 @@ definition:
   name: ECS Worker
   description: Background worker ECS service for running private container workloads without exposed ports.
 release:
-  version: 2.4.0
-  description: Automatically size sandbox build CPU, memory, and disk from previous builds, with an option to use fixed sizes.
+  version: 2.5.0
+  description: Let pipeline deploy steps set allowlisted environment variables, such as a release version computed earlier in the pipeline.
 module:
   inputs:
     - id: section_cluster
@@ -100,7 +100,12 @@ module:
         > 0 ? module.input.image_start_command : nil), "cpu": (module.input.capacity_provider == "ec2" ?
         int(float(module.input.task_cpu) * 1024) : int(float(module.input.fargate_size.vcpu) * 1024)),
         "depends_on": [{"container_name": "log_router", "condition": "START"}], "environment":
-        (module.input.environment_variables != nil ? module.input.environment_variables : []), "essential": true,
+        ((module.input.environment_variables != nil ? module.input.environment_variables : []) |
+        concat(module.input.deploy_environment_variable_names != nil && deploy.input.environment_variables != nil ?
+        map(filter(deploy.input.environment_variables, #.name in module.input.deploy_environment_variable_names &&
+        #.value != nil && !(#.name in map(module.input.environment_variables ?? [], .name)) &&
+        !(#.name in map(module.input.secrets ?? [], .name))),
+        {"name": #.name, "value": string(#.value)}) : [])), "essential": true,
         "image": (module.input.build_source == "image_registry" ? (deploy.input.image_ref contains
         "sha256:" ? module.input.image_repository + "@" + deploy.input.image_ref : module.input.image_repository
         + ":" + deploy.input.image_ref) : (deploy.input.image_ref contains "sha256:" ?
@@ -131,8 +136,13 @@ module:
         "enable-ecs-log-metadata": string(module.input.firelens_ecs_metadata_enabled)}}}] : [{"command":
         (len(module.input.image_start_command) > 0 ? module.input.image_start_command : nil), "cpu":
         (module.input.capacity_provider == "ec2" ? int(float(module.input.task_cpu) * 1024) :
-        int(float(module.input.fargate_size.vcpu) * 1024)), "environment": (module.input.environment_variables != nil ?
-        module.input.environment_variables : []), "essential": true, "image": (module.input.build_source ==
+        int(float(module.input.fargate_size.vcpu) * 1024)), "environment":
+        ((module.input.environment_variables != nil ? module.input.environment_variables : []) |
+        concat(module.input.deploy_environment_variable_names != nil && deploy.input.environment_variables != nil ?
+        map(filter(deploy.input.environment_variables, #.name in module.input.deploy_environment_variable_names &&
+        #.value != nil && !(#.name in map(module.input.environment_variables ?? [], .name)) &&
+        !(#.name in map(module.input.secrets ?? [], .name))),
+        {"name": #.name, "value": string(#.value)}) : [])), "essential": true, "image": (module.input.build_source ==
         "image_registry" ? (deploy.input.image_ref contains "sha256:" ? module.input.image_repository + "@" +
         deploy.input.image_ref : module.input.image_repository + ":" + deploy.input.image_ref) :
         (deploy.input.image_ref contains "sha256:" ? stack.output.ecr_repository_url + "@" + deploy.input.image_ref :
@@ -225,6 +235,14 @@ module:
 
     The running task alarm uses the Container Insights `RunningTaskCount` metric and stays silent when Container Insights is disabled on the cluster. Every alarm treats missing data as not breaching and must breach for the configured number of consecutive periods before it fires. Add SNS topic ARNs as Alarm actions and OK actions to be notified.
 
+    ## Application configuration
+
+    Use **Runtime environment variables** for plain values, as an array of `{name, value}` objects.
+
+    Use **Deploy-provided environment variables** to allow a pipeline deploy step to set selected plain values. They cannot replace a Runtime environment variable or Runtime secret.
+
+    Use **Runtime secrets** for sensitive values, as an array of `{name, value_from}` objects. `value_from` is an SSM Parameter Store or Secrets Manager ARN.
+
     ## Configuration
 
     | Field | Required | Default | Description |
@@ -242,6 +260,7 @@ module:
     | Custom metric scaling policies | No | [] | Queue/backlog-oriented target tracking policies |
     | Runtime environment variables | No | - | Environment variables passed to the worker container |
     | Runtime secrets | No | - | Secrets injected from SSM Parameter Store or Secrets Manager |
+    | Deploy-provided environment variables | No | - | Names a pipeline deploy step may set to plain values |
     | Stop timeout (secs) | No | 30 | Seconds between SIGTERM and SIGKILL when a worker task stops; Fargate allows up to 120 |
     | Sidecars | No | [] | Additional containers in the worker task |
     | EFS file system | No | false | Mount an EFS file system into the worker container |

compute/eks_service/rvn-eks-cron-definition.yml

@@ -365,6 +365,8 @@ module:
 
     Use **Runtime environment variables** for plain values, as an array of `{name, value}` objects.
 
+    Use **Deploy-provided environment variables** to allow a pipeline deploy step to set selected plain values. They cannot replace a Runtime environment variable or Runtime secret.
+
     Use **Runtime secrets** for sensitive values, as an array of `{name, value_from}` objects - the same shape as ECS. `value_from` must be a full Secrets Manager or SSM Parameter Store ARN, including the JSON-key extraction suffix when you want a single key out of a JSON secret.
 
     Ravion never reads these values. The ARNs travel to the chart as references, the chart renders an `ExternalSecret`, and the External Secrets Operator materializes the Kubernetes Secret in the cluster. No secret value reaches Ravion, the Helm values document, or the Helm release history.
@@ -435,6 +437,7 @@ module:
     | Memory limit | Yes | 512Mi | Memory ceiling per run |
     | Runtime environment variables | No | - | Array of {name, value} objects |
     | Runtime secrets | No | - | Array of {name, value_from} ARN references |
+    | Deploy-provided environment variables | No | - | Names a pipeline deploy step may set to plain values |
     | Create IAM role | No | false | Create an IAM role for the workload and bind it to its pods through EKS Pod Identity |
     | IAM role name | No | {service}-task | Explicit role name, for a service whose ECS task role already holds the default |
     | IAM policy ARNs | No | [] | Managed policy ARNs attached to the workload role |