KMS Permission Verification Report

Date: 2026-06-30
Cluster: kms-verify (infra ID: kms-verify-2xs8c)
HyperShift: commit 73da66e (main)
OCP Version: 4.22.3 (selected by defaulting webhook)
KMS Key: arn:aws:kms:us-east-1:820196288204:key/8062a2b1-5212-4326-8075-2e5de8f1bc58
Storage Role: kms-verify-2xs8c-aws-ebs-csi-driver-controller
Region: us-east-1

Summary

Empirically verified that exactly three KMS IAM actions are required for the EBS CSI
driver to create encrypted volumes with a customer-managed KMS key:

1. kms:Decrypt
2. kms:GenerateDataKeyWithoutPlaintext
3. kms:CreateGrant

Starting from a ROSA-equivalent storage role (zero KMS actions), permissions were added
one at a time. Only when all three are present does volume creation succeed. The
kms:ViaService: ec2.<region>.amazonaws.com condition is compatible and does not break
functionality.

Results

Error Analysis

Phase 2 (zero KMS) — Error

rpc error: code = Internal desc = Could not create volume "pvc-...":
timed out waiting for volume to create: operation error EC2: DescribeVolumes,
api error InvalidVolume.NotFound: The volume 'vol-...' does not exist.

Interpretation: EBS CreateVolume is called with KmsKeyId and Encrypted=true.
Without KMS permissions, EBS attempts to create the volume but the encryption setup
fails. The volume ID is allocated but the volume never becomes available — it's
immediately invalidated. The CSI driver then polls DescribeVolumes and gets
InvalidVolume.NotFound because the volume was rolled back by EBS.

Phase 3 (Decrypt only) — Same error

Same InvalidVolume.NotFound pattern. kms:Decrypt alone is insufficient — EBS
needs GenerateDataKeyWithoutPlaintext to create the data encryption key during
volume creation.

Phase 4 (Decrypt + GenerateDataKeyWithoutPlaintext) — Same error

Same InvalidVolume.NotFound pattern. Volume creation likely progresses further
(data key generation succeeds) but without kms:CreateGrant, EBS cannot establish
the grant that allows the EBS service principal to decrypt the data key during
volume attachment. EBS rolls back the volume.

Phase 5 (all three) — Success

Encrypted: true
KmsKeyId: arn:aws:kms:us-east-1:820196288204:key/8062a2b1-5212-4326-8075-2e5de8f1bc58

Volume created, encrypted with the specified key, attached, pod running.

Phase 6 (all three + ViaService condition) — Success

Same successful result. The kms:ViaService: ec2.us-east-1.amazonaws.com condition
correctly scopes the KMS actions to calls made through the EC2/EBS service without
breaking functionality.

CloudTrail Evidence

CloudTrail shows the KMS API calls made by the EBS service during successful volume
creation (Phases 5-6):

Key observation: Encrypt appears in CloudTrail even though the storage role does
NOT have kms:Encrypt permission. This confirms that EBS calls kms:Encrypt under
its own service principal (via the grant), not under the CSI driver's role. The driver
does not need kms:Encrypt — this validates the upstream driver's documented policy.

Recommended Policy for ROSA Managed Policy Submission

{
  "Effect": "Allow",
  "Action": [
    "kms:Decrypt",
    "kms:GenerateDataKeyWithoutPlaintext",
    "kms:CreateGrant"
  ],
  "Resource": "arn:aws:kms:*:*:key/*",
  "Condition": {
    "StringEquals": {
      "kms:ViaService": "ec2.*.amazonaws.com"
    }
  }
}

This statement should be added to ROSAAmazonEBSCSIDriverOperatorPolicy.

Methodology

1. Fresh HyperShift operator installed from main with --private-platform AWS
2. Fresh HostedCluster created with --public-only, defaulting webhook selected 4.22.3
3. ClusterCSIDriver.spec.driverConfig.aws.kmsKeyARN manually patched on guest cluster
4. Cluster-storage-operator automatically reconfigured gp3-csi StorageClass with kmsKeyId and encrypted=true
5. Storage role inline policy swapped via aws iam put-role-policy for each phase
6. Test PVC + pod created for each phase, waited 180s for bind
7. Evidence captured from PVC events, pod events, CSI driver logs, and aws ec2 describe-volumes
8. CloudTrail queried for KMS API calls
9. Original policy restored after verification

References

• Epic #54 — Export KMS key ARN for initial StorageClass in HyperShift
• Story #183 — This verification story
• Upstream EBS CSI KMS policy — "Encrypted EBS Volumes via KMS" section
• HyperShift dev storage role — inline policy with 8 KMS actions
• ROSA managed policy — zero KMS actions (the gap)