Core Cloud MSK Module

This MSK Child Module is written and maintained by the Core Cloud Platform team and includes the following checks and scans: Terraform validate, Terraform fmt, TFLint, Checkov scan, Sonarqube scan and Semantic versioning - MAJOR.MINOR.PATCH.

Module Structure

---| .github
    ---| dependabot.yaml - Checks repository daily for any dependency updates and raises a PR into main for review.
    ---| workflows
        ---| pull-request-sast.yaml - Workflow containing terraform init, terraform validate, terraform fmt - referencing Core Cloud TFLint, Checkov scan and Sonarqube scan shared workflows. Runs on pull request and merge to main branch.
        ---| pull-request-semver-label-check.yaml - Verifies all PRs to main raised in the module must have an appropriate semver label: major/minor/patch.
        ---| pull-request-semver-tag-merge.yaml - Calculates the new semver value depending on the PR label and tags the repository with the correct tag.
---| tests
  ---| msk.tftest.hcl
  ---| msk.tftest.hcl
  ---| msk.tftest.hcl
---| CHANGELOG.md - Contains all significant changes in relation to a semver tag made to this module.
---| CODEOWNERS
---| CODE_OF_CONDUCT
---| CONTRIBUTING.md
---| LICENSE.md
---| README.md
---| main.tf - Contains the main set of configuration for this module.
---| outputs.tf - Contain the output definitions for this module.
---| variables.tf - Contains the declarations for module variables, all variables have a defined type and short description outlining their purpose.
---| versions.tf - Contains the providers needed by the module.

Terraform Tests

All module tests are located in the test/ folder and uses Terraform test. These are written and maintained by the Core Cloud QA team.
The test files found in this folder validate the msk module configuration.
Please refer to the Official Hashicorp Terraform Test documentation.

Usage

Recommended settings:

• Opt into Open Monitoring (prometheus_jmx_exporter and prometheus_node_exporter)

• Adhere to Core Cloud mandatory tags.

• Opt into TLS client authentication through AWS Certificate Manager.

• Note: When creating a PCA, once created via the console select the option to 'Install the CA Certificate' - the CA status will then update from 'Pending' to 'Active' for a successful Terraform apply.

terraform {
  source = "git::https://github.com/UKHomeOffice/core-cloud-msk-tf-module.git?ref={tag}"
}

inputs = {
  account_id                       = "xxx"
  certificate_authority            = true
  client_unauthenticated           = false
  cluster_name                     = "msk-test"
  ebs_volume_size                  = 1000
  environment                      = "test"
  instance_type                    = "kafka.t3.small"
  jmx_exporter_monitoring_enabled  = true
  node_exporter_monitoring_enabled = true
  number_of_broker_nodes           = 2
  project_name                     = "xxx"
  storage_autoscaling_max_capacity = 1050
  subnet_ids                       = ["subnet-xxx", "subnet-xxx"]
  tls_authentication               = true
  vpc_id                           = "vpc-xxx"
  vpc_cidr                         = ["xxx"]


  # Tags for all resources
  tags = {
    cost-centre      = "xxx"
    account-code     = "xxx"
    portfolio-id     = "xxx"
    project-id       = "xxx"
    service-id       = "xxx"
    environment-type = "test"
    owner-business   = "xxx"
    budget-holder    = "xxx"
    source-repo      = "xxx"
  }
}

Requirements

Name	Version
terraform	>= 1.9.3
aws	>= 5.88.0

Providers

Name	Version
aws	>= 5.88.0

Modules

No modules.

Resources

Name	Type
aws_acmpca_certificate_authority.msk_with_ca	resource
aws_appautoscaling_policy.msk_appautoscaling_policy	resource
aws_appautoscaling_target.msk_appautoscaling_target	resource
aws_cloudwatch_log_group.msk_broker_logs	resource
aws_iam_policy.msk_cloudwatch_logs_write	resource
aws_iam_policy.msk_iam_ca_policy	resource
aws_iam_policy.msk_permissions	resource
aws_iam_role.msk_role	resource
aws_iam_role_policy_attachment.attach_cloudwatch_logs	resource
aws_iam_role_policy_attachment.attach_msk_permissions	resource
aws_iam_role_policy_attachment.msk_ca_policy_attachment	resource
aws_msk_cluster.msk_cluster	resource
aws_security_group.sg_msk	resource
aws_iam_policy_document.msk_ca_policy	data source

Inputs

Name	Description	Type	Default	Required
account_id	The AWS Account ID.	string	n/a	yes
ca_arn	ARN of the AWS managed CA to attach to the MSK cluster	list(string)	[]	no
ca_type	The type of the certificate authority	string	"ROOT"	no
certificate_authority	True if PCA should be created on cluster creation and there is not an existing CA to use	bool	null	no
client_unauthenticated	True if no client authentication. Should be false if TLS authentication enabled.	bool	true	no
cluster_name	Name of the MSK Cluster	string	""	no
days_after_initiation	Specifies the number of days after initiating a multipart upload when the multipart upload must be completed.	number	15	no
ebs_volume_size	MSK EBS Volume Size	number	1000	no
environment	Environment name	string	n/a	yes
instance_type	MSK Cluster Instance Type	string	"kafka.t3.small"	no
jmx_exporter_monitoring_enabled	Whether to enable JMX Exporter Open Monitoring	bool	false	no
kafka_version	n/a	string	"3.9.x"	no
node_exporter_monitoring_enabled	Whether to enable Node Exporter Open Monitoring	bool	false	no
number_of_broker_nodes	n/a	number	3	no
project_name	Name of the project	string	""	no
region	AWS region	string	"eu-west-2"	no
storage_autoscaling_max_capacity	The MSK cluster EBS maximum volume size for each broker. Value between 1 and 16384.	number	1	no
storage_autoscaling_threshold	The percentage threshold that needs to be exceeded to trigger a scale up. Value between 10 and 80.	number	65	no
storage_mode	Specify the storage mode for MSK brokers. Valid values: LOCAL (default) or TIERED.	string	"LOCAL"	no
subnet_ids	A list of subnets that the MSK cluster should run in	list(string)	n/a	yes
tags	Tags to be applied to the msk	map(string)	{}	no
tls_authentication	Enables TLS client authentication	bool	false	no
vpc_cidr	VPC CIDR Range	list(string)	n/a	yes
vpc_id	The MSK cluster's VPC ID	string	n/a	yes

Outputs

Name	Description
bootstrap_brokers_tls	n/a
msk_cluster_arn	The MSK cluster arn
msk_cluster_ca_arn	The MSK cluster CA arn
msk_sg_id	The MSK security group ID
zookeeper_connect_string	n/a