Skip to content

Commit 8707bcb

Browse files
fdmsantosfdmsantos
committed
fix: Fix Incorrect ARNs in MSK Policy
1 parent 767f2c5 commit 8707bcb

File tree

8 files changed

+177
-55
lines changed

8 files changed

+177
-55
lines changed

‎README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -998,13 +998,15 @@ No modules.
998998
| [aws_iam_policy_document.kinesis](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
999999
| [aws_iam_policy_document.lambda](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
10001000
| [aws_iam_policy_document.msk](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
1001+
| [aws_iam_policy_document.msk_source_assume_role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
10011002
| [aws_iam_policy_document.opensearch](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
10021003
| [aws_iam_policy_document.opensearchserverless](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
10031004
| [aws_iam_policy_document.s3](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
10041005
| [aws_iam_policy_document.s3_kms](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
10051006
| [aws_iam_policy_document.secretsmanager](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
10061007
| [aws_iam_policy_document.secretsmanager_cmk_encryption](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
10071008
| [aws_iam_policy_document.vpc](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
1009+
| [aws_msk_cluster.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/msk_cluster) | data source |
10081010
| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
10091011
| [aws_subnet.subnet](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/subnet) | data source |
10101012

‎examples/s3/msk-to-s3/README.md

Lines changed: 42 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,35 @@ $ terraform apply
1414

1515
Note that this example may create resources which cost money. Run `terraform destroy` when you don't need these resources.
1616

17+
* Send Message to Kafka
18+
19+
[Documentation](https://docs.aws.amazon.com/msk/latest/developerguide/create-serverless-cluster-client.html)
20+
21+
```sh
22+
# Create Client Machine
23+
sudo su -
24+
sudo yum -y install java-11
25+
wget https://archive.apache.org/dist/kafka/2.8.1/kafka_2.12-2.8.1.tgz
26+
tar -xzf kafka_2.12-2.8.1.tgz
27+
wget https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar
28+
mv aws-msk-iam-auth-1.1.1-all.jar kafka_2.12-2.8.1/libs/
29+
vi kafka_2.12-2.8.1/bin/client.properties
30+
security.protocol=SASL_SSL
31+
sasl.mechanism=AWS_MSK_IAM
32+
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
33+
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
34+
35+
# Create Topic
36+
export BS=my-endpoint
37+
./kafka_2.12-2.8.1/bin/kafka-topics.sh --bootstrap-server $BS --command-config kafka_2.12-2.8.1/bin/client.properties --create --topic demo-topic --partitions 6
38+
39+
# Produce data
40+
./kafka_2.12-2.8.1/bin/kafka-console-producer.sh --broker-list $BS --producer.config kafka_2.12-2.8.1/bin/client.properties --topic demo-topic
41+
42+
# Consume Data
43+
./kafka_2.12-2.8.1/bin/kafka-console-consumer.sh --bootstrap-server $BS --consumer.config kafka_2.12-2.8.1/bin/client.properties --topic demo-topic --from-beginning
44+
```
45+
1746
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
1847
## Requirements
1948

@@ -34,6 +63,7 @@ Note that this example may create resources which cost money. Run `terraform des
3463

3564
| Name | Source | Version |
3665
|------|--------|---------|
66+
| <a name="module_ec2"></a> [ec2](#module\_ec2) | terraform-aws-modules/ec2-instance/aws | n/a |
3767
| <a name="module_firehose"></a> [firehose](#module\_firehose) | ../../../ | n/a |
3868
| <a name="module_msk_cluster"></a> [msk\_cluster](#module\_msk\_cluster) | terraform-aws-modules/msk-kafka-cluster/aws | 2.3.0 |
3969
| <a name="module_security_group"></a> [security\_group](#module\_security\_group) | terraform-aws-modules/security-group/aws | ~> 5.0 |
@@ -53,8 +83,19 @@ Note that this example may create resources which cost money. Run `terraform des
5383
| Name | Description | Type | Default | Required |
5484
|------|-------------|------|---------|:--------:|
5585
| <a name="input_name_prefix"></a> [name\_prefix](#input\_name\_prefix) | Name prefix to use in resources | `string` | `"msk-to-s3-basic"` | no |
86+
| <a name="input_tags"></a> [tags](#input\_tags) | Default Tags to be added to all resources. | `map(string)` | `{}` | no |
5687

5788
## Outputs
5889

59-
No outputs.
90+
| Name | Description |
91+
|------|-------------|
92+
| <a name="output_kinesis_data_stream_name"></a> [kinesis\_data\_stream\_name](#output\_kinesis\_data\_stream\_name) | The name of the Kinesis Firehose Stream |
93+
| <a name="output_kinesis_firehose_arn"></a> [kinesis\_firehose\_arn](#output\_kinesis\_firehose\_arn) | The ARN of the Kinesis Firehose Stream |
94+
| <a name="output_kinesis_firehose_destination_id"></a> [kinesis\_firehose\_destination\_id](#output\_kinesis\_firehose\_destination\_id) | The Destination id of the Kinesis Firehose Stream |
95+
| <a name="output_kinesis_firehose_role_arn"></a> [kinesis\_firehose\_role\_arn](#output\_kinesis\_firehose\_role\_arn) | The ARN of the IAM role created for Kinesis Firehose Stream |
96+
| <a name="output_kinesis_firehose_version_id"></a> [kinesis\_firehose\_version\_id](#output\_kinesis\_firehose\_version\_id) | The Version id of the Kinesis Firehose Stream |
97+
| <a name="output_msk_arn"></a> [msk\_arn](#output\_msk\_arn) | MSK Topic Endpoint |
98+
| <a name="output_msk_brokers_endpoint"></a> [msk\_brokers\_endpoint](#output\_msk\_brokers\_endpoint) | Brokers endpoints |
99+
| <a name="output_s3_bucket_arn"></a> [s3\_bucket\_arn](#output\_s3\_bucket\_arn) | S3 Bucket ARN |
100+
| <a name="output_topic_name"></a> [topic\_name](#output\_topic\_name) | MSK Topic Name |
60101
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

‎examples/s3/msk-to-s3/main.tf

Lines changed: 50 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -3,43 +3,62 @@ data "aws_availability_zones" "available" {}
33
locals {
44
vpc_cidr = "10.0.0.0/16"
55
azs = slice(data.aws_availability_zones.available.names, 0, 2)
6+
topic = "demo-topic"
67
}
78

89
resource "random_pet" "this" {
910
length = 2
1011
}
1112

13+
### S3 Bucket ###
1214
resource "aws_s3_bucket" "s3" {
1315
bucket = "${var.name_prefix}-destination-bucket-${random_pet.this.id}"
1416
force_destroy = true
1517
}
1618

19+
### Networking ###
1720
module "vpc" {
1821
source = "terraform-aws-modules/vpc/aws"
1922
version = "~> 5.0"
2023
name = "${var.name_prefix}-vpc"
2124
cidr = local.vpc_cidr
2225
azs = local.azs
2326
public_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k)]
24-
private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 3)]
25-
database_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 6)]
26-
create_database_subnet_group = true
27-
enable_nat_gateway = true
28-
single_nat_gateway = true
27+
create_database_subnet_group = false
28+
enable_nat_gateway = false
29+
single_nat_gateway = false
2930
}
3031

3132
module "security_group" {
32-
source = "terraform-aws-modules/security-group/aws"
33-
version = "~> 5.0"
34-
35-
name = "${var.name_prefix}-sg"
36-
description = "Security group for ${var.name_prefix}-sg"
37-
vpc_id = module.vpc.vpc_id
38-
39-
ingress_cidr_blocks = module.vpc.private_subnets_cidr_blocks
33+
source = "terraform-aws-modules/security-group/aws"
34+
version = "~> 5.0"
35+
name = "${var.name_prefix}-sg"
36+
description = "Security group for ${var.name_prefix}-sg"
37+
vpc_id = module.vpc.vpc_id
38+
ingress_cidr_blocks = ["0.0.0.0/0"]
4039
ingress_rules = [
41-
"kafka-broker-tcp",
42-
"kafka-broker-tls-tcp"
40+
"all-all"
41+
]
42+
egress_cidr_blocks = ["0.0.0.0/0"]
43+
egress_rules = ["all-all"]
44+
}
45+
46+
### MSK ###
47+
module "ec2" {
48+
source = "terraform-aws-modules/ec2-instance/aws"
49+
name = "test-instance"
50+
instance_type = "t2.micro"
51+
subnet_id = module.vpc.public_subnets[0]
52+
vpc_security_group_ids = [module.security_group.security_group_id]
53+
create_iam_instance_profile = true
54+
create_eip = true
55+
iam_role_description = "IAM role for EC2 instance"
56+
iam_role_policies = {
57+
AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
58+
AdministratorAccess = "arn:aws:iam::aws:policy/AdministratorAccess"
59+
}
60+
depends_on = [
61+
module.msk_cluster
4362
]
4463
}
4564

@@ -71,8 +90,8 @@ module "msk_cluster" {
7190
sasl = { iam = true }
7291
}
7392
enable_storage_autoscaling = false
74-
create_cloudwatch_log_group = false
75-
cloudwatch_logs_enabled = false
93+
create_cloudwatch_log_group = true
94+
cloudwatch_logs_enabled = true
7695
s3_logs_enabled = false
7796
configuration_name = "${var.name_prefix}-msk-configuration"
7897
configuration_description = "${var.name_prefix} MSK configuration"
@@ -92,22 +111,27 @@ resource "aws_msk_cluster_policy" "this" {
92111
"Service" = "firehose.amazonaws.com"
93112
}
94113
Action = [
95-
"kafka:Describe*",
96-
"kafka:Get*",
97114
"kafka:CreateVpcConnection",
98115
"kafka:GetBootstrapBrokers",
116+
"kafka:DescribeCluster",
117+
"kafka:DescribeClusterV2"
99118
]
100119
Resource = module.msk_cluster.arn
101120
}]
102121
})
103122
}
104123

105124
module "firehose" {
106-
source = "../../../"
107-
name = "${var.name_prefix}-delivery-stream"
108-
input_source = "msk"
109-
msk_source_cluster_arn = module.msk_cluster.arn
110-
msk_source_topic_name = "test"
111-
destination = "s3"
112-
s3_bucket_arn = aws_s3_bucket.s3.arn
125+
source = "../../../"
126+
name = "${var.name_prefix}-delivery-stream"
127+
input_source = "msk"
128+
msk_source_cluster_arn = module.msk_cluster.arn
129+
msk_source_connectivity_type = "PRIVATE"
130+
msk_source_topic_name = local.topic
131+
destination = "s3"
132+
s3_bucket_arn = aws_s3_bucket.s3.arn
133+
buffering_interval = 10
134+
depends_on = [
135+
aws_msk_cluster_policy.this
136+
]
113137
}

‎examples/s3/msk-to-s3/outputs.tf

Lines changed: 44 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,44 @@
1-
#output "kinesis_firehose_arn" {
2-
# description = "The ARN of the Kinesis Firehose Stream"
3-
# value = module.firehose.kinesis_firehose_arn
4-
#}
5-
#
6-
#output "kinesis_data_stream_name" {
7-
# description = "The name of the Kinesis Firehose Stream"
8-
# value = module.firehose.kinesis_firehose_name
9-
#}
10-
#
11-
#output "kinesis_firehose_destination_id" {
12-
# description = "The Destination id of the Kinesis Firehose Stream"
13-
# value = module.firehose.kinesis_firehose_destination_id
14-
#}
15-
#
16-
#output "kinesis_firehose_version_id" {
17-
# description = "The Version id of the Kinesis Firehose Stream"
18-
# value = module.firehose.kinesis_firehose_version_id
19-
#}
20-
#
21-
#output "kinesis_firehose_role_arn" {
22-
# description = "The ARN of the IAM role created for Kinesis Firehose Stream"
23-
# value = module.firehose.kinesis_firehose_role_arn
24-
#}
1+
output "msk_arn" {
2+
description = "MSK Topic Endpoint"
3+
value = module.msk_cluster.arn
4+
}
5+
6+
output "kinesis_firehose_arn" {
7+
description = "The ARN of the Kinesis Firehose Stream"
8+
value = module.firehose.kinesis_firehose_arn
9+
}
10+
11+
output "kinesis_data_stream_name" {
12+
description = "The name of the Kinesis Firehose Stream"
13+
value = module.firehose.kinesis_firehose_name
14+
}
15+
16+
output "kinesis_firehose_destination_id" {
17+
description = "The Destination id of the Kinesis Firehose Stream"
18+
value = module.firehose.kinesis_firehose_destination_id
19+
}
20+
21+
output "kinesis_firehose_version_id" {
22+
description = "The Version id of the Kinesis Firehose Stream"
23+
value = module.firehose.kinesis_firehose_version_id
24+
}
25+
26+
output "kinesis_firehose_role_arn" {
27+
description = "The ARN of the IAM role created for Kinesis Firehose Stream"
28+
value = module.firehose.kinesis_firehose_role_arn
29+
}
30+
31+
output "msk_brokers_endpoint" {
32+
description = "Brokers endpoints"
33+
value = module.msk_cluster.bootstrap_brokers
34+
}
35+
36+
output "topic_name" {
37+
description = "MSK Topic Name"
38+
value = local.topic
39+
}
40+
41+
output "s3_bucket_arn" {
42+
description = "S3 Bucket ARN"
43+
value = aws_s3_bucket.s3.arn
44+
}

‎examples/s3/msk-to-s3/providers.tf

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
provider "aws" {
2+
skip_metadata_api_check = true
3+
skip_region_validation = true
4+
skip_credentials_validation = true
5+
default_tags {
6+
tags = var.tags
7+
}
8+
}

‎examples/s3/msk-to-s3/variables.tf

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,9 @@ variable "name_prefix" {
33
type = string
44
default = "msk-to-s3-basic"
55
}
6+
7+
variable "tags" {
8+
description = "Default Tags to be added to all resources."
9+
type = map(string)
10+
default = {}
11+
}

‎iam.tf

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,12 @@ locals {
1717
add_vpc_policy = var.create && var.create_role && var.enable_vpc && var.vpc_use_existing_role && local.is_search_destination
1818
add_secretsmanager_policy = var.create && var.create_role && var.enable_secrets_manager
1919
add_secretsmanager_decrypt_policy = local.add_secretsmanager_policy && var.secret_kms_key_arn != null
20+
msk_source_topic_arn = local.add_msk_source_policy ? "arn:aws:kafka:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:topic/${data.aws_msk_cluster.this[0].cluster_name}/${data.aws_msk_cluster.this[0].cluster_uuid}/${var.msk_source_topic_name}" : null
21+
msk_source_group_arn = local.add_msk_source_policy ? "arn:aws:kafka:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:group/${data.aws_msk_cluster.this[0].cluster_name}/${data.aws_msk_cluster.this[0].cluster_uuid}/*" : null
2022
}
2123

2224
data "aws_iam_policy_document" "assume_role" {
23-
count = var.create && var.create_role ? 1 : 0
25+
count = var.create && var.create_role && !local.is_msk_source ? 1 : 0
2426

2527
statement {
2628
effect = "Allow"
@@ -41,14 +43,28 @@ data "aws_iam_policy_document" "assume_role" {
4143
}
4244
}
4345

46+
data "aws_iam_policy_document" "msk_source_assume_role" {
47+
count = var.create && var.create_role && local.is_msk_source ? 1 : 0
48+
49+
statement {
50+
effect = "Allow"
51+
actions = ["sts:AssumeRole"]
52+
53+
principals {
54+
type = "Service"
55+
identifiers = ["firehose.amazonaws.com"]
56+
}
57+
}
58+
}
59+
4460
resource "aws_iam_role" "firehose" {
4561
count = var.create && var.create_role ? 1 : 0
4662
name = local.role_name
4763
description = var.role_description
4864
path = var.role_path
4965
force_detach_policies = var.role_force_detach_policies
5066
permissions_boundary = var.role_permissions_boundary
51-
assume_role_policy = data.aws_iam_policy_document.assume_role[0].json
67+
assume_role_policy = !local.is_msk_source ? data.aws_iam_policy_document.assume_role[0].json : data.aws_iam_policy_document.msk_source_assume_role[0].json
5268
tags = merge(var.tags, var.role_tags)
5369
}
5470

@@ -128,7 +144,7 @@ data "aws_iam_policy_document" "msk" {
128144
"kafka-cluster:ReadData"
129145
]
130146
resources = [
131-
"${var.msk_source_cluster_arn}/${var.msk_source_topic_name}"
147+
local.msk_source_topic_arn
132148
]
133149
}
134150

@@ -138,7 +154,7 @@ data "aws_iam_policy_document" "msk" {
138154
"kafka-cluster:DescribeGroup"
139155
]
140156
resources = [
141-
"${var.msk_source_cluster_arn}/*"
157+
local.msk_source_group_arn
142158
]
143159
}
144160
}

‎main.tf

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,11 @@ data "aws_subnet" "subnet" {
77
id = var.vpc_subnet_ids[0]
88
}
99

10+
data "aws_msk_cluster" "this" {
11+
count = local.is_msk_source ? 1 : 0
12+
cluster_name = split("/", var.msk_source_cluster_arn)[1]
13+
}
14+
1015
resource "aws_kinesis_firehose_delivery_stream" "this" {
1116
count = var.create ? 1 : 0
1217
name = local.is_waf_source ? "aws-waf-logs-${var.name}" : var.name

0 commit comments

Comments
 (0)