docs: Add Managed Kafka Connect terraform sample for Clusters #876
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| /** | ||
| * Copyright 2025 Google LLC | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| # [START managedkafka_create_connect_cluster_parent] | ||
| resource "google_managed_kafka_cluster" "example-kafka-cluster" { | ||
salmany marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| project = data.google_project.default.project_id | ||
| cluster_id = "my-cluster-id" | ||
| location = "us-central1" | ||
| capacity_config { | ||
| vcpu_count = 3 | ||
| memory_bytes = 3221225472 | ||
salmany marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
| gcp_config { | ||
| access_config { | ||
| network_configs { | ||
| subnet = "projects/${data.google_project.default.number}/regions/us-central1/subnetworks/default" | ||
|
There was a problem hiding this comment. Should this value be linked to a subnet resource? the hardcoding of the region and subnet name is suboptimal There was a problem hiding this comment. Although I acknowledge this is suboptimal, I'm hardcoding the subnet resource due to a current limitation with deleting network attachments associated with Managed Kafka Connect clusters. I had previously attempted creating a Subnet resource as part of the code example, but it was causing test failures. (For more information, please see here) However, hardcoding the subnet resource also seems to be no longer working (although it initially passed the test cases). This could be due to the changing test setup across runs. Do you have any suggestions on how we may proceed to pass the tests given this limitation? There was a problem hiding this comment. I've committed a workaround using a provisioner for deleting leaked resources: 612a5ac Seems to be working with test cases. Marking as resolved. There was a problem hiding this comment. (moved my review comment to this thread) We do not support local-exec in samples. We do not want to encourage our users to use this functionality. If there's an upstream issue, it needs to be fixed at the resource level. If the sample can't be tested (either now, or in general), then we can disable the terraform apply/destroy part of tests. The google_managed_kafka_cluster takes 30 minutes to create, the google_managed_kafka_connect_cluster takes 20 minutes, unsure if this is expected. There was a problem hiding this comment. Understood. Given that there is a known issue which prevents resource deletion, I believe the best path forward for now would be to disable the terraform apply/destroy part of the tests as you have suggested. Once the issue is resolved, we can update the examples to enable terraform apply/destroy. (Also, it is indeed expected that the google_managed_kafka_cluster and google_managed_kafka_connect_cluster take 30 & 20 minutes to create, respectively. For reference, see here.) There was a problem hiding this comment. To give you some context, there's an issue with GKE networking that leaks Network Attachment resources in a Subnet even after a ConnectCluster is deleted. This means that if we add a Subnet resource to the same Terraform config, upon TF delete action the Subnet resource's deletion will hang, as there's still a dangling Network Attachment left. The leak eventually goes away, but its indefinite - perhaps hours, perhaps days. The leak issue is pretty complex to fix and doesn't have staffing yet - so it may not land for months/quarters. I think the best way forward is to include the subnet resources hardcoded as strings in the Terraform samples. It allows us to exercise the apply/delete logic for the ConnectCluster, which is what we care about here. For most customers, their network resources will be managed in a separate file, often by separate teams altogether. And the lifetime of Network/Subnet resources is often longer than those of using resources like ConnectClusters. So our customers may still be ok, even with this unfortunate leak issue preventing them from putting network + ConnectCluster resources in the same file. How does that sound? There was a problem hiding this comment. Generally: Our samples should reflect best practices in real world applications. We have some samples where the configuration required to make the sample run cleanly is only useful in a CI testing instance, and not code we would suggest a user configure. In these cases, we can disable the You can disable this by adding a test.yaml file with There was a problem hiding this comment. If the best-practice sample is not going to run without modification (ie: applying the Network resources separate from the ConnectCluster resource), is it still okay to recommend that? Should we add a comment with the limitation? We don't want to mislead users by showing them a sample that won't work out of the box. There was a problem hiding this comment. If that staggered application is required, you can describe that in the devsite page. You can also use additional |
||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| # [START managedkafka_create_connect_cluster] | ||
| resource "google_managed_kafka_connect_cluster" "example-kafka-connect-cluster" { | ||
| provider = google-beta | ||
| project = data.google_project.default.project_id # Replace this with your project ID in quotes | ||
salmany marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| connect_cluster_id = "my-connect-cluster-id" | ||
| location = "us-central1" | ||
| kafka_cluster = google_managed_kafka_cluster.example-kafka-cluster.id # Replace this with the ID of the primary Managed Service for Apache Kafka cluster associated with your Connect cluster in quotes | ||
| capacity_config { | ||
| vcpu_count = 3 | ||
salmany marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| memory_bytes = 3221225472 | ||
| } | ||
| gcp_config { | ||
| access_config { | ||
| network_configs { | ||
| primary_subnet = "projects/${data.google_project.default.number}/regions/us-central1/subnetworks/default" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| # [END managedkafka_create_connect_cluster] | ||
|
|
||
| data "google_project" "default" { | ||
| provider = google-beta | ||
salmany marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
| # [END managedkafka_create_connect_cluster_parent] | ||
Uh oh!
There was an error while loading. Please reload this page.