[Question] Why does metricDescriptor have to be created from the app

[Jeyob]

Hi,

Couldn't find any better place to ask this question (so sorry in advance)

I was just curious to understand what the relationship is between opentelemetry instrumentation and the Terraform monitoring resource?

The CloudMonitoringMetricsExporter when sending the metrics also appears to be making requests to create metric descriptors thus requiring the "roles/monitoring.metricWriter" permission on the running identity.

but then what is the purpose of the terraform resource? It would have been quite neat if I could just define the metric descriptors through terraform and then have my application use that without having to add additional permissions.

Thanks!
Jonas

[Jeyob]

On second thought, one use case for having the terraform resource being created beforehand is that it does give us something to reference in Cloud monitoring dashboards I suppose, without data necessarily having to be present..

[dashpole]

roles/monitoring.metricWriter is also required to call createTimeSeries, which writes the actual metric data. Even if you don't createMetricDescriptor, you will still need that permission.

If you call createTimeSeries, that will actually create a metric descriptor for you if one doesn't already exist. However, the unit and description of the metric are not populated (since createTimeSeries does not include those fields). We call createMetricDescriptor in order to populate those fields using the unit and description from the OpenTelemetry metric definition.

Using createMetricDescriptor isn't perfect. Requests often fail initially because of rate limiting, especially if a bunch of new metric descriptors are created at once, such as when starting a many replicas of the same pod (which each produce the same metrics). This just causes log spam, but can be confusing for users.

It can be tedious to create a terraform resource for each metric, since customers often have hundreds of metrics. Also, a typo or incorrect value type (int vs float) can cause metric writes to fail. So we generally recommend leaving managing descriptors to the exporter.