Cold starts double to quadruple when layer is included

[ianwremmel]

Is your feature request related to a problem? Please describe.
I'm just getting started with OpenTelemetry and just the act of adding the ADOT NodeJS layer to my functions makes the simplest ping function from 1-2 to 4-5 seconds and my remix app's cold start goes from 8 seconds to more than 30 (I've got the function set to timeout at 30).

Is there any configuration that I can do to improve that or is that just the way it has to be?

Describe the solution you'd like
Minimal impact on cold start times.

Describe alternatives you've considered

• use standard xray libs
• use otel with non-AWS providers and send directly from application code

[NathanielRN]

Hi @ianwremmel, thanks for raising this issue!

We've actually gotten issues on things like this before. A user on the OpenTelemetry CNCF Slack channel did a useful deep dive into investigating cold start impacts of Lambda with the ADOT NodeJS Lambda Layer.

Unfortunately it seems that the best solution is to bump up the memory capacity of your Lambda which does improve Cold Start time. If your Lambda is being used a lot this should prove to be a good idea.

Let me know if that helps!

[ianwremmel]

Gotcha, thanks!

Could you share that post somehow? I don't seem to be able to access that Slack.

[ianwremmel]

After a bit of experimenting (and seeing a serendipitous tweet this morning), I've determined that my cold start times without the layer were bad due to running the the --source-map-support flag. Once that was removed, my ping function cold start time was reduced to about 700ms with a MemorySize of 128MB, which seems pretty tolerable to me.

With the unexpected overhead of --source-map-support removed and the layer added, I still see ping taking 4 to 5 seconds and my remix app (MemorySize of 1024MB) still times out. If I up MemorySize to 8192 for all functions, I end up with ping at 3.6 seconds and the remix app at 11 seconds.

So, unless you've got other suggestions, it sounds like the AWS otel lambda support isn't viable for my project.

[NathanielRN]

Could you share that post somehow? I don't seem to be able to access that Slack.

Here is the Discussion on OTel JS that the user linked.

[ianwremmel]

Thanks! yea, it looks I'm in kind of the same boat. I didn't see the kind of speedup he did by increasing memory, but even if I had, that would be way too expensive. I don't think there's any way I can use AWS otel, given these numbers, but maybe there's still a chance to use otel instrumentation and ship directly to third-party receivers (e.g., honeycomb).

[NathanielRN]

One improvement you might try to reduce the cold start time is to use ESBuild instead of CommonJS build for your Node JS app on Lambda. ESBuild is nicer because it does operations like tree-shaking to seriously reduce the size of imported libraries which reduces memory and improves performance.

There are discussions upstream on OpenTelemetry JS Contrib to improve support for ESBuild because it requires more careful configuration. You might find the comment thread at open-telemetry/opentelemetry-js-contrib#647 (comment) helpful towards this goal and improve your Node App's memory impact in a way that would make ADOT Lambda layers usable for your purpose 🙂

The solution utilizes esbuild's "external" option and a custom plugin. "external" is an array of packages you tell esbuild to not bundle, and the plugin picks up those external packages and installs them into a raw node_modules directory when building. OpenTelemetry needs to monkey patch methods, so you can't bundle the libraries it needs to instrument (e.g. http, express, graphql, etc.), but you can instrument everything else.

[ianwremmel]

I'm already using rollup to do pretty aggressive treeshaking and I've tried putting all the otel packages on my dependencies layer. Unfortunately, as soon as I include the AWS Open Telemetry layer, cold boot times skyrocket.

I've made a few attempts at switch my lambdas over to mjs, but things go wrong with dependencies every time.

Note that the ping lambda I've been talking about above is about 40k after being processed by rollup.

[ianwremmel]

came back to poke at this a little bit more and it seems like most of the overhead comes from adding http: to protocols in opentelemetry.yml. If I stick to only grpc:, then cold starts are only about 10% worse rather than 400% worse.

[ianwremmel]

Another interesting detail. This line from the docs site, when used with a remix app, seems to mostly prevent the function from returning. From what I can tell, initialization takes more than 30 seconds. (For reasons not entirely clear to me, I couldn't get the lambda timeout to be more than 30 seconds, but that might be a sleep-deprivation issue on my part).

const tracerProvider = new NodeTracerProvider(tracerConfig);

[ianwremmel]

Actually, that's the wrong line. I think it's this line:

import {NodeTracerProvider} from '@opentelemetry/node';

Apparently there's some side effect somewhere in @opentelemetry/node that completely tanks lambda initialization when Remix is involved.

[ianwremmel]

For reasons not entirely clear to me, I couldn't get the lambda timeout to be more than 30 seconds, but that might be a sleep-deprivation issue on my part

It's the API gateway timeout that has hard limit at 30 seconds. Looking at the corresponding xray trace, it looks like the lambda takes 1.2 minutes to initialize when both remix and @opentelemetry/node are present.

Admittedly, remix is only the most obvious package in this lambda. There are any number of frontend packages that I could imagine might not play well with otel; the bundle file is 3MB.

[ianwremmel]

After a bunch of refactoring to get AWSXRayIdGenerator and AWSXRayPropagator working, both of the problems I mentioned above seem to have gone away: the http receiver and @opentelementr/node + remix are no longer making initialization times skyrocket. I don't really know what meaningfully changed.

[willarmiros]

After a bunch of refactoring to get AWSXRayIdGenerator and AWSXRayPropagator working

@ianwremmel why did you need to refactor to get these components to work? They should work out of the box with the Lambda layer instrumentation