Skip to content

Conversation

aleks-p
Copy link
Contributor

@aleks-p aleks-p commented Sep 10, 2025

Currently, pprof data is being sent as raw bytes in responses from query backend nodes, to backend and frontend nodes.

This has the benefit of a tightly controlled serialization and potentially smaller payloads, however tests have shown that letting connectrpc handle serialization results in less overhead for large requests, when combined together with #4422. I am yet to confirm what exactly causes the improvement, my guess is that the manual compression is more resource intense compared to the one from connectrpc but I will try to capture a diff profile to fully understand this.

End-to-end benchmark, with individual profiles having ~100-500k samples, ~50-100k locations, ~10-20k unique strings.

goos: darwin
goarch: arm64
pkg: github.com/grafana/pyroscope/pkg/test/integration
cpu: Apple M2 Pro
                                │ flat-bytes.txt │           flat-typed.txt           │
                                │     sec/op     │   sec/op    vs base                │
_ReadPath/SelectMergeProfile-10       9.311 ± 2%   6.558 ± 1%  -29.57% (p=0.000 n=10)

                                │ flat-bytes.txt │            flat-typed.txt            │
                                │      B/op      │     B/op      vs base                │
_ReadPath/SelectMergeProfile-10     14.73Gi ± 2%   11.63Gi ± 1%  -21.05% (p=0.000 n=10)

                                │ flat-bytes.txt │           flat-typed.txt            │
                                │   allocs/op    │  allocs/op   vs base                │
_ReadPath/SelectMergeProfile-10      81.27M ± 2%   72.63M ± 0%  -10.63% (p=0.000 n=10)

TODO:

Note before merging:
Once #4422 is merged, we should adjust the new aggregator introduced there (pkg/frontend/readpath/queryfrontend/report_aggregator.go) to support typed responses.

@aleks-p aleks-p added the performance If there's anything we have to be really good at it's this label Sep 10, 2025
@aleks-p aleks-p force-pushed the feat/typed-pprof-output branch from 586a65a to 81c39b0 Compare September 12, 2025 12:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance If there's anything we have to be really good at it's this
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant