Skip to content

core,server: report defrag metrics #5560

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

core,server: report defrag metrics #5560

wants to merge 5 commits into from
+279 −34

Conversation

abhijat
Copy link
Contributor

@abhijat abhijat commented Jul 24, 2025 •
edited
Loading

The stats collected during defragmentation are reported in this PR. The following counts are reported (summed up over all shards):

  • pages scanned
  • pages marked for reallocation (data was moved from these pages because their usage was lower than threshold)
  • pages which were full
  • pages which were reserved for malloc
  • pages where heap mismatch was observed so nothing was done
  • pages which were not full but were above the threshold

Additionally per shard, we report a histogram of pages which were not full but above threshold, the histogram is for p50, p90 and p99 (ie 50% were below or equal to X threshold and so on). This can guide the next defragment attempt, if the current attempt did not touch enough pages.

Internally hdr histogram is used to store the histogram, we also use bloom filter (SBF) to collect counts, because accurate counts are not necessary and storing hash sets of uintptr_t takes around 8 bytes per item whereas sbf uses much lower amount of memory with a sufficiently small false positive rate.

Scanning/reporting is done only for the interactive memory defragment command, for the scheduled task we do not collect stats.

@abhijat abhijat force-pushed the abhijat/feat/report-defrag-metrics branch 14 times, most recently from af4a388 to 26b1361 Compare July 25, 2025 12:01
@abhijat abhijat changed the title [WIP]: core,server: report defrag metrics core,server: report defrag metrics Jul 28, 2025
@abhijat abhijat requested review from romange and kostasrim July 28, 2025 07:02
@abhijat abhijat marked this pull request as ready for review July 28, 2025 07:02
@abhijat abhijat requested a review from dranikpg July 30, 2025 07:28
dranikpg
dranikpg previously approved these changes Jul 30, 2025
View reviewed changes
Copy link
Contributor

@dranikpg dranikpg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, small comments

src/core/page_usage_stats.h Outdated
#include <mimalloc/types.h>

#include <vector>
#include "core/bloom.h"
#include "server/tx_base.h"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what specifically do you need from tx_base? It just creates a two-way dependency

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I needed the ShardId typedef, but I have changed the map to just use the underlying type now.

src/core/page_usage_stats.cc Outdated
Comment on lines 87 to 93
UniquePages::UniquePages()
: pages_scanned{MakeSBF()},
pages_marked_for_realloc{MakeSBF()},
pages_full{MakeSBF()},
pages_reserved_for_malloc{MakeSBF()},
pages_with_heap_mismatch{MakeSBF()},
pages_above_threshold{MakeSBF()} {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be just a default constructor on FilterWithSize

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed

src/core/page_usage_stats.h Outdated
absl::btree_map<ShardId, ShardUsageSummary> shard_wide_summary;

void Merge(CollectedPageStats&& other, ShardId shard_id);
static CollectedPageStats Merge(CollectedPageStats* stats, size_t size, float threshold);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge(absl::Span<CollectedPageStats>, float...)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed to use span

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed it to take vector by rvalue ref to address #5560 (comment)

src/core/page_usage_stats.h Outdated
std::string ToString() const;
};

struct UniquePages {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this type can be hidden in PageUsage

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved to private type

src/core/page_usage_stats.cc Outdated
Comment on lines 69 to 76
std::vector<std::string> rows;
rows.push_back(absl::StrFormat("Page usage threshold: %f", threshold * 100));
rows.push_back(absl::StrFormat("Pages scanned: %d", pages_scanned));
rows.push_back(absl::StrFormat("Pages marked for reallocation: %d", pages_marked_for_realloc));
rows.push_back(absl::StrFormat("Pages full: %d", pages_full));
rows.push_back(absl::StrFormat("Pages reserved for malloc: %d", pages_reserved_for_malloc));
rows.push_back(
absl::StrFormat("Pages skipped due to heap mismatch: %d", pages_with_heap_mismatch));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could've been made a little simpler:
absl::StrAppend(&out, "Page scanned: ", pages_scanned, "\n");

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed

src/server/memory_cmd.cc Outdated
if (auto* shard = EngineShard::tlocal(); shard)
shard->DoDefrag(CollectPageStats::YES, threshold);

CollectedPageStats results[shard_set->size()];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We usually use heap arrays to not overflow the fiber stack (small) with large structures

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed to vector with initial size

abhijat added 4 commits July 31, 2025 15:16
@abhijat
build: Move hdr histogram to dfly core
ce72099 
The histogram dependency is used in page usage to store percentage of
page usage in a histogram.

Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
@abhijat
core: Collect stats while defragmenting
c90f2d3 
Store the page addresses in bloom filters to count unique pages per
category. Also store the page usage (used block/capacity) in histogram,
where the page usage is above threshold but would otherwise be fit for
reallocation, ie it is not full, not a malloc target etc.

Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
@abhijat
server: Collect page defrag results
eb5a7ba 
Collect and summarize defrag results from shards when command is run
interactively.

Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
@abhijat
core: Tests for stat collection
815b25a 
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
@abhijat abhijat force-pushed the abhijat/feat/report-defrag-metrics branch from 09c1a94 to 4fbcbfb Compare July 31, 2025 09:48
@abhijat
core: Address feedback
a75a525 
Signed-off-by: Abhijat Malviya <abhijat@dragonflydb.io>
@abhijat abhijat force-pushed the abhijat/feat/report-defrag-metrics branch from 4fbcbfb to a75a525 Compare July 31, 2025 10:03
@abhijat abhijat requested a review from dranikpg July 31, 2025 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@romange romange Awaiting requested review from romange

@kostasrim kostasrim Awaiting requested review from kostasrim

@dranikpg dranikpg Awaiting requested review from dranikpg

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@abhijat @dranikpg