Skip to content

Data size based aggregation #4548

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 49 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
9f29c29
Add a test to invoke data-size based aggregation
scottwittenburg May 9, 2025
71d6788
experiment with getting data size from all ranks
scottwittenburg May 13, 2025
896a36b
add partitioner class to helper, and greedy implementation
scottwittenburg May 14, 2025
56696cc
Fix compile warning and issue w/ serial tests, add a partitioning test
scottwittenburg May 14, 2025
5ab0958
Remove badly conceived test
scottwittenburg May 14, 2025
7dda794
Move data-size exchange logic to MPIChain
scottwittenburg Jun 6, 2025
0827c09
Begin working to make some writer state aggregator-specific
scottwittenburg Jun 11, 2025
9c7b465
A few obvious fixes:
scottwittenburg Jun 11, 2025
a6f7c5c
Prefer references to raw pointers
scottwittenburg Jun 11, 2025
80a1a17
Scale back the avoidance of init for cache hits, work on a test
scottwittenburg Jun 12, 2025
9ecbd31
Fix a couple issues, make the test simpler
scottwittenburg Jun 13, 2025
f3aad59
apply clang formatting
scottwittenburg Jun 13, 2025
b93fb9a
Testing reading back in, print partitioning summary, etc...
scottwittenburg Jun 13, 2025
619139f
fix some compiler warnings
scottwittenburg Jun 13, 2025
e012e8b
Try to fix remaining compiler warnings in ci
scottwittenburg Jun 16, 2025
3c62b47
Fix another warning caught by windows ci
scottwittenburg Jun 16, 2025
bed4ce5
Use a number of aggregators to match ci capabilities
scottwittenburg Jun 16, 2025
0c0e2a0
Test improvements
scottwittenburg Jun 17, 2025
b6a15ac
apply clang-format
scottwittenburg Jun 17, 2025
99a161a
fix compiler warnings, parameterize number of steps
scottwittenburg Jun 17, 2025
b8c1e04
FIx more compiler warnings
scottwittenburg Jun 17, 2025
0c7f5e5
Handle steps with rank data size shuffling
scottwittenburg Jun 17, 2025
4a82b23
fix unused variable warning
scottwittenburg Jun 17, 2025
12e5cf2
Try to get multiple time steps working
scottwittenburg Jun 18, 2025
93d9fe6
Properly finish a debug print
scottwittenburg Jun 18, 2025
ea016ac
adjust some debug prints
scottwittenburg Jun 25, 2025
75d07fa
Track data position for each substream/file between steps
scottwittenburg Jun 25, 2025
6fc3380
fix compiler warning
scottwittenburg Jun 25, 2025
9e07962
Only update metadata file names on step 0
scottwittenburg Jun 26, 2025
edccc6b
Fix debug data loc info dump
scottwittenburg Jun 26, 2025
51ddb1e
Fix data-size based aggregator bug
scottwittenburg Jun 26, 2025
c2ec010
1. build mpi chains at most once per step, 2. comment some debugs
scottwittenburg Jun 26, 2025
ceb6b0c
clang format
scottwittenburg Jun 27, 2025
316af62
Synchronize opening files for data-size based aggregation
scottwittenburg Jun 30, 2025
422340b
add back debug stmts, but guard by verbose check
scottwittenburg Jun 30, 2025
48a37b2
clang format
scottwittenburg Jun 30, 2025
cfb8d0d
Each rank reads/validates a block, instead of all
scottwittenburg Jun 30, 2025
d4cd0bf
Add some profiling around dsb-specific operations
scottwittenburg Jul 2, 2025
6229e50
avoid long chains of ranks w/ no data
scottwittenburg Jul 2, 2025
26c295e
clang format
scottwittenburg Jul 2, 2025
0a38aa0
Use captured variable, add a comment
scottwittenburg Jul 2, 2025
03f854a
refactor partitioning logic into the writer
scottwittenburg Jul 2, 2025
d576b8b
improve debug print management
scottwittenburg Jul 2, 2025
e174d3c
Fix broken test
scottwittenburg Jul 2, 2025
b7aefc2
Fix compiler warning on windows
scottwittenburg Jul 2, 2025
c41db8a
remove some unnecessary changes
scottwittenburg Jul 2, 2025
8bba19e
Clean up partitioning test
scottwittenburg Jul 2, 2025
9a1ca59
Ensure IO for write goes out of scope before read
scottwittenburg Jul 3, 2025
bc366af
Try size-based aggregation as the default
scottwittenburg Jul 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions source/adios2/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ add_library(adios2_core
helper/adiosMath.cpp
helper/adiosMemory.cpp
helper/adiosNetwork.cpp
helper/adiosPartitioner.cpp
helper/adiosPluginManager.cpp
helper/adiosString.cpp helper/adiosString.tcc
helper/adiosSystem.cpp
Expand Down
4 changes: 4 additions & 0 deletions source/adios2/engine/bp5/BP5Engine.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,10 @@ void BP5Engine::ParseParams(IO &io, struct BP5Params &Params)
{
parameter = (int)AggregationType::TwoLevelShm;
}
else if (value == "datasizebased")
{
parameter = (int)AggregationType::DataSizeBased;
}
else
{
helper::Throw<std::invalid_argument>(
Expand Down
3 changes: 2 additions & 1 deletion source/adios2/engine/bp5/BP5Engine.h
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,7 @@ class BP5Engine
EveryoneWrites,
EveryoneWritesSerial,
TwoLevelShm,
DataSizeBased,
Auto
};

Expand All @@ -148,7 +149,7 @@ class BP5Engine
MACRO(DirectIO, Bool, bool, false) \
MACRO(DirectIOAlignOffset, UInt, unsigned int, 512) \
MACRO(DirectIOAlignBuffer, UInt, unsigned int, 0) \
MACRO(AggregationType, AggregationType, int, (int)AggregationType::TwoLevelShm) \
MACRO(AggregationType, AggregationType, int, (int)AggregationType::DataSizeBased) \
MACRO(AsyncOpen, Bool, bool, true) \
MACRO(AsyncWrite, AsyncWrite, int, (int)AsyncWrite::Sync) \
MACRO(GrowthFactor, Float, float, DefaultBufferGrowthFactor) \
Expand Down
Loading
Loading