Skip to content

Adds the * functionality for filenames, multiple filenames and --iterations #1176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ribalba
Copy link
Member

@ribalba ribalba commented May 6, 2025

This is a patch I often use when running benchmarks. Basis for a discussion

@ribalba ribalba requested a review from ArneTR May 6, 2025 21:14
@ArneTR
Copy link
Member

ArneTR commented May 7, 2025

I am a bit skeptical of this PR.

I do see the need to run scenarios one after another. We also do have a functionality called "Variance" in our API that inserts a run 3 times.

However on the CLI I wonder if the gain is really given bc users could just do:

for i in {1..3}; do python3 runner.py --uri XXX --filename XXX ; done

if they really want to do this.

On the other hand I see other tools like perf that also have a repetition parameter to repeat tests and thus nudge the user to use it, but that is for quick introspection of stuff.

With the GMT we want users to only use CLI functionality to develop and setup stuff quickly. Proper measurements must be done in Cluster mode.

So why nudge the user to use behaviour that is good for deriving StdDev and Means of stuff, if you are connected via SSH and on the CLI, which is skewing results in the first place?

@ribalba
Copy link
Member Author

ribalba commented May 7, 2025

Then maybe we should have a script ./submit_to_cluster that does these things for you. For the work I am doing now I have 15 usage scenario files that I need to run and I want 5 repeats. Of course I could do this with shell magic but like this I can just it all in the GMT.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants