Introduce runner group definition. (via --runner-group-member-id/--runner-group-total-members or env vars) (#397)

* WIP on benchmark runner groups (splitting tests among workers)

* Introduce runner group definition. (via --runner-group-member-id/--runner-group-total-members or env vars)

* Addressed org change from RedisLabsModules to redis-performance

* Addressed org change from RedisLabsModules to redis-performance

Author: filipecosta90

README.md

@@ -1,13 +1,13 @@
-[![codecov](https://codecov.io/gh/RedisLabsModules/redisbench-admin/branch/master/graph/badge.svg)](https://codecov.io/gh/RedisLabsModules/redisbench-admin)
-![Actions](https://github.com/RedisLabsModules/redisbench-admin/workflows/Run%20Tests/badge.svg?branch=master)
+[![codecov](https://codecov.io/gh/redis-performance/redisbench-admin/branch/master/graph/badge.svg)](https://codecov.io/gh/redis-performance/redisbench-admin)
+![Actions](https://github.com/redis-performance/redisbench-admin/workflows/Run%20Tests/badge.svg?branch=master)
 ![Actions](https://badge.fury.io/py/redisbench-admin.svg)
 
-# [redisbench-admin](https://github.com/RedisLabsModules/redisbench-admin)
+# [redisbench-admin](https://github.com/redis-performance/redisbench-admin)
 
 Redis benchmark run helper can help you with the following tasks:
 
 - Setup abd teardown of benchmarking infrastructure specified
-  on [RedisLabsModules/testing-infrastructure](https://github.com/RedisLabsModules/testing-infrastructure)
+  on [redis-performance/testing-infrastructure](https://github.com/redis-performance/testing-infrastructure)
 - Setup and teardown of an Redis and Redis Modules DBs for benchmarking
 - Management of benchmark data and specifications across different setups
 - Running benchmarks and recording results
@@ -19,7 +19,7 @@ Current supported benchmark tools:
 
 - [redis-benchmark](https://github.com/redis/redis)
 - [memtier_benchmark](https://github.com/RedisLabs/memtier_benchmark)
-- [redis-benchmark-go](https://github.com/filipecosta90/redis-benchmark-go)
+- [redis-benchmark-go](https://github.com/redis-performance/redis-benchmark-go)
 - [YCSB](https://github.com/RediSearch/YCSB)
 - [tsbs](https://github.com/RedisTimeSeries/tsbs)
 - [redisgraph-benchmark-go](https://github.com/RedisGraph/redisgraph-benchmark-go)

docs/Readme.md

@@ -3,7 +3,7 @@
 
 The automated benchmark definitions provides a framework for evaluating and comparing feature branches and catching regressions prior letting them into the master branch.
 
-To be able to run local benchmarks you need `redisbench_admin>=0.1.64` [[tool repo for full details](https://github.com/RedisLabsModules/redisbench-admin)] and the benchmark tool specified on each configuration file . You can install redisbench-admin via PyPi as any other package. 
+To be able to run local benchmarks you need `redisbench_admin>=0.1.64` [[tool repo for full details](https://github.com/redis-performance/redisbench-admin)] and the benchmark tool specified on each configuration file . You can install redisbench-admin via PyPi as any other package. 
 ```
 pip3 install redisbench_admin>=0.1.64
 ```
@@ -21,7 +21,7 @@ A benchmark definition will then consist of:
 
 - mandatory client configuration (`clientconfig`) specifing the parameters to pass to the benchmark tool tool. The properties allowed here are: `tool`, `min-tool-version`, `tool_source`, `parameters`. If you don't have the required tools and the `tool_source` property is specified then the benchmark client will be downloaded once to a local path `./binaries/<tool>`. 
 
-- optional ci remote definition (`remote`), with the proper terraform deployment configurations definition. The properties allowed here are `type` and `setup`. Both properties are used to find the proper benchmark specification folder within [RedisLabsModules/testing-infrastructure](https://github.com/RedisLabsModules/testing-infrastructure). As an example, if you specify ` - type: oss-standalone` and `- setup: redistimeseries-m5` the used terraform setup will be described by the setup at [`testing-infrastructure/tree/terraform/oss-standalone-redistimeseries-m5`](https://github.com/RedisLabsModules/testing-infrastructure/tree/master/terraform/oss-standalone-redistimeseries-m5)
+- optional ci remote definition (`remote`), with the proper terraform deployment configurations definition. The properties allowed here are `type` and `setup`. Both properties are used to find the proper benchmark specification folder within [redis-performance/testing-infrastructure](https://github.com/redis-performance/testing-infrastructure). As an example, if you specify ` - type: oss-standalone` and `- setup: redistimeseries-m5` the used terraform setup will be described by the setup at [`testing-infrastructure/tree/terraform/oss-standalone-redistimeseries-m5`](https://github.com/redis-performance/testing-infrastructure/tree/master/terraform/oss-standalone-redistimeseries-m5)
 
 - optional KPIs definition (`kpis`), specifying the target upper or lower bounds for each relevant performance metric. If specified the KPIs definitions constraints the tests passing/failing. 
 

docs/export.md

@@ -1,22 +1,22 @@
 
 
 
-# [redisbench-admin export](https://github.com/RedisLabsModules/redisbench-admin)
+# [redisbench-admin export](https://github.com/redis-performance/redisbench-admin)
 
 Redis benchmark exporter can help you exporting performance results based on several formats input (CSV, JSON) and
 pushing them to data sinks in a time-series format. 
 
 Ultimately it provides a framework for evaluating and comparing feature branches and catching regressions prior letting them into the master branch,
-as shown on the bellow sample dashboard produced from exported data via [redisbench-admin export](https://github.com/RedisLabsModules/redisbench-admin).
+as shown on the bellow sample dashboard produced from exported data via [redisbench-admin export](https://github.com/redis-performance/redisbench-admin).
 
 ![RedisTimeSeries Sample CI dashboard detail](screenshot_RedisTimeSeries_CI_benchmarks_Grafana.png)
 
 Current supported benchmark tools to export data from:
 
 - [redis-benchmark](https://github.com/redis/redis)
 - [memtier_benchmark](https://github.com/RedisLabs/memtier_benchmark)
-- [redis-benchmark-go](https://github.com/filipecosta90/redis-benchmark-go)
-- [YCSB](https://github.com/RediSearch/YCSB)
+- [redis-benchmark-go](https://github.com/redis-performance/redis-benchmark-go)
+- [YCSB](https://github.com/redis-performance/go-ycsb)
 - [tsbs](https://github.com/RedisTimeSeries/tsbs)
 - [redisgraph-benchmark-go](https://github.com/RedisGraph/redisgraph-benchmark-go)
 - [ftsb_redisearch](https://github.com/RediSearch/ftsb)
@@ -161,7 +161,7 @@ memtier_benchmark --json-out-file results.json
 
 After the benchmark ends we will have the `results.json` file.
 Together with the "Sample exporter definition for memtier_benchmark" YAML above, that you can download at 
-[exporter-memtier-metrics.yml](https://github.com/RedisLabsModules/redisbench-admin/blob/master/docs/exporter-memtier-metrics.yml)
+[exporter-memtier-metrics.yml](https://github.com/redis-performance/redisbench-admin/blob/master/docs/exporter-memtier-metrics.yml)
 you can push the memtier results to redistimeseries via:
 
 **Command:**

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "redisbench-admin"
-version = "0.9.44"
+version = "0.9.55"
 description = "Redis benchmark run helper. A wrapper around Redis and Redis Modules benchmark tools ( ftsb_redisearch, memtier_benchmark, redis-benchmark, aibench, etc... )."
 authors = ["filipecosta90 <filipecosta.90@gmail.com>","Redis Performance Group <performance@redis.com>"]
 readme = "README.md"

redisbench_admin/run/args.py

@@ -25,6 +25,8 @@
 SETUP = os.getenv("SETUP", "")
 BENCHMARK_GLOB = os.getenv("BENCHMARK_GLOB", "*.yml")
 BENCHMARK_REGEX = os.getenv("BENCHMARK_REGEX", ".*")
+BENCHMARK_RUNNER_GROUP_TOTAL = int(os.getenv("BENCHMARK_RUNNER_GROUP_TOTAL", "1"))
+BENCHMARK_RUNNER_GROUP_M_ID = int(os.getenv("BENCHMARK_RUNNER_GROUP_M_ID", "1"))
 S3_BUCKET_NAME = os.getenv("S3_BUCKET_NAME", "ci.benchmarks.redislabs")
 PUSH_S3 = bool(os.getenv("PUSH_S3", False))
 REDIS_7 = bool(os.getenv("REDIS_7", True))
@@ -89,7 +91,18 @@ def common_run_args(parser):
         default=BENCHMARK_REGEX,
         help="specify a test regex pattern to use on the tests directory. by default uses '.*'. If --test is defined this options has no effect.",
     )
-
+    parser.add_argument(
+        "--runner-group-member-id",
+        type=str,
+        default=BENCHMARK_RUNNER_GROUP_M_ID,
+        help="Split test files evenly among a runner group. This is the id of the runner. Non-zero remainder of the division of tests will be attributed to the last member.",
+    )
+    parser.add_argument(
+        "--runner-group-total-members",
+        type=str,
+        default=BENCHMARK_RUNNER_GROUP_TOTAL,
+        help="Split test files evenly among a runner group. This is the total number of elements of the runner group",
+    )
     parser.add_argument(
         "--defaults_filename",
         type=str,

redisbench_admin/run_remote/terraform.py

@@ -33,7 +33,7 @@ def terraform_spin_or_reuse_env(
 ):
     (remote_setup, deployment_type, remote_id,) = fetch_remote_setup_from_config(
         benchmark_config["remote"],
-        "https://github.com/RedisLabsModules/testing-infrastructure.git",
+        "https://github.com/redis-performance/testing-infrastructure.git",
         "master",
         tf_folder_path,
     )

redisbench_admin/utils/benchmark_config.py

@@ -57,7 +57,12 @@ def prepare_benchmark_definitions(args):
     result = True
     defaults_filename = args.defaults_filename
     files = get_testfiles_to_process(
-        args.test_glob, args.test, defaults_filename, args.test_regex
+        args.test_glob,
+        args.test,
+        defaults_filename,
+        args.test_regex,
+        args.runner_group_member_id,
+        args.runner_group_total_members,
     )
 
     (
@@ -485,7 +490,14 @@ def min_ver_check(
     )
 
 
-def get_testfiles_to_process(test_glob, test_name, defaults_filename, test_regex=".*"):
+def get_testfiles_to_process(
+    test_glob,
+    test_name,
+    defaults_filename,
+    test_regex=".*",
+    runner_group_member_id=1,
+    runner_group_total_members=1,
+):
     if test_name == "":
         files = pathlib.Path().glob(test_glob)
         files = [str(x) for x in files]
@@ -507,6 +519,22 @@ def get_testfiles_to_process(test_glob, test_name, defaults_filename, test_regex
     else:
         files = test_name.split(",")
         logging.info("Running specific benchmark in file: {}".format(files))
+
+    if runner_group_total_members > 1:
+        tests_per_member = round(len(files) / runner_group_total_members)
+        member_test_start_pos = (runner_group_member_id - 1) * tests_per_member
+        member_test_end_pos = runner_group_member_id * tests_per_member
+        if runner_group_member_id == runner_group_total_members:
+            member_test_end_pos = len(files)
+        final_files = files[member_test_start_pos:member_test_end_pos]
+        logging.info(
+            "Detected a benchmark runner group. Splitting tests evenly. Non-zero remainder will be attributed to the last member. Member ID: {}. Total members: {}. Benchmarks per runner {}.".format(
+                runner_group_member_id, runner_group_total_members, tests_per_member
+            )
+        )
+        logging.info("Tests for this runner: {}".format(",".join(final_files)))
+        files = final_files
+
     return files
 
 

redisbench_admin/utils/remote.py

@@ -540,7 +540,7 @@ def common_tf(branch, path, repo, temporary_dir=None, destroy=False):
 
 def fetch_remote_setup_from_config(
     remote_setup_config,
-    repo="https://github.com/RedisLabsModules/testing-infrastructure.git",
+    repo="https://github.com/redis-performance/testing-infrastructure.git",
     branch="master",
     path=None,
 ):

tests/test_benchmark_config.py

@@ -237,10 +237,10 @@ def test_process_benchmark_definitions_remote_timeouts():
 def test_get_testfiles_to_process():
     test_glob_pattern_all = "./tests/test_data/benchmark_definitions/*.yml"
     test_glob_pattern_graph500 = "./tests/test_data/benchmark_definitions/graph500*.yml"
-    test_files_to_process = get_testfiles_to_process(
+    test_files_to_process_all = get_testfiles_to_process(
         test_glob_pattern_all, "", "defaults.yml"
     )
-    assert 6 == len(test_files_to_process)
+    assert 6 == len(test_files_to_process_all)
     test_files_to_process_graph500_glob = get_testfiles_to_process(
         test_glob_pattern_graph500, "", "defaults.yml"
     )
@@ -263,3 +263,56 @@ def test_get_testfiles_to_process():
     assert 3 == len(test_files_to_process)
     for test_graph in test_files_to_process_graph500_glob:
         assert test_graph not in test_files_to_process
+
+    test_files_to_process_group_member_1 = get_testfiles_to_process(
+        test_glob_pattern_all, "", "defaults.yml", ".*", 1, 2
+    )
+    test_files_to_process_group_member_2 = get_testfiles_to_process(
+        test_glob_pattern_all, "", "defaults.yml", ".*", 2, 2
+    )
+    assert 3 == len(test_files_to_process_group_member_1)
+    assert 3 == len(test_files_to_process_group_member_2)
+
+    test_files_to_process_graph500_glob_group_member_1 = get_testfiles_to_process(
+        test_glob_pattern_graph500, "", "defaults.yml", ".*", 1, 2
+    )
+    assert 2 == len(test_files_to_process_graph500_glob_group_member_1)
+    test_files_to_process_graph500_glob_group_member_2 = get_testfiles_to_process(
+        test_glob_pattern_graph500, "", "defaults.yml", ".*", 2, 2
+    )
+    assert 1 == len(test_files_to_process_graph500_glob_group_member_2)
+
+    test_files_to_process_graph500_glob_group_member_1 = get_testfiles_to_process(
+        test_glob_pattern_graph500, "", "defaults.yml", ".*", 1, 3
+    )
+    test_files_to_process_graph500_glob_group_member_2 = get_testfiles_to_process(
+        test_glob_pattern_graph500, "", "defaults.yml", ".*", 2, 3
+    )
+    test_files_to_process_graph500_glob_group_member_3 = get_testfiles_to_process(
+        test_glob_pattern_graph500, "", "defaults.yml", ".*", 3, 3
+    )
+    assert 1 == len(test_files_to_process_graph500_glob_group_member_1)
+    assert 1 == len(test_files_to_process_graph500_glob_group_member_2)
+    assert 1 == len(test_files_to_process_graph500_glob_group_member_3)
+
+    test_files_to_process_all_glob_group_member_1 = get_testfiles_to_process(
+        test_glob_pattern_all, "", "defaults.yml", ".*", 1, 5
+    )
+    assert 1 == len(test_files_to_process_all_glob_group_member_1)
+    assert (
+        test_files_to_process_all[0] == test_files_to_process_all_glob_group_member_1[0]
+    )
+    test_files_to_process_all_glob_group_member_4 = get_testfiles_to_process(
+        test_glob_pattern_all, "", "defaults.yml", ".*", 4, 5
+    )
+    assert 1 == len(test_files_to_process_all_glob_group_member_4)
+    assert (
+        test_files_to_process_all[3] == test_files_to_process_all_glob_group_member_4[0]
+    )
+    test_files_to_process_all_glob_group_member_5 = get_testfiles_to_process(
+        test_glob_pattern_all, "", "defaults.yml", ".*", 5, 5
+    )
+    assert 2 == len(test_files_to_process_all_glob_group_member_5)
+    assert (
+        test_files_to_process_all[4:6] == test_files_to_process_all_glob_group_member_5
+    )

tests/test_remote.py

@@ -34,7 +34,7 @@ def test_extract_git_vars():
         github_branch,
         github_branch_detached,
     ) = extract_git_vars(".")
-    assert github_org_name == "RedisLabsModules"
+    assert github_org_name == "redis-performance"
     assert github_repo_name == "redisbench-admin"
     assert github_sha != None and github_branch != ""
     if github_branch_detached is False:
@@ -51,9 +51,9 @@ def test_extract_git_vars_passing_repo():
         github_branch,
         github_branch_detached,
     ) = extract_git_vars(
-        ".", github_url="https://github.com/RedisLabsModules/redisbench-admin"
+        ".", github_url="https://github.com/redis-performance/redisbench-admin"
     )
-    assert github_org_name == "RedisLabsModules"
+    assert github_org_name == "redis-performance"
     assert github_repo_name == "redisbench-admin"
     assert github_sha != None and github_branch != ""
     if github_branch_detached is False:
@@ -70,9 +70,9 @@ def test_extract_git_vars_passing_repo2():
         github_branch,
         github_branch_detached,
     ) = extract_git_vars(
-        ".", github_url="https://github.com/RedisLabsModules/redisbench-admin/"
+        ".", github_url="https://github.com/redis-performance/redisbench-admin/"
     )
-    assert github_org_name == "RedisLabsModules"
+    assert github_org_name == "redis-performance"
     assert github_repo_name == "redisbench-admin"
     assert github_sha != None and github_branch != ""
     if github_branch_detached is False:
@@ -89,9 +89,9 @@ def test_extract_git_vars_passing_repo3():
         github_branch,
         github_branch_detached,
     ) = extract_git_vars(
-        ".", github_url="git@github.com:RedisLabsModules/redisbench-admin.git"
+        ".", github_url="git@github.com:redis-performance/redisbench-admin.git"
     )
-    assert github_org_name == "RedisLabsModules"
+    assert github_org_name == "redis-performance"
     assert github_repo_name == "redisbench-admin"
     assert github_sha != None and github_branch != ""
     if github_branch_detached is False: