Skip to content

Commit 2125e9c

Browse files
arjunsuresharjunsuresh
authored
Merge pull request #82 from alxdsptr/mlperf-inference-results-scc24
Mlperf inference results scc24 pku 2
2 parents 8149ec6 + 4ff08c5 commit 2125e9c

35 files changed

+40715
-0
lines changed

‎open/pku/code/stable-diffusion-xl/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
TBD

‎open/pku/measurements/mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main/README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
| Model | Scenario | Accuracy | Throughput | Latency (in ms) |
2+
|---------------------|------------|----------------------|--------------|-------------------|
3+
| stable-diffusion-xl | offline | (14.02827, 84.33062) | 8.281 | - |

‎open/pku/measurements/mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main/stable-diffusion-xl/offline/README.md

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
This experiment is generated using the [MLCommons Collective Mind automation framework (CM)](https://github.com/mlcommons/cm4mlops).
2+
3+
*Check [CM MLPerf docs](https://docs.mlcommons.org/inference) for more details.*
4+
5+
## Host platform
6+
7+
* OS version: Linux-5.14.0-427.33.1.el9_4.x86_64-x86_64-with-glibc2.29
8+
* CPU version: x86_64
9+
* Python version: 3.8.10 (default, Sep 11 2024, 16:02:53)
10+
[GCC 9.4.0]
11+
* MLCommons CM version: 3.4.1
12+
13+
## CM Run Command
14+
15+
See [CM installation guide](https://docs.mlcommons.org/inference/install/).
16+
17+
```bash
18+
pip install -U cmind
19+
20+
cm rm cache -f
21+
22+
cm pull repo mlcommons@cm4mlops --checkout=852b297c18a90edb8a9c975dd7ee7cf731e1e347
23+
24+
cm run script \
25+
--tags=run-mlperf,inference,_r4.1-dev,_scc24-main \
26+
--model=sdxl \
27+
--implementation=nvidia \
28+
--max_query_count=5000 \
29+
--min_query_count=504 \
30+
--framework=tensorrt \
31+
--category=datacenter \
32+
--scenario=Offline \
33+
--execution_mode=test \
34+
--device=cuda \
35+
--max_batchsize=8 \
36+
--quiet \
37+
--rerun
38+
```
39+
*Note that if you want to use the [latest automation recipes](https://docs.mlcommons.org/inference) for MLPerf (CM scripts),
40+
you should simply reload mlcommons@cm4mlops without checkout and clean CM cache as follows:*
41+
42+
```bash
43+
cm rm repo mlcommons@cm4mlops
44+
cm pull repo mlcommons@cm4mlops
45+
cm rm cache -f
46+
47+
```
48+
49+
## Results
50+
51+
Platform: mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main
52+
53+
Model Precision: int8
54+
55+
### Accuracy Results
56+
`CLIP_SCORE`: `14.02827`, Required accuracy for closed division `>= 31.68632` and `<= 31.81332`
57+
`FID_SCORE`: `84.33062`, Required accuracy for closed division `>= 23.01086` and `<= 23.95008`
58+
59+
### Performance Results
60+
`Samples per second`: `8.2807`

‎open/pku/measurements/mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main/stable-diffusion-xl/offline/accuracy_console.out

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
[2024-11-18 22:46:22,137 systems.py:197 INFO] Found unknown device in GPU connection topology: NIC0. Skipping.
2+
[2024-11-18 22:46:22,217 main.py:229 INFO] Detected system ID: KnownSystem.sc1
3+
[2024-11-18 22:46:24,991 generate_conf_files.py:107 INFO] Generated measurements/ entries for sc1_TRT/stable-diffusion-xl/Offline
4+
[2024-11-18 22:46:24,991 __init__.py:46 INFO] Running command: python3 -m code.stable-diffusion-xl.tensorrt.harness --logfile_outdir="/home/lry/CM/repos/local/cache/6c0ba4746fa74e77/test_results/mlperf_inference_lry_40-nvidia_original-gpu-tensorrt-vdefault-scc24-main/stable-diffusion-xl/offline/accuracy" --logfile_prefix="mlperf_log_" --performance_sample_count=5000 --test_mode="AccuracyOnly" --gpu_batch_size=8 --mlperf_conf_path="/home/lry/CM/repos/local/cache/3e2d12440d5a4a93/inference/mlperf.conf" --tensor_path="build/preprocessed_data/coco2014-tokenized-sdxl/5k_dataset_final/" --use_graphs=true --user_conf_path="/home/lry/CM/repos/mlcommons@cm4mlops/script/generate-mlperf-inference-user-conf/tmp/0fe769204cb64955852be59f43b33ad5.conf" --gpu_inference_streams=1 --gpu_copy_streams=1 --gpu_engines="./build/engines/sc1/stable-diffusion-xl/Offline/stable-diffusion-xl-CLIP-Offline-gpu-b8-fp16.custom_k_99_MaxP.plan,./build/engines/sc1/stable-diffusion-xl/Offline/stable-diffusion-xl-CLIPWithProj-Offline-gpu-b8-fp16.custom_k_99_MaxP.plan,./build/engines/sc1/stable-diffusion-xl/Offline/stable-diffusion-xl-UNetXL-Offline-gpu-b8-int8.custom_k_99_MaxP.plan,./build/engines/sc1/stable-diffusion-xl/Offline/stable-diffusion-xl-VAE-Offline-gpu-b8-fp32.custom_k_99_MaxP.plan" --scenario Offline --model stable-diffusion-xl
5+
[2024-11-18 22:46:24,991 __init__.py:53 INFO] Overriding Environment
6+
[2024-11-18 22:46:27,765 systems.py:197 INFO] Found unknown device in GPU connection topology: NIC0. Skipping.
7+
2024-11-18 22:46:30,481 INFO worker.py:1567 -- Connecting to existing Ray cluster at address: 10.0.0.1:6379...
8+
2024-11-18 22:46:30,489 INFO worker.py:1743 -- Connected to Ray cluster. View the dashboard at http://127.0.0.1:8265 
9+
[2024-11-18 22:46:30,733 harness.py:207 INFO] Start Warm Up!
10+
(SDXLCore pid=220850) [2024-11-18 22:46:34,300 backend.py:428 INFO] initialized
11+
(SDXLCore pid=220850) [2024-11-18 22:46:34,402 backend.py:72 INFO] Loading TensorRT engine: ./build/engines/sc1/stable-diffusion-xl/Offline/stable-diffusion-xl-CLIP-Offline-gpu-b8-fp16.custom_k_99_MaxP.plan.
12+
(SDXLCore pid=220850) [2024-11-18 22:46:34,654 backend.py:72 INFO] Loading TensorRT engine: ./build/engines/sc1/stable-diffusion-xl/Offline/stable-diffusion-xl-CLIPWithProj-Offline-gpu-b8-fp16.custom_k_99_MaxP.plan.
13+
(SDXLCore pid=220850) [2024-11-18 22:46:35,018 backend.py:72 INFO] Loading TensorRT engine: ./build/engines/sc1/stable-diffusion-xl/Offline/stable-diffusion-xl-UNetXL-Offline-gpu-b8-int8.custom_k_99_MaxP.plan.
14+
(SDXLCore pid=220850) [2024-11-18 22:46:39,168 backend.py:97 INFO] Enabling cuda graphs for unet
15+
(SDXLCore pid=220850) [2024-11-18 22:46:39,604 backend.py:155 INFO] captured graph for BS=1
16+
(SDXLCore pid=18778, ip=10.0.0.3) [2024-11-18 22:46:32,459 backend.py:428 INFO] initialized [repeated 7x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)
17+
(SDXLCore pid=220848) [2024-11-18 22:46:37,641 backend.py:72 INFO] Loading TensorRT engine: ./build/engines/sc1/stable-diffusion-xl/Offline/stable-diffusion-xl-VAE-Offline-gpu-b8-fp32.custom_k_99_MaxP.plan. [repeated 29x across cluster]
18+
(SDXLCore pid=220850) [2024-11-18 22:46:40,416 backend.py:155 INFO] captured graph for BS=2
19+
(SDXLCore pid=18776, ip=10.0.0.3) [2024-11-18 22:46:40,449 backend.py:97 INFO] Enabling cuda graphs for unet [repeated 8x across cluster]
20+
(SDXLCore pid=220852) [2024-11-18 22:46:44,655 backend.py:155 INFO] captured graph for BS=6 [repeated 45x across cluster]
21+
(SDXLCore pid=18776, ip=10.0.0.3) [2024-11-18 22:46:36,615 backend.py:428 INFO] initialized
22+
(SDXLCore pid=18776, ip=10.0.0.3) [2024-11-18 22:46:38,113 backend.py:72 INFO] Loading TensorRT engine: ./build/engines/sc1/stable-diffusion-xl/Offline/stable-diffusion-xl-VAE-Offline-gpu-b8-fp32.custom_k_99_MaxP.plan. [repeated 4x across cluster]
23+
[2024-11-18 22:47:06,644 harness.py:209 INFO] Warm Up Done!
24+
[2024-11-18 22:47:06,644 harness.py:211 INFO] Start Test!
25+
[2024-11-18 22:47:06,794 backend.py:852 INFO] 500
26+
(SDXLCore pid=18774, ip=10.0.0.3) [2024-11-18 22:47:03,586 backend.py:630 INFO] generate_images
27+
(SDXLCore pid=18776, ip=10.0.0.3) [2024-11-18 22:46:45,509 backend.py:155 INFO] captured graph for BS=8 [repeated 25x across cluster]
28+
(SDXLCore pid=18776, ip=10.0.0.3) [2024-11-18 22:47:11,266 backend.py:630 INFO] generate_images [repeated 9x across cluster]
29+
(SDXLCore pid=18776, ip=10.0.0.3) [2024-11-18 22:47:18,975 backend.py:630 INFO] generate_images [repeated 9x across cluster]
30+
(SDXLCore pid=18776, ip=10.0.0.3) [2024-11-18 22:47:26,675 backend.py:630 INFO] generate_images [repeated 9x across cluster]
31+
(SDXLCore pid=220848) [2024-11-18 22:47:35,394 backend.py:630 INFO] generate_images [repeated 4x across cluster]
32+
(SDXLCore pid=220848) [2024-11-18 22:47:45,024 backend.py:630 INFO] generate_images [repeated 9x across cluster]
33+
(SDXLCore pid=18776, ip=10.0.0.3) [2024-11-18 22:47:49,957 backend.py:630 INFO] generate_images [repeated 9x across cluster]
34+
(SDXLCore pid=220848) [2024-11-18 22:48:04,367 backend.py:630 INFO] generate_images [repeated 9x across cluster]
35+
[2024-11-18 22:48:16,996 backend.py:901 INFO] [Server] Received 500 total samples
36+
[2024-11-18 22:48:16,999 backend.py:911 INFO] [Device 0] Reported 56 samples
37+
[2024-11-18 22:48:17,001 backend.py:911 INFO] [Device 1] Reported 56 samples
38+
[2024-11-18 22:48:17,002 backend.py:911 INFO] [Device 2] Reported 56 samples
39+
[2024-11-18 22:48:17,004 backend.py:911 INFO] [Device 3] Reported 56 samples
40+
[2024-11-18 22:48:17,006 backend.py:911 INFO] [Device 4] Reported 56 samples
41+
[2024-11-18 22:48:17,008 backend.py:911 INFO] [Device 5] Reported 55 samples
42+
[2024-11-18 22:48:17,009 backend.py:911 INFO] [Device 6] Reported 55 samples
43+
[2024-11-18 22:48:17,011 backend.py:911 INFO] [Device 7] Reported 55 samples
44+
[2024-11-18 22:48:17,013 backend.py:911 INFO] [Device 8] Reported 55 samples
45+
[2024-11-18 22:48:17,013 harness.py:214 INFO] Test Done!
46+
[2024-11-18 22:48:17,013 harness.py:216 INFO] Destroying SUT...
47+
[2024-11-18 22:48:17,013 harness.py:219 INFO] Destroying QSL...
48+
(SDXLCore pid=220847) [2024-11-18 22:48:06,100 backend.py:630 INFO] generate_images [repeated 4x across cluster]
49+
benchmark : Benchmark.SDXL
50+
buffer_manager_thread_count : 0
51+
data_dir : /home/lry/CM/repos/local/cache/d2b9079c1073417b/data
52+
gpu_batch_size : 8
53+
gpu_copy_streams : 1
54+
gpu_inference_streams : 1
55+
input_dtype : int32
56+
input_format : linear
57+
log_dir : /home/lry/CM/repos/local/cache/3443882dd9374096/repo/closed/NVIDIA/build/logs/2024.11.18-22.46.18
58+
mlperf_conf_path : /home/lry/CM/repos/local/cache/3e2d12440d5a4a93/inference/mlperf.conf
59+
model_path : /home/lry/CM/repos/local/cache/d2b9079c1073417b/models/SDXL/
60+
offline_expected_qps : 0.0
61+
precision : int8
62+
preprocessed_data_dir : /home/lry/CM/repos/local/cache/d2b9079c1073417b/preprocessed_data
63+
scenario : Scenario.Offline
64+
system : SystemConfiguration(host_cpu_conf=CPUConfiguration(layout={CPU(name='AMD EPYC 9684X 96-Core Processor', architecture=<CPUArchitecture.x86_64: AliasedName(name='x86_64', aliases=(), patterns=())>, core_count=96, threads_per_core=2): 2}), host_mem_conf=MemoryConfiguration(host_memory_capacity=Memory(quantity=791.59486, byte_suffix=<ByteSuffix.GB: (1000, 3)>, _num_bytes=791594860000), comparison_tolerance=0.05), accelerator_conf=AcceleratorConfiguration(layout=defaultdict(<class 'int'>, {GPU(name='NVIDIA H100 80GB HBM3', accelerator_type=<AcceleratorType.Discrete: AliasedName(name='Discrete', aliases=(), patterns=())>, vram=Memory(quantity=79.6474609375, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=85520809984), max_power_limit=700.0, pci_id='0x233010DE', compute_sm=90): 5})), numa_conf=NUMAConfiguration(numa_nodes={}, num_numa_nodes=2), system_id='sc1')
65+
tensor_path : build/preprocessed_data/coco2014-tokenized-sdxl/5k_dataset_final/
66+
test_mode : AccuracyOnly
67+
use_graphs : True
68+
user_conf_path : /home/lry/CM/repos/mlcommons@cm4mlops/script/generate-mlperf-inference-user-conf/tmp/0fe769204cb64955852be59f43b33ad5.conf
69+
system_id : sc1
70+
config_name : sc1_stable-diffusion-xl_Offline
71+
workload_setting : WorkloadSetting(HarnessType.Custom, AccuracyTarget.k_99, PowerSetting.MaxP)
72+
optimization_level : plugin-enabled
73+
num_profiles : 1
74+
config_ver : custom_k_99_MaxP
75+
accuracy_level : 99%
76+
inference_server : custom
77+
skip_file_checks : False
78+
power_limit : None
79+
cpu_freq : None
80+
(SDXLCore pid=220850) [I] Loading bytes from ./build/engines/sc1/stable-diffusion-xl/Offline/stable-diffusion-xl-CLIP-Offline-gpu-b8-fp16.custom_k_99_MaxP.plan
81+
(SDXLCore pid=220850) [I] Loading bytes from ./build/engines/sc1/stable-diffusion-xl/Offline/stable-diffusion-xl-CLIPWithProj-Offline-gpu-b8-fp16.custom_k_99_MaxP.plan
82+
(SDXLCore pid=220850) [I] Loading bytes from ./build/engines/sc1/stable-diffusion-xl/Offline/stable-diffusion-xl-UNetXL-Offline-gpu-b8-int8.custom_k_99_MaxP.plan
83+
(SDXLCore pid=18776, ip=10.0.0.3) [I] Loading bytes from ./build/engines/sc1/stable-diffusion-xl/Offline/stable-diffusion-xl-CLIP-Offline-gpu-b8-fp16.custom_k_99_MaxP.plan [repeated 30x across cluster]
84+
(SDXLCore pid=18776, ip=10.0.0.3) [I] Loading bytes from ./build/engines/sc1/stable-diffusion-xl/Offline/stable-diffusion-xl-VAE-Offline-gpu-b8-fp32.custom_k_99_MaxP.plan [repeated 3x across cluster]
85+
[2024-11-18 22:48:18,480 run_harness.py:166 INFO] Result: Accuracy run detected.
86+
87+
======================== Result summaries: ========================
88+

0 commit comments

Comments
 (0)