Wait frame timeout with yolo8 or yolo11 detector

[mahxn0]

with cuda12.0，cudnn8.9，tensorrt8.6
./MultitargetTracker -e=6 -s=../data/settings_yolov11.ini

OpenCL not used
ParseTrackerSettings: ../data/settings_yolov11.ini ...
ParseTrackerSettings - readed
ParseTrackerSettings: 1
Video ../data/atrium.avi was started from 0 frame with 30 fps, frame size [640 x 360] and length 431
YoloTensorRTDetector::Init: tensor_rt::Detector
File does not exist : /home/mahxn0/workspace/pc_x64/Multitarget-tracker-master/model/yolo11/yolov11n.onnx-kHALF-batch1.engine
[04/06/2025-10:52:44] [I] [TRT] [MemUsageChange] Init CUDA: CPU +18, GPU +0, now: CPU 48, GPU 422 (MiB)
[04/06/2025-10:52:48] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +1453, GPU +268, now: CPU 1581, GPU 690 (MiB)
[04/06/2025-10:52:48] [I] Parsing ONNX file: /home/mahxn0/workspace/pc_x64/Multitarget-tracker-master/model/yolo11/yolov11n.onnx
[04/06/2025-10:52:48] [I] [TRT] ----------------------------------------------------------------
[04/06/2025-10:52:48] [I] [TRT] Input filename:   /home/mahxn0/workspace/pc_x64/Multitarget-tracker-master/model/yolo11/yolov11n.onnx
[04/06/2025-10:52:48] [I] [TRT] ONNX IR version:  0.0.10
[04/06/2025-10:52:48] [I] [TRT] Opset version:    12
[04/06/2025-10:52:48] [I] [TRT] Producer name:    pytorch
[04/06/2025-10:52:48] [I] [TRT] Producer version: 1.10
[04/06/2025-10:52:48] [I] [TRT] Domain:
[04/06/2025-10:52:48] [I] [TRT] Model version:    0
[04/06/2025-10:52:48] [I] [TRT] Doc string:
[04/06/2025-10:52:48] [I] [TRT] ----------------------------------------------------------------
[04/06/2025-10:52:48] [W] [TRT] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
m_params.videoMemory = 0, workspaceSize = 8196784128, dlaManagedSRAMSize = 0, dlaLocalDRAMSize = 1073741824, dlaGlobalDRAMSize = 536870912
[04/06/2025-10:52:48] [I] Building TensorRT engine: /home/mahxn0/workspace/pc_x64/Multitarget-tracker-master/model/yolo11/yolov11n.onnx-kHALF-batch1.engine
[04/06/2025-10:52:48] [I] [TRT] Graph optimization time: 0.215797 seconds.
[04/06/2025-10:52:48] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
--- Wait frame timeout!

[Nuzhny007]

Hello!
The conversation weights from onnx to TensorRT engine may be too long: several minutes on some computers. Wait it, plz, and after run program again.

[mahxn0]

Hello! The conversation weights from onnx to TensorRT engine may be too long: several minutes on some computers. Wait it, plz, and after run program again.

Hi, I've tried multiple times, but the file /home/mahxn0/workspace/pc_x64/Multitarget-tracker-master/model/yolo11/yolov11n.onnx-kHALF-batch1.engine has not been saved, and the same error occurs every time.

[Nuzhny007]

For fisrt start it's normally:

`OpenCL not used
ParseTrackerSettings: settings_yolov8.ini ...
ParseTrackerSettings - readed
ParseTrackerSettings: 1
Video C:\work\mtracking\Multitarget-tracker\data\for_it_pan.mp4 was started from 0 frame with 30 fps, frame size [1920 x 1080] and length 10522
YoloTensorRTDetector::Init: tensor_rt::Detector
File does not exist : C:\work\mtracking\Multitarget-tracker\data\coco\yolov8s.onnx-kHALF-batch1.engine
[04/07/2025-12:53:33] [I] [TRT] [MemUsageChange] Init CUDA: CPU +94, GPU +0, now: CPU 8053, GPU 1640 (MiB
[04/07/2025-12:53:40] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +3172, GPU +396, now: CPU 11558, GPU 2036 (MiB)
[04/07/2025-12:53:40] [I] Parsing ONNX file: C:\work\mtracking\Multitarget-tracker\data\coco\yolov8s.onnx
[04/07/2025-12:53:40] [I] [TRT] ----------------------------------------------------------------
[04/07/2025-12:53:40] [I] [TRT] Input filename: C:\work\mtracking\Multitarget-tracker\data\coco\yolov8s.onnx
[04/07/2025-12:53:40] [I] [TRT] ONNX IR version: 0.0.8
[04/07/2025-12:53:40] [I] [TRT] Opset version: 17
[04/07/2025-12:53:40] [I] [TRT] Producer name: pytorch
[04/07/2025-12:53:40] [I] [TRT] Producer version: 1.13.1
[04/07/2025-12:53:40] [I] [TRT] Domain:
[04/07/2025-12:53:40] [I] [TRT] Model version: 0
[04/07/2025-12:53:40] [I] [TRT] Doc string:
[04/07/2025-12:53:40] [I] [TRT] ----------------------------------------------------------------
m_params.videoMemory = 0, workspaceSize = 8589410304, dlaManagedSRAMSize = 0, dlaLocalDRAMSize = 1073741824, dlaGlobalDRAMSize = 536870912
[04/07/2025-12:53:40] [I] Building TensorRT engine: C:\work\mtracking\Multitarget-tracker\data\coco\yolov8s.onnx-kHALF-batch1.engine
[04/07/2025-12:53:40] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.

--- Wait frame timeout!

[04/07/2025-12:57:05] [I] [TRT] Detected 1 inputs and 3 output network tensors.
[04/07/2025-12:57:07] [I] [TRT] Total Host Persistent Memory: 377136 bytes
[04/07/2025-12:57:07] [I] [TRT] Total Device Persistent Memory: 614912 bytes
[04/07/2025-12:57:07] [I] [TRT] Max Scratch Memory: 0 bytes[04/07/2025-12:57:07] [I] [TRT] [BlockAssignment] Started assigning block shifts. This will take 120 steps to complete.
[04/07/2025-12:57:07] [I] [TRT] [BlockAssignment] Algorithm ShiftNTopDown took 4.2805ms to assign 9 blocks to 120 nodes requiring 30926336 bytes.
[04/07/2025-12:57:07] [I] [TRT] Total Activation Memory: 30924800 bytes
[04/07/2025-12:57:07] [I] [TRT] Total Weights Memory: 22435968 bytes
[04/07/2025-12:57:08] [I] [TRT] Engine generation completed in 207.798 seconds.
[04/07/2025-12:57:08] [I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 2 MiB, GPU 261 MiB[04/07/2025-12:57:08] [I] [TRT] [MemUsageStats] Peak memory usage during Engine building and serialization: CPU: 4057 MiB
[04/07/2025-12:57:08] [I] [TRT] Loaded engine size: 25 MiB
[04/07/2025-12:57:08] [I] TRT Engine file saved to: C:\work\mtracking\Multitarget-tracker\data\coco\yolov8s.onnx-kHALF
batch1.engine
4
** Bindings: 2 **
0: name: images, size: 1x3x640x640
1: name: output0, size: 1x84x8400
[04/07/2025-12:57:09] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +31, now: CPU 0, GPU 52 (MiB)
YoloTensorRTDetector::Init: Detector created - 1
--- algorithms time = 0, work time = 218.596
`

But the engine was created and the second start runs successfully. Logs say us then TensorRT creates engine about 4 minutes after timeout.

[Nuzhny007]

A long long time works function buildSerializedNetwork from TensorRT: https://github.com/Smorodov/Multitarget-tracker/blob/master/src/Detector/tensorrt_yolo/YoloONNX.cpp#L210

On my Jetson it works about 10 minutes.

[mahxn0]

[04/07/2025-21:08:37] [W] [TRT] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
m_params.videoMemory = 0, workspaceSize = 8196784128, dlaManagedSRAMSize = 0, dlaLocalDRAMSize = 1073741824, dlaGlobalDRAMSize = 536870912
[04/07/2025-21:08:37] [I] Building TensorRT engine: /home/mahxn0/workspace/pc_x64/Multitarget-tracker-master/model/yolo11/yolov11n.onnx-kHALF-batch1.engine
[04/07/2025-21:08:37] [I] [TRT] Graph optimization time: 0.215115 seconds.
[04/07/2025-21:08:37] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored.
--- Wait frame timeout!
[04/07/2025-21:12:32] [I] [TRT] Detected 1 inputs and 3 output network tensors.
[04/07/2025-21:12:33] [I] [TRT] Total Host Persistent Memory: 518512
[04/07/2025-21:12:33] [I] [TRT] Total Device Persistent Memory: 0
[04/07/2025-21:12:33] [I] [TRT] Total Scratch Memory: 640000
[04/07/2025-21:12:33] [I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 5 MiB, GPU 261 MiB
[04/07/2025-21:12:33] [I] [TRT] [BlockAssignment] Started assigning block shifts. This will take 190 steps to complete.
[04/07/2025-21:12:33] [I] [TRT] [BlockAssignment] Algorithm ShiftNTopDown took 10.3925ms to assign 10 blocks to 190 nodes requiring 10573824 bytes.
[04/07/2025-21:12:33] [I] [TRT] Total Activation Memory: 10572800
[04/07/2025-21:12:33] [W] [TRT] TensorRT encountered issues when converting weights between types and that could affect accuracy.
[04/07/2025-21:12:33] [W] [TRT] If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
[04/07/2025-21:12:33] [W] [TRT] Check verbose logs for the list of affected weights.
[04/07/2025-21:12:33] [W] [TRT] - 75 weights are affected by this issue: Detected subnormal FP16 values.
[04/07/2025-21:12:33] [W] [TRT] - 1 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.
[04/07/2025-21:12:33] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +5, GPU +6, now: CPU 5, GPU 6 (MiB)
[04/07/2025-21:12:33] [I] [TRT] Loaded engine size: 8 MiB
[04/07/2025-21:12:33] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +5, now: CPU 0, GPU 5 (MiB)
Segmentation Fault