From PyTorch to ONNX & TensorRT: Understanding Vision Inference Trade-offs

01 — The Problem

Why native PyTorch runtimes introduce unacceptable latency on multi-camera streams

Deploying native PyTorch models (`.pt`) directly in production pipelines carries significant Python interpreter overhead, dynamic memory allocations, and non-fused CUDA kernel invocations.

When scaling from 1 video feed to 4 concurrent RTSP streams on an edge workstation, frame turnaround degraded from 28ms to 85ms per frame, causing video buffer backpressure and frame dropping.

02 — The Optimization Strategy

Translating computation graphs into hardware-optimized kernels

We converted the model graph through ONNX into TensorRT FP16 engines. TensorRT inspects the target GPU architecture and fuses adjacent Convolution, BatchNorm, and SiLU activation layers into single unified CUDA kernels.

┌──────────────┐      ┌──────────────┐      ┌─────────────────────────┐
│ PyTorch .pt  │ ───► │  ONNX Graph  │ ───► │ TensorRT Engine (.engine)│
│ Dynamic DAG  │      │ Static Model │      │ Layer Fusion & FP16     │
└──────────────┘      └──────────────┘      └─────────────────────────┘
   85ms latency          42ms latency               18ms latency

03 — Export & Execution Pipeline

Generating hardware-specific execution plans

The export pipeline explicitly defines fixed tensor shapes to eliminate dynamic buffer allocations during live video inference loops.

from ultralytics import YOLO

# Load production trained weights
model = YOLO('retail_best.pt')

# Export to TensorRT engine with half-precision (FP16)
model.export(
    format='engine',
    imgsz=640,
    half=True,          # FP16 precision
    device=0,           # Target CUDA GPU ID
    workspace=4         # 4GB dedicated scratchpad workspace
)

Why this mattersEnabling half-precision (FP16) cut GPU VRAM consumption by 50% and doubled tensor core throughput with a negligible mAP loss of under 0.003.