Skip to content

DeepStream tracker library causes OOM error when compiled with -DCMAKE_BUILD_TYPE=Release option #253

@Klomi

Description

@Klomi

Description
Build libByteTracker.so as per the instruction given in (README.md), but replace cmake .. with cmake -DCMAKE_BUILD_TYPE=Release to enable optimizations. Running a GStreamer pipeline with the built library file takes up all system memory immediately and crashes with OOM error.

To Reproduce

  1. Launch Docker container, set the memory limit to avoid screwing up the host system.
docker run --rm -it --gpus=all --memory=4G nvcr.io/nvidia/deepstream:6.1-devel
  1. Install dependencies.
apt update && apt install -y libeigen3-dev
  1. Clone repo and build libByteTracker.so.
cd /root
git clone --depth 1 https://github.com/ifzhang/ByteTrack/
cd ByteTrack/deploy/DeepStream
mkdir build && cd build
# cmake -> /usr/local/bin/cmake version is 3.15.2
# /usr/bin/cmake version is 3.16.3
/usr/bin/cmake -DCMAKE_BUILD_TYPE=Release ..
make
  1. Run GStreamer pipeline.
gst-launch-1.0 -v \
    filesrc location=/opt/nvidia/deepstream/deepstream-6.1/samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder \
    ! m.sink_0 nvstreammux name=m batch-size=1 width=1920 height=1080 \
    ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/config_infer_primary.txt \
    ! nvtracker \
        ll-lib-file=/root/ByteTrack/deploy/DeepStream/lib/libByteTracker.so \
        tracker-width=640 \
        tracker-height=384 \
    ! nvvideoconvert ! nvdsosd ! fpsdisplaysink sync=0 video-sink=fakesink text-overlay=0
  1. Observe memory usage. After a while, the program is killed and exit code is 137.

Additional Information
The issue is probably caused by missing return value in NvMOTContext::processFrame, which is an undefined behaviour that could lead to bad compiler optimizations.
https://github.com/ifzhang/ByteTrack/blob/72ca8b45d36caf5a39e949c6aa815d9abffd1ab5/deploy/DeepStream/src/NvMOTContext.cpp#L9-L56
Simply append return NvMOTStatus_OK; at the end of the function, rebuild the library, and the issue is fixed (tested on my local machine).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions