Skip to content

Commit

Permalink
Added more information regarding the failing hybrid case
Browse files Browse the repository at this point in the history
  • Loading branch information
@JarnoRalli
JarnoRalli committed Feb 23, 2025
1 parent 6968617 commit 12535d2
Show file tree
Hide file tree
Showing 8 changed files with 193 additions and 8 deletions.
44 changes: 36 additions & 8 deletions deepstream-examples/deepstream-hybrid/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,24 +12,31 @@ First you need to start the Docker container, this can be done with the followin
docker compose up --build
```

, you only need to use the `--build` option when you bring up the service for the first time. After
that you can verify the id of the container using
, you only need to use the `--build` option when you bring up the service for the first time. Once
the Docker container is up and running, you can check the ID of the container using:

```bash
docker ps
```

, and then you can execute `bash` in the container
, and then in order to join the running container, execute the following:

```bash
xhost +
docker exec -it <ID> bash
```

Once inside the docker, you need to change the directory to `/home/gstreamer_examples/`

```bash
cd /home/gstreamer_examples
ls -la
```

### 1.1.1 Single Pipeline

Following example executes a single image processing pipeline so that the detector (primary mode) runs locally in Triton and the classifier (secondary mode) that classifies
the car model runs without Triton.
the car mmake/brand runs without Triton. You can launch the example by executing the following inside the `/home/gstreamer_examples/` directory:

```bash
gst-launch-1.0 \
Expand All @@ -43,14 +50,19 @@ nvdsosd display-clock=1 ! nvvideoconvert ! nveglglessink

### 1.1.2 Pipeline with Two Branches

In this example we test a pipeline where process two different streams in the same detector (primary mode) running locally in Triton

In this pipeline we process two different streams in the same detector (primary mode) running locally in Triton. After the tracker we split the streams using `nvstreamdemux` and
add a classifier (secondary mode) that classifies the car model. If the classifier is in the `src_0` branch then everything works as expected. However, if the classifier is in
the `src_1` branch then this doesn't work.
add a classifier (secondary mode) that classifies the car make/brand. The expectation is that the car model is displayed in the output in the branch that has the car make/brand classifier.
If the classifier is in the `src_0` branch then everything works as expected. However, if the classifier is in the `src_1` branch then this doesn't work. Figure 1. shows the pipeline.

<figure align="center">
<img src="./figures/hybrid_pipeline.png" width="900">
<figcaption>Figure 1. Hybrid pipeline with two branches.</figcaption>
</figure>

**This works in Deepstream 6.3**

In this case the classifier is in the `src_0` output of the `nvstreamdemuxer` and we can see the car make in the sink. You can launch the example by executing the following inside the `/home/gstreamer_examples/` directory:

```bash
gst-launch-1.0 -e \
nvstreammux name=mux width=1280 height=720 batch-size=2 ! \
Expand All @@ -63,8 +75,17 @@ demux.src_0 ! queue ! nvinfer config-file-path=dstest2_sgie2_config.txt ! nvdsos
demux.src_1 ! queue ! nvdsosd ! nvvideoconvert ! nveglglessink
```

Figure 2. shows the pipeline that works.

<figure align="center">
<img src="./figures/pipeline_working.png" width="900">
<figcaption>Figure 2. Pipeline that works.</figcaption>
</figure>

**This doesn't work in Deepstream 6.3**

In this case the classifier is in the `src_1` output of the `nvstreamdemuxer` and we don't' see the car make in the sink. You can launch the example by executing the following inside the `/home/gstreamer_examples/` directory:

```bash
gst-launch-1.0 -e \
nvstreammux name=mux width=1280 height=720 batch-size=2 ! \
Expand All @@ -77,3 +98,10 @@ demux.src_0 ! queue ! nvdsosd ! nvvideoconvert ! nveglglessink \
demux.src_1 ! queue ! nvinfer config-file-path=dstest2_sgie2_config.txt ! nvdsosd ! nvvideoconvert ! nveglglessink
```

Figure 3. shows the pipeline that does not work.

<figure align="center">
<img src="./figures/pipeline_not_working.png" width="900">
<figcaption>Figure 3. Pipeline that doesn not work.</figcaption>
</figure>

157 changes: 157 additions & 0 deletions deepstream-examples/deepstream-hybrid/figures/hybrid_pipeline.drawio
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
<mxfile host="app.diagrams.net" agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:135.0) Gecko/20100101 Firefox/135.0" version="26.0.14">
<diagram name="Page-1" id="-Sh_f1pxR6C-SZwm3D4O">
<mxGraphModel dx="1295" dy="837" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1700" pageHeight="1100" math="0" shadow="0">
<root>
<mxCell id="0" />
<mxCell id="1" parent="0" />
<mxCell id="bV7FN9ZMzJkyV8n1XKku-1" value="nvurisrcbin" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" vertex="1">
<mxGeometry x="40" y="99" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="bV7FN9ZMzJkyV8n1XKku-2" value="queue" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" vertex="1">
<mxGeometry x="190" y="99" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="bV7FN9ZMzJkyV8n1XKku-3" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" source="bV7FN9ZMzJkyV8n1XKku-1" target="bV7FN9ZMzJkyV8n1XKku-2" edge="1">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="400" y="519" as="sourcePoint" />
<mxPoint x="450" y="469" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="bV7FN9ZMzJkyV8n1XKku-13" value="nvurisrcbin" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" vertex="1">
<mxGeometry x="40" y="240" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="bV7FN9ZMzJkyV8n1XKku-14" value="queue" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" vertex="1">
<mxGeometry x="190" y="240" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="bV7FN9ZMzJkyV8n1XKku-15" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" source="bV7FN9ZMzJkyV8n1XKku-13" target="bV7FN9ZMzJkyV8n1XKku-14" edge="1">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="400" y="660" as="sourcePoint" />
<mxPoint x="450" y="610" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="bV7FN9ZMzJkyV8n1XKku-16" value="nvstreammux" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" vertex="1">
<mxGeometry x="380" y="168" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="bV7FN9ZMzJkyV8n1XKku-18" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" source="bV7FN9ZMzJkyV8n1XKku-2" target="bV7FN9ZMzJkyV8n1XKku-16" edge="1">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="400" y="460" as="sourcePoint" />
<mxPoint x="450" y="410" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="bV7FN9ZMzJkyV8n1XKku-21" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;fillColor=#d5e8d4;strokeColor=#82b366;entryX=0;entryY=1;entryDx=0;entryDy=0;" parent="1" source="bV7FN9ZMzJkyV8n1XKku-14" edge="1" target="bV7FN9ZMzJkyV8n1XKku-16">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="400" y="460" as="sourcePoint" />
<mxPoint x="390" y="270" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="bV7FN9ZMzJkyV8n1XKku-22" value="nvinferserver" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" vertex="1">
<mxGeometry x="520" y="168" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="bV7FN9ZMzJkyV8n1XKku-23" value="nvtracker" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" vertex="1">
<mxGeometry x="660" y="168" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="bV7FN9ZMzJkyV8n1XKku-24" value="nvstreamdemux" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" vertex="1">
<mxGeometry x="800" y="168" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="bV7FN9ZMzJkyV8n1XKku-35" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" source="bV7FN9ZMzJkyV8n1XKku-16" target="bV7FN9ZMzJkyV8n1XKku-22" edge="1">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="520" y="468" as="sourcePoint" />
<mxPoint x="570" y="418" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="bV7FN9ZMzJkyV8n1XKku-36" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" source="bV7FN9ZMzJkyV8n1XKku-22" target="bV7FN9ZMzJkyV8n1XKku-23" edge="1">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="520" y="468" as="sourcePoint" />
<mxPoint x="570" y="418" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="bV7FN9ZMzJkyV8n1XKku-37" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;" parent="1" source="bV7FN9ZMzJkyV8n1XKku-23" target="bV7FN9ZMzJkyV8n1XKku-24" edge="1">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="520" y="468" as="sourcePoint" />
<mxPoint x="570" y="418" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-1" value="queue" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" vertex="1" parent="1">
<mxGeometry x="970" y="99" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-2" value="nvdsosd" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" vertex="1" parent="1">
<mxGeometry x="1109" y="99" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-3" value="nvvideoconvert" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" vertex="1" parent="1">
<mxGeometry x="1250" y="99" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-4" value="nveglglessink" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" vertex="1" parent="1">
<mxGeometry x="1390" y="99" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-5" value="queue" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" vertex="1" parent="1">
<mxGeometry x="971" y="238" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-6" value="nvdsosd" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" vertex="1" parent="1">
<mxGeometry x="1251" y="238" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-7" value="nvvideoconvert" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" vertex="1" parent="1">
<mxGeometry x="1392" y="238" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-8" value="nveglglessink" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" vertex="1" parent="1">
<mxGeometry x="1531" y="238" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-9" value="nvinfer" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;" vertex="1" parent="1">
<mxGeometry x="1111" y="238" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-10" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;strokeColor=#82B366;" edge="1" parent="1" source="bV7FN9ZMzJkyV8n1XKku-24" target="nKkdr1dbNrpamliiyeTE-1">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="970" y="460" as="sourcePoint" />
<mxPoint x="1020" y="410" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-11" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;strokeColor=#82B366;" edge="1" parent="1" source="nKkdr1dbNrpamliiyeTE-1" target="nKkdr1dbNrpamliiyeTE-2">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="980" y="460" as="sourcePoint" />
<mxPoint x="1030" y="410" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-12" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;strokeColor=#82B366;" edge="1" parent="1" source="nKkdr1dbNrpamliiyeTE-2" target="nKkdr1dbNrpamliiyeTE-3">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="1239" y="170" as="sourcePoint" />
<mxPoint x="1019" y="410" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-13" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;strokeColor=#82B366;" edge="1" parent="1" source="nKkdr1dbNrpamliiyeTE-3" target="nKkdr1dbNrpamliiyeTE-4">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="960" y="460" as="sourcePoint" />
<mxPoint x="1010" y="410" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-14" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;strokeColor=#82B366;" edge="1" parent="1" source="nKkdr1dbNrpamliiyeTE-5" target="nKkdr1dbNrpamliiyeTE-9">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="980" y="460" as="sourcePoint" />
<mxPoint x="1030" y="410" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-15" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;strokeColor=#82B366;" edge="1" parent="1" source="nKkdr1dbNrpamliiyeTE-9" target="nKkdr1dbNrpamliiyeTE-6">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="971" y="460" as="sourcePoint" />
<mxPoint x="1021" y="410" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-16" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;strokeColor=#82B366;" edge="1" parent="1" source="nKkdr1dbNrpamliiyeTE-6" target="nKkdr1dbNrpamliiyeTE-7">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="958" y="460" as="sourcePoint" />
<mxPoint x="1008" y="410" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-17" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;strokeColor=#82B366;" edge="1" parent="1" source="nKkdr1dbNrpamliiyeTE-7" target="nKkdr1dbNrpamliiyeTE-8">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="961" y="460" as="sourcePoint" />
<mxPoint x="1011" y="410" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="nKkdr1dbNrpamliiyeTE-18" value="" style="endArrow=classic;html=1;rounded=0;exitX=1;exitY=1;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;strokeColor=#82B366;" edge="1" parent="1" source="bV7FN9ZMzJkyV8n1XKku-24" target="nKkdr1dbNrpamliiyeTE-5">
<mxGeometry width="50" height="50" relative="1" as="geometry">
<mxPoint x="970" y="460" as="sourcePoint" />
<mxPoint x="1020" y="410" as="targetPoint" />
</mxGeometry>
</mxCell>
</root>
</mxGraphModel>
</diagram>
</mxfile>
Binary file added deepstream-examples/deepstream-hybrid/figures/hybrid_pipeline.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added deepstream-examples/deepstream-hybrid/figures/output_not_working.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added deepstream-examples/deepstream-hybrid/figures/output_working.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added deepstream-examples/deepstream-hybrid/figures/pipeline_not_working.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added deepstream-examples/deepstream-hybrid/figures/pipeline_working.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed deepstream-examples/deepstream-hybrid/pipeline.png
View file
Binary file not shown.

0 comments on commit 12535d2

Please sign in to comment.