apollo perception doesn't work

[wangzhensuo]

hello,my friend.
many thanks for your carla_apollo_bridge!
I created the carla_apollo_bridge environment as readme.md, all is running perfectly.
then, I want to use the Apollo's perception, and changed the code as below step,

To use Apollo perception, remove obstacle_sensor from config/settings.yaml and enable the 'Transform' and 'Perception' modules.

restart the dreamview , reload the map , restart run_bridge.py and manual_control.py , and spawn some cars.

It seems that apollo perception doesn't work（couldnot see other green object like box,but I can see the cars in pygame window and cars pointcloud in the browser）.
It is there any other settings that I did not done?

[AngelJA]

Hi there. If everything else is working this points to a possible hardware compatibility issue. What gpu are you using? Apollo perception isn't compatible with different gpu architectures such as that on the GTX 2080.

[wangzhensuo]

thank you for your reply. today I do the same thing in another PC. it worked OK.

[wangzhensuo]

I am sorry to reopen this issue as something still unknown.
I built the carla_apollo_bridge environment in 3 PCs,the result is shown as below:

PC1: in fact ,it is a AWS server，apollo's perception runs OK.
PC2: a local computer. apollo's perception runs failed,crashed.
PC3: another local computer. apollo's perception runs failed,crashed.

when I click the "perception" button on the webpage, the terminal shows error
"[/apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/common/data/global_flagfile.txt] E1109 13:26:55.701213 19258 hmi_worker.cc:128] [dreamview] FAILED(-1): nohup mainboard -d /apollo/modules/perception/production/dag/dag_streaming_perception.dag &"

I run cmd(gdb --args mainboard -d /apollo/modules/perception/production/dag/dag_streaming_perception.dag) to start perception,and got the below core dump.

(gdb) bt
#0 0x00007fff154d0fc9 in apollo::perception::inference::RTNet::Init(std::map<std::string, std::vector<int, std::allocator >, std::lessstd::string, std::allocator<std::pair<std::string const, std::vector<int, std::allocator > > > > const&) ()
from /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_k8/libmodules_Sperception_Sinference_Stensorrt_Slibrt_Unet.so
#1 0x00007fff27df6ceb in apollo::perception::lidar::CNNSegmentation::Init(apollo::perception::lidar::SegmentationInitOptions const&) ()
from /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_k8/libmodules_Sperception_Slidar_Slib_Ssegmentation_Scnnseg_Slibcnn_Usegmentation.so
#2 0x00007fff2fdfbe0a in apollo::perception::lidar::LidarObstacleSegmentation::Init(apollo::perception::lidar::LidarObstacleSegmentationInitOptions const&) ()
from /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_k8/libmodules_Sperception_Slidar_Sapp_Sliblidar_Uobstacle_Usegmentation.so
#3 0x00007fff74587b44 in apollo::perception::onboard::SegmentationComponent::InitAlgorithmPlugin() ()
from /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_k8/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ulidar.so
#4 0x00007fff7459e93b in apollo::perception::onboard::SegmentationComponent::Init() ()
from /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_k8/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ulidar.so
#5 0x00007fff745a4094 in apollo::cyber::Component<apollo::drivers::PointCloud, apollo::cyber::NullType, apollo::cyber::NullType, apollo::cyber::NullType>::Initialize(apollo::cyber::proto::ComponentConfig const&) ()
from /apollo/bazel-bin/modules/perception/onboard/component/../../../../_solib_k8/libmodules_Sperception_Sonboard_Scomponent_Slibperception_Ucomponent_Uinner_Ulidar.so
#6 0x000000000040adea in apollo::cyber::mainboard::ModuleController::LoadModule(apollo::cyber::proto::DagConfig const&) ()
#7 0x000000000040b43b in apollo::cyber::mainboard::ModuleController::LoadModule(std::string const&) ()
#8 0x000000000040b70d in apollo::cyber::mainboard::ModuleController::LoadAll() ()
#9 0x000000000040539d in main ()

three PC s are the same code,but one( AWS server) is OK ,and other two is failed.

compare the three PC's hardware info:
run cmd: lspci | grep -i vga
PC1 (AWS server)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)

PC 2
01:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A] (rev a1)

PC3 :
00:02.0 VGA compatible controller: Intel Corporation Device 9bc4 (rev 05)
01:00.0 VGA compatible controller: NVIDIA Corporation Device 2191 (rev a1)
TU116M [GeForce GTX 1660 Ti Mobile](I use this web to tranfer "2191" to this hardware info.)

compare the three PC's "nvidia-smi" info:
run cmd : nvidia-smi
PC1(AWS):
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.100 Driver Version: 440.100 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla M60 Off | 00000000:00:1E.0 Off | 0 |
| N/A 26C P8 16W / 150W | 186MiB / 7618MiB | 1% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1778 G /usr/lib/xorg/Xorg 90MiB |
| 0 5665 G /usr/bin/gnome-shell 92MiB |
+-----------------------------------------------------------------------------+

PC2:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.82 Driver Version: 440.82 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 208... Off | 00000000:01:00.0 On | N/A |
| 37% 47C P0 94W / 250W | 2136MiB / 11016MiB | 38% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1292 G /usr/lib/xorg/Xorg 16MiB |
| 0 1385 G /usr/bin/gnome-shell 57MiB |
| 0 3826 G /usr/lib/xorg/Xorg 109MiB |
| 0 3997 G /usr/bin/gnome-shell 116MiB |
| 0 10917 C+G .../Binaries/Linux/CarlaUE4-Linux-Shipping 1096MiB |
| 0 11931 G /usr/lib/firefox/firefox 734MiB |
+-----------------------------------------------------------------------------+

PC3:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.100 Driver Version: 440.100 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 166... Off | 00000000:01:00.0 Off | N/A |
| N/A 41C P0 24W / N/A | 332MiB / 5944MiB | 28% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2821 G /usr/lib/xorg/Xorg 213MiB |
| 0 3529 G /usr/bin/gnome-shell 99MiB |
| 0 16894 G ...AAAAAAAAAAAACAAAAAAAAAA= --shared-files 16MiB |
+-----------------------------------------------------------------------------+

Does anyone meet the same problem(perception crashed)?
thank you!

[AngelJA]

Hi there, no need to apologize. The Nvidia 2080 and 1660 are both Turing architecture gpus which Apollo 5.0's perception doesn't support. ApolloAuto/apollo#6916

It seems there were plans to support different gpu architectures, but I don't know the current status of those efforts.