Python Kernel Crashing with GPU support

[manideepabc]

Hello,

I am trying to run the tutorial example Ray tracing part 1 using GPU support in the docker container. I see that the Python kernel crashes as soon as I try to load the scene. But everything works fine when using the CPU to run the code. Did anyone face this problem while running it? Can someone help us with this?

We pretty much used the DOCKERFILE and Makefile provided in the repository and haven't changed any Python versions while running the sample codes.

What have I tried?: We tried to see if this issue persists while running other code samples. So, we tried running the Realistic Multiuser MIMO channel code with GPU support, and it worked perfectly. We ensured that it ran on GPU by observing that the GPU utilization has increased when running with GPU support.

[pablosreyero]

I'm not a Sionna developer, but here is my basic advice:

First, check if tensorflow detects your GPU: print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU'))). Next, you should check your GPU drivers and keep them up to date (just in case). Later, check if your CUDA and CUDNN installation, is compatible with the version of tensorflow you are running with Sionna, I guess you shouldn't worry too much about this part, as it is already specified in the docker file, but it is really important. At first, most of the problems you will run into will be GPU related, or tensorflow versions, then luckily you will only run into some tensor dimension mismatches, if you are implementing your own code (at least I am).

In my experience, I had a hard time installing a compatible version of CUDA and CUDNNN and making everything compatible within my system, but in the end, everything works fine and it's worth it. Don't hesitate and keep going.

[manideepabc]

Hi,
Thanks for the suggestion.
I have tried printing GPUs using tensorflow commands and it was printing the GPU information correctly. I did some testing using breakpoints in python code and I figured that the Mitsuba is the one that is causing the kernel crash when using GPU.

Can you share the version of Mitsuba, CUDA, and cuDNN you used and any dependencies? I would like to try out the same versions and make it work on my end.

[merlinND]

Hello @manideepabc,

Could you please share any error message or stack trace that you can see in the terminal when the crash occurs?

There were two new releases of Mitsuba last week, there might be compatibility issues. Could you try again by installing either:

pip install mitsuba===3.4.1

or

pip install mitsuba===3.4.0

[manideepabc]

Hello @merlinND .
Thanks for getting back. Some of my colleagues helped me fix this issue.
Looks like the problem is with the NVIDIA Optix library using the GPUs. The fix is to add
RUN ln -s /usr/lib/x86_64-linux-gnu/libcuda.so.1 /usr/lib/libcuda.so at the end of the Dockerfile