How to profile model training?

[systemdevart]

In the official PyTorch documentation, there is a recommendation to use nvprof, but after starting the training script using nvprof, it simply exits with the warning:

nvprof --profile-from-start off -o trace_name.prof -- python3 examples/asr/asr_ctc/speech_to_text_ctc.py   --config-path=...
======== Warning: nvprof is not supported on devices with compute capability 8.0 and higher.
                  Use NVIDIA Nsight Systems for GPU tracing and CPU sampling and NVIDIA Nsight Compute for GPU profiling.
                  Refer https://developer.nvidia.com/tools-overview for more details.

For dlprof, there is no clear way to use it with Pytorch Lightning, so my question is: what is the recommended way to profile the model training?

[titu1994]

Nvprof is deprecated since a year or two. The recommended way is below - https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html

You can also use Nsys profiler directly by setting up the code locally in editable mode, and pytorchs tools - https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59

[systemdevart]

But those methods don't allow integration into Pytorch Lightning, and because of that, you can't distinguish the train step and validation step with nice trace visualization charts with the dlprofviewer tool.