Inference takes large GPU memory

[plusgrey]

Hi,

Thanks for your great work. However, when I run your demo on an RTX 3090 (24GB) it takes about 22GB of GPU memory. Is this normal? Compared to LoFTR, which only consumes about 1.7GB, it is really surprising.

[AmrSheta22]

Same problem.. I have been trying to run it on a 16 GB GPU with no luck. the models themselves aren't that large so I am very confused why they take that much space.
I also needed to move around some files and change some imports for the code to run properly.

[arjunkarpur]

@plusgrey We're looking into this and will get back to you soon.

@AmrSheta22 could you share the necessary changes? I will update the repo instructions. Thanks!

[hwjiang1510]

We suspect the reason is we are using default transformer layers without flash attention or some other memory-efficient implementations. The reason is that we were not able to compile these implementations, e.g., xFormers, under google's code base due to conflicts. This can be easily solved by using the original DINOv2 implementation.

[paolovic]

For me the same, it basically takes the whole GPU, in my case almost 11GB

this is how I could reduce it

import tensorflow as tf
physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)

[kke19]

@paolovic Hi, can you tell me in more detail how you solved it?

[paolovic]

@paolovic Hi, can you tell me in more detail how you solved it?

Hi,
I inserted the snippet from my previous post after the imports to the omniglue_demo script, which reduces my GPU VRAM consumption to

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.161.03   Driver Version: 470.161.03   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:05:00.0 Off |                  N/A |
| 39%   65C    P2    62W / 250W |   5499MiB / 11176MiB |      0%      Default |

But I was not able to enable GPU acceleration (that's why the 0% GPU-Util), although CUDA and cuDNN are installed and recognized.
Accordingly, it's more or less useless, and my experiment runs for days already..