About training code and differentiable ba implementation

[mkocabas]

Hi @jytime,

Awesome work! Do you have any plans to release the training code?

I am specifically interested in how you used the theseus library to implement differentiable BA.

[jytime]

Hi @mkocabas,

We do plan to release the training code, but it is not our top priority at the moment. Our recent focus is to resolve the memory issue and provide a version that supports video input (as mentioned in #9). We hope they will facilitate easier off-the-shelf usage.

Regarding Theseus, their team provides an official example for differentiable Bundle Adjustment, available here:
https://github.com/facebookresearch/theseus/blob/e07569138cafa2d5da3e16ff2586d2495e77817c/examples/bundle_adjustment.py#L4

While this implementation isn’t fast, it functions correctly.

Please let me know if there is anything else I can help.

[mkocabas]

Thanks a lot for your prompt response!

I am looking forward to your training code.

I am aware of the example implementation. But as you mentioned it is quite slow. I wonder which tricks you used to make it work faster. It would be immensely helpful if you can share your code snippet implementing those changes.

[jytime]

Hi, as far as I can remember now (they were conducted months ago sry), these points helped:

(a) Use the linear solver BaSpaCho. BaSpaCho was specially designed by the Theseus team for BA. You need to compile it from the source code.
(b) A very time-consuming step is using the for loop to convert pytorch tensors to Theseus variables. it will be much faster if you can make it a batch operation. A trial I used before was facebookresearch/theseus#565, please check the github issue discussion for details. (But plz notice that we did not use this implementation finally. Instead we modified the source codes of Theseus to achieve something similar)
(c) During training, use a small number of 3D points and camera parameters. Subsampling 25% of the parameters for BA does not significantly impact performance but greatly improves speed.

[jytime]

Closed now. Feel free to reopen this issue.