Support for COLMAP rig bundle adjustment (multi-camera rig optimization)

[jahad9819jjj]

Summary

I would like to apply COLMAP's rig bundle adjustment to fastmap for optimizing multi-camera systems (e.g. stereo rigs). Currently, fastmap appears to treat all camera poses independently, but in my use case, the stereo cameras have a fixed relative transform.

Motivation

In many real-world settings, such as stereo setups or mobile robot camera rigs, cameras are rigidly attached. Applying rig bundle adjustment helps to:

• Maintain fixed relative poses between cameras
• Reduce drift in trajectory estimation
• Improve photometric consistency across multiple views

Requested Feature

Could you advise which part of fastmap’s COLMAP pipeline needs to be modified in order to:

If there's already support for this (or partial support), could you point me to the relevant part of the codebase?

Additional Notes

If this is something that might require major changes, I’m also open to contributing a PR for basic rig support. Any design guidance would be appreciated.

Thanks for the excellent work on fastmap!

[jiahaoli95]

Hi @jahad9819jjj , thank you for the suggestion. I think this would be a great feature to add. We can start from designing the input format. I'm not familiar with the stereo rig use case myself, and so need your opinion on this. What do you think is a convenient way for the user to specify the relative pose relationship between cameras? For example, how does colmap allow you to input this information? Thanks.

[jahad9819jjj]

Hi @jiahaoli95, thank you for the quick response and for considering this feature and sorry for late reply.
You're absolutely right that defining a good input format is the key starting point. COLMAP supports rig configurations through the --rig_config_path parameter with a JSON format like these:

--rig_config_path colmap_input/rig_config.json`

[
  {
    "ref_camera_id": 1,
    "cameras": [
      {
        "camera_id": 1,
        "image_prefix": "left",
        "rel_tvec": [0, 0, 0],
        "rel_qvec": [1, 0, 0, 0]
      },
      {
        "camera_id": 2,
        "image_prefix": "right", 
        "rel_tvec": [-0.15, 0, 0],
        "rel_qvec": [1, 0, 0, 0]
      }
    ]
  }
]

The key components are:

ref_camera_id: The reference camera in the rig (typically camera 1)
camera_id: Individual camera identifier
image_prefix: How to match images to cameras (e.g., "left_001.jpg", "right_001.jpg")
rel_tvec/rel_qvec: Relative translation/rotation from reference camera

For fastmap integration, I think we could either:

Use a similar JSON format as input

• Auto-detect stereo pairs from filename patterns (simpler for common cases)
• Allow both approaches

Some relevant COLMAP issues that discuss common challenges: Rig Bundle Adjuster Failing and rig_bundle_adjuster converging but not constraining cam extrinsics

Would this input format work for fastmap's use cases? I'm happy to help with the design and implementation if you think this direction makes sense.
Thanks again for considering this enhancement!

[jiahaoli95]

Thanks @jahad9819jjj . Yes the json input format looks good. Another question: could you provide (or point me to) one or more sets of images taken by a rig, along with the corresponding json file? I will use this for testing the implementation, and let you know how it works along the way. It would be better if the images densely cover the scene so that fastmap won't fail (see the limitation section in the readme). Thank you!

[jahad9819jjj]

@jiahaoli95
Thanks.
The data at hand is private, so I need to look at public data sets that we can use.
I think the following are the ones we could use at the moment.

https://lamar.ethz.ch/
・grayscale
https://hilti-challenge.com/dataset-2023.html
・need to parse ROS bag
https://sites.google.com/view/graco-dataset/system
・need to parse ROS bag

At the very least, I might adopt the lamar dataset. colmap feature extraction works with grayscale, right?

[jiahaoli95]

No worries. I think it makes sense to start with some synthetic data. Currently I'm occupied by something else, and will let you know when I have a chance to put my hands on it. Thank you!