CodeSLAM

PyTorch implementation of CodeSLAM: Learning a Compact, Optimisable Representation for Dense Visual SLAM.

This repository now implements the complete algorithmic pipeline described in the paper:

What Is In The Repo

Core implementation

Entry points

Legacy utilities

Paper Coverage

The table below maps the main technical pieces from the paper to this codebase.

Paper concept Where it is implemented
Image-conditioned compact depth representation codeslam/network.py
128-D optimizable code codeslam/config.py, codeslam/network.py
Hybrid proximity depth parametrization codeslam/proximity.py
Learned depth uncertainty codeslam/network.py, codeslam/losses.py
Multi-scale negative log-likelihood training codeslam/training.py, codeslam/losses.py
Decoder linear in the latent code codeslam/network.py
Precomputable Jacobian wrt code CodeSLAMDepthModel.precompute_linear_jacobian(...)
Dense photometric residuals codeslam/optimization.py
Dense geometric residuals codeslam/optimization.py
Affine brightness correction codeslam/optimization.py
Robust and weighted residual handling codeslam/losses.py, codeslam/optimization.py
Joint optimization of pose and latent geometry codeslam/optimization.py
Two-frame initialization codeslam/system.py, scripts/run_pair_optimization.py
Coarse-to-fine tracking codeslam/optimization.py
Sliding-window monocular SLAM codeslam/system.py
Schur-complement marginalization prior codeslam/optimization.py, codeslam/system.py

Installation

1. Create an environment

Use Python 3.11 or 3.12. The current torch wheels used here do not support Python 3.14.

python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

2. Optional: generate the SceneNet protobuf module

make

This generates scenenet_pb2.py from scenenet.proto, which is only needed if you want to inspect SceneNet metadata with read_protobuf.py.

3. Quick sanity checks

make compile
make test

Data Layout

The RGB-D training/evaluation loader expects a SceneNet-style directory tree like this:

data/train/
  0/
    some_scene/
      photo/
        0.jpg
        1.jpg
      depth/
        0.png
        1.png

The loader pairs files by frame stem. It also tolerates intensity filenames created by the preprocessing utilities, such as 0_intensity.jpg and 0_resized.jpg.

Preprocessing

If your SceneNet export still contains RGB images and you want grayscale/intensity preprocessing:

python3 preprocessing.py i r --data-root data/train --width 256 --height 192

Supported actions:

Training

Train the image-conditioned depth model on RGB-D data:

python3 scripts/train_codeslam.py \
  --data-root data/train \
  --batch-size 8 \
  --epochs 30 \
  --learning-rate 1e-4 \
  --checkpoint-dir checkpoints

What training does:

  1. Loads grayscale intensity images and metric depth.
  2. Encodes depth into a compact posterior code conditioned on the image.
  3. Decodes multi-scale proximity maps and uncertainty maps.
  4. Optimizes a multi-scale Laplace likelihood plus KL regularization.
  5. Saves checkpoints/latest.pt and checkpoints/best.pt.

Important training details implemented from the paper:

Depth Evaluation

Evaluate the learned single-image prior on RGB-D data:

python3 scripts/evaluate_depth.py \
  --data-root data/val \
  --checkpoint checkpoints/best.pt \
  --batch-size 8

This reports:

Single-Image Inference

Predict a dense depth prior from one image:

python3 scripts/infer_depth.py \
  frame.png \
  --checkpoint checkpoints/best.pt \
  --output-prefix outputs/frame

Outputs:

Notes:

Two-Frame Pair Optimization

Reproduce the paper's joint initialization for two overlapping frames:

python3 scripts/run_pair_optimization.py \
  frame_0001.png \
  frame_0002.png \
  --checkpoint checkpoints/best.pt \
  --fx 320 --fy 320 --cx 128 --cy 96

This jointly optimizes:

using:

Monocular SLAM / Sequence Inference

Run the full sliding-window system on a folder of images:

python3 scripts/run_slam.py \
  /path/to/sequence \
  --checkpoint checkpoints/best.pt \
  --fx 320 --fy 320 --cx 128 --cy 96

The sequence loader accepts .png and .jpg images and processes them in sorted order.

The system flow is:

  1. Insert the first frame as a seed keyframe.
  2. Bootstrap the map from the next frame by jointly optimizing pose and both keyframe codes.
  3. Track each new frame against the current keyframe with coarse-to-fine direct alignment.
  4. Insert a new keyframe once the motion threshold is exceeded.
  5. Run sliding-window mapping over active keyframes with joint pose/code optimization.
  6. Marginalize old keyframes into a compact linear prior.

The default SLAM configuration now follows the paper more closely:

Implementation Notes

Camera model

The direct optimization code assumes a calibrated pinhole camera model:

Residual design

The optimizer implements:

Optimization

The code uses dense Levenberg-Marquardt with:

Proximity parametrization

The paper's depth encoding is implemented as:

p = a / (d + a)

where a is the average depth hyperparameter, exposed here as ModelConfig.proximity_average_depth.

Jacobians wrt code

The model exposes:

model.precompute_linear_jacobian(intensity, level=-1)

for the decoder path that remains linear in the latent code. This is the optimization-friendly design used in the paper.

Testing

Run all tests with:

make test

Current coverage includes:

Some tests are skipped automatically when torch is not installed, because the geometry/optimization/model code depends on PyTorch.

What "Full Paper Implementation" Means Here

This repository includes every major algorithmic component described in the paper and wires them together into a train/eval/inference/SLAM workflow.

What still depends on runtime validation rather than static code inspection:

In other words: the paper's system is implemented here, but benchmark parity still requires trained weights, the intended datasets, and empirical evaluation.