longitudinal/REGISTRATION_BENCHMARK.md

# Registration Pipeline Benchmark — CT → T1c (M6)

Benchmarks **four rigid, cross-modality registration pipelines** that align a
head-and-neck CT to a tightly-cropped T1c MRI so the GTV can be warped onto the
T1c (MR) native grid. This is the registration step of
`scripts/02_build_m6_dataset.py`. The pairs are hard: different modality (CT
Hounsfield vs MR intensity) and the patient is repositioned between the CT and
MR sessions, so the inter-session head tilt can reach ~30°.

- Pipelines: [`src/registration_pipelines.py`](src/registration_pipelines.py)
- Benchmark harness: [`scripts/bench_registration.py`](scripts/bench_registration.py)
- Results: `results/bench_registration/` (per-pipeline JSON, `summary.json`,
  `summary.md`, `bench_registration.png`)

## Unified contract

Every pipeline returns a SimpleITK transform `T` such that

```python
sitk.Resample(ct, t1c, T, sitk.sitkLinear)              # CT  onto the MR grid
sitk.Resample(tv,  t1c, T, sitk.sitkNearestNeighbor, 0) # GTV onto the MR grid
```

places the CT / GTV concentric with the MR head. In `Resample`'s
destination→source convention this `T` maps **T1c → CT**. All pipelines register
on the same CT-space head box (MI computed on head tissue, not neck/air) and,
for the single-init tools, from the **same centroid-aligned start**
(`centroid_offset`), so the comparison is on the optimizer / metric / transform
handling — not on who finds the ~30–60 mm gross offset first.

## The four pipelines

| # | pipeline | toolkit | metric | transform | status in this env |
|---|----------|---------|--------|-----------|--------------------|
| 1 | `elastix`   | ITK-Elastix | Mattes MI | `Rigid3DTransform` (ASGD) | **not installed** (conda-forge unreachable) |
| 2 | `simpleitk` | SimpleITK 2.5.6 | Mattes MI (32 bins) | `Euler3D` rigid, pyramid `[4,2,1]` | ✅ ran |
| 3 | `antspyx`   | ANTsPy / ANTs 0.6.3 | Mattes MI (multi-res) | ANTs `Rigid` | ✅ ran (isolated py3.12 env) |
| 4 | `deepreg`   | Oxford DeepReg (CycleGAN) | CycleGAN + reg. loss (pretrained) | 3D displacement field | **not installed** (see below) |

`simpleitk` has two budgets: `--simpleitk-mode benchmark` (single pyramid,
centered start — the default, comparable to the other tools) and
`--simpleitk-mode production` (the full multi-start tilt grid from
`02_build_m6_dataset.py`, ~20× slower, the most robust but not used for the
head-to-head).

## Test cases

First 10 labeled M6 cases (`--n-cases 10 --sample first`), reproducible. 9 have
a complete CT; 1 (`m6_22GD5VL2_20230724`) has an incomplete CT head and is
reported as `incomplete-CT` for every pipeline (a data issue, not a pipeline
failure).

```
m6_22GD5VL2_20230724  (incomplete CT)
m6_22IRRKU6_20240115  m6_23XIMO4M_20230222  m6_24V62QQM_20230223
m6_24V62QQM_20231017  m6_24V62QQM_20240820  m6_26TYQI3C_20221027
m6_2BBN5XA4_20250411  m6_2C4TKI6H_20240506  m6_2CDNN7BH_20240102
```

## Metrics (identical for every pipeline)

Production QA, crop-independent:

- **`skull_ring_gap_mm`** — march rays from the MR head center; the gap between
  the MR head silhouette boundary and the warped-CT skull ring (p97 of positive
  CT) at mid and low levels, plus the angular std. **Aligned ⇔
  `0 ≤ mid ≤ 20 mm` and `std ≤ 12 mm`.** `mid` measures translational
  concentricity; `std` measures rotational concentricity.
- **`tv_in_ct_frac`** — fraction of the dilated warped GTV inside the warped-CT
  head tissue (the slab must straddle the lesion). Acceptance needs `≥ 0.5`.
- **`tv_nvox`** — warped GTV volume.
- **`accepted`** — `tv_nvox > 0 and tv_in_ct_frac ≥ 0.5 and aligned` (the
  production acceptance rule).
- **`mi_ct_mr`** — mean Mattes MI (nats) between the **CT inverse-transformed
  into MR coordinates** (the warped CT on the T1c grid) and the **original MR**,
  restricted to the MR head and each image percentile-binned independently
  (SimpleITK Mattes-MI style). This is the registration's *achieved* metric read
  off as a standalone alignment score: higher = the warped-CT and MR intensities
  are better in-sync = better aligned; it collapses toward 0 when the CT is
  rotated/translated away from the MR.
- **`reg_wall_s`** — wall-clock registration time (excludes I/O + scoring).

## Results (10 cases)

| pipeline | toolkit | registered | accepted | gap mid (med/mean) | gap std (med/mean) | s/reg (med) | tv-in-CT | MI (med/mean, nats) |
|---|---|---|---|---|---|---|---|---|
| **simpleitk** | SimpleITK 2.5.6 | 9 | **4/10 (0.40)** | 12.6 / 16.1 | 15.4 / 16.0 | 8.1 | 0.93 | **0.643 / 0.630** |
| **antspyx** | ANTsPy 0.6.3 | 9 | **1/10 (0.10)** | 14.2 / 17.3 | 24.8 / 22.4 | 12.7 | 0.93 | **0.600 / 0.523** |
| elastix | — | — | — | — | — | — | — | — |
| deepreg | — | — | — | — | — | — | — | — |

Per-case (mid / std, `OK` = accepted):

| case | simpleitk | antspyx |
|---|---|---|
| m6_22GD5VL2_20230724 | incomplete-CT | incomplete-CT |
| m6_22IRRKU6_20240115 | 25.7 / 29.3 | 26.3 / 29.2 |
| m6_23XIMO4M_20230222 | 22.2 / 23.9 | 23.8 / 25.0 |
| m6_24V62QQM_20230223 | 10.7 / 16.9 | 9.4 / 18.2 |
| m6_24V62QQM_20231017 | 11.5 / 7.3 **OK** | 12.5 / 12.8 |
| m6_24V62QQM_20240820 | 11.6 / 9.6 **OK** | 14.8 / 11.7 **OK** |
| m6_26TYQI3C_20221027 | 12.2 / 3.7 **OK** | 6.1 / 37.2 |
| m6_2BBN5XA4_20250411 | 18.6 / 8.0 **OK** | 14.3 / 12.7 |
| m6_2C4TKI6H_20240506 | 12.6 / 15.4 | 13.6 / 14.7 |
| m6_2CDNN7BH_20240102 | 19.4 / 30.2 | 18.2 / 33.6 |

### Interpretation

- **Translation is recovered well by both** (mid gaps mostly 6–20 mm, within the
  `≤ 20 mm` band). `tv-in-CT` is identical (0.93): the warped GTV lands inside
  the CT head for both, so the target is in the right place.
- **The binding failure is rotation** — `std > 12 mm` on the repositioned cases
  (the ~30° inter-session tilt produces MI local minima that a single centered
  start can lock onto). This is exactly why production `02_build_m6_dataset.py`
  uses a **multi-start tilt grid**.
- **SimpleITK is the more robust of the two** (4 vs 1 accepted, lower median
  std 15.4 vs 18.2): its RSGD + physical-step-scales + 32-bin Mattes-MI
  landscape handles the tilt better than ANTs' Rigid. Note `m6_26TYQI3C`: ANTs
  nailed the position (mid 6.1) but missed the orientation (std 37.2) — a
  textbook tilted local minimum.
- **Speed**: SimpleITK ~8.1 s/case, ANTs ~12.7 s/case (ANTs includes the
  pre-shift resample + its affine-MI init stages). Both are CPU-bound and ran
  4 cases in parallel.
- **Mattes MI (warped-CT ~ MR) is an independent, intensity-based confirmation**:
  SimpleITK median 0.643 nats vs ANTs 0.600, and it diverges sharply on the
  tilted cases — ANTs `m6_26TYQI3C` drops to 0.094 nats (rotation ~37 mm off)
  while SimpleITK holds 0.670. A high MI means the CT skull/tissue intensities
  fall on the matching MR structures after the inverse transform.
- **ANTs MI is stochastic**: ANTs' Mattes-MI subsamples the image at random, so
  its per-case numbers carry run-to-run variance (this table is one draw);
  SimpleITK's MI (and registration) is deterministic.

## Reproducing

```bash
# main env — SimpleITK baseline (elastix/deepreg report unavailable here)
source /opt/conda/etc/profile.d/conda.sh && conda activate longitudinal
python scripts/bench_registration.py --pipelines simpleitk --n-cases 10 \
    --workers 4 --threads 24 --simpleitk-mode benchmark

# isolated py3.12 env with antspyx — ANTs (+ optional SimpleITK cross-check)
/home/xfr/.conda/envs/ants312/bin/python scripts/bench_registration.py \
    --pipelines antspyx --n-cases 10 --workers 4 --threads 24

# combine the per-pipeline JSONs accumulated across envs into one table + plot
python scripts/bench_registration.py --merge
```

Other flags: `--sample random --seed 0` (reproducible random 10),
`--keys k1,k2,...` (explicit cases), `--n-cases N`, `--workers W`,
`--threads T`, `--simpleitk-mode {benchmark,production}`,
`--deepreg-ckpt <.ckpt>` (DeepReg), `--out-dir <dir>`,
`--qa-images` / `--qa-out <dir>` (visual QA, below).

## Visual QA images (checkerboard + color overlay)

`--qa-images` writes a `checkreg`-style PNG per (pipeline, case) to
`<out-dir>/qa/<pipeline>__<case>.png` for eyeballing the alignment.  Each image
is a 3×2 grid: **rows** are the three orthogonal mid-slices through the head
center (**mid-axial, mid-coronal, mid-sagittal**), and **columns** are:

- **left — checkerboard blend:** alternating blocks of the CT inverse-transformed
  into MR coordinates and the original MR; a clean registration gives a seamless
  mosaic, a misregistration shows hard edges at the block boundaries.
- **right — additive color overlay:** CT in red, MR in green. Structures that
  coincide blend to yellow; rotation/translation error appears as red/green
  fringing at the skull and brain edges.

The CT is shown in a **brain window (L40/W80 = 0–80 HU)** so the brain
parenchyma is visible and comparable to the MR (the skull, ~800+ HU, clips to
white); the MR uses a 2–98 % head window. The coronal and sagittal views are
oriented superior-up to match the axial view.

The title carries the pipeline, case key, Mattes MI, and the skull-ring gap
(mid/std), so a glance tells you both the achieved-metric score and whether the
visuals agree. Generate for the two available pipelines with:

```bash
/home/xfr/.conda/envs/longitudinal/bin/python scripts/bench_registration.py \
    --pipelines simpleitk --n-cases 10 --qa-images
/home/xfr/.conda/envs/ants312/bin/python scripts/bench_registration.py \
    --pipelines antspyx --n-cases 10 --qa-images
```

(9 images each; the incomplete-CT case is skipped.)

## Enabling the two blocked pipelines

**ITK-Elastix** — not on PyPI; needs conda-forge (which this host cannot reach).
Once conda-forge is available:

```bash
conda install -c conda-forge elastix        # provides the elastix/transformix CLI
python scripts/bench_registration.py --pipelines elastix --n-cases 10
```

The adapter registers `fixed=T1c, moving=CT` with `Rigid3DTransform` + Mattes MI
on the head box and reads the result transform.

**DeepReg (Oxford)** — the PyPI `deepreg` (0.1.2) is an unrelated stub that pins
`numpy==1.18.5`; do not install it. DeepReg is not a one-shot rigid tool: it is a
CycleGAN image-to-image registration that must be **trained or run from a
pretrained checkpoint**. To benchmark it:

```bash
git clone https://github.com/OB-CVHU/DeepReg && cd DeepReg && pip install -e .
python scripts/bench_registration.py --pipelines deepreg --n-cases 10 \
    --deepreg-ckpt /path/to/pretrained_3d.ckpt
```

The adapter runs `deepreg test` to predict a voxel displacement field, converts
it to a SimpleITK `DisplacementFieldTransform`, and scores it like the others.

## Notes / caveats

- The benchmark compares **standard single-init** configurations (centered
  start). Production uses `--simpleitk-mode production`'s multi-start tilt grid,
  which raises SimpleITK's acceptance at ~20× the cost; that is out of scope for
  the head-to-head but is the config the dataset actually ships with.
- `elastix` / `deepreg` could not be executed here, so their rows are
  "not installed", not measured. Their code paths are implemented and
  unit-verified for import/convention; run them where the toolkits are present.
- `fix_orientation` (elastix/ants/deepreg) chooses between a transform and its
  inverse by skull-ring concentricity — a correctness gate that makes the result
  independent of each toolkit's transform-direction convention.