longitudinal/REGISTRATION_BENCHMARK.md

Registration Pipeline Benchmark — CT → T1c (M6)

Benchmarks four rigid, cross-modality registration pipelines that align a head-and-neck CT to a tightly-cropped T1c MRI so the GTV can be warped onto the T1c (MR) native grid. This is the registration step of scripts/02_build_m6_dataset.py. The pairs are hard: different modality (CT Hounsfield vs MR intensity) and the patient is repositioned between the CT and MR sessions, so the inter-session head tilt can reach ~30°.

• Pipelines: src/registration_pipelines.py
• Benchmark harness: scripts/bench_registration.py
• Results: results/bench_registration/ (per-pipeline JSON, summary.json, summary.md, bench_registration.png)

Unified contract

Every pipeline returns a SimpleITK transform T such that

sitk.Resample(ct, t1c, T, sitk.sitkLinear)              # CT  onto the MR grid
sitk.Resample(tv,  t1c, T, sitk.sitkNearestNeighbor, 0) # GTV onto the MR grid

places the CT / GTV concentric with the MR head. In Resample's destination→source convention this T maps T1c → CT. All pipelines register on the same CT-space head box (MI computed on head tissue, not neck/air) and, for the single-init tools, from the same centroid-aligned start (centroid_offset), so the comparison is on the optimizer / metric / transform handling — not on who finds the ~30–60 mm gross offset first.

The four pipelines

#	pipeline	toolkit	metric	transform	status in this env
1	elastix	ITK-Elastix	Mattes MI	Rigid3DTransform (ASGD)	not installed (conda-forge unreachable)
2	simpleitk	SimpleITK 2.5.6	Mattes MI (32 bins)	Euler3D rigid, pyramid [4,2,1]	✅ ran
3	antspyx	ANTsPy / ANTs 0.6.3	Mattes MI (multi-res)	ANTs Rigid	✅ ran (isolated py3.12 env)
4	deepreg	Oxford DeepReg (CycleGAN)	CycleGAN + reg. loss (pretrained)	3D displacement field	not installed (see below)

simpleitk has two budgets: --simpleitk-mode benchmark (single pyramid, centered start — the default, comparable to the other tools) and --simpleitk-mode production (the full multi-start tilt grid from 02_build_m6_dataset.py, ~20× slower, the most robust but not used for the head-to-head).

Test cases

First 10 labeled M6 cases (--n-cases 10 --sample first), reproducible. 9 have a complete CT; 1 (m6_22GD5VL2_20230724) has an incomplete CT head and is reported as incomplete-CT for every pipeline (a data issue, not a pipeline failure).

m6_22GD5VL2_20230724  (incomplete CT)
m6_22IRRKU6_20240115  m6_23XIMO4M_20230222  m6_24V62QQM_20230223
m6_24V62QQM_20231017  m6_24V62QQM_20240820  m6_26TYQI3C_20221027
m6_2BBN5XA4_20250411  m6_2C4TKI6H_20240506  m6_2CDNN7BH_20240102

Metrics (identical for every pipeline)

Production QA, crop-independent:

• skull_ring_gap_mm — march rays from the MR head center; the gap between the MR head silhouette boundary and the warped-CT skull ring (p97 of positive CT) at mid and low levels, plus the angular std. Aligned ⇔ 0 ≤ mid ≤ 20 mm and std ≤ 12 mm. mid measures translational concentricity; std measures rotational concentricity.
• tv_in_ct_frac — fraction of the dilated warped GTV inside the warped-CT head tissue (the slab must straddle the lesion). Acceptance needs ≥ 0.5.
• tv_nvox — warped GTV volume.
• accepted — tv_nvox > 0 and tv_in_ct_frac ≥ 0.5 and aligned (the production acceptance rule).
• reg_wall_s — wall-clock registration time (excludes I/O + scoring).

Results (10 cases)

pipeline	toolkit	registered	accepted	gap mid (med/mean)	gap std (med/mean)	s/reg (med)	tv-in-CT
simpleitk	SimpleITK 2.5.6	9	4/10 (0.40)	12.6 / 16.1	15.4 / 16.0	8.3	0.93
antspyx	ANTsPy 0.6.3	9	1/10 (0.10)	14.3 / 15.4	18.2 / 21.7	13.4	0.93
elastix	—	—	—	—	—	—	—
deepreg	—	—	—	—	—	—	—

Per-case (mid / std, OK = accepted):

case	simpleitk	antspyx
m6_22GD5VL2_20230724	incomplete-CT	incomplete-CT
m6_22IRRKU6_20240115	25.7 / 29.3	26.3 / 29.2
m6_23XIMO4M_20230222	22.2 / 23.9	23.8 / 25.0
m6_24V62QQM_20230223	10.7 / 16.9	9.4 / 18.2
m6_24V62QQM_20231017	11.5 / 7.3 OK	12.5 / 12.8
m6_24V62QQM_20240820	11.6 / 9.6 OK	14.8 / 11.7 OK
m6_26TYQI3C_20221027	12.2 / 3.7 OK	6.1 / 37.2
m6_2BBN5XA4_20250411	18.6 / 8.0 OK	14.3 / 12.7
m6_2C4TKI6H_20240506	12.6 / 15.4	13.6 / 14.7
m6_2CDNN7BH_20240102	19.4 / 30.2	18.2 / 33.6

Interpretation

• Translation is recovered well by both (mid gaps mostly 6–20 mm, within the ≤ 20 mm band). tv-in-CT is identical (0.93): the warped GTV lands inside the CT head for both, so the target is in the right place.
• The binding failure is rotation — std > 12 mm on the repositioned cases (the ~30° inter-session tilt produces MI local minima that a single centered start can lock onto). This is exactly why production 02_build_m6_dataset.py uses a multi-start tilt grid.
• SimpleITK is the more robust of the two (4 vs 1 accepted, lower median std 15.4 vs 18.2): its RSGD + physical-step-scales + 32-bin Mattes-MI landscape handles the tilt better than ANTs' Rigid. Note m6_26TYQI3C: ANTs nailed the position (mid 6.1) but missed the orientation (std 37.2) — a textbook tilted local minimum.
• Speed: SimpleITK ~8.3 s/case, ANTs ~13.4 s/case (ANTs includes the pre-shift resample + its affine-MI init stages). Both are CPU-bound and ran 4 cases in parallel.

Reproducing

# main env — SimpleITK baseline (elastix/deepreg report unavailable here)
source /opt/conda/etc/profile.d/conda.sh && conda activate longitudinal
python scripts/bench_registration.py --pipelines simpleitk --n-cases 10 \
    --workers 4 --threads 24 --simpleitk-mode benchmark

# isolated py3.12 env with antspyx — ANTs (+ optional SimpleITK cross-check)
/home/xfr/.conda/envs/ants312/bin/python scripts/bench_registration.py \
    --pipelines antspyx --n-cases 10 --workers 4 --threads 24

# combine the per-pipeline JSONs accumulated across envs into one table + plot
python scripts/bench_registration.py --merge

Other flags: --sample random --seed 0 (reproducible random 10), --keys k1,k2,... (explicit cases), --n-cases N, --workers W, --threads T, --simpleitk-mode {benchmark,production}, --deepreg-ckpt <.ckpt> (DeepReg), --out-dir <dir>.

Enabling the two blocked pipelines

ITK-Elastix — not on PyPI; needs conda-forge (which this host cannot reach). Once conda-forge is available:

conda install -c conda-forge elastix        # provides the elastix/transformix CLI
python scripts/bench_registration.py --pipelines elastix --n-cases 10

The adapter registers fixed=T1c, moving=CT with Rigid3DTransform + Mattes MI on the head box and reads the result transform.

DeepReg (Oxford) — the PyPI deepreg (0.1.2) is an unrelated stub that pins numpy==1.18.5; do not install it. DeepReg is not a one-shot rigid tool: it is a CycleGAN image-to-image registration that must be trained or run from a pretrained checkpoint. To benchmark it:

git clone https://github.com/OB-CVHU/DeepReg && cd DeepReg && pip install -e .
python scripts/bench_registration.py --pipelines deepreg --n-cases 10 \
    --deepreg-ckpt /path/to/pretrained_3d.ckpt

The adapter runs deepreg test to predict a voxel displacement field, converts it to a SimpleITK DisplacementFieldTransform, and scores it like the others.

Notes / caveats

• The benchmark compares standard single-init configurations (centered start). Production uses --simpleitk-mode production's multi-start tilt grid, which raises SimpleITK's acceptance at ~20× the cost; that is out of scope for the head-to-head but is the config the dataset actually ships with.
• elastix / deepreg could not be executed here, so their rows are "not installed", not measured. Their code paths are implemented and unit-verified for import/convention; run them where the toolkits are present.
• fix_orientation (elastix/ants/deepreg) chooses between a transform and its inverse by skull-ring concentricity — a correctness gate that makes the result independent of each toolkit's transform-direction convention.