xfr/longitudinal
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
longitudinal/REGISTRATION_BENCHMARK.md

11 KiB
Raw Blame History

Registration Pipeline Benchmark — CT → T1c (M6)

Benchmarks four rigid, cross-modality registration pipelines that align a head-and-neck CT to a tightly-cropped T1c MRI so the GTV can be warped onto the T1c (MR) native grid. This is the registration step of scripts/02_build_m6_dataset.py. The pairs are hard: different modality (CT Hounsfield vs MR intensity) and the patient is repositioned between the CT and MR sessions, so the inter-session head tilt can reach ~30°.

Unified contract

Every pipeline returns a SimpleITK transform T such that

sitk.Resample(ct, t1c, T, sitk.sitkLinear)              # CT  onto the MR grid
sitk.Resample(tv,  t1c, T, sitk.sitkNearestNeighbor, 0) # GTV onto the MR grid

places the CT / GTV concentric with the MR head. In Resample's destination→source convention this T maps T1c → CT. All pipelines register on the same CT-space head box (MI computed on head tissue, not neck/air) and, for the single-init tools, from the same centroid-aligned start (centroid_offset), so the comparison is on the optimizer / metric / transform handling — not on who finds the ~30–60 mm gross offset first.

The four pipelines

# pipeline toolkit metric transform status in this env
1 elastix ITK-Elastix Mattes MI Rigid3DTransform (ASGD) not installed (conda-forge unreachable)
2 simpleitk SimpleITK 2.5.6 Mattes MI (32 bins) Euler3D rigid, pyramid [4,2,1] ✅ ran
3 antspyx ANTsPy / ANTs 0.6.3 Mattes MI (multi-res) ANTs Rigid ✅ ran (isolated py3.12 env)
4 deepreg Oxford DeepReg (CycleGAN) CycleGAN + reg. loss (pretrained) 3D displacement field not installed (see below)

simpleitk has two budgets: --simpleitk-mode benchmark (single pyramid, centered start — the default, comparable to the other tools) and --simpleitk-mode production (the full multi-start tilt grid from 02_build_m6_dataset.py, ~20× slower, the most robust but not used for the head-to-head).

Test cases

First 10 labeled M6 cases (--n-cases 10 --sample first), reproducible. 9 have a complete CT; 1 (m6_22GD5VL2_20230724) has an incomplete CT head and is reported as incomplete-CT for every pipeline (a data issue, not a pipeline failure).

m6_22GD5VL2_20230724  (incomplete CT)
m6_22IRRKU6_20240115  m6_23XIMO4M_20230222  m6_24V62QQM_20230223
m6_24V62QQM_20231017  m6_24V62QQM_20240820  m6_26TYQI3C_20221027
m6_2BBN5XA4_20250411  m6_2C4TKI6H_20240506  m6_2CDNN7BH_20240102

Metrics (identical for every pipeline)

Production QA, crop-independent:

  • skull_ring_gap_mm — march rays from the MR head center; the gap between the MR head silhouette boundary and the warped-CT skull ring (p97 of positive CT) at mid and low levels, plus the angular std. Aligned ⇔ 0 ≤ mid ≤ 20 mm and std ≤ 12 mm. mid measures translational concentricity; std measures rotational concentricity.
  • tv_in_ct_frac — fraction of the dilated warped GTV inside the warped-CT head tissue (the slab must straddle the lesion). Acceptance needs ≥ 0.5.
  • tv_nvox — warped GTV volume.
  • accepted — tv_nvox > 0 and tv_in_ct_frac ≥ 0.5 and aligned (the production acceptance rule).
  • mi_ct_mr — mean Mattes MI (nats) between the CT inverse-transformed into MR coordinates (the warped CT on the T1c grid) and the original MR, restricted to the MR head and each image percentile-binned independently (SimpleITK Mattes-MI style). This is the registration's achieved metric read off as a standalone alignment score: higher = the warped-CT and MR intensities are better in-sync = better aligned; it collapses toward 0 when the CT is rotated/translated away from the MR.
  • reg_wall_s — wall-clock registration time (excludes I/O + scoring).

Results (10 cases)

pipeline toolkit registered accepted gap mid (med/mean) gap std (med/mean) s/reg (med) tv-in-CT MI (med/mean, nats)
simpleitk SimpleITK 2.5.6 9 4/10 (0.40) 12.6 / 16.1 15.4 / 16.0 8.1 0.93 0.643 / 0.630
antspyx ANTsPy 0.6.3 9 1/10 (0.10) 14.2 / 17.3 24.8 / 22.4 12.7 0.93 0.600 / 0.523
elastix — — — — — — — —
deepreg — — — — — — — —

Per-case (mid / std, OK = accepted):

case simpleitk antspyx
m6_22GD5VL2_20230724 incomplete-CT incomplete-CT
m6_22IRRKU6_20240115 25.7 / 29.3 26.3 / 29.2
m6_23XIMO4M_20230222 22.2 / 23.9 23.8 / 25.0
m6_24V62QQM_20230223 10.7 / 16.9 9.4 / 18.2
m6_24V62QQM_20231017 11.5 / 7.3 OK 12.5 / 12.8
m6_24V62QQM_20240820 11.6 / 9.6 OK 14.8 / 11.7 OK
m6_26TYQI3C_20221027 12.2 / 3.7 OK 6.1 / 37.2
m6_2BBN5XA4_20250411 18.6 / 8.0 OK 14.3 / 12.7
m6_2C4TKI6H_20240506 12.6 / 15.4 13.6 / 14.7
m6_2CDNN7BH_20240102 19.4 / 30.2 18.2 / 33.6

Interpretation

  • Translation is recovered well by both (mid gaps mostly 6–20 mm, within the ≤ 20 mm band). tv-in-CT is identical (0.93): the warped GTV lands inside the CT head for both, so the target is in the right place.
  • The binding failure is rotation — std > 12 mm on the repositioned cases (the ~30° inter-session tilt produces MI local minima that a single centered start can lock onto). This is exactly why production 02_build_m6_dataset.py uses a multi-start tilt grid.
  • SimpleITK is the more robust of the two (4 vs 1 accepted, lower median std 15.4 vs 18.2): its RSGD + physical-step-scales + 32-bin Mattes-MI landscape handles the tilt better than ANTs' Rigid. Note m6_26TYQI3C: ANTs nailed the position (mid 6.1) but missed the orientation (std 37.2) — a textbook tilted local minimum.
  • Speed: SimpleITK ~8.1 s/case, ANTs ~12.7 s/case (ANTs includes the pre-shift resample + its affine-MI init stages). Both are CPU-bound and ran 4 cases in parallel.
  • Mattes MI (warped-CT ~ MR) is an independent, intensity-based confirmation: SimpleITK median 0.643 nats vs ANTs 0.600, and it diverges sharply on the tilted cases — ANTs m6_26TYQI3C drops to 0.094 nats (rotation ~37 mm off) while SimpleITK holds 0.670. A high MI means the CT skull/tissue intensities fall on the matching MR structures after the inverse transform.
  • ANTs MI is stochastic: ANTs' Mattes-MI subsamples the image at random, so its per-case numbers carry run-to-run variance (this table is one draw); SimpleITK's MI (and registration) is deterministic.

Reproducing

# main env — SimpleITK baseline (elastix/deepreg report unavailable here)
source /opt/conda/etc/profile.d/conda.sh && conda activate longitudinal
python scripts/bench_registration.py --pipelines simpleitk --n-cases 10 \
    --workers 4 --threads 24 --simpleitk-mode benchmark

# isolated py3.12 env with antspyx — ANTs (+ optional SimpleITK cross-check)
/home/xfr/.conda/envs/ants312/bin/python scripts/bench_registration.py \
    --pipelines antspyx --n-cases 10 --workers 4 --threads 24

# combine the per-pipeline JSONs accumulated across envs into one table + plot
python scripts/bench_registration.py --merge

Other flags: --sample random --seed 0 (reproducible random 10), --keys k1,k2,... (explicit cases), --n-cases N, --workers W, --threads T, --simpleitk-mode {benchmark,production}, --deepreg-ckpt <.ckpt> (DeepReg), --out-dir <dir>, --qa-images / --qa-out <dir> (visual QA, below).

Visual QA images (checkerboard + color overlay)

--qa-images writes a checkreg-style PNG per (pipeline, case) to <out-dir>/qa/<pipeline>__<case>.png for eyeballing the alignment. Each image is a 3×2 grid: rows are the three orthogonal mid-slices through the head center (mid-axial, mid-coronal, mid-sagittal), and columns are:

  • left — checkerboard blend: alternating blocks of the CT inverse-transformed into MR coordinates and the original MR; a clean registration gives a seamless mosaic, a misregistration shows hard edges at the block boundaries.
  • right — additive color overlay: CT in red, MR in green. Structures that coincide blend to yellow; rotation/translation error appears as red/green fringing at the skull and brain edges.

The CT is shown in a brain window (L40/W80 = 0–80 HU) so the brain parenchyma is visible and comparable to the MR (the skull, ~800+ HU, clips to white); the MR uses a 2–98 % head window. The coronal and sagittal views are oriented superior-up to match the axial view.

The title carries the pipeline, case key, Mattes MI, and the skull-ring gap (mid/std), so a glance tells you both the achieved-metric score and whether the visuals agree. Generate for the two available pipelines with:

/home/xfr/.conda/envs/longitudinal/bin/python scripts/bench_registration.py \
    --pipelines simpleitk --n-cases 10 --qa-images
/home/xfr/.conda/envs/ants312/bin/python scripts/bench_registration.py \
    --pipelines antspyx --n-cases 10 --qa-images

(9 images each; the incomplete-CT case is skipped.)

Enabling the two blocked pipelines

ITK-Elastix — not on PyPI; needs conda-forge (which this host cannot reach). Once conda-forge is available:

conda install -c conda-forge elastix        # provides the elastix/transformix CLI
python scripts/bench_registration.py --pipelines elastix --n-cases 10

The adapter registers fixed=T1c, moving=CT with Rigid3DTransform + Mattes MI on the head box and reads the result transform.

DeepReg (Oxford) — the PyPI deepreg (0.1.2) is an unrelated stub that pins numpy==1.18.5; do not install it. DeepReg is not a one-shot rigid tool: it is a CycleGAN image-to-image registration that must be trained or run from a pretrained checkpoint. To benchmark it:

git clone https://github.com/OB-CVHU/DeepReg && cd DeepReg && pip install -e .
python scripts/bench_registration.py --pipelines deepreg --n-cases 10 \
    --deepreg-ckpt /path/to/pretrained_3d.ckpt

The adapter runs deepreg test to predict a voxel displacement field, converts it to a SimpleITK DisplacementFieldTransform, and scores it like the others.

Notes / caveats

  • The benchmark compares standard single-init configurations (centered start). Production uses --simpleitk-mode production's multi-start tilt grid, which raises SimpleITK's acceptance at ~20× the cost; that is out of scope for the head-to-head but is the config the dataset actually ships with.
  • elastix / deepreg could not be executed here, so their rows are "not installed", not measured. Their code paths are implemented and unit-verified for import/convention; run them where the toolkits are present.
  • fix_orientation (elastix/ants/deepreg) chooses between a transform and its inverse by skull-ring concentricity — a correctness gate that makes the result independent of each toolkit's transform-direction convention.