xfr/longitudinal
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
longitudinal/REGISTRATION_BENCHMARK_DEFORMABLE.md

20 KiB
Raw Blame History

Registration Pipeline Benchmark — CT → T1c (M6) — Deformable

Companion to REGISTRATION_BENCHMARK_RIGID.md: the deformable variant of the same six-pipeline benchmark. The rigid benchmark asked "which toolkit aligns the head best with 6 DOF?"; this one asks "which toolkit's native deformable workflow aligns the head (and the tissue) best?" on the same 10 M6 CT→T1c test cases, the same unified contract, and the same metrics.

Run date: 2026-10-02 (main run 05:43–07:52 local; serial synthmorph re-run 09:58–10:07 local; accelerated simpleitk re-run 15:31–16:26 local). 10 M6 test cases, 3× NVIDIA A100 80GB PCIe, 96-core host, conda env longitudinal-313. Main run: 6 workers (24 threads each, 2 workers per GPU). synthmorph was re-run serially (1 worker, one container per GPU at a time) because the concurrent run lost 6/9 cases to cuDNN scratch-space allocation failures (see the SynthMorph note under Interpretation); the re-run registered 9/9, matching the concurrent run's scores to 4 decimals on the 3 cases it completed. simpleitk was re-run after its spline grid was accelerated from 16 mm to 24 mm (see the simpleitk row of the pipeline table; the original 16 mm results are archived as simpleitk_16mm_legacy.json).

Unified contract

Identical to the rigid benchmark: the returned transform T must satisfy sitk.Resample(ct, t1c, T, ...) = CT on the T1c grid, i.e. T maps T1c → CT coordinates. fixed = T1c, moving = CT, full volumes (no head box, no mask, no crop, no pad), each toolkit's native automatic transform initialization, CT clipped to [0, 80] HU only for the pipelines where the rigid A/B showed clipping helps (niftyreg, synthmorph).

One deformable-specific change: every pipeline's output (elastix BSpline combination, ANTs warp + affine composite, NiftyReg affine + F3D spline, SimpleITK Euler + BSpline, SynthMorph joint field, IMPACT RecursiveBSpline) is rasterised to a SimpleITK DisplacementFieldTransform on the T1c grid before scoring, so all six pipelines return the same transform type and the shared scoring / GTV warping / QA code is used verbatim.

The six pipelines (deformable configurations)

Each pipeline runs its toolkit's native deformable workflow — a linear init stage + a deformable stage, or a single deformable stage where that is the toolkit's canonical setup:

pipeline deformable configuration metric
antspyx type_of_transform='SyN' (ANTs' native deformable): Affine stage (2100×1200×1200×0, shrink 4×2×2×1) + SyN[0.2,3,0] deformable stage (40×20×0, shrink 4×2×1), center-of-mass auto init Mattes MI (32 bins), both stages
elastix two-stage (the toolkit's canonical deformable workflow): EulerTransform rigid stage (2000 iters × 3 resolutions, CenterOfGravity auto init) → BSplineTransform stage (16 mm grid, spline order 3, 2000 iters × 3 resolutions) initialized from stage 1 via -t0 Mattes MI (32 bins), 5000 samples/level
impact the ImpactReg MR_CT_TS reference map in its native deformable form: RecursiveBSplineTransform (14 mm final grid, order 3) driven by the MIND + TotalSegmentator static feature combination (4 per-level scales 400/300/200/200 iters, 2000 samples) with the reference's three-metric combination: IMPACT (w 1.0) + Mattes MI (w 0.4) + TransformBendingEnergyPenalty (w 10). Two M6 adaptations, same as the rigid benchmark: CenterOfGravity auto init (the reference's GeometricalCenter start leaves the heads ~100 mm apart and aborts) and RequiredRatioOfValidSamples 0.1 IMPACT semantic (L1 + Dice) + Mattes MI + bending energy
niftyreg reg_aladin rigid init (3-level pyramid, 15 TLS iters/level) → reg_f3d non-rigid (F3D B-spline Free-Form deformation, Modat et al. 2010; NMI metric, symmetric bending/linear regularisation, 3-level pyramid, 150 iters at the final level, 8 mm final grid) NCC block matching (init) + NMI (F3D)
simpleitk two-stage (the toolkit's canonical deformable demo): Euler3D stage (2000 iters/level, pyramid [4,2,1]) → BSplineTransform3D stage (24 mm grid, 50 iters/level on pyramid [4,2]), the BSpline's control points initialised to reproduce the Euler stage exactly. The 24 mm grid (5577 parameters) is the accelerated setting (the 2026-10-02 original used 16 mm / 13005 parameters, ~61 min/case under the harness): this build's spline per-iteration cost scales ~linearly with the control-point count (~2.8 s/iter at the shrink-4 level and ~20 s/iter at shrink-2 for 24 mm vs ~6.4 / ~45.8 s for 16 mm — independent of the metric sample count, which is fixed in this SimpleITK build; LBFGS2 was tested and fails to move the transform on this noisy MI), and the full re-run keeps the 16 mm quality within 0.003–0.029 MI per case (median 0.698 vs 0.705, ImpactLoss median unchanged) while retaining simpleitk's m6_26TYQI3C MI/ImpactLoss win — see the acceleration note under Interpretation. A full-resolution polish pass (10 iters at shrink-1) was tested and dropped the achieved MI (0.6891 at 32 mm), so the pyramid stays [4,2] Mattes MI (32 bins), both stages
synthmorph joint model (affine + deformable in one symmetric network pass — the toolkit's canonical deformable; the deformable model requires a pre-existing field and would bias the comparison) anatomy-aware contrast-invariant network (pretrained)

Test cases

First 10 M6 labelled cases of m6.discover() (all with a GTV label): m6_22GD5VL2_20230724, m6_22IRRKU6_20240115, m6_23XIMO4M_20230222, m6_24V62QQM_20230223, m6_24V62QQM_20231017, m6_24V62QQM_20240820, m6_26TYQI3C_20221027, m6_2BBN5XA4_20250411, m6_2C4TKI6H_20240506, m6_2CDNN7BH_20240102.

(m6_22GD5VL2_20230724 has an incomplete CT head — a known data issue, not a pipeline failure; it is scored as ct_incomplete in every pipeline, as in the rigid benchmark.)

(Ground-truth reference transforms are not part of this benchmark — scoring is by achieved image similarity only, as in the rigid benchmark.)

Metrics (identical for every pipeline)

  • Mattes MI (nats) between the warped CT and the T1c over the MR head (largest connected component of the T1c mask) — achieved alignment, higher = better.
  • IMPACT loss (vboussot/ImpactLoss; pretrained TotalSegmentator TS/M730 deep features — weighted L1 over the two deepest layers, full volumes; the same scorer as the rigid benchmark) between the warped CT and the T1c — semantic-anatomy similarity, lower = better.
  • Registration wall time (seconds, excludes I/O + scoring).

Per-case JSONs additionally retain the production skull-ring QA (gap + concentricity, and the acceptance verdict) for reference.

Result

Alignment is scored by the achieved Mattes MI (nats, warped-CT ~ T1c over the MR head; higher = better) and the ImpactLoss semantic score (warped-CT ~ T1c over the full volumes; lower = better), on the same unclipped scoring volumes as the rigid benchmark, so all rows are comparable on the same scales and against the rigid numbers. Full volumes, no mask / crop / pad, fixed = T1c / moving = CT (shared rule). niftyreg and synthmorph clip the CT (the moving image) to [0, 80] HU before registration (per the rigid benchmark's clipping A/B); antspyx, elastix, impact and simpleitk register the CT as-is. All six transforms are deformable (each toolkit's native workflow) and are rasterised to T1c-grid displacement fields before scoring.

All six pipelines registered 9/9 cases (the 10th is the known incomplete-CT data issue, unscored in every pipeline). Bold = best for the column (MI higher is better, ImpactLoss lower is better).

pipeline toolkit registered s/reg (med) ImpactLoss (med/mean) MI (med/mean, nats)
antspyx ANTsPy 0.6.3 9 31.7 0.158 / 0.156 0.723 / 0.729
elastix ITK-Elastix 0.25.4 9 45.8 0.164 / 0.164 0.622 / 0.627
impact Docker vboussot/elastix_impact (Elastix 5.2.0 + IMPACT, deformable MIND+TS reference map) 9 273.2 0.159 / 0.159 0.553 / 0.547
niftyreg NiftyReg 2.1.1 (reg_aladin + reg_f3d, Docker + CUDA) 9 38.1 0.161 / 0.161 0.666 / 0.686
simpleitk SimpleITK 2.5.6 9 1369.9 0.158 / 0.159 0.698 / 0.704
synthmorph Docker freesurfer/synthmorph (TensorFlow, pretrained) 9 49.1 0.156 / 0.156 0.627 / 0.640

Per-case MI (nats, unclipped score; bold = best for the case). The synthmorph column is from the serial re-run (see the SynthMorph note under Interpretation):

case antspyx elastix impact niftyreg simpleitk synthmorph
m6_22GD5VL2_20230724 incomplete-CT incomplete-CT incomplete-CT incomplete-CT incomplete-CT incomplete-CT
m6_22IRRKU6_20240115 0.679 0.580 0.548 0.637 0.654 0.620
m6_23XIMO4M_20230222 0.723 0.622 0.573 0.711 0.693 0.618
m6_24V62QQM_20230223 0.789 0.570 0.635 0.729 0.738 0.701
m6_24V62QQM_20231017 0.720 0.622 0.580 0.665 0.692 0.628
m6_24V62QQM_20240820 0.738 0.674 0.462 0.707 0.700 0.666
m6_26TYQI3C_20221027 0.799 0.694 0.537 0.793 0.801 0.709
m6_2BBN5XA4_20250411 0.666 0.585 0.436 0.612 0.641 0.569
m6_2C4TKI6H_20240506 0.719 0.635 0.553 0.656 0.698 0.627
m6_2CDNN7BH_20240102 0.726 0.660 0.602 0.666 0.718 0.624

Per-case ImpactLoss (semantic, unclipped score; bold = best (lowest) for the case):

case antspyx elastix impact niftyreg simpleitk synthmorph
m6_22GD5VL2_20230724 incomplete-CT incomplete-CT incomplete-CT incomplete-CT incomplete-CT incomplete-CT
m6_22IRRKU6_20240115 0.1541 0.1595 0.1532 0.1530 0.1572 0.1512
m6_23XIMO4M_20230222 0.1581 0.1657 0.1567 0.1605 0.1603 0.1573
m6_24V62QQM_20230223 0.1473 0.1634 0.1497 0.1541 0.1502 0.1470
m6_24V62QQM_20231017 0.1518 0.1591 0.1547 0.1620 0.1548 0.1518
m6_24V62QQM_20240820 0.1545 0.1646 0.1694 0.1706 0.1584 0.1562
m6_26TYQI3C_20221027 0.1580 0.1638 0.1656 0.1672 0.1562 0.1602
m6_2BBN5XA4_20250411 0.1624 0.1663 0.1669 0.1648 0.1631 0.1616
m6_2C4TKI6H_20240506 0.1594 0.1640 0.1585 0.1552 0.1626 0.1564
m6_2CDNN7BH_20240102 0.1590 0.1662 0.1597 0.1605 0.1633 0.1583

Interpretation

  • antspyx (Affine + SyN) leads on achieved MI (median 0.723; best on 7 of 9 cases and within 0.02 of best on the other 2) at the best median ImpactLoss of the gradient-based tools (0.158, within 0.002 of synthmorph's 0.156), and it is the fastest full-workflow pipeline (31.7 s/case) — the best time/quality trade of the six.
  • simpleitk is second on MI (0.698; it keeps its per-case win on m6_26TYQI3C — 0.801, still the highest MI of the whole benchmark, though now only 0.001 ahead of antspyx — and cedes its m6_2CDNN7BH win to antspyx, 0.718 vs 0.726) at a cost of ~23 min/case under the harness — ~43× the fastest pipeline after the spline-grid acceleration below (this build's BSpline per-iteration cost scales ~linearly with the control-point count, independent of the metric sample count — see the pipeline table).
  • simpleitk acceleration (2026-10-02): the original setting (16 mm spline grid, 13005 parameters) ran 3658.9 s/case (median) under the harness. The spline stage's per-iteration cost in this build scales ~linearly with the control-point count (16 mm: ~6.4 s/iter at the shrink-4 level, ~45.8 s at shrink-2; 24 mm: ~2.8 / ~20 s; 32 mm: ~1.2 / ~9.6 s), and the metric sample count is fixed in this SimpleITK build (SetMetricAsMattesMutualInformation exposes only the histogram bins); LBFGS2 was tested and its line search rejects every step on this noisy MI (the transform does not move). Switching to the 24 mm grid (5577 parameters) cut the wall to 1369.9 s median (2.7×) at a cost of −0.003…−0.029 MI on 8 of 9 cases (+0.017 on m6_2C4TKI6H; median 0.698 vs 0.705), with the median ImpactLoss unchanged (0.158) and simpleitk's m6_26TYQI3C MI/ImpactLoss wins intact. A 32 mm grid (a further ~1.9×) was rejected: it drops below antspyx's 0.7994 on m6_26TYQI3C (0.7902, and 0.7963 even with 100 fine-level iters). A full-resolution polish pass (10 iters at shrink-1) was also tested and dropped the achieved MI (0.6891 at 32 mm). The original 16 mm results are archived as simpleitk_16mm_legacy.json.
  • niftyreg's reg_f3d stage delivers the largest deformable gain of any pipeline (0.292 → 0.666 median MI, +0.374): it recovers the two gross-offset cases where the rigid block-matching locked onto local minima (m6_24V62QQM_20240820 0.118 → 0.707, m6_2BBN5XA4 0.069 → 0.612), lifts the toolkit to third on MI, and it is the most concentric of the six (median skull-ring gap 5.5 mm mid / 3.8 mm std, 6/9 accepted).
  • The deformable stage does not uniformly pay off. Against the rigid benchmark's per-case MI it improves antspyx (0.652 → 0.723 median), simpleitk (0.647 → 0.698, accelerated 24 mm setting) and niftyreg (0.292 → 0.666), but degrades elastix (0.646 → 0.622; mixed per case, worst regression m6_24V62QQM_20230223 0.698 → 0.570) and impact (0.652 → 0.553 — lower than its own rigid init on all 9 cases). The two degradations are the coarse-grid BSpline configurations (14–16 mm) with heavy bending regularisation — for impact the reference map's TransformBendingEnergyPenalty at weight 10 — so the deformable stage smooths rather than refines, and (for impact) its IMPACT-dominant objective is simply not the achieved-MI objective.
  • impact's deformable reference map is the weakest on the intensity scale (MI 0.553) even though its semantic score is competitive (ImpactLoss 0.159, within 0.003 of the best median) — a metric-objective/achieved-MI divergence the rigid benchmark did not show (there the two scores agreed, with impact in the top tier on both). At 273 s/case it is the worst time/quality trade of the six.
  • synthmorph (serial re-run, 9/9) splits the leaderboards: best on the semantic scale, mid-pack on the intensity scale. Median ImpactLoss 0.156 — the best of the six — with the per-case best on 4 of 9 (plus a 4-decimal tie with antspyx on m6_24V62QQM_20231017), while its MI (0.627) is fourth, statistically tied with elastix (0.622). The pretrained contrast-invariant network optimises learned anatomy rather than raw CT/MR intensity synchrony, so it wins where anatomy matters and loses where the intensity histogram matters. Fast (49.1 s/case) and the read-back cross-check against the tool's own resampling is NCC 1.0 on all 9 cases.
  • The two score families discriminate differently here than in the rigid benchmark. The ImpactLoss scale is nearly saturated (medians 0.156–0.164, ~5 % spread; per-case bests spread across five tools, elastix with 0/9), whereas the MI scale spans 0.553–0.723 (0.17 nats absolute) and the per-case bests go 8/9 antspyx, 1/9 simpleitk, 0/9 the rest. The deformable stage's main effect is fine intensity alignment on top of the gross anatomy the rigid stages already place — exactly what MI measures and ImpactLoss is less sensitive to. For ranking the deformable workflows, achieved MI is the more discriminative score.
  • synthmorph infrastructure note. In the main concurrent run (6 workers, 2 containers per GPU) 6 of 9 cases failed with CUDNN failed to allocate the scratch space ... [Op:Conv3D] inside the joint model's 256-channel 128³ hyperconvolution. Each case is a fresh docker run --rm container, so the failures are cuDNN scratch-space contention when two SynthMorph containers share one A100, not a pipeline fault: the serial re-run registered 9/9, reproduced the concurrent run's MI/ImpactLoss to 4 decimals on its 3 completed cases (deterministic model), and ran ~17 s/case faster without the contention (49.1 s vs 65.7 s median). The concurrent-run JSON is archived as synthmorph_concurrent_run.json.
  • Production skull-ring QA (reference only, kept in the per-case JSONs): acceptance 6/9 niftyreg, 5/9 elastix / synthmorph, 4/9 antspyx / simpleitk, 2/9 impact. Concentricity: niftyreg best (median gap 5.5 mm mid / 3.8 mm std), simpleitk (19.1 mm std, accelerated setting) and impact (18.9 mm) worst on the rotational term. The same weak correlation with perceived alignment noted in the rigid benchmark, so it is reported for reference only.

Transform conventions (deformable-specific)

Unlike the rigid benchmark (where the elastix-style toolkits came out in the opposite direction 8/9 cases and needed the best-of-T vs T⁻¹ orientation gate), every deformable output here is consumed in the documented fixed→moving = T1c→CT direction; the gate is replaced by an analytic read-back plus a skull-ring sanity check. Two non-obvious conventions were established empirically (validated against each tool's own resampling, NCC ≥ 0.998 on the head):

  • NiftyReg and surfa (FreeSurfer's NIfTI stack) decode RAS+ NIfTI qforms naively, i.e. they work in an x/y-mirrored world. A displacement delta (dx, dy, dz) stored by such a tool is (−dx, −dy, dz) in the true RAS world; the field grid (file indices) is unaffected. The read-back flips the x/y components of the dense field. (ANTs writes correct RAS fields — no flip needed.)
  • NiftyReg reg_f3d writes only the control-point grid (outputCPP.nii); reg_transform -disp evaluates it to a dense field on the reference grid. The reg_aladin affine is folded into the control points before optimization (reg_f3d initialises the spline with the affine's deformation field), so the dense field is the full transform — no separate composition step. (The -comp subcommand was tested and does not reproduce reg_f3d's internal transform; it is not used.)
  • SynthMorph joint: driven as register -m joint moving fixed (moving = clipped CT first, fixed = T1c second); the -t output is the displacement field on the fixed (T1c) grid — the contract transform directly. The tool's own -o resampling is written as a cross-check (per-case own_resample_ncc_head in the JSONs).
  • ANTs SyN: the two stage outputs (001Warp.nii.gz + 000GenericAffine.mat) are composed as warp-then-affine (verified to reproduce ANTs' own warpedmovout to < 0.001 MI) and rasterised.
  • Elastix / IMPACT: the combination transform (auto-init folded in) is rasterised — elastix in-process via itk.TransformToDisplacementFieldFilter (exact, machine precision on affine fields), IMPACT by evaluating the BSpline control-point file directly (SimpleITK cannot read elastix's BSpline .txt format; scipy's order-3 B-spline = ITK's kernel).

Reproducing

source /opt/conda/etc/profile.d/conda.sh && conda activate longitudinal-313
cd /mnt/b4/xfr/git26/longitudinal

# all six deformable pipelines (results go to results/bench_registration_deformable/)
# the 2026-10-02 main run used --workers 6 --threads 24 (~2 h 9 min wall,
# dominated by simpleitk's ~111 min phase)
python scripts/bench_registration.py --deformable --workers 6 --threads 24 --qa-images

# single pipeline / case subset
python scripts/bench_registration.py --deformable --pipelines simpleitk impact --n-cases 3

# serial SynthMorph re-run (the 2026-10-02 final numbers): one container per
# GPU at a time; two concurrent containers on one A100 hit cuDNN
# scratch-space OOM in the joint model's 256-ch 128^3 hyperconvolution
python scripts/bench_registration.py --deformable --pipelines synthmorph --workers 1 --threads 24 --qa-images

# combine the per-pipeline JSONs into summary.json / summary.md / plot
python scripts/bench_registration.py --deformable --merge

Visual QA images

--qa-images writes a checkreg-style PNG per (pipeline, case) to results/bench_registration_deformable/qa/<pipeline>__<case>.png (9 per pipeline; the incomplete-CT case is skipped) — same format as the rigid benchmark: a 3×2 grid of the mid-axial / mid-coronal / mid-sagittal slices through the head center, left column checkerboard blend of the warped CT vs the T1c (clean registration = seamless mosaic), right column additive color overlay (CT red, T1c green; agreement = yellow, misregistration = red/green fringing). CT is brain-windowed 0–80 HU, T1c 2–98 % head window; the title carries the pipeline, case key and the achieved Mattes MI.