Furen Xiao
a9a4d9bba0
Replace the conda-forge-dependent Elastix CLI path with the pip-installable ITK-Elastix Python components so the benchmark can run elastix in the main environment. Add SimpleITK/ITK image and transform conversion helpers, and use the elastix 5.x EulerTransform rigid pipeline with Mattes MI and ASGD. Update REGISTRATION_BENCHMARK.md and bench_registration.py documentation to reflect the new elastix availability, results, and multi-environment merge workflow. Ignore local elastix output directories in .gitignore.
250 lines
No EOL
14 KiB
Markdown
250 lines
No EOL
14 KiB
Markdown
# Registration Pipeline Benchmark — CT → T1c (M6)
|
||
|
||
Benchmarks **four rigid, cross-modality registration pipelines** that align a
|
||
head-and-neck CT to a tightly-cropped T1c MRI so the GTV can be warped onto the
|
||
T1c (MR) native grid. This is the registration step of
|
||
`scripts/02_build_m6_dataset.py`. The pairs are hard: different modality (CT
|
||
Hounsfield vs MR intensity) and the patient is repositioned between the CT and
|
||
MR sessions, so the inter-session head tilt can reach ~30°.
|
||
|
||
- Pipelines: [`src/registration_pipelines.py`](src/registration_pipelines.py)
|
||
- Benchmark harness: [`scripts/bench_registration.py`](scripts/bench_registration.py)
|
||
- Results: `results/bench_registration/` (per-pipeline JSON, `summary.json`,
|
||
`summary.md`, `bench_registration.png`)
|
||
|
||
## Unified contract
|
||
|
||
Every pipeline returns a SimpleITK transform `T` such that
|
||
|
||
```python
|
||
sitk.Resample(ct, t1c, T, sitk.sitkLinear) # CT onto the MR grid
|
||
sitk.Resample(tv, t1c, T, sitk.sitkNearestNeighbor, 0) # GTV onto the MR grid
|
||
```
|
||
|
||
places the CT / GTV concentric with the MR head. In `Resample`'s
|
||
destination→source convention this `T` maps **T1c → CT**. All pipelines register
|
||
on the same CT-space head box (MI computed on head tissue, not neck/air) and,
|
||
for the single-init tools, from the **same centroid-aligned start**
|
||
(`centroid_offset`), so the comparison is on the optimizer / metric / transform
|
||
handling — not on who finds the ~30–60 mm gross offset first.
|
||
|
||
## The four pipelines
|
||
|
||
| # | pipeline | toolkit | metric | transform | status in this env |
|
||
|---|----------|---------|--------|-----------|--------------------|
|
||
| 1 | `elastix` | ITK-Elastix 0.25.4 (pip `itk` + `itk-elastix`, elastix 5.3.1) | Mattes MI (10 bins) | `EulerTransform` rigid (5.x replacement for `Rigid3DTransform`), ASGD, 3 resolutions | ✅ ran (main env, in-process python API — no CLI) |
|
||
| 2 | `simpleitk` | SimpleITK 2.5.6 | Mattes MI (32 bins) | `Euler3D` rigid, pyramid `[4,2,1]` | ✅ ran |
|
||
| 3 | `antspyx` | ANTsPy / ANTs 0.6.3 | Mattes MI (multi-res) | ANTs `Rigid` | ✅ ran (isolated py3.12 env) |
|
||
| 4 | `deepreg` | Oxford DeepReg (CycleGAN) | CycleGAN + reg. loss (pretrained) | 3D displacement field | **not installed** (see below) |
|
||
|
||
`simpleitk` has two budgets: `--simpleitk-mode benchmark` (single pyramid,
|
||
centered start — the default, comparable to the other tools) and
|
||
`--simpleitk-mode production` (the full multi-start tilt grid from
|
||
`02_build_m6_dataset.py`, ~20× slower, the most robust but not used for the
|
||
head-to-head).
|
||
|
||
## Test cases
|
||
|
||
First 10 labeled M6 cases (`--n-cases 10 --sample first`), reproducible. 9 have
|
||
a complete CT; 1 (`m6_22GD5VL2_20230724`) has an incomplete CT head and is
|
||
reported as `incomplete-CT` for every pipeline (a data issue, not a pipeline
|
||
failure).
|
||
|
||
```
|
||
m6_22GD5VL2_20230724 (incomplete CT)
|
||
m6_22IRRKU6_20240115 m6_23XIMO4M_20230222 m6_24V62QQM_20230223
|
||
m6_24V62QQM_20231017 m6_24V62QQM_20240820 m6_26TYQI3C_20221027
|
||
m6_2BBN5XA4_20250411 m6_2C4TKI6H_20240506 m6_2CDNN7BH_20240102
|
||
```
|
||
|
||
## Metrics (identical for every pipeline)
|
||
|
||
Production QA, crop-independent:
|
||
|
||
- **`skull_ring_gap_mm`** — march rays from the MR head center; the gap between
|
||
the MR head silhouette boundary and the warped-CT skull ring (p97 of positive
|
||
CT) at mid and low levels, plus the angular std. **Aligned ⇔
|
||
`0 ≤ mid ≤ 20 mm` and `std ≤ 12 mm`.** `mid` measures translational
|
||
concentricity; `std` measures rotational concentricity.
|
||
- **`tv_in_ct_frac`** — fraction of the dilated warped GTV inside the warped-CT
|
||
head tissue (the slab must straddle the lesion). Acceptance needs `≥ 0.5`.
|
||
- **`tv_nvox`** — warped GTV volume.
|
||
- **`accepted`** — `tv_nvox > 0 and tv_in_ct_frac ≥ 0.5 and aligned` (the
|
||
production acceptance rule).
|
||
- **`mi_ct_mr`** — mean Mattes MI (nats) between the **CT inverse-transformed
|
||
into MR coordinates** (the warped CT on the T1c grid) and the **original MR**,
|
||
restricted to the MR head and each image percentile-binned independently
|
||
(SimpleITK Mattes-MI style). This is the registration's *achieved* metric read
|
||
off as a standalone alignment score: higher = the warped-CT and MR intensities
|
||
are better in-sync = better aligned; it collapses toward 0 when the CT is
|
||
rotated/translated away from the MR.
|
||
- **`reg_wall_s`** — wall-clock registration time (excludes I/O + scoring).
|
||
|
||
## Results (10 cases)
|
||
|
||
| pipeline | toolkit | registered | accepted | gap mid (med/mean) | gap std (med/mean) | s/reg (med) | tv-in-CT | MI (med/mean, nats) |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| **simpleitk** | SimpleITK 2.5.6 | 9 | **4/10 (0.40)** | 12.6 / 16.1 | 15.4 / 16.0 | 8.6 | 0.93 | **0.643 / 0.630** |
|
||
| **antspyx** | ANTsPy 0.6.3 | 9 | **1/10 (0.10)** | 14.2 / 17.3 | 24.8 / 22.4 | 12.7 | 0.93 | **0.600 / 0.523** |
|
||
| **elastix** | ITK-Elastix 0.25.4 | 9 | **5/10 (0.50)** | 15.3 / 16.1 | 9.9 / 13.8 | 3.2 | 0.93 | **0.562 / 0.549** |
|
||
| deepreg | — | — | — | — | — | — | — | — |
|
||
|
||
Per-case (mid / std, `OK` = accepted):
|
||
|
||
| case | simpleitk | antspyx | elastix |
|
||
|---|---|---|---|
|
||
| m6_22GD5VL2_20230724 | incomplete-CT | incomplete-CT | incomplete-CT |
|
||
| m6_22IRRKU6_20240115 | 25.7 / 29.3 | 26.3 / 29.2 | 26.4 / 30.3 |
|
||
| m6_23XIMO4M_20230222 | 22.2 / 23.9 | 23.8 / 25.0 | 21.1 / 23.3 |
|
||
| m6_24V62QQM_20230223 | 10.7 / 16.9 | 9.4 / 18.2 | 10.6 / 8.2 **OK** |
|
||
| m6_24V62QQM_20231017 | 11.5 / 7.3 **OK** | 12.5 / 12.8 | 11.0 / 6.1 **OK** |
|
||
| m6_24V62QQM_20240820 | 11.6 / 9.6 **OK** | 14.8 / 11.7 **OK** | 15.3 / 14.0 |
|
||
| m6_26TYQI3C_20221027 | 12.2 / 3.7 **OK** | 6.1 / 37.2 | 11.4 / 3.2 **OK** |
|
||
| m6_2BBN5XA4_20250411 | 18.6 / 8.0 **OK** | 14.3 / 12.7 | 18.1 / 9.9 **OK** |
|
||
| m6_2C4TKI6H_20240506 | 12.6 / 15.4 | 13.6 / 14.7 | 12.0 / 6.8 **OK** |
|
||
| m6_2CDNN7BH_20240102 | 19.4 / 30.2 | 18.2 / 33.6 | 18.9 / 22.7 |
|
||
|
||
### Interpretation
|
||
|
||
- **Translation is recovered well by all three** (mid gaps mostly 6–20 mm,
|
||
within the `≤ 20 mm` band). `tv-in-CT` is identical (~0.93): the warped GTV
|
||
lands inside the CT head for every pipeline, so the target is in the right
|
||
place.
|
||
- **The binding failure is rotation** — `std > 12 mm` on the repositioned cases
|
||
(the ~30° inter-session tilt produces MI local minima that a single centered
|
||
start can lock onto). This is exactly why production `02_build_m6_dataset.py`
|
||
uses a **multi-start tilt grid**.
|
||
- **Elastix is the most accurate of the three standard single-init configs**
|
||
(5/10 accepted, lowest median std 9.9 vs SimpleITK 15.4 and ANTs 24.8) and
|
||
the fastest (3.2 s/case median wall, ~1.8 s pure registration vs 8.6 s
|
||
SimpleITK / 12.7 s ANTs): ASGD's adaptive step sizes climb out of small
|
||
misalignments better than RSGD/ANTs on these boxes. It accepts `m6_24V62QQM_20230223`
|
||
(std 8.2) and `m6_2C4TKI6H` (std 6.8), which the other two tools fail on
|
||
(std 16.9 / 15.4), and misses `m6_24V62QQM_20240820` (std 14.0) that both
|
||
others accept — different optimizers settle in different local minima on the
|
||
hard tilted cases.
|
||
- **ANTs' Rigid is the weakest** (1/10 accepted): it frequently locks onto the
|
||
tilted local minimum. Note `m6_26TYQI3C`: ANTs nailed the position (mid 6.1)
|
||
but missed the orientation (std 37.2) — a textbook tilted local minimum.
|
||
- **Speed** (median wall, 4 cases in parallel, 24 threads): Elastix ~3.2 s
|
||
(warm workers; a worker's first case pays a ~10 s warm-up), SimpleITK ~8.6 s,
|
||
ANTs ~12.7 s (includes the pre-shift resample + its affine-MI init stages).
|
||
All CPU-bound.
|
||
- **Mattes MI (warped-CT ~ MR) is an independent, intensity-based confirmation**:
|
||
medians 0.643 (SimpleITK) / 0.600 (ANTs) / 0.562 (Elastix), and it diverges
|
||
sharply on the tilted cases — ANTs `m6_26TYQI3C` drops to 0.094 nats
|
||
(rotation ~37 mm off) while SimpleITK holds 0.670. A high MI means the CT
|
||
skull/tissue intensities fall on the matching MR structures after the inverse
|
||
transform.
|
||
- **Stochasticity**: ANTs' Mattes-MI subsamples the image at random, so its
|
||
per-case numbers carry run-to-run variance (this table is one draw);
|
||
SimpleITK's MI (and registration) is deterministic at fixed thread count
|
||
(RSGD wall time is thread-sensitive, convergence is not); Elastix's ASGD is
|
||
deterministic given its fixed seed.
|
||
|
||
## Reproducing
|
||
|
||
```bash
|
||
# main env — SimpleITK + Elastix (itk-elastix is pip-installed here;
|
||
# antspyx/deepreg report unavailable and are recorded as such)
|
||
source /opt/conda/etc/profile.d/conda.sh && conda activate longitudinal
|
||
python scripts/bench_registration.py --n-cases 10 \
|
||
--workers 4 --threads 24 --simpleitk-mode benchmark
|
||
|
||
# isolated py3.12 env with antspyx — ANTs (+ optional SimpleITK cross-check)
|
||
/home/xfr/.conda/envs/ants312/bin/python scripts/bench_registration.py \
|
||
--pipelines antspyx --n-cases 10 --workers 4 --threads 24
|
||
|
||
# combine the per-pipeline JSONs accumulated across envs into one table + plot
|
||
python scripts/bench_registration.py --merge
|
||
```
|
||
|
||
Other flags: `--sample random --seed 0` (reproducible random 10),
|
||
`--keys k1,k2,...` (explicit cases), `--n-cases N`, `--workers W`,
|
||
`--threads T`, `--simpleitk-mode {benchmark,production}`,
|
||
`--deepreg-ckpt <.ckpt>` (DeepReg), `--out-dir <dir>`,
|
||
`--qa-images` / `--qa-out <dir>` (visual QA, below).
|
||
|
||
## Visual QA images (checkerboard + color overlay)
|
||
|
||
`--qa-images` writes a `checkreg`-style PNG per (pipeline, case) to
|
||
`<out-dir>/qa/<pipeline>__<case>.png` for eyeballing the alignment. Each image
|
||
is a 3×2 grid: **rows** are the three orthogonal mid-slices through the head
|
||
center (**mid-axial, mid-coronal, mid-sagittal**), and **columns** are:
|
||
|
||
- **left — checkerboard blend:** alternating blocks of the CT inverse-transformed
|
||
into MR coordinates and the original MR; a clean registration gives a seamless
|
||
mosaic, a misregistration shows hard edges at the block boundaries.
|
||
- **right — additive color overlay:** CT in red, MR in green. Structures that
|
||
coincide blend to yellow; rotation/translation error appears as red/green
|
||
fringing at the skull and brain edges.
|
||
|
||
The CT is shown in a **brain window (L40/W80 = 0–80 HU)** so the brain
|
||
parenchyma is visible and comparable to the MR (the skull, ~800+ HU, clips to
|
||
white); the MR uses a 2–98 % head window. The coronal and sagittal views are
|
||
oriented superior-up to match the axial view.
|
||
|
||
The title carries the pipeline, case key, Mattes MI, and the skull-ring gap
|
||
(mid/std), so a glance tells you both the achieved-metric score and whether the
|
||
visuals agree. Generate for the available pipelines with:
|
||
|
||
```bash
|
||
/home/xfr/.conda/envs/longitudinal/bin/python scripts/bench_registration.py \
|
||
--pipelines simpleitk elastix --n-cases 10 --qa-images
|
||
/home/xfr/.conda/envs/ants312/bin/python scripts/bench_registration.py \
|
||
--pipelines antspyx --n-cases 10 --qa-images
|
||
```
|
||
|
||
(9 images each; the incomplete-CT case is skipped.)
|
||
|
||
## Enabling the blocked pipeline
|
||
|
||
**DeepReg (Oxford)** — the PyPI `deepreg` (0.1.2) is an unrelated stub that pins
|
||
`numpy==1.18.5`; do not install it. DeepReg is not a one-shot rigid tool: it is a
|
||
CycleGAN image-to-image registration that must be **trained or run from a
|
||
pretrained checkpoint**. To benchmark it:
|
||
|
||
```bash
|
||
git clone https://github.com/OB-CVHU/DeepReg && cd DeepReg && pip install -e .
|
||
python scripts/bench_registration.py --pipelines deepreg --n-cases 10 \
|
||
--deepreg-ckpt /path/to/pretrained_3d.ckpt
|
||
```
|
||
|
||
The adapter runs `deepreg test` to predict a voxel displacement field, converts
|
||
it to a SimpleITK `DisplacementFieldTransform`, and scores it like the others.
|
||
|
||
## Notes / caveats
|
||
|
||
- The benchmark compares **standard single-init** configurations (centered
|
||
start). Production uses `--simpleitk-mode production`'s multi-start tilt grid,
|
||
which raises SimpleITK's acceptance at ~20× the cost; that is out of scope for
|
||
the head-to-head but is the config the dataset actually ships with.
|
||
- `deepreg` could not be executed here, so its row is "not installed", not
|
||
measured; its code path is implemented and unit-verified for import/convention.
|
||
- `fix_orientation` (elastix/ants/deepreg) chooses between a transform and its
|
||
inverse by skull-ring concentricity — a correctness gate that makes the result
|
||
independent of each toolkit's transform-direction convention. In practice it
|
||
never fired (all `auto_inverted: False`): elastix/ants returned the T1c→CT
|
||
direction as expected.
|
||
- **Elastix specifics** (ITK-Elastix 0.25.4, elastix 5.3.1, in-process python
|
||
API — no CLI binary involved):
|
||
- Elastix 5.x renamed components vs the classic parameter files: the rigid
|
||
3D transform is `EulerTransform` (6-DOF, `Rigid3DTransform` is gone from
|
||
the component set), the metric key is `Metric` (not `ImageMetric`), the
|
||
random sampler is `Random` (not `RandomSampler`). The adapter uses the 5.x
|
||
names and is not compatible with elastix ≤ 4.x parameter files.
|
||
- The initial transform (the shared `centroid_offset` start) is passed as an
|
||
`InitialTransformParameterObject` (a `TranslationTransform` with
|
||
`NumberOfParameters 3`); the C++ filter only wires it up when an
|
||
**output directory is set** (`SetOutputDirectory`), so the adapter uses a
|
||
scratch dir under `data/m6reg/elastix/itkelastix/`.
|
||
- `WriteResultImage` must be the string `"false"` (Python `False`); the
|
||
default (absent) makes the filter raise "Could not read result image"
|
||
after a *successful* registration.
|
||
- The result transform is recovered by point-mapping
|
||
(`GetCombinationTransform().TransformPoint`) into an affine — independent
|
||
of elastix's Euler-angle parameterization — and wrapped in a SimpleITK
|
||
`AffineTransform(3)`.
|
||
- Registration wall time is ~1.8 s/case at full budget (ASGD 500/500/250
|
||
over 3 resolutions); each worker's first case pays a ~10 s warm-up, so the
|
||
median wall is 3.2 s and the mean is inflated by warm-up. |