longitudinal/BENCHMARKING.md

# Pipeline benchmark (A / B / C)

Benchmark of the three segmentation pipelines — A in-house Unet3D, B nnU-Net v2
(NTUHLPLTrainer), C MONAI UNet — driven by `scripts/benchmark_pipelines.py`
(DICE, training time, inference time, params, VRAM). Protocol details live in
the script docstring; this file records what has been run and what the machine
currently supports.

## Data (2026-10-04 curation)

M6 labeled set rebuilt 2026-10-04 (`scripts/02_build_m6_dataset.py`): T1c
fixed, best-of-ANTs (`ants_rigid` / `ants_affine` / `ants_syn`, winner by
Mattes MI), TV warped onto the T1c grid → 1312 labeled volumes, all 1541
processed (`data/proc/<key>{,_label}.nii.gz`, 1 mm). Row-level curation lives
in `src/exclusions.py` (applied at `scripts/make_tv_slices.py`, re-applied
after split by `05_build_splits`): total TV > 1e6 mm3 (1 row),
single-component TV < 100 mm3 (109), non-brain images (33, all ntuh
spine/neck/torso; 6 skull-base/posterior-fossa cases kept after expert
review — protocol: classify at the mid-cranial slice, not just the TV-max
slice). Audit trail: `data/manifests/excluded.jsonl` (143 rows);
pre-exclusion manifests in `data/manifests/backup_20261004/`.

Current splits (seed 0): **4057** labeled rows → train 3447 / val 205 / test
405 (3771 subjects); unlabeled pool 13280. Tumor volume: median 3212 mm3,
p95 23.4k, max 163k (m6 meningioma), zero_frac 0.12%. Benchmark budget:
**2750** train rows (80% of split_train), **40** eval rows (10% of
split_test), TTA on.

## Environment

- `monai==1.6.1` in `longitudinal-313` (pip `--no-deps`, torch untouched).
- **B's nnU-Net tree is on local disk, not CIFS**: `nnu_common.nnu_root()`
  honors `NNU_ROOT=/home/xfr/longitudinal_nnu` (local ZFS) and B's pred I/O
  honors `BENCH_B_DIR=/home/xfr/longitudinal_benchB` — the 2026-10-05 CIFS
  flapping (`OSError [Errno 112]`, stale `rmtree` → `FileExistsError`) had
  killed four B attempts. CIFS is then touched only for the one-time raw
  reads from `/mnt/pve` and small log/JSON appends. Hardening:
  `nnu_common.rmtree_verified()`, plan retries 2→5, full `traceback` in
  failure JSONs.

## Headline correction: A's low dice was an inference TTA bug (2026-10-06)

A's persistent low test dice (pmax median ~0.25, "under-confidence state") in
the 2026-10-05/06 10–100-epoch runs was a bug in
`src/training.sliding_window_probs`, not a training problem: the old code
flipped the **whole volume** and windowed at the **original grid positions**,
so each region averaged its own model response with its mirror regions'
responses — for an off-mirror target the four-view average collapses to ~1/4
of the identity-view probability. C's flip→window→flip-back scheme was
already correct, hence unaffected. Evidence (A 100ep model): p = 0.998 at
the tumor voxel with TTA off (top-10 activations 10/10 inside the tumor,
mean p inside 0.91 vs 0.004 outside), pmax = 0.2498 with TTA on; a
BN→InstanceNorm conversion changed nothing (0.2498 → 0.2499). Fix: flip the
**patch** (`patch = tf(vol[sl])`).

All A checkpoints re-scored on 40 held-out rows
(`results/benchmark_full/A_reeval_fixed_tta.json`):

| A checkpoint | dice (old TTA) | dice (fixed) | pmax median | rows > 0.001 |
|---|---|---|---|---|
| 100ep nn-loss (AdamW, BN) | 0.0047 | **0.5313** | 0.9994 | 30/40 |
| 10ep baseline, uniform crops, std loss | 0.0026 | **0.5102** | 1.0000 | 31/40 |
| 20ep nn-loss (AdamW, BN) | 0.0000 | 0.3644 | 0.9532 | 26/40 |
| 20ep std-loss + CE w=500 | 0.0076 | 0.1466 | 0.9982 | 33/40 |

The 2026-10-05 fg-crop / CE-weight / loss-parity investigations were chasing
a symptom of this eval bug — the plain uniform-crop 10ep baseline (0.5102)
already beats everything else on record at that budget. (Caveat: that
checkpoint predates the current split by a few moved patients — same leakage
caveat as `--skip-train`.)

## What pipeline B actually does (2026-10-06 audit)

From the installed nnunetv2 2.8.1 source + B's `fold_0/debug.json`:

- **Loss**: `DC_and_CE_loss({'batch_dice': True, 'smooth': 1e-5, 'do_bg':
  False})` = CE + tumor-class-only batch-level soft dice, `loss = CE − dice`
  (`src.losses.b_dc_and_ce` matches nnunetv2's classes to max abs diff 0.0).
- **Deep supervision**: 4 seg outputs at 1x/1/2/1/4, weights 4/7, 2/7, 1/7
  (raw [1, 1/2, 1/4, 1/8] with the bottleneck scale zeroed, renormalized),
  nearest-neighbor downsampled targets.
- **Optimizer**: SGD lr 1e-2, momentum 0.99, nesterov, weight_decay 3e-5;
  PolyLR exponent 0.9 on the global epoch index; grad clip 12.
- **Network**: PlainConvUNet, InstanceNorm3d(affine, eps 1e-5),
  LeakyReLU(0.01), conv bias, init kaiming_normal_(a=0.01) + zero bias.
- **Data**: resampled to spacing [1.0, 0.586, 0.586], ZScore (no mask),
  patch 64×192×192, batch 2, 250 iterations/epoch, 33% fg patch oversampling.
- **Augmentation** (per patch): rotation ±30° p=0.2/axis; zoom 0.7–1.4
  p=0.2; Gaussian noise p=0.1; blur p=0.2; brightness/contrast p=0.15;
  low-resolution sim p=0.25; gamma p=0.1 + p=0.3; mirror p=0.5; no elastic.
- **"EMA"**: an EMA (0.9/0.1) of the per-epoch val pseudo-dice, used only to
  select `checkpoint_best.pth` — not a weight EMA.
- **Epoch semantics**: B's epoch = 250 iters × batch 2 = 500 patches; A/C's
  epoch = full pass = 2750 patches (B @100ep ≈ A/C @~18ep in data seen).

## B extended to 100 epochs (2026-10-06, complete 14:59 UTC)

Staged extension from the 20ep checkpoint via
`scripts/run_b_100ep_stages.sh`: `--resume --nnu-skip-prep
--train-epochs 50` then `100`, scoring `checkpoint_final.pth` (the
last-trained model, A/C-comparable) via `--nnu-ckpt`; stage 3 additionally
scored `checkpoint_best.pth` with `--skip-train`.

| dice @20 | dice @50 | dice @100 (final) | dice @100 (best) |
|---|---|---|---|
| 0.2927 | 0.3333 | **0.3912** | 0.3912 (== final) |

B improved monotonically; 32/39 rows > 0.001 @100 (vs 29/39 @20);
138.5 s/epoch stage pace. Artifacts: `results/benchmark_full/
benchmark_full_B_{50ep,100ep,100ep_best}.json`.

## B-aligned A/C (C's default; A's opt-in `--b-aligned`)

Training C with `benchmark_pipelines.py` (no config flags) — and training A
with `--b-aligned` — uses B's full protocol on top of the backbone (A:
`Unet3D(16,4)` + LeakyReLU; C: MONAI `UNet(16→128)`), keeping the benchmark
data budget (2750 train / 40 eval rows, 96³, batch 3, bf16):
InstanceNorm3d(affine) + kaiming-normal(0.01) init, B's true loss
(`b_dc_and_ce`) with deep supervision (4/7, 2/7, 1/7 on 1x/1/2/1/4, NN
downsampled targets), SGD 1e-2 (0.99, nesterov, wd 3e-5) + PolyLR(0.9), B's
augmentation + Z-score + 33% fg sampling (`src/augment_b.py`, patch-level,
shared by A and C), grad clip 12, and B-style per-epoch val pseudo-dice (50
val patches, EMA 0.9/0.1) selecting `runs_bench/{A,C}/best_ema.pt`. Code:
`src/unet3d.py` (norm/deep-sup/init), `src/losses.py` (b_dc_and_ce,
deep_sup_loss), `src/dataset.py` (zscore/b_aug),
`scripts_monai/monai_common.py`, plus the TTA fix above. A's default is its
own A-native config (see the ablation section below); opt-outs: `--legacy`
(pre-2026-10-07 config, approx nn-loss default; `best_legacy.pt`),
`--b-loss-only` (B's loss, rest legacy; `best_bloss.pt`).

From-scratch 100-epoch runs (40 held-out test rows, TTA on; A/C epoch = full
pass, B epoch = 500 patches):

| run (100ep, from scratch) | dice @5 | @10 | @20 | @50 | @100 |
|---|---|---|---|---|---|
| **A B-aligned** | 0.2017 | 0.3176 | 0.4368 | 0.4998 | **0.5417** |
| **A all-default** (`--legacy --std-loss`) | 0.3161 | 0.4578 | 0.4249 | 0.5089 | **0.5508** (A record; see ablation below) |
| **C B-aligned** | 0.0587 | 0.1748 | 0.4073 | 0.4396 | **0.4002** |
| **C all-default** (`--legacy --std-loss`) | 0.2025 | 0.2209 | 0.0974 | 0.2872 | **0.2083** (see C ablation below) |
| C b-loss-only (true B loss, rest legacy) | 0.2051 | 0.3267 | 0.2824 | 0.1710 | 0.2523 |
| A legacy (BN, AdamW; re-scored, fixed TTA) | — | — | 0.3644 | — | 0.5313 |
| C legacy (BN, AdamW, approx nn-loss) | 0.1889 | 0.2562 | 0.3358 | 0.1798 | 0.2649 |
| B nnU-Net (reference) | — | — | 0.2927 | 0.3333 | 0.3912 |

- **Both B-aligned models beat B (and their legacy counterparts) at every
  point ≥20ep.** A: 0.5417 vs 0.5313 legacy — the TTA fix was the big A win,
  the B protocol adds a smaller one on top. C: 0.4002 vs 0.2649 legacy
  (+0.14) and the legacy mid-training collapse is GONE — monotone
  0.4073 → 0.4396 → 0.4002 (legacy: 0.3358 → 0.1798 → 0.2649). C's slow
  start (0.0587 @5) is the SGD/PolyLR + instance-norm warm-up.
- **A 100→200ep extension** (`--resume --train-epochs 200 --eval-at "150
200"`, complete 2026-10-07 05:14 UTC, 139.9 s/epoch): @150 0.5072, @200
0.5397 — no improvement over the @100 B-aligned (0.5417); A is converged.
   The resume restarts PolyLR at a higher LR over the 200-ep total, costing
   a @150 dip; val pseudo-dice EMA kept rising (0.3622 @200) while test dice
   plateaued — the val-EMA proxy over-estimates later epochs. The @200 final
   save overwrote `best_bal.pt`/`best.pt` with the @200 model; it is 0.002
   from the @100 B-aligned (within noise), so the @200 weights on disk are
   the best A B-aligned model.
- **The loss alone does NOT fix C's instability**: b-loss-only (B's exact
  loss, legacy BN/AdamW/no deep-sup/no B aug) repeats the legacy dip —
  statistically the same trajectory as C-legacy. The full package (instance
  norm + deep supervision + SGD/PolyLR + B augmentation + Z-score) is what
  stabilizes C.
- **B's best-checkpoint heuristic does not transfer**: for the B-aligned
  models the EMA-selected ckpt is WORSE than the final model — A's EMA-best
  happens to be epoch 100 (= final), but C's EMA-best (epoch 97, val-ema
  0.2442) scores 0.2829 on test vs 0.4002 for the epoch-100 model. Use the
  final checkpoint for A/C.
- pmax saturation is immediate under B-aligned training (A: 0.999 @50; no
  under-confidence at any point — the old pmax ~0.25 was purely the TTA bug).
- Pace: A-bal ~150 s/epoch, C-bal ~175 s/epoch (patch-level B augmentation;
  the initial volume-level C attempt ran ~600 s/epoch and was restarted — B's
  SpatialTransform acts on the sampled patch, so patch-level is the faithful
  and fast variant).

Artifacts: `results/benchmark_full/benchmark_full_{A,C}_bal_100ep.json`
(+ `_per_row.jsonl`, progress PNGs), `benchmark_full_A_bal_200ep.json`;
checkpoints `runs_bench/C/best.pt` (C @100 B-aligned 0.4002, C's record) /
`best_ema.pt` (EMA-selected); A's B-aligned models: `runs_bench/A/
best_bal.pt` (@200, 0.5397 — the best A B-aligned model) /
`best_ema_bal.pt`.

## A ablation: all-default config — now A's default (2026-10-07)

With the TTA fixed, A B-aligned (0.5417 @100ep) vs A legacy (0.5313 @100ep,
approx nn-loss) differs by many components at once. This run trained A with
**every hyperparameter at A's default, none nnU-derived** — the fully
A-native baseline, whose mid-training trajectory was never recorded (the old
100ep legacy run's @5/@10/@50 evals were scored with the buggy TTA):

BatchNorm + PyTorch default init (no kaiming), ReLU, per-sample
0.5CE+0.5mean-dice loss (no deep supervision — the `--legacy` default loss
is the nnU-style approx `nn_dc_and_ce`, so the run used `--std-loss` to make
the loss A-native too), AdamW 3e-4 (wd 1e-4) + warmup/cosine, grad clip 5.0,
percentile [0,1] input (no Z-score), legacy augmentation
(flip/rot90/brightness/noise), 50% fg-centered crops (not B's 33%).
Benchmark constants unchanged for comparability: 2750 train / 40 eval rows,
batch 3, 96³, bf16, TTA on, eval at 5/10/20/50. (Run as
`--legacy --std-loss`; since the promotion below it is A's plain default.)

**Complete 2026-10-07 ~09:45 UTC** (persistent background process on GPU 0,
shared with co-tenant jobs; 125.1 s/epoch, train VRAM 1.98 GB, 2.78 s/vol
TTA; log `logs/bench_A_legacy_std_100ep.log`):

| dice (40 held-out rows, TTA on) | @5 | @10 | @20 | @50 | @100 |
|---|---|---|---|---|---|
| **A all-default** (`--legacy --std-loss`) | 0.3161 | 0.4578 | 0.4249 | 0.5089 | **0.5508** (A record) |
| A B-aligned | 0.2017 | 0.3176 | 0.4368 | 0.4998 | 0.5417 |

- **A's own defaults beat B's protocol at every eval point** — final
  0.5508 (30/40 rows > 0.001, pmax median 0.9989) vs 0.5417 B-aligned;
  0.5508 is the best result on record for any pipeline (B 0.3912, C
  0.4002). Loss 0.386 → 0.161, flat from ~epoch 40. The earlier "B
  protocol adds +0.01 for A" (0.5417 vs 0.5313) was against the legacy
  config with the nnU-style approx loss: the only difference between the
  0.5313 model and this run is the loss, so A's per-sample 0.5CE+0.5dice
  alone is worth +0.0195 — and the full B package (IN + deep sup +
  SGD/PolyLR + B aug + Z-score + 33% fg) nets −0.009 on this backbone.
  Contrast C, where the same B package was worth +0.14 (0.4002 vs 0.2649)
  and remains C's record and default.
- pmax saturation is immediate (0.81 @5 → 0.9998 @10) — no
  under-confidence at any point; with the TTA fix, A's A-native config is
  the best A result on record.
- Checkpoint: the 100ep all-default model (17.6 MB). The prior legacy 100ep
  model (0.5313) had already been lost to a smoke-run overwrite; its number
  is recorded in `results/benchmark_full/A_reeval_fixed_tta.json`.

**Promoted to A's default (2026-10-07).** Running `benchmark_pipelines.py`
with no config flags now trains A with this A-native config and writes
`runs_bench/A/best.pt` (the 0.5508 model is there); `--b-aligned` opts A
into B's protocol (`best_bal.pt`), `--legacy` gives the pre-2026-10-07 A
config (approx nn-loss; `best_legacy.pt`), `--b-loss-only` the B-loss
ablation (`best_bloss.pt`), and deviating A knobs (`--a-leaky`,
`--a-fg-*`) write `best_alt.pt` so the record checkpoint is never
clobbered. C's default is unchanged (B-aligned, `runs_bench/C/best.pt`).
Verified: all four A modes + C `--legacy` smoke-tested (1 epoch each,
config/loss/ckpt recorded in the JSONs), and a no-op
`--resume --train-epochs 100` on the new default re-scored 0.5508 exactly
(30/40 rows, pmax 0.9989).

## C ablation: all-default config (2026-10-07)

The mirror of A's ablation on the MONAI backbone: C trained with every
hyperparameter at C's own default (its original study protocol from
`scripts_monai/07_train.py`, nothing nnU-derived) — BatchNorm
`UNet(16→128)` with MONAI default init, LeakyReLU (C's own default act),
per-sample 0.5CE+0.5mean-dice loss (WeightedDiceCELoss, no deep
supervision), AdamW 3e-4 (wd 1e-4) + warmup/cosine, grad clip 5.0,
percentile [0,1] input (no Z-score), legacy augmentation
(flip/rot90/brightness/noise), 50% fg-centered crops. Benchmark constants
unchanged: 2750 train / 40 eval rows, batch 3, 96³, bf16, TTA on, eval at
5/10/20/50. Run as `--legacy --std-loss` (C's default remains B-aligned).

**Complete 2026-10-07 ~18:31 UTC** (persistent background process on GPU 0;
254.3 s/epoch, train VRAM 0.48 GB, 0.39 s/vol TTA; 1-epoch smoke first,
config recorded in `benchmark_full_C_legacy_std_smoke1.json`):

| dice (40 held-out rows, TTA on) | @5 | @10 | @20 | @50 | @100 |
|---|---|---|---|---|---|
| **C all-default** (`--legacy --std-loss`) | 0.2025 | 0.2209 | 0.0974 | 0.2872 | **0.2083** |
| C B-aligned (record) | 0.0587 | 0.1748 | 0.4073 | 0.4396 | **0.4002** |
| C legacy (approx nn-loss) | 0.1889 | 0.2562 | 0.3358 | 0.1798 | 0.2649 |
| C b-loss-only | 0.2051 | 0.3267 | 0.2824 | 0.1710 | 0.2523 |

- **The non-B package is unstable for C whatever loss it carries**: this
  run dips @10→@20 (0.2209 → 0.0974) and ends 0.2083 — the worst of the
  three legacy-family C configs (nn-loss 0.2649, b-loss-only 0.2523) — even
  though the TRAIN loss is smooth (0.35 → 0.15, no collapse; min 0.153 @88):
  the test dice oscillates, the same family as C-legacy's
  0.3358 → 0.1798 @20→@50 dip. B-aligned beats it at every point ≥20ep
  (+0.31 @20, +0.15 @50, +0.19 @100).
- **Mirror image of A**: on A's backbone the all-default config is A's best
  (0.5508; every A variant sits in 0.53–0.55), on C's backbone it is C's
  worst (0.2083; every non-B C variant sits in 0.21–0.27). The B package
  (instance norm + deep supervision + SGD/PolyLR + B augmentation + Z-score)
  is not a preference — it is what stabilizes the MONAI UNet. And A's
  winning loss switch (per-sample 0.5CE+0.5dice, +0.0195 for A) does not
  transfer: for C it is the worst legacy-family result.
- Checkpoint: the @100 model (14.4 MB) is `runs_bench/C/best_legacy.pt` —
  it overwrote the prior C legacy nn-loss model (0.2649; number recorded in
  `benchmark_full_C_nnloss_100ep.json`). C's record B-aligned model
  (`best.pt`, 0.4002) is untouched.

Artifacts: `results/benchmark_full/benchmark_full_C_legacy_std_100ep.json`
(+ `_per_row.jsonl`, progress PNG).

## Resuming training (`--resume`)

`--train-epochs` is the **total budget** (not epochs added).

- **A / C**: resume from the latest checkpoint of the **same config mode**
  (model + optimizer + scheduler when present; A's B-aligned PolyLR is
  stateless — the LR is recomputed from the epoch index over the new total,
  same staged convention as B's extensions). A: `best.pt` (A-native default)
  / `best_bal.pt` (`--b-aligned`) / `best_legacy.pt` (`--legacy`) /
  `best_bloss.pt` (`--b-loss-only`); C: `best.pt` (B-aligned default) /
  `best_legacy.pt` / `best_bloss.pt`. Checkpoints written before
  2026-10-04 carry model + epoch only and fall back to fresh
  optimizer/scheduler. A run is a no-op train (eval only) when the
  checkpoint already reached the total.
- **B**: appends nnU-Net's `--c` (continue `fold_0` from
  `checkpoint_latest.pth`, native optimizer + LR-schedule restore) and
  `NNUNet_pl_EPOCHS = --train-epochs`; pair with `--nnu-skip-prep` so the
  preprocessed tree isn't `--clean`ed. Completed runs delete
  `checkpoint_latest.pth` — copy `checkpoint_final.pth` →
  `checkpoint_latest.pth` first to resume (same state dict; used for B's
  20→50→100ep stages).

## Notes

- Per-pipeline defaults (since 2026-10-07): A = A-native all-defaults (the
  0.5508 @100ep record; B's protocol is `--b-aligned`), C = B-aligned (the
  0.4002 @100ep record; `--legacy` reverts). Checkpoint conventions: see the
  A and C ablation sections.
- Study entrypoints match the record configs (2026-10-07): `scripts/07_train.py`
  (A) was already A-native all-defaults; `scripts_monai/07_train.py`
  (C) now trains B-aligned by default (its record config) — `--legacy` reverts
  to the pre-2026-10-07 config. C checkpoints carry a `b_aligned` marker that
  03/04 read for the Z-score input; in C's B-aligned study mode pseudo rows
  enter at full weight (row "w" and `--lr` unused — the same treatment
  Pipeline B's study gives them; per-sample down-weighting is `--legacy` only).
- GPUs: 3× 80 GB available; the benchmark uses one (`--gpu`).
- Budget knobs: `--train-epochs` (default 100), `--max-rows` (default 80% of
  split_train = 2750, also B's `--nnu-cases`), `--max-eval-rows` (default 10%
  of split_test = 40), `--no-tta`.
- `--skip-train` is A only: `runs/round0/best.pt` (trained 2026-09-25 on the
  previous split — a few patients moved between train/test, small leakage
  caveat).
- B's one-time cost scales with `--nnu-cases` (plan_and_preprocess ≈ 3.5
  s/case at 24 cases with 8 preprocess procs).
- Historical runs (10/20-epoch legacy A/B/C, nn-loss experiments, smoke run)
  remain in `results/benchmark_full/` and `results/backup_20261004/`;
  superseded analyses (fg-crop/CE-weight, loss parity) are omitted from this
  file — see the TTA-bug section for the correction.