longitudinal/BENCHMARKING.md

Pipeline benchmark (A / B / C)

Benchmark of the three segmentation pipelines — A in-house Unet3D, B nnU-Net v2 (NTUHLPLTrainer), C MONAI UNet — driven by scripts/benchmark_pipelines.py (DICE, training time, inference time, params, VRAM). Protocol details live in the script docstring; this file records what has been run and what the machine currently supports.

Data (2026-10-04 curation)

M6 labeled set rebuilt 2026-10-04 (scripts/02_build_m6_dataset.py): T1c fixed, best-of-ANTs (ants_rigid / ants_affine / ants_syn, winner by Mattes MI), TV warped onto the T1c grid → 1312 labeled volumes, all 1541 processed (data/proc/<key>{,_label}.nii.gz, 1 mm). Row-level curation lives in src/exclusions.py (applied at scripts/make_tv_slices.py, re-applied after split by 05_build_splits): total TV > 1e6 mm3 (1 row), single-component TV < 100 mm3 (109), non-brain images (33, all ntuh spine/neck/torso; 6 skull-base/posterior-fossa cases kept after expert review — protocol: classify at the mid-cranial slice, not just the TV-max slice). Audit trail: data/manifests/excluded.jsonl (143 rows); pre-exclusion manifests in data/manifests/backup_20261004/.

Current splits (seed 0): 4057 labeled rows → train 3447 / val 205 / test 405 (3771 subjects); unlabeled pool 13280. Tumor volume: median 3212 mm3, p95 23.4k, max 163k (m6 meningioma), zero_frac 0.12%. Benchmark budget: 2750 train rows (80% of split_train), 40 eval rows (10% of split_test), TTA on.

Environment

• monai==1.6.1 in longitudinal-313 (pip --no-deps, torch untouched).
• B's nnU-Net tree is on local disk, not CIFS: nnu_common.nnu_root() honors NNU_ROOT=/home/xfr/longitudinal_nnu (local ZFS) and B's pred I/O honors BENCH_B_DIR=/home/xfr/longitudinal_benchB — the 2026-10-05 CIFS flapping (OSError [Errno 112], stale rmtree → FileExistsError) had killed four B attempts. CIFS is then touched only for the one-time raw reads from /mnt/pve and small log/JSON appends. Hardening: nnu_common.rmtree_verified(), plan retries 2→5, full traceback in failure JSONs.

Headline correction: A's low dice was an inference TTA bug (2026-10-06)

A's persistent low test dice (pmax median ~0.25, "under-confidence state") in the 2026-10-05/06 10–100-epoch runs was a bug in src/training.sliding_window_probs, not a training problem: the old code flipped the whole volume and windowed at the original grid positions, so each region averaged its own model response with its mirror regions' responses — for an off-mirror target the four-view average collapses to ~1/4 of the identity-view probability. C's flip→window→flip-back scheme was already correct, hence unaffected. Evidence (A 100ep model): p = 0.998 at the tumor voxel with TTA off (top-10 activations 10/10 inside the tumor, mean p inside 0.91 vs 0.004 outside), pmax = 0.2498 with TTA on; a BN→InstanceNorm conversion changed nothing (0.2498 → 0.2499). Fix: flip the patch (patch = tf(vol[sl])).

All A checkpoints re-scored on 40 held-out rows (results/benchmark_full/A_reeval_fixed_tta.json):

A checkpoint	dice (old TTA)	dice (fixed)	pmax median	rows > 0.001
100ep nn-loss (AdamW, BN)	0.0047	0.5313	0.9994	30/40
10ep baseline, uniform crops, std loss	0.0026	0.5102	1.0000	31/40
20ep nn-loss (AdamW, BN)	0.0000	0.3644	0.9532	26/40
20ep std-loss + CE w=500	0.0076	0.1466	0.9982	33/40

The 2026-10-05 fg-crop / CE-weight / loss-parity investigations were chasing a symptom of this eval bug — the plain uniform-crop 10ep baseline (0.5102) already beats everything else on record at that budget. (Caveat: that checkpoint predates the current split by a few moved patients — same leakage caveat as --skip-train.)

What pipeline B actually does (2026-10-06 audit)

From the installed nnunetv2 2.8.1 source + B's fold_0/debug.json:

• Loss: DC_and_CE_loss({'batch_dice': True, 'smooth': 1e-5, 'do_bg': False}) = CE + tumor-class-only batch-level soft dice, loss = CE − dice (src.losses.b_dc_and_ce matches nnunetv2's classes to max abs diff 0.0).
• Deep supervision: 4 seg outputs at 1x/1/2/1/4, weights 4/7, 2/7, 1/7 (raw [1, 1/2, 1/4, 1/8] with the bottleneck scale zeroed, renormalized), nearest-neighbor downsampled targets.
• Optimizer: SGD lr 1e-2, momentum 0.99, nesterov, weight_decay 3e-5; PolyLR exponent 0.9 on the global epoch index; grad clip 12.
• Network: PlainConvUNet, InstanceNorm3d(affine, eps 1e-5), LeakyReLU(0.01), conv bias, init kaiming_normal_(a=0.01) + zero bias.
• Data: resampled to spacing [1.0, 0.586, 0.586], ZScore (no mask), patch 64×192×192, batch 2, 250 iterations/epoch, 33% fg patch oversampling.
• Augmentation (per patch): rotation ±30° p=0.2/axis; zoom 0.7–1.4 p=0.2; Gaussian noise p=0.1; blur p=0.2; brightness/contrast p=0.15; low-resolution sim p=0.25; gamma p=0.1 + p=0.3; mirror p=0.5; no elastic.
• "EMA": an EMA (0.9/0.1) of the per-epoch val pseudo-dice, used only to select checkpoint_best.pth — not a weight EMA.
• Epoch semantics: B's epoch = 250 iters × batch 2 = 500 patches; A/C's epoch = full pass = 2750 patches (B @100ep ≈ A/C @~18ep in data seen).

B extended to 100 epochs (2026-10-06, complete 14:59 UTC)

Staged extension from the 20ep checkpoint via scripts/run_b_100ep_stages.sh: --resume --nnu-skip-prep --train-epochs 50 then 100, scoring checkpoint_final.pth (the last-trained model, A/C-comparable) via --nnu-ckpt; stage 3 additionally scored checkpoint_best.pth with --skip-train.

dice @20	dice @50	dice @100 (final)	dice @100 (best)
0.2927	0.3333	0.3912	0.3912 (== final)

B improved monotonically; 32/39 rows > 0.001 @100 (vs 29/39 @20); 138.5 s/epoch stage pace. Artifacts: results/benchmark_full/ benchmark_full_B_{50ep,100ep,100ep_best}.json.

B-aligned A/C (C's default; A's opt-in --b-aligned)

Training C with benchmark_pipelines.py (no config flags) — and training A with --b-aligned — uses B's full protocol on top of the backbone (A: Unet3D(16,4) + LeakyReLU; C: MONAI UNet(16→128)), keeping the benchmark data budget (2750 train / 40 eval rows, 96³, batch 3, bf16): InstanceNorm3d(affine) + kaiming-normal(0.01) init, B's true loss (b_dc_and_ce) with deep supervision (4/7, 2/7, 1/7 on 1x/1/2/1/4, NN downsampled targets), SGD 1e-2 (0.99, nesterov, wd 3e-5) + PolyLR(0.9), B's augmentation + Z-score + 33% fg sampling (src/augment_b.py, patch-level, shared by A and C), grad clip 12, and B-style per-epoch val pseudo-dice (50 val patches, EMA 0.9/0.1) selecting runs_bench/{A,C}/best_ema.pt. Code: src/unet3d.py (norm/deep-sup/init), src/losses.py (b_dc_and_ce, deep_sup_loss), src/dataset.py (zscore/b_aug), scripts_monai/monai_common.py, plus the TTA fix above. A's default is its own A-native config (see the ablation section below); opt-outs: --legacy (pre-2026-10-07 config, approx nn-loss default; best_legacy.pt), --b-loss-only (B's loss, rest legacy; best_bloss.pt).

From-scratch 100-epoch runs (40 held-out test rows, TTA on; A/C epoch = full pass, B epoch = 500 patches):

run (100ep, from scratch)	dice @5	@10	@20	@50	@100
A B-aligned	0.2017	0.3176	0.4368	0.4998	0.5417
A all-default (--legacy --std-loss)	0.3161	0.4578	0.4249	0.5089	0.5508 (A record; see ablation below)
C B-aligned	0.0587	0.1748	0.4073	0.4396	0.4002
C all-default (--legacy --std-loss)	0.2025	0.2209	0.0974	0.2872	0.2083 (see C ablation below)
C b-loss-only (true B loss, rest legacy)	0.2051	0.3267	0.2824	0.1710	0.2523
A legacy (BN, AdamW; re-scored, fixed TTA)	—	—	0.3644	—	0.5313
C legacy (BN, AdamW, approx nn-loss)	0.1889	0.2562	0.3358	0.1798	0.2649
B nnU-Net (reference)	—	—	0.2927	0.3333	0.3912

• Both B-aligned models beat B (and their legacy counterparts) at every point ≥20ep. A: 0.5417 vs 0.5313 legacy — the TTA fix was the big A win, the B protocol adds a smaller one on top. C: 0.4002 vs 0.2649 legacy (+0.14) and the legacy mid-training collapse is GONE — monotone 0.4073 → 0.4396 → 0.4002 (legacy: 0.3358 → 0.1798 → 0.2649). C's slow start (0.0587 @5) is the SGD/PolyLR + instance-norm warm-up.
• A 100→200ep extension (--resume --train-epochs 200 --eval-at "150 200", complete 2026-10-07 05:14 UTC, 139.9 s/epoch): @150 0.5072, @200 0.5397 — no improvement over the @100 B-aligned (0.5417); A is converged. The resume restarts PolyLR at a higher LR over the 200-ep total, costing a @150 dip; val pseudo-dice EMA kept rising (0.3622 @200) while test dice plateaued — the val-EMA proxy over-estimates later epochs. The @200 final save overwrote best_bal.pt/best.pt with the @200 model; it is 0.002 from the @100 B-aligned (within noise), so the @200 weights on disk are the best A B-aligned model.
• The loss alone does NOT fix C's instability: b-loss-only (B's exact loss, legacy BN/AdamW/no deep-sup/no B aug) repeats the legacy dip — statistically the same trajectory as C-legacy. The full package (instance norm + deep supervision + SGD/PolyLR + B augmentation + Z-score) is what stabilizes C.
• B's best-checkpoint heuristic does not transfer: for the B-aligned models the EMA-selected ckpt is WORSE than the final model — A's EMA-best happens to be epoch 100 (= final), but C's EMA-best (epoch 97, val-ema 0.2442) scores 0.2829 on test vs 0.4002 for the epoch-100 model. Use the final checkpoint for A/C.
• pmax saturation is immediate under B-aligned training (A: 0.999 @50; no under-confidence at any point — the old pmax ~0.25 was purely the TTA bug).
• Pace: A-bal ~150 s/epoch, C-bal ~175 s/epoch (patch-level B augmentation; the initial volume-level C attempt ran ~600 s/epoch and was restarted — B's SpatialTransform acts on the sampled patch, so patch-level is the faithful and fast variant).

Artifacts: results/benchmark_full/benchmark_full_{A,C}_bal_100ep.json (+ _per_row.jsonl, progress PNGs), benchmark_full_A_bal_200ep.json; checkpoints runs_bench/C/best.pt (C @100 B-aligned 0.4002, C's record) / best_ema.pt (EMA-selected); A's B-aligned models: runs_bench/A/ best_bal.pt (@200, 0.5397 — the best A B-aligned model) / best_ema_bal.pt.

A ablation: all-default config — now A's default (2026-10-07)

With the TTA fixed, A B-aligned (0.5417 @100ep) vs A legacy (0.5313 @100ep, approx nn-loss) differs by many components at once. This run trained A with every hyperparameter at A's default, none nnU-derived — the fully A-native baseline, whose mid-training trajectory was never recorded (the old 100ep legacy run's @5/@10/@50 evals were scored with the buggy TTA):

BatchNorm + PyTorch default init (no kaiming), ReLU, per-sample 0.5CE+0.5mean-dice loss (no deep supervision — the --legacy default loss is the nnU-style approx nn_dc_and_ce, so the run used --std-loss to make the loss A-native too), AdamW 3e-4 (wd 1e-4) + warmup/cosine, grad clip 5.0, percentile [0,1] input (no Z-score), legacy augmentation (flip/rot90/brightness/noise), 50% fg-centered crops (not B's 33%). Benchmark constants unchanged for comparability: 2750 train / 40 eval rows, batch 3, 96³, bf16, TTA on, eval at 5/10/20/50. (Run as --legacy --std-loss; since the promotion below it is A's plain default.)

Complete 2026-10-07 ~09:45 UTC (persistent background process on GPU 0, shared with co-tenant jobs; 125.1 s/epoch, train VRAM 1.98 GB, 2.78 s/vol TTA; log logs/bench_A_legacy_std_100ep.log):

dice (40 held-out rows, TTA on)	@5	@10	@20	@50	@100
A all-default (--legacy --std-loss)	0.3161	0.4578	0.4249	0.5089	0.5508 (A record)
A B-aligned	0.2017	0.3176	0.4368	0.4998	0.5417

• A's own defaults beat B's protocol at every eval point — final 0.5508 (30/40 rows > 0.001, pmax median 0.9989) vs 0.5417 B-aligned; 0.5508 is the best result on record for any pipeline (B 0.3912, C 0.4002). Loss 0.386 → 0.161, flat from ~epoch 40. The earlier "B protocol adds +0.01 for A" (0.5417 vs 0.5313) was against the legacy config with the nnU-style approx loss: the only difference between the 0.5313 model and this run is the loss, so A's per-sample 0.5CE+0.5dice alone is worth +0.0195 — and the full B package (IN + deep sup + SGD/PolyLR + B aug + Z-score + 33% fg) nets −0.009 on this backbone. Contrast C, where the same B package was worth +0.14 (0.4002 vs 0.2649) and remains C's record and default.
• pmax saturation is immediate (0.81 @5 → 0.9998 @10) — no under-confidence at any point; with the TTA fix, A's A-native config is the best A result on record.
• Checkpoint: the 100ep all-default model (17.6 MB). The prior legacy 100ep model (0.5313) had already been lost to a smoke-run overwrite; its number is recorded in results/benchmark_full/A_reeval_fixed_tta.json.

Promoted to A's default (2026-10-07). Running benchmark_pipelines.py with no config flags now trains A with this A-native config and writes runs_bench/A/best.pt (the 0.5508 model is there); --b-aligned opts A into B's protocol (best_bal.pt), --legacy gives the pre-2026-10-07 A config (approx nn-loss; best_legacy.pt), --b-loss-only the B-loss ablation (best_bloss.pt), and deviating A knobs (--a-leaky, --a-fg-*) write best_alt.pt so the record checkpoint is never clobbered. C's default is unchanged (B-aligned, runs_bench/C/best.pt). Verified: all four A modes + C --legacy smoke-tested (1 epoch each, config/loss/ckpt recorded in the JSONs), and a no-op --resume --train-epochs 100 on the new default re-scored 0.5508 exactly (30/40 rows, pmax 0.9989).

C ablation: all-default config (2026-10-07)

The mirror of A's ablation on the MONAI backbone: C trained with every hyperparameter at C's own default (its original study protocol from scripts_monai/07_train.py, nothing nnU-derived) — BatchNorm UNet(16→128) with MONAI default init, LeakyReLU (C's own default act), per-sample 0.5CE+0.5mean-dice loss (WeightedDiceCELoss, no deep supervision), AdamW 3e-4 (wd 1e-4) + warmup/cosine, grad clip 5.0, percentile [0,1] input (no Z-score), legacy augmentation (flip/rot90/brightness/noise), 50% fg-centered crops. Benchmark constants unchanged: 2750 train / 40 eval rows, batch 3, 96³, bf16, TTA on, eval at 5/10/20/50. Run as --legacy --std-loss (C's default remains B-aligned).

Complete 2026-10-07 ~18:31 UTC (persistent background process on GPU 0; 254.3 s/epoch, train VRAM 0.48 GB, 0.39 s/vol TTA; 1-epoch smoke first, config recorded in benchmark_full_C_legacy_std_smoke1.json):

dice (40 held-out rows, TTA on)	@5	@10	@20	@50	@100
C all-default (--legacy --std-loss)	0.2025	0.2209	0.0974	0.2872	0.2083
C B-aligned (record)	0.0587	0.1748	0.4073	0.4396	0.4002
C legacy (approx nn-loss)	0.1889	0.2562	0.3358	0.1798	0.2649
C b-loss-only	0.2051	0.3267	0.2824	0.1710	0.2523

• The non-B package is unstable for C whatever loss it carries: this run dips @10→@20 (0.2209 → 0.0974) and ends 0.2083 — the worst of the three legacy-family C configs (nn-loss 0.2649, b-loss-only 0.2523) — even though the TRAIN loss is smooth (0.35 → 0.15, no collapse; min 0.153 @88): the test dice oscillates, the same family as C-legacy's 0.3358 → 0.1798 @20→@50 dip. B-aligned beats it at every point ≥20ep (+0.31 @20, +0.15 @50, +0.19 @100).
• Mirror image of A: on A's backbone the all-default config is A's best (0.5508; every A variant sits in 0.53–0.55), on C's backbone it is C's worst (0.2083; every non-B C variant sits in 0.21–0.27). The B package (instance norm + deep supervision + SGD/PolyLR + B augmentation + Z-score) is not a preference — it is what stabilizes the MONAI UNet. And A's winning loss switch (per-sample 0.5CE+0.5dice, +0.0195 for A) does not transfer: for C it is the worst legacy-family result.
• Checkpoint: the @100 model (14.4 MB) is runs_bench/C/best_legacy.pt — it overwrote the prior C legacy nn-loss model (0.2649; number recorded in benchmark_full_C_nnloss_100ep.json). C's record B-aligned model (best.pt, 0.4002) is untouched.

Artifacts: results/benchmark_full/benchmark_full_C_legacy_std_100ep.json (+ _per_row.jsonl, progress PNG).

Resuming training (--resume)

--train-epochs is the total budget (not epochs added).

• A / C: resume from the latest checkpoint of the same config mode (model + optimizer + scheduler when present; A's B-aligned PolyLR is stateless — the LR is recomputed from the epoch index over the new total, same staged convention as B's extensions). A: best.pt (A-native default) / best_bal.pt (--b-aligned) / best_legacy.pt (--legacy) / best_bloss.pt (--b-loss-only); C: best.pt (B-aligned default) / best_legacy.pt / best_bloss.pt. Checkpoints written before 2026-10-04 carry model + epoch only and fall back to fresh optimizer/scheduler. A run is a no-op train (eval only) when the checkpoint already reached the total.
• B: appends nnU-Net's --c (continue fold_0 from checkpoint_latest.pth, native optimizer + LR-schedule restore) and NNUNet_pl_EPOCHS = --train-epochs; pair with --nnu-skip-prep so the preprocessed tree isn't --cleaned. Completed runs delete checkpoint_latest.pth — copy checkpoint_final.pth → checkpoint_latest.pth first to resume (same state dict; used for B's 20→50→100ep stages).

Notes

• Per-pipeline defaults (since 2026-10-07): A = A-native all-defaults (the 0.5508 @100ep record; B's protocol is --b-aligned), C = B-aligned (the 0.4002 @100ep record; --legacy reverts). Checkpoint conventions: see the A and C ablation sections.
• Study entrypoints match the record configs (2026-10-07): scripts/07_train.py (A) was already A-native all-defaults; scripts_monai/07_train.py (C) now trains B-aligned by default (its record config) — --legacy reverts to the pre-2026-10-07 config. C checkpoints carry a b_aligned marker that 03/04 read for the Z-score input; in C's B-aligned study mode pseudo rows enter at full weight (row "w" and --lr unused — the same treatment Pipeline B's study gives them; per-sample down-weighting is --legacy only).
• GPUs: 3× 80 GB available; the benchmark uses one (--gpu).
• Budget knobs: --train-epochs (default 100), --max-rows (default 80% of split_train = 2750, also B's --nnu-cases), --max-eval-rows (default 10% of split_test = 40), --no-tta.
• --skip-train is A only: runs/round0/best.pt (trained 2026-09-25 on the previous split — a few patients moved between train/test, small leakage caveat).
• B's one-time cost scales with --nnu-cases (plan_and_preprocess ≈ 3.5 s/case at 24 cases with 8 preprocess procs).
• Historical runs (10/20-epoch legacy A/B/C, nn-loss experiments, smoke run) remain in results/benchmark_full/ and results/backup_20261004/; superseded analyses (fg-crop/CE-weight, loss parity) are omitted from this file — see the TTA-bug section for the correction.