longitudinal/BENCHMARKING.md

Pipeline benchmark (A / B / C)

Benchmark of the three segmentation pipelines — A in-house Unet3D, B nnU-Net v2 (NTUHLPLTrainer), C MONAI UNet — driven by scripts/benchmark_pipelines.py (DICE, training time, inference time, params, VRAM). Protocol details live in the script docstring; this file records what has been run and what the machine currently supports.

Data chain status (2026-10-04)

The M6 labeled set was fully rebuilt on 2026-10-04 (see scripts/02_build_m6_dataset.py): T1c fixed, best-of-ANTs (ants_rigid / ants_affine / ants_syn, winner by Mattes MI), TV warped onto the T1c grid with SMD+linear. After the repair pass:

• data/manifests/m6_labeled.jsonl: 1312 volumes (was 1094 pre-repair), m6_unlabeled.jsonl: 229
• all 1541 m6 processed volumes (data/proc/<key>{,_label}.nii.gz, 1mm cropped/normalized) regenerated, 0 errors
• patient-level splits rebuilt (scripts/05_build_splits.py, seed 0): 4200 labeled rows (ntuh + m6) → train 3570 / val 211 / test 419 volumes, 3893 subjects; unlabeled pool 13280
• tumor volume stats: median 3116 mm3, p95 24.8k, max 3.2M, zero_frac 0.12%

Dataset exclusions (2026-10-04)

Row-level curation lives in src/exclusions.py and is applied at the first stage by scripts/make_tv_slices.py (right after the rows are fetched from the sources, before any split): it classifies every labeled volume and writes its TV-slice QA figure to data/tv_slices/inclusion/ (kept) or data/tv_slices/exclusion/ (excluded, reason in the title). 05_build_splits re-applies the same rules after split assignment (kept rows retain their split; the subject-level shuffle boundaries are unchanged):

• total TV > 1e6 mm3: 1 row (ntuh_6019528, label outlines the full skull)
• single target (1 connected component) with TV < 100 mm3: 109 rows
• non-brain images: 33 rows (all ntuh — spine/neck/torso CyberKnife cases whose series names are indistinguishable from brain T1c; identified by visual review of the TV-slice figures, now in data/tv_slices/exclusion/)

Audit trail: data/manifests/excluded.jsonl (143 rows, reason + split); pre-exclusion manifests in data/manifests/backup_20261004/. New dataset: 4057 labeled rows → train 3447 / val 205 / test 405 (3771 subjects); unlabeled pool unchanged (13280); volume stats now median 3212 mm3, p95 23.4k, max 163k (m6 meningioma), zero_frac still 0.12% (5 rows, e.g. ntuh_3436472 ×3).

After the initial 39-row non-brain list, expert inspection overturned 6 (skull-base / posterior-fossa brain targets that look like cervical spine at the TV-max slice: ntuh_5902230, ntuh_5231609, ntuh_5083686, ntuh_5157194, ntuh_5303758, ntuh_6373987). The remaining 33 were re-audited at the mid-cranial slice (widest head-mask axial) — all still non-brain. Review protocol now recorded in src/exclusions.py: classify at full resolution on the mid-cranial slice, not just the TV-max slice.

Smoke run (2026-10-04)

Small/fast end-to-end check that all three pipelines run on the rebuilt data (1 epoch, 24 train rows, 3 eval rows, TTA on, GPU 0):

python scripts/benchmark_pipelines.py --gpu 0 --max-rows 24 --nnu-cases 24 \
    --max-eval-rows 3 --train-epochs 1 --out results/benchmark_pipelines_smoke.json

pipeline	wall	s/epoch	tr VRAM	s/vol (TTA)	in VRAM	params	weights	dice (1 epoch)
A Unet3D(16, depth 4)	17 s	2.5 (8 steps)	2.0 GB	4.12	0.5 GB	1.46 M	5.9 MB	0.0000
B nnU-Net 3d_fullres	285 s (incl. 83 s plan&preprocess, 4 s raw build)	73.3	9.8 GB	34.73	1.8 GB	30.8 M	246 MB	0.0000
C MONAI UNet(16→128)	5 s	2.5 (8 steps)	1.7 GB	0.33	0.5 GB	1.19 M	4.8 MB	0.0005

DICE ≈ 0 is expected (models trained for a single epoch on 24 volumes); the point of the run was validating the full chain — new m6 native/processed manifests, split membership, B's ntuh + m6_labeled native join, nnU-Net isolated dataset 221 build, inference + dice scoring. All three passed. Eval rows for the smoke run were the first three split_test volumes (all ntuh). Artifacts: results/benchmark_pipelines_smoke.json, _per_row.jsonl, _bench.png.

Environment fixes made during the check

• monai was missing from longitudinal-313 (pipeline C failed with ModuleNotFoundError) — installed pip install monai==1.6.1 --no-deps (torch 2.14+cu126 untouched); import monai verified.

Full-dataset 10-epoch run (2026-10-04) — interim

--train-epochs 10, default budgets (2856 train rows = 80% of split_train, 42 eval rows = 10% of split_test, TTA on), A/C/GPU0·GPU2 finished, B still preprocessing at time of writing:

python scripts/benchmark_pipelines.py --pipelines A --gpu 0 --train-epochs 10 \
    --out results/benchmark_full_A.json      # done
python scripts/benchmark_pipelines.py --pipelines C --gpu 2 --train-epochs 10 \
    --out results/benchmark_full_C.json      # done
python scripts/benchmark_pipelines.py --pipelines B --gpu 1 --train-epochs 10 \
    --out results/benchmark_full_B.json      # in progress (nnU-Net prep)

pipeline	train s/epoch	wall	tr VRAM	s/vol (TTA)	in VRAM	params	weights	dice @10ep
A Unet3D(16, depth 4)	258.8 (952 steps)	46 min	2.0 GB	3.51 ± 0.93	0.5 GB	1.46 M	17.6 MB	0.0008
C MONAI UNet(16→128)	284.3 (952 steps)	48 min	0.4 GB	0.38 ± 0.06	0.5 GB	1.19 M	14.3 MB	0.2119
B nnU-Net 3d_fullres	(pending)	(pending)	—	—	—	30.8 M	246 MB	(pending)

Dice distribution over 41 timed test vols: A p25/p75 = 0.0000/0.0000 (98% ≈ 0); C p25/p75 = 0.0000/0.4048 (29% ≈ 0). Training loss (epoch 1 → 10): A 0.335 → 0.209, C 0.403 → 0.188. 10 epochs is far below convergence for these sparse targets (median tumor 3.1 k mm³); the A/C numbers here are intermediate, and C is already clearly ahead at this budget. Checkpoints at runs_bench/{A,C}/best.pt (extend with --resume --train-epochs <total>). Note: B's first plan_and_preprocess attempt died and the built-in 60 s retry re-ran it successfully.

Resuming training (--resume)

All three pipelines can continue from their latest benchmark checkpoint:

python scripts/benchmark_pipelines.py --pipelines "A B C" --resume \
    --train-epochs 200 --nnu-skip-prep

• --train-epochs is the total budget (not epochs added). Each pipeline trains from where its checkpoint left off up to that total.
• A / C: load runs_bench/{A,C}/best.pt (model + optimizer + scheduler state — checkpoints written before 2026-10-04 carry model + epoch only and fall back to fresh optimizer/scheduler) and run the remaining epochs; the run is a no-op train (eval only) when the checkpoint already reached the total.
• B: appends nnU-Net's --c flag (continue fold_0 from checkpoint_latest.pth, native optimizer + LR-schedule restore) and NNUNet_pl_EPOCHS = --train-epochs. Pair with --nnu-skip-prep so the preprocessed tree isn't --cleaned; the isolated Dataset221 is rebuilt otherwise and a finished run's checkpoint_latest.pth is gone, so a completed run + a bigger budget retrains B from scratch (interrupted runs keep checkpoint_latest.pth and resume correctly).
• Verified 2026-10-04: A/C resumed epoch 1 → 3 (model restored, fresh opt/sched for the old-format ckpt), re-run at 3/3 trained 0 epochs, B continued from epoch 1 with LR restored mid-decay (0.00536 @ start of epoch 2).

Readiness notes for a full run

• Budget knobs: --train-epochs (default 100), --max-rows (default 80% of split_train ≈ 2753 for A/C and for B's --nnu-cases, post-exclusion; ≈ 2856 pre-exclusion), --max-eval-rows (default 10% of split_test ≈ 40 post-exclusion), --no-tta.
• --skip-train is only usable for A (runs/round0/best.pt, trained 2026-09-25 on the previous split — a few patients moved between train/test in the rebuild, so its test dice carries a small leakage caveat). No study checkpoints exist for B (Dataset210) or C (runs_monai/round0/best.pt); they must be trained.
• GPUs: 3× 80 GB available; the benchmark uses one (--gpu).
• B's one-time cost scales with --nnu-cases (plan_and_preprocess ≈ 3.5 s/case at 24 cases with 8 preprocess procs, plus training time at the planned batch/patch).