xfr/longitudinal
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
longitudinal/BENCHMARKING.md

14 KiB
Raw Blame History

Pipeline benchmark (A / B / C)

Benchmark of the three segmentation pipelines — A in-house Unet3D, B nnU-Net v2 (NTUHLPLTrainer), C MONAI UNet — driven by scripts/benchmark_pipelines.py (DICE, training time, inference time, params, VRAM). Protocol details live in the script docstring; this file records what has been run and what the machine currently supports.

Data chain status (2026-10-04)

The M6 labeled set was fully rebuilt on 2026-10-04 (see scripts/02_build_m6_dataset.py): T1c fixed, best-of-ANTs (ants_rigid / ants_affine / ants_syn, winner by Mattes MI), TV warped onto the T1c grid with SMD+linear. After the repair pass:

  • data/manifests/m6_labeled.jsonl: 1312 volumes (was 1094 pre-repair), m6_unlabeled.jsonl: 229
  • all 1541 m6 processed volumes (data/proc/<key>{,_label}.nii.gz, 1mm cropped/normalized) regenerated, 0 errors
  • patient-level splits rebuilt (scripts/05_build_splits.py, seed 0): 4200 labeled rows (ntuh + m6) → train 3570 / val 211 / test 419 volumes, 3893 subjects; unlabeled pool 13280
  • tumor volume stats: median 3116 mm3, p95 24.8k, max 3.2M, zero_frac 0.12%

Dataset exclusions (2026-10-04)

Row-level curation lives in src/exclusions.py and is applied at the first stage by scripts/make_tv_slices.py (right after the rows are fetched from the sources, before any split): it classifies every labeled volume and writes its TV-slice QA figure to data/tv_slices/inclusion/ (kept) or data/tv_slices/exclusion/ (excluded, reason in the title). 05_build_splits re-applies the same rules after split assignment (kept rows retain their split; the subject-level shuffle boundaries are unchanged):

  • total TV > 1e6 mm3: 1 row (ntuh_6019528, label outlines the full skull)
  • single target (1 connected component) with TV < 100 mm3: 109 rows
  • non-brain images: 33 rows (all ntuh — spine/neck/torso CyberKnife cases whose series names are indistinguishable from brain T1c; identified by visual review of the TV-slice figures, now in data/tv_slices/exclusion/)

Audit trail: data/manifests/excluded.jsonl (143 rows, reason + split); pre-exclusion manifests in data/manifests/backup_20261004/. New dataset: 4057 labeled rows → train 3447 / val 205 / test 405 (3771 subjects); unlabeled pool unchanged (13280); volume stats now median 3212 mm3, p95 23.4k, max 163k (m6 meningioma), zero_frac still 0.12% (5 rows, e.g. ntuh_3436472 ×3).

After the initial 39-row non-brain list, expert inspection overturned 6 (skull-base / posterior-fossa brain targets that look like cervical spine at the TV-max slice: ntuh_5902230, ntuh_5231609, ntuh_5083686, ntuh_5157194, ntuh_5303758, ntuh_6373987). The remaining 33 were re-audited at the mid-cranial slice (widest head-mask axial) — all still non-brain. Review protocol now recorded in src/exclusions.py: classify at full resolution on the mid-cranial slice, not just the TV-max slice.

Smoke run (2026-10-04)

Small/fast end-to-end check that all three pipelines run on the rebuilt data (1 epoch, 24 train rows, 3 eval rows, TTA on, GPU 0):

python scripts/benchmark_pipelines.py --gpu 0 --max-rows 24 --nnu-cases 24 \
    --max-eval-rows 3 --train-epochs 1 --out results/benchmark_pipelines_smoke.json
pipeline wall s/epoch tr VRAM s/vol (TTA) in VRAM params weights dice (1 epoch)
A Unet3D(16, depth 4) 17 s 2.5 (8 steps) 2.0 GB 4.12 0.5 GB 1.46 M 5.9 MB 0.0000
B nnU-Net 3d_fullres 285 s (incl. 83 s plan&preprocess, 4 s raw build) 73.3 9.8 GB 34.73 1.8 GB 30.8 M 246 MB 0.0000
C MONAI UNet(16→128) 5 s 2.5 (8 steps) 1.7 GB 0.33 0.5 GB 1.19 M 4.8 MB 0.0005

DICE ≈ 0 is expected (models trained for a single epoch on 24 volumes); the point of the run was validating the full chain — new m6 native/processed manifests, split membership, B's ntuh + m6_labeled native join, nnU-Net isolated dataset 221 build, inference + dice scoring. All three passed. Eval rows for the smoke run were the first three split_test volumes (all ntuh). Artifacts: results/benchmark_pipelines_smoke.json, _per_row.jsonl, _bench.png.

Environment fixes made during the check

  • monai was missing from longitudinal-313 (pipeline C failed with ModuleNotFoundError) — installed pip install monai==1.6.1 --no-deps (torch 2.14+cu126 untouched); import monai verified.

Full-dataset 10-epoch run (2026-10-05, updated dataset)

--train-epochs 10, default budgets on the current curated splits: 2750 train rows = 80% of split_train (3438 labeled), 40 eval rows = 10% of split_test (404), TTA on. The 2026-10-04 interim run (pre-exclusion budget 2856/42: A 0.0008, C 0.2119) is superseded; its JSONs are kept in results/backup_20261004/ (smoke B checkpoint in nnu/results/backup_20261004_Dataset221_smoke/).

python scripts/benchmark_pipelines.py --pipelines C --gpu 2 --train-epochs 10 \
    --out results/benchmark_full_C.json      # done
python scripts/benchmark_pipelines.py --pipelines A --gpu 0 --train-epochs 10 \
    --a-fg-ce-weight 500 --out results/benchmark_full_A.json   # done
# A baseline (uniform crops) on this dataset: dice 0.0026, see table;
# checkpoint runs_bench/A/best_baseline_10ep_uniform_crop.pt
NNU_ROOT=/home/xfr/longitudinal_nnu BENCH_B_DIR=/home/xfr/longitudinal_benchB \
python scripts/benchmark_pipelines.py --pipelines B --gpu 1 --train-epochs 10 \
    --out results/benchmark_full_B.json      # done (local-disk NNU_ROOT, see below)
pipeline train s/epoch wall tr VRAM s/vol (TTA) in VRAM params weights dice @10ep
C MONAI UNet(16→128) 256.7 (916 steps) 43 min 0.4 GB 0.37 ± 0.06 0.5 GB 1.19 M 14.3 MB 0.2162
A Unet3D(16, depth 4), improved (fg crops + CE w=500) 232.3 (916 steps) 41 min 2.0 GB 3.60 ± 0.76 0.5 GB 1.46 M 17.6 MB 0.0369
A baseline (uniform crops, unweighted loss) 216.6 (916 steps) 45 min 2.0 GB 3.33 ± 0.86 0.5 GB 1.46 M 17.6 MB 0.0026
B nnU-Net 3d_fullres 331.5 (incl. 275-case val/epoch) ~2.5 h (raw 6 m, prep 62 m, train 55 m, infer 14 m) 9.3 GB 21.84 1.8 GB 30.8 M 246 MB 0.1424 (hard 0.1424)

C: 25/39 timed test vols with dice > 0.001 (max 0.7392); loss 0.403 → 0.182. 10 epochs is far below convergence for these sparse targets (median tumor 3.2 k mm³); A/C numbers at this budget are intermediate. Checkpoints at runs_bench/{A,C}/best.pt (extend with --resume --train-epochs <total>).

A: why dice ≈ 0, and the fix (2026-10-05)

The A baseline predicts all-background (38/39 test rows dice exactly 0.0). Root-cause work:

  1. Uniform random 96³ crops starve the loss of foreground. A 96³ patch is 885 k voxels; the median tumor is ~4.6 k mm³ (0.02–0.05% of a head volume). Simulated uniform crops contain the whole blob in only 8–37% of patches and a median crop often has 0 tumor voxels. Pipeline C's RandCropByPosNegLabeld(pos=1, neg=1) centers half its patches on the tumor — ~4× more foreground supervision per epoch.
  2. Fg-aware crops alone are not enough: A under-confides. With the crop protocol fixed (src/dataset.py), A's loss matches C's at every epoch (0.193 vs 0.182 @10ep) yet p_tumor maxes at 0.43 (C: ~1.0): it localizes (top-k dice 0.48 on best rows) but never crosses the 0.5 threshold, so dice@0.5 stays 0.
  3. Fix: class-weighted CE — tumor-class weight w in the CE part of the per-sample loss (--a-fg-ce-weight, src/losses.py). 3-epoch probes on this dataset: w=20 → pmax median 0.33, dice 0.0; w=100 → 0.41, dice 0.0002; w=500 → 0.62, dice 0.0188 (34/39 rows > 0.5). Official 10-epoch w=500 run: dice 0.0369 (vs 0.0026 baseline, 14×; 8/39 rows > 0.001, best 0.555, pmax median 0.55, 29/39 rows > 0.5) — A now segments, but still trails C (0.2162) at this budget; its loss is still descending at epoch 10 (0.78 → 0.51), i.e. under-converged vs C (0.41 → 0.18), so the gap should narrow with more epochs.

Changes: src/dataset.py (pos/neg crop sampling pos_neg=0.5, per-worker RNG reseed), src/losses.py (soft_dice_fg, fg_ce_weight), benchmark flags --a-fg-dice / --a-fg-ce-weight, per-row pmax in A's dice_per_row, full traceback in failure JSONs. A/B probes: results/bench_A_*probe*.json.

B: CIFS failures and local-disk NNU_ROOT (2026-10-05)

The CIFS /mnt/b4 mount dropped I/O repeatedly on 2026-10-05 (NAS flapping: OSError [Errno 112] Host is down, RuntimeError: Error while setting the slice, one silent rmtree failure that made the raw rebuild collide with stale entries → FileExistsError). Four B attempts died (07:52 plan&prep; 11:12 raw EEXIST; ~12:40 / ~13:50 / ~14:00 plan&prep). Fixes:

  • nnu_common.rmtree_verified() + defensive per-link unlink in make_raw_dataset (EEXIST on rebuild)
  • failure JSONs now carry a full traceback key
  • plan step retries raised 2 → 5 (N.run(..., retries=5))
  • B's nnU-Net tree moved off CIFS: nnu_common.nnu_root() honors NNU_ROOT (set to /home/xfr/longitudinal_nnu, local ZFS ~1.8 T free) and the benchmark's B pred I/O honors BENCH_B_DIR (/home/xfr/longitudinal_benchB). CIFS is then touched only for the one-time raw image reads from /mnt/pve and small log/JSON appends. The pre-NNU_ROOT attempt's half-built preprocessed tree under nnu/preprocessed/ is stale — the local tree is the one being used.

The local-disk run succeeded on attempt 1 (2026-10-05 14:07→16:38 UTC): raw build 6.3 min, plan&preprocess 62 min (140 GB preprocessed), 10 epochs in 55 min (epoch times 101–167 s train + 275-case validation — nnUNet_compile is enabled, vs the Oct-4 eager-mode pace of ~2.4 h/epoch), inference 21.8 s/vol TTA. Val pseudo dice 0.091 → 0.459, best EMA 0.285; test dice 0.1424 (29/39 rows > 0.001, max 0.675). Checkpoints in /home/xfr/longitudinal_nnu/results/Dataset221_T1C_BENCH/...fold_0/ (copy to nnu/results/ for the record if local space is reclaimed).

Extended to 20 epochs (2026-10-05/06)

All three continued from their 10-epoch checkpoints to a 20-epoch total (--train-epochs 20 = total budget; 10-epoch JSONs snapshotted as results/benchmark_full_{A,B,C}_10ep.json):

python scripts/benchmark_pipelines.py --pipelines A --gpu 0 --resume --train-epochs 20 \
    --a-fg-ce-weight 500 --out results/benchmark_full_A.json
python scripts/benchmark_pipelines.py --pipelines C --gpu 2 --resume --train-epochs 20 \
    --out results/benchmark_full_C.json
NNU_ROOT=/home/xfr/longitudinal_nnu BENCH_B_DIR=/home/xfr/longitudinal_benchB \
python scripts/benchmark_pipelines.py --pipelines B --gpu 1 --resume --nnu-skip-prep \
    --train-epochs 20 --out results/benchmark_full_B.json

A/C resume from runs_bench/{A,C}/best.pt (model + optimizer + scheduler, epochs 11–20). B: the completed run had deleted checkpoint_latest.pth (see Resuming training), so it was restored from checkpoint_final.pth (current_epoch: 10) before --c + --nnu-skip-prep — optimizer state and LR schedule restored (initial validation 0.1253). bench_B's resume s/epoch now divides by epochs actually trained (reads current_epoch from the checkpoint).

pipeline dice @10ep dice @20ep Δ median max rows > 0.001
B nnU-Net 3d_fullres 0.1424 0.2927 (hard 0.2927) +0.150 0.289 0.839 29/39
C MONAI UNet(16→128) 0.2162 0.1621 −0.054 0.126 0.680 31/39
A Unet3D improved (fg crops + CE w=500) 0.0369 0.0076 −0.029 0.000 0.196 6/39

Resume window: A 229.2 s/epoch, C 253.9 s/epoch, B 280.1 s/epoch (10 epochs each, same data budget). B was far from convergence at 10 epochs and improved a lot; C's loss was flat (0.184) and its extra epochs slightly degraded test dice; the A w=500 config oscillates (loss 0.49–0.55, pmax median 0.55 → 0.49, rows >0.5: 29 → 19), so its 10-epoch point is its best under this config. At 20 epochs B leads (0.2927). Checkpoints in runs_bench/{A,C}/best.pt and B's fold_0 now hold the 20-epoch models — the 10-epoch models were overwritten (JSONs preserved). B: 21.9 s/vol TTA, 9.8 GB train VRAM.

Resuming training (--resume)

All three pipelines can continue from their latest benchmark checkpoint:

python scripts/benchmark_pipelines.py --pipelines "A B C" --resume \
    --train-epochs 200 --nnu-skip-prep
  • --train-epochs is the total budget (not epochs added). Each pipeline trains from where its checkpoint left off up to that total.
  • A / C: load runs_bench/{A,C}/best.pt (model + optimizer + scheduler state — checkpoints written before 2026-10-04 carry model + epoch only and fall back to fresh optimizer/scheduler) and run the remaining epochs; the run is a no-op train (eval only) when the checkpoint already reached the total.
  • B: appends nnU-Net's --c flag (continue fold_0 from checkpoint_latest.pth, native optimizer + LR-schedule restore) and NNUNet_pl_EPOCHS = --train-epochs. Pair with --nnu-skip-prep so the preprocessed tree isn't --cleaned; the isolated Dataset221 is rebuilt otherwise and a finished run's checkpoint_latest.pth is gone, so a completed run + a bigger budget retrains B from scratch (interrupted runs keep checkpoint_latest.pth and resume correctly). A finished run CAN still be resumed by copying checkpoint_final.pth → checkpoint_latest.pth first (same state dict; used for the 20-epoch extension above).
  • Verified 2026-10-04: A/C resumed epoch 1 → 3 (model restored, fresh opt/sched for the old-format ckpt), re-run at 3/3 trained 0 epochs, B continued from epoch 1 with LR restored mid-decay (0.00536 @ start of epoch 2).

Readiness notes for a full run

  • Budget knobs: --train-epochs (default 100), --max-rows (default 80% of split_train = 2750 for A/C and for B's --nnu-cases on the current curated splits; was 2856 pre-exclusion), --max-eval-rows (default 10% of split_test = 40), --no-tta.
  • --skip-train is only usable for A (runs/round0/best.pt, trained 2026-09-25 on the previous split — a few patients moved between train/test in the rebuild, so its test dice carries a small leakage caveat). No study checkpoints exist for B (Dataset210) or C (runs_monai/round0/best.pt); they must be trained.
  • GPUs: 3× 80 GB available; the benchmark uses one (--gpu).
  • B's one-time cost scales with --nnu-cases (plan_and_preprocess ≈ 3.5 s/case at 24 cases with 8 preprocess procs, plus training time at the planned batch/patch).