xfr/longitudinal
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
longitudinal/BENCHMARKING.md

4.8 KiB
Raw Blame History

Pipeline benchmark (A / B / C)

Benchmark of the three segmentation pipelines — A in-house Unet3D, B nnU-Net v2 (NTUHLPLTrainer), C MONAI UNet — driven by scripts/benchmark_pipelines.py (DICE, training time, inference time, params, VRAM). Protocol details live in the script docstring; this file records what has been run and what the machine currently supports.

Data chain status (2026-10-04)

The M6 labeled set was fully rebuilt on 2026-10-04 (see scripts/02_build_m6_dataset.py): T1c fixed, best-of-ANTs (ants_rigid / ants_affine / ants_syn, winner by Mattes MI), TV warped onto the T1c grid with SMD+linear. After the repair pass:

  • data/manifests/m6_labeled.jsonl: 1312 volumes (was 1094 pre-repair), m6_unlabeled.jsonl: 229
  • all 1541 m6 processed volumes (data/proc/<key>{,_label}.nii.gz, 1mm cropped/normalized) regenerated, 0 errors
  • patient-level splits rebuilt (scripts/05_build_splits.py, seed 0): 4200 labeled rows (ntuh + m6) → train 3570 / val 211 / test 419 volumes, 3893 subjects; unlabeled pool 13280
  • tumor volume stats: median 3116 mm3, p95 24.8k, max 3.2M, zero_frac 0.12%

Smoke run (2026-10-04)

Small/fast end-to-end check that all three pipelines run on the rebuilt data (1 epoch, 24 train rows, 3 eval rows, TTA on, GPU 0):

python scripts/benchmark_pipelines.py --gpu 0 --max-rows 24 --nnu-cases 24 \
    --max-eval-rows 3 --train-epochs 1 --out results/benchmark_pipelines_smoke.json
pipeline wall s/epoch tr VRAM s/vol (TTA) in VRAM params weights dice (1 epoch)
A Unet3D(16, depth 4) 17 s 2.5 (8 steps) 2.0 GB 4.12 0.5 GB 1.46 M 5.9 MB 0.0000
B nnU-Net 3d_fullres 285 s (incl. 83 s plan&preprocess, 4 s raw build) 73.3 9.8 GB 34.73 1.8 GB 30.8 M 246 MB 0.0000
C MONAI UNet(16→128) 5 s 2.5 (8 steps) 1.7 GB 0.33 0.5 GB 1.19 M 4.8 MB 0.0005

DICE ≈ 0 is expected (models trained for a single epoch on 24 volumes); the point of the run was validating the full chain — new m6 native/processed manifests, split membership, B's ntuh + m6_labeled native join, nnU-Net isolated dataset 221 build, inference + dice scoring. All three passed. Eval rows for the smoke run were the first three split_test volumes (all ntuh). Artifacts: results/benchmark_pipelines_smoke.json, _per_row.jsonl, _bench.png.

Environment fixes made during the check

  • monai was missing from longitudinal-313 (pipeline C failed with ModuleNotFoundError) — installed pip install monai==1.6.1 --no-deps (torch 2.14+cu126 untouched); import monai verified.

Resuming training (--resume)

All three pipelines can continue from their latest benchmark checkpoint:

python scripts/benchmark_pipelines.py --pipelines "A B C" --resume \
    --train-epochs 200 --nnu-skip-prep
  • --train-epochs is the total budget (not epochs added). Each pipeline trains from where its checkpoint left off up to that total.
  • A / C: load runs_bench/{A,C}/best.pt (model + optimizer + scheduler state — checkpoints written before 2026-10-04 carry model + epoch only and fall back to fresh optimizer/scheduler) and run the remaining epochs; the run is a no-op train (eval only) when the checkpoint already reached the total.
  • B: appends nnU-Net's --c flag (continue fold_0 from checkpoint_latest.pth, native optimizer + LR-schedule restore) and NNUNet_pl_EPOCHS = --train-epochs. Pair with --nnu-skip-prep so the preprocessed tree isn't --cleaned; the isolated Dataset221 is rebuilt otherwise and a finished run's checkpoint_latest.pth is gone, so a completed run + a bigger budget retrains B from scratch (interrupted runs keep checkpoint_latest.pth and resume correctly).
  • Verified 2026-10-04: A/C resumed epoch 1 → 3 (model restored, fresh opt/sched for the old-format ckpt), re-run at 3/3 trained 0 epochs, B continued from epoch 1 with LR restored mid-decay (0.00536 @ start of epoch 2).

Readiness notes for a full run

  • Budget knobs: --train-epochs (default 100), --max-rows (default 80% of split_train ≈ 2856 for A/C and for B's --nnu-cases), --max-eval-rows (default 10% of split_test ≈ 42), --no-tta.
  • --skip-train is only usable for A (runs/round0/best.pt, trained 2026-09-25 on the previous split — a few patients moved between train/test in the rebuild, so its test dice carries a small leakage caveat). No study checkpoints exist for B (Dataset210) or C (runs_monai/round0/best.pt); they must be trained.
  • GPUs: 3× 80 GB available; the benchmark uses one (--gpu).
  • B's one-time cost scales with --nnu-cases (plan_and_preprocess ≈ 3.5 s/case at 24 cases with 8 preprocess procs, plus training time at the planned batch/patch).