# Pipeline benchmark (A / B / C)

Benchmark of the three segmentation pipelines — A in-house Unet3D, B nnU-Net v2
(NTUHLPLTrainer), C MONAI UNet — driven by `scripts/benchmark_pipelines.py`
(DICE, training time, inference time, params, VRAM). Protocol details live in
the script docstring; this file records what has been run and what the machine
currently supports.

## Data chain status (2026-10-04)

The M6 labeled set was fully rebuilt on 2026-10-04 (see
`scripts/02_build_m6_dataset.py`): T1c fixed, best-of-ANTs
(`ants_rigid` / `ants_affine` / `ants_syn`, winner by Mattes MI), TV warped
onto the T1c grid with SMD+linear. After the repair pass:

- `data/manifests/m6_labeled.jsonl`: **1312** volumes (was 1094 pre-repair),
  `m6_unlabeled.jsonl`: **229**
- all 1541 m6 processed volumes (`data/proc/<key>{,_label}.nii.gz`, 1mm
  cropped/normalized) regenerated, 0 errors
- patient-level splits rebuilt (`scripts/05_build_splits.py`, seed 0):
  **4200** labeled rows (ntuh + m6) → train 3570 / val 211 / test 419
  volumes, 3893 subjects; unlabeled pool 13280
- tumor volume stats: median 3116 mm3, p95 24.8k, max 3.2M, zero_frac 0.12%

## Smoke run (2026-10-04)

Small/fast end-to-end check that all three pipelines run on the rebuilt data
(1 epoch, 24 train rows, 3 eval rows, TTA on, GPU 0):

```
python scripts/benchmark_pipelines.py --gpu 0 --max-rows 24 --nnu-cases 24 \
    --max-eval-rows 3 --train-epochs 1 --out results/benchmark_pipelines_smoke.json
```

| pipeline | wall | s/epoch | tr VRAM | s/vol (TTA) | in VRAM | params | weights | dice (1 epoch) |
|---|---|---|---|---|---|---|---|---|
| A Unet3D(16, depth 4) | 17 s | 2.5 (8 steps) | 2.0 GB | 4.12 | 0.5 GB | 1.46 M | 5.9 MB | 0.0000 |
| B nnU-Net 3d_fullres | 285 s (incl. 83 s plan&preprocess, 4 s raw build) | 73.3 | 9.8 GB | 34.73 | 1.8 GB | 30.8 M | 246 MB | 0.0000 |
| C MONAI UNet(16→128) | 5 s | 2.5 (8 steps) | 1.7 GB | 0.33 | 0.5 GB | 1.19 M | 4.8 MB | 0.0005 |

DICE ≈ 0 is expected (models trained for a single epoch on 24 volumes); the
point of the run was validating the full chain — new m6 native/processed
manifests, split membership, B's `ntuh + m6_labeled` native join, nnU-Net
isolated dataset 221 build, inference + dice scoring. All three passed.
Eval rows for the smoke run were the first three `split_test` volumes (all
ntuh). Artifacts: `results/benchmark_pipelines_smoke.json`,
`_per_row.jsonl`, `_bench.png`.

### Environment fixes made during the check

- `monai` was missing from `longitudinal-313` (pipeline C failed with
  ModuleNotFoundError) — installed `pip install monai==1.6.1 --no-deps`
  (torch 2.14+cu126 untouched); `import monai` verified.

## Resuming training (`--resume`)

All three pipelines can continue from their latest benchmark checkpoint:

```
python scripts/benchmark_pipelines.py --pipelines "A B C" --resume \
    --train-epochs 200 --nnu-skip-prep
```

- `--train-epochs` is the **total budget** (not epochs added). Each pipeline
  trains from where its checkpoint left off up to that total.
- **A / C**: load `runs_bench/{A,C}/best.pt` (model + optimizer + scheduler
  state — checkpoints written before 2026-10-04 carry model + epoch only and
  fall back to fresh optimizer/scheduler) and run the remaining epochs; the
  run is a no-op train (eval only) when the checkpoint already reached the
  total.
- **B**: appends nnU-Net's `--c` flag (continue `fold_0` from
  `checkpoint_latest.pth`, native optimizer + LR-schedule restore) and
  `NNUNet_pl_EPOCHS = --train-epochs`. Pair with `--nnu-skip-prep` so the
  preprocessed tree isn't `--clean`ed; the isolated Dataset221 is rebuilt
  otherwise and a finished run's `checkpoint_latest.pth` is gone, so a
  completed run + a bigger budget retrains B from scratch (interrupted runs
  keep `checkpoint_latest.pth` and resume correctly).
- Verified 2026-10-04: A/C resumed epoch 1 → 3 (model restored, fresh
  opt/sched for the old-format ckpt), re-run at 3/3 trained 0 epochs, B
  continued from epoch 1 with LR restored mid-decay (0.00536 @ start of
  epoch 2).

## Readiness notes for a full run

- Budget knobs: `--train-epochs` (default 100), `--max-rows` (default 80% of
  split_train ≈ 2856 for A/C and for B's `--nnu-cases`), `--max-eval-rows`
  (default 10% of split_test ≈ 42), `--no-tta`.
- `--skip-train` is only usable for **A** (`runs/round0/best.pt`, trained
  2026-09-25 on the *previous* split — a few patients moved between
  train/test in the rebuild, so its test dice carries a small leakage
  caveat). No study checkpoints exist for B (Dataset210) or C
  (`runs_monai/round0/best.pt`); they must be trained.
- GPUs: 3× 80 GB available; the benchmark uses one (`--gpu`).
- B's one-time cost scales with `--nnu-cases` (plan_and_preprocess ≈ 3.5 s/case
  at 24 cases with 8 preprocess procs, plus training time at the planned
  batch/patch).