longitudinal/BENCHMARKING.md

# Pipeline benchmark (A / B / C)

Benchmark of the three segmentation pipelines — A in-house Unet3D, B nnU-Net v2
(NTUHLPLTrainer), C MONAI UNet — driven by `scripts/benchmark_pipelines.py`
(DICE, training time, inference time, params, VRAM). Protocol details live in
the script docstring; this file records what has been run and what the machine
currently supports.

## Data chain status (2026-10-04)

The M6 labeled set was fully rebuilt on 2026-10-04 (see
`scripts/02_build_m6_dataset.py`): T1c fixed, best-of-ANTs
(`ants_rigid` / `ants_affine` / `ants_syn`, winner by Mattes MI), TV warped
onto the T1c grid with SMD+linear. After the repair pass:

- `data/manifests/m6_labeled.jsonl`: **1312** volumes (was 1094 pre-repair),
  `m6_unlabeled.jsonl`: **229**
- all 1541 m6 processed volumes (`data/proc/<key>{,_label}.nii.gz`, 1mm
  cropped/normalized) regenerated, 0 errors
- patient-level splits rebuilt (`scripts/05_build_splits.py`, seed 0):
  **4200** labeled rows (ntuh + m6) → train 3570 / val 211 / test 419
  volumes, 3893 subjects; unlabeled pool 13280
- tumor volume stats: median 3116 mm3, p95 24.8k, max 3.2M, zero_frac 0.12%

## Dataset exclusions (2026-10-04)

Row-level curation lives in `src/exclusions.py` and is applied at the **first
stage** by `scripts/make_tv_slices.py` (right after the rows are fetched from
the sources, before any split): it classifies every labeled volume and writes
its TV-slice QA figure to `data/tv_slices/inclusion/` (kept) or
`data/tv_slices/exclusion/` (excluded, reason in the title). `05_build_splits`
re-applies the same rules **after** split assignment (kept rows retain their
split; the subject-level shuffle boundaries are unchanged):

- total TV > 1e6 mm3: **1** row (ntuh_6019528, label outlines the full skull)
- single target (1 connected component) with TV < 100 mm3: **109** rows
- non-brain images: **33** rows (all ntuh — spine/neck/torso CyberKnife
  cases whose series names are indistinguishable from brain T1c; identified
  by visual review of the TV-slice figures, now in `data/tv_slices/exclusion/`)

Audit trail: `data/manifests/excluded.jsonl` (143 rows, reason + split);
pre-exclusion manifests in `data/manifests/backup_20261004/`. New dataset:
**4057** labeled rows → train 3447 / val 205 / test 405 (3771 subjects);
unlabeled pool unchanged (13280); volume stats now median 3212 mm3, p95
23.4k, max 163k (m6 meningioma), zero_frac still 0.12% (5 rows, e.g.
ntuh_3436472 ×3).

After the initial 39-row non-brain list, expert inspection overturned 6
(skull-base / posterior-fossa brain targets that look like cervical spine at
the TV-max slice: ntuh_5902230, ntuh_5231609, ntuh_5083686, ntuh_5157194,
ntuh_5303758, ntuh_6373987). The remaining 33 were re-audited at the
mid-cranial slice (widest head-mask axial) — all still non-brain. Review
protocol now recorded in `src/exclusions.py`: classify at full resolution
on the mid-cranial slice, not just the TV-max slice.

## Smoke run (2026-10-04)

Small/fast end-to-end check that all three pipelines run on the rebuilt data
(1 epoch, 24 train rows, 3 eval rows, TTA on, GPU 0):

```
python scripts/benchmark_pipelines.py --gpu 0 --max-rows 24 --nnu-cases 24 \
    --max-eval-rows 3 --train-epochs 1 --out results/benchmark_pipelines_smoke.json
```

| pipeline | wall | s/epoch | tr VRAM | s/vol (TTA) | in VRAM | params | weights | dice (1 epoch) |
|---|---|---|---|---|---|---|---|---|
| A Unet3D(16, depth 4) | 17 s | 2.5 (8 steps) | 2.0 GB | 4.12 | 0.5 GB | 1.46 M | 5.9 MB | 0.0000 |
| B nnU-Net 3d_fullres | 285 s (incl. 83 s plan&preprocess, 4 s raw build) | 73.3 | 9.8 GB | 34.73 | 1.8 GB | 30.8 M | 246 MB | 0.0000 |
| C MONAI UNet(16→128) | 5 s | 2.5 (8 steps) | 1.7 GB | 0.33 | 0.5 GB | 1.19 M | 4.8 MB | 0.0005 |

DICE ≈ 0 is expected (models trained for a single epoch on 24 volumes); the
point of the run was validating the full chain — new m6 native/processed
manifests, split membership, B's `ntuh + m6_labeled` native join, nnU-Net
isolated dataset 221 build, inference + dice scoring. All three passed.
Eval rows for the smoke run were the first three `split_test` volumes (all
ntuh). Artifacts: `results/benchmark_pipelines_smoke.json`,
`_per_row.jsonl`, `_bench.png`.

### Environment fixes made during the check

- `monai` was missing from `longitudinal-313` (pipeline C failed with
  ModuleNotFoundError) — installed `pip install monai==1.6.1 --no-deps`
  (torch 2.14+cu126 untouched); `import monai` verified.

## Full-dataset 10-epoch run (2026-10-05, updated dataset)

`--train-epochs 10`, default budgets on the current curated splits: **2750**
train rows = 80% of split_train (3438 labeled), **40** eval rows = 10% of
split_test (404), TTA on. The 2026-10-04 interim run (pre-exclusion budget
2856/42: A 0.0008, C 0.2119) is superseded; its JSONs are kept in
`results/backup_20261004/` (smoke B checkpoint in
`nnu/results/backup_20261004_Dataset221_smoke/`).

```
python scripts/benchmark_pipelines.py --pipelines C --gpu 2 --train-epochs 10 \
    --out results/benchmark_full_C.json      # done
python scripts/benchmark_pipelines.py --pipelines A --gpu 0 --train-epochs 10 \
    --a-fg-ce-weight 500 --out results/benchmark_full_A.json   # done
# A baseline (uniform crops) on this dataset: dice 0.0026, see table;
# checkpoint runs_bench/A/best_baseline_10ep_uniform_crop.pt
NNU_ROOT=/home/xfr/longitudinal_nnu BENCH_B_DIR=/home/xfr/longitudinal_benchB \
python scripts/benchmark_pipelines.py --pipelines B --gpu 1 --train-epochs 10 \
    --out results/benchmark_full_B.json      # done (local-disk NNU_ROOT, see below)
```

| pipeline | train s/epoch | wall | tr VRAM | s/vol (TTA) | in VRAM | params | weights | dice @10ep |
|---|---|---|---|---|---|---|---|---|
| C MONAI UNet(16→128) | 256.7 (916 steps) | 43 min | 0.4 GB | 0.37 ± 0.06 | 0.5 GB | 1.19 M | 14.3 MB | **0.2162** |
| A Unet3D(16, depth 4), improved (fg crops + CE w=500) | 232.3 (916 steps) | 41 min | 2.0 GB | 3.60 ± 0.76 | 0.5 GB | 1.46 M | 17.6 MB | **0.0369** |
| A baseline (uniform crops, unweighted loss) | 216.6 (916 steps) | 45 min | 2.0 GB | 3.33 ± 0.86 | 0.5 GB | 1.46 M | 17.6 MB | **0.0026** |
| B nnU-Net 3d_fullres | 331.5 (incl. 275-case val/epoch) | ~2.5 h (raw 6 m, prep 62 m, train 55 m, infer 14 m) | 9.3 GB | 21.84 | 1.8 GB | 30.8 M | 246 MB | **0.1424** (hard 0.1424) |

C: 25/39 timed test vols with dice > 0.001 (max 0.7392); loss 0.403 → 0.182.
10 epochs is far below convergence for these sparse targets (median tumor
3.2 k mm³); A/C numbers at this budget are intermediate. Checkpoints at
`runs_bench/{A,C}/best.pt` (extend with `--resume --train-epochs <total>`).

### A: why dice ≈ 0, and the fix (2026-10-05)

The A baseline predicts all-background (38/39 test rows dice exactly 0.0).
Root-cause work:

1. **Uniform random 96³ crops starve the loss of foreground.** A 96³ patch is
   885 k voxels; the median tumor is ~4.6 k mm³ (0.02–0.05% of a head volume).
   Simulated uniform crops contain the whole blob in only 8–37% of patches and
   a *median* crop often has 0 tumor voxels. Pipeline C's
   `RandCropByPosNegLabeld(pos=1, neg=1)` centers half its patches on the
   tumor — ~4× more foreground supervision per epoch.
2. **Fg-aware crops alone are not enough: A under-confides.** With the crop
   protocol fixed (`src/dataset.py`), A's loss matches C's at every epoch
   (0.193 vs 0.182 @10ep) yet p_tumor maxes at 0.43 (C: ~1.0): it localizes
   (top-k dice 0.48 on best rows) but never crosses the 0.5 threshold, so
   dice@0.5 stays 0.
3. **Fix: class-weighted CE** — tumor-class weight `w` in the CE part of the
   per-sample loss (`--a-fg-ce-weight`, `src/losses.py`). 3-epoch probes on
   this dataset: w=20 → pmax median 0.33, dice 0.0; w=100 → 0.41, dice
   0.0002; w=500 → 0.62, dice 0.0188 (34/39 rows > 0.5). Official 10-epoch
   w=500 run: **dice 0.0369** (vs 0.0026 baseline, 14×; 8/39 rows > 0.001,
   best 0.555, pmax median 0.55, 29/39 rows > 0.5) — A now segments, but
   still trails C (0.2162) at this budget; its loss is still descending at
   epoch 10 (0.78 → 0.51), i.e. under-converged vs C (0.41 → 0.18), so the
   gap should narrow with more epochs.

Changes: `src/dataset.py` (pos/neg crop sampling `pos_neg=0.5`, per-worker
RNG reseed), `src/losses.py` (`soft_dice_fg`, `fg_ce_weight`), benchmark flags
`--a-fg-dice` / `--a-fg-ce-weight`, per-row `pmax` in A's `dice_per_row`,
full traceback in failure JSONs. A/B probes: `results/bench_A_*probe*.json`.

### B: CIFS failures and local-disk NNU_ROOT (2026-10-05)

The CIFS `/mnt/b4` mount dropped I/O repeatedly on 2026-10-05 (NAS flapping:
`OSError [Errno 112] Host is down`, `RuntimeError: Error while setting the
slice`, one silent `rmtree` failure that made the raw rebuild collide with
stale entries → `FileExistsError`). Four B attempts died (07:52 plan&prep;
11:12 raw EEXIST; ~12:40 / ~13:50 / ~14:00 plan&prep). Fixes:

- `nnu_common.rmtree_verified()` + defensive per-link unlink in
  `make_raw_dataset` (EEXIST on rebuild)
- failure JSONs now carry a full `traceback` key
- plan step retries raised 2 → 5 (`N.run(..., retries=5)`)
- **B's nnU-Net tree moved off CIFS**: `nnu_common.nnu_root()` honors
  `NNU_ROOT` (set to `/home/xfr/longitudinal_nnu`, local ZFS ~1.8 T free) and
  the benchmark's B pred I/O honors `BENCH_B_DIR`
  (`/home/xfr/longitudinal_benchB`). CIFS is then touched only for the
  one-time raw image reads from `/mnt/pve` and small log/JSON appends.
  The pre-`NNU_ROOT` attempt's half-built preprocessed tree under
  `nnu/preprocessed/` is stale — the local tree is the one being used.

The local-disk run succeeded on attempt 1 (2026-10-05 14:07→16:38 UTC):
raw build 6.3 min, plan&preprocess 62 min (140 GB preprocessed), 10 epochs
in 55 min (epoch times 101–167 s train + 275-case validation —
`nnUNet_compile is enabled`, vs the Oct-4 eager-mode pace of ~2.4 h/epoch),
inference 21.8 s/vol TTA. Val pseudo dice 0.091 → 0.459, best EMA 0.285;
test dice 0.1424 (29/39 rows > 0.001, max 0.675). Checkpoints in
`/home/xfr/longitudinal_nnu/results/Dataset221_T1C_BENCH/...fold_0/`
(copy to `nnu/results/` for the record if local space is reclaimed).

## Extended to 20 epochs (2026-10-05/06)

All three continued from their 10-epoch checkpoints to a 20-epoch total
(`--train-epochs 20` = total budget; 10-epoch JSONs snapshotted as
`results/benchmark_full_{A,B,C}_10ep.json`):

```
python scripts/benchmark_pipelines.py --pipelines A --gpu 0 --resume --train-epochs 20 \
    --a-fg-ce-weight 500 --out results/benchmark_full_A.json
python scripts/benchmark_pipelines.py --pipelines C --gpu 2 --resume --train-epochs 20 \
    --out results/benchmark_full_C.json
NNU_ROOT=/home/xfr/longitudinal_nnu BENCH_B_DIR=/home/xfr/longitudinal_benchB \
python scripts/benchmark_pipelines.py --pipelines B --gpu 1 --resume --nnu-skip-prep \
    --train-epochs 20 --out results/benchmark_full_B.json
```

A/C resume from `runs_bench/{A,C}/best.pt` (model + optimizer + scheduler,
epochs 11–20). B: the completed run had deleted `checkpoint_latest.pth`
(see Resuming training), so it was restored from `checkpoint_final.pth`
(`current_epoch: 10`) before `--c` + `--nnu-skip-prep` — optimizer state and
LR schedule restored (initial validation 0.1253). `bench_B`'s resume
s/epoch now divides by epochs actually trained (reads `current_epoch` from
the checkpoint).

| pipeline | dice @10ep | dice @20ep | Δ | median | max | rows > 0.001 |
|---|---|---|---|---|---|---|
| B nnU-Net 3d_fullres | 0.1424 | **0.2927** (hard 0.2927) | +0.150 | 0.289 | 0.839 | 29/39 |
| C MONAI UNet(16→128) | 0.2162 | **0.1621** | −0.054 | 0.126 | 0.680 | 31/39 |
| A Unet3D improved (fg crops + CE w=500) | 0.0369 | **0.0076** | −0.029 | 0.000 | 0.196 | 6/39 |

Resume window: A 229.2 s/epoch, C 253.9 s/epoch, B 280.1 s/epoch
(10 epochs each, same data budget). B was far from convergence at 10 epochs
and improved a lot; C's loss was flat (0.184) and its extra epochs slightly
degraded test dice; the A w=500 config oscillates (loss 0.49–0.55, pmax
median 0.55 → 0.49, rows >0.5: 29 → 19), so its 10-epoch point is its best
under this config. At 20 epochs **B leads (0.2927)**. Checkpoints in
`runs_bench/{A,C}/best.pt` and B's fold_0 now hold the 20-epoch models — the
10-epoch models were overwritten (JSONs preserved). B: 21.9 s/vol TTA,
9.8 GB train VRAM.

## Resuming training (`--resume`)

All three pipelines can continue from their latest benchmark checkpoint:

```
python scripts/benchmark_pipelines.py --pipelines "A B C" --resume \
    --train-epochs 200 --nnu-skip-prep
```

- `--train-epochs` is the **total budget** (not epochs added). Each pipeline
  trains from where its checkpoint left off up to that total.
- **A / C**: load `runs_bench/{A,C}/best.pt` (model + optimizer + scheduler
  state — checkpoints written before 2026-10-04 carry model + epoch only and
  fall back to fresh optimizer/scheduler) and run the remaining epochs; the
  run is a no-op train (eval only) when the checkpoint already reached the
  total.
- **B**: appends nnU-Net's `--c` flag (continue `fold_0` from
  `checkpoint_latest.pth`, native optimizer + LR-schedule restore) and
  `NNUNet_pl_EPOCHS = --train-epochs`. Pair with `--nnu-skip-prep` so the
  preprocessed tree isn't `--clean`ed; the isolated Dataset221 is rebuilt
  otherwise and a finished run's `checkpoint_latest.pth` is gone, so a
  completed run + a bigger budget retrains B from scratch (interrupted runs
  keep `checkpoint_latest.pth` and resume correctly). A finished run CAN
  still be resumed by copying `checkpoint_final.pth` → `checkpoint_latest.pth`
  first (same state dict; used for the 20-epoch extension above).
- Verified 2026-10-04: A/C resumed epoch 1 → 3 (model restored, fresh
  opt/sched for the old-format ckpt), re-run at 3/3 trained 0 epochs, B
  continued from epoch 1 with LR restored mid-decay (0.00536 @ start of
  epoch 2).

## Readiness notes for a full run

- Budget knobs: `--train-epochs` (default 100), `--max-rows` (default 80% of
  split_train = 2750 for A/C and for B's `--nnu-cases` on the current curated
  splits; was 2856 pre-exclusion), `--max-eval-rows` (default 10% of
  split_test = 40), `--no-tta`.
- `--skip-train` is only usable for **A** (`runs/round0/best.pt`, trained
  2026-09-25 on the *previous* split — a few patients moved between
  train/test in the rebuild, so its test dice carries a small leakage
  caveat). No study checkpoints exist for B (Dataset210) or C
  (`runs_monai/round0/best.pt`); they must be trained.
- GPUs: 3× 80 GB available; the benchmark uses one (`--gpu`).
- B's one-time cost scales with `--nnu-cases` (plan_and_preprocess ≈ 3.5 s/case
  at 24 cases with 8 preprocess procs, plus training time at the planned
  batch/patch).