Furen Xiao
2453fb48ba
Introduce src/exclusions.py (rules: total TV > 1e6 mm3, single target
< 100 mm3, non-brain NTUH keys) and scripts/make_tv_slices.py, a
first-stage curation script that classifies every labeled row right
after the source fetch and writes TV-slice QA figures to
data/tv_slices/{inclusion,exclusion}/.
05_build_splits.py re-applies the same rules after split assignment:
kept rows retain their split, excluded subjects still count in the
subject-level shuffle boundaries, and the audit trail is written to
data/manifests/excluded.jsonl. Label stats run in a
ProcessPoolExecutor (32 workers).
Move proc_row into src/common.py and add labeled_source_rows() so the
curation stage and the split builder share one processed-row view.
Document the 143-row exclusion audit and the interim 10-epoch
benchmark results in BENCHMARKING.md; update the README data layout
and script tables.
158 lines
No EOL
8.2 KiB
Markdown
158 lines
No EOL
8.2 KiB
Markdown
# Pipeline benchmark (A / B / C)
|
||
|
||
Benchmark of the three segmentation pipelines — A in-house Unet3D, B nnU-Net v2
|
||
(NTUHLPLTrainer), C MONAI UNet — driven by `scripts/benchmark_pipelines.py`
|
||
(DICE, training time, inference time, params, VRAM). Protocol details live in
|
||
the script docstring; this file records what has been run and what the machine
|
||
currently supports.
|
||
|
||
## Data chain status (2026-10-04)
|
||
|
||
The M6 labeled set was fully rebuilt on 2026-10-04 (see
|
||
`scripts/02_build_m6_dataset.py`): T1c fixed, best-of-ANTs
|
||
(`ants_rigid` / `ants_affine` / `ants_syn`, winner by Mattes MI), TV warped
|
||
onto the T1c grid with SMD+linear. After the repair pass:
|
||
|
||
- `data/manifests/m6_labeled.jsonl`: **1312** volumes (was 1094 pre-repair),
|
||
`m6_unlabeled.jsonl`: **229**
|
||
- all 1541 m6 processed volumes (`data/proc/<key>{,_label}.nii.gz`, 1mm
|
||
cropped/normalized) regenerated, 0 errors
|
||
- patient-level splits rebuilt (`scripts/05_build_splits.py`, seed 0):
|
||
**4200** labeled rows (ntuh + m6) → train 3570 / val 211 / test 419
|
||
volumes, 3893 subjects; unlabeled pool 13280
|
||
- tumor volume stats: median 3116 mm3, p95 24.8k, max 3.2M, zero_frac 0.12%
|
||
|
||
## Dataset exclusions (2026-10-04)
|
||
|
||
Row-level curation lives in `src/exclusions.py` and is applied at the **first
|
||
stage** by `scripts/make_tv_slices.py` (right after the rows are fetched from
|
||
the sources, before any split): it classifies every labeled volume and writes
|
||
its TV-slice QA figure to `data/tv_slices/inclusion/` (kept) or
|
||
`data/tv_slices/exclusion/` (excluded, reason in the title). `05_build_splits`
|
||
re-applies the same rules **after** split assignment (kept rows retain their
|
||
split; the subject-level shuffle boundaries are unchanged):
|
||
|
||
- total TV > 1e6 mm3: **1** row (ntuh_6019528, label outlines the full skull)
|
||
- single target (1 connected component) with TV < 100 mm3: **109** rows
|
||
- non-brain images: **33** rows (all ntuh — spine/neck/torso CyberKnife
|
||
cases whose series names are indistinguishable from brain T1c; identified
|
||
by visual review of the TV-slice figures, now in `data/tv_slices/exclusion/`)
|
||
|
||
Audit trail: `data/manifests/excluded.jsonl` (143 rows, reason + split);
|
||
pre-exclusion manifests in `data/manifests/backup_20261004/`. New dataset:
|
||
**4057** labeled rows → train 3447 / val 205 / test 405 (3771 subjects);
|
||
unlabeled pool unchanged (13280); volume stats now median 3212 mm3, p95
|
||
23.4k, max 163k (m6 meningioma), zero_frac still 0.12% (5 rows, e.g.
|
||
ntuh_3436472 ×3).
|
||
|
||
After the initial 39-row non-brain list, expert inspection overturned 6
|
||
(skull-base / posterior-fossa brain targets that look like cervical spine at
|
||
the TV-max slice: ntuh_5902230, ntuh_5231609, ntuh_5083686, ntuh_5157194,
|
||
ntuh_5303758, ntuh_6373987). The remaining 33 were re-audited at the
|
||
mid-cranial slice (widest head-mask axial) — all still non-brain. Review
|
||
protocol now recorded in `src/exclusions.py`: classify at full resolution
|
||
on the mid-cranial slice, not just the TV-max slice.
|
||
|
||
## Smoke run (2026-10-04)
|
||
|
||
Small/fast end-to-end check that all three pipelines run on the rebuilt data
|
||
(1 epoch, 24 train rows, 3 eval rows, TTA on, GPU 0):
|
||
|
||
```
|
||
python scripts/benchmark_pipelines.py --gpu 0 --max-rows 24 --nnu-cases 24 \
|
||
--max-eval-rows 3 --train-epochs 1 --out results/benchmark_pipelines_smoke.json
|
||
```
|
||
|
||
| pipeline | wall | s/epoch | tr VRAM | s/vol (TTA) | in VRAM | params | weights | dice (1 epoch) |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| A Unet3D(16, depth 4) | 17 s | 2.5 (8 steps) | 2.0 GB | 4.12 | 0.5 GB | 1.46 M | 5.9 MB | 0.0000 |
|
||
| B nnU-Net 3d_fullres | 285 s (incl. 83 s plan&preprocess, 4 s raw build) | 73.3 | 9.8 GB | 34.73 | 1.8 GB | 30.8 M | 246 MB | 0.0000 |
|
||
| C MONAI UNet(16→128) | 5 s | 2.5 (8 steps) | 1.7 GB | 0.33 | 0.5 GB | 1.19 M | 4.8 MB | 0.0005 |
|
||
|
||
DICE ≈ 0 is expected (models trained for a single epoch on 24 volumes); the
|
||
point of the run was validating the full chain — new m6 native/processed
|
||
manifests, split membership, B's `ntuh + m6_labeled` native join, nnU-Net
|
||
isolated dataset 221 build, inference + dice scoring. All three passed.
|
||
Eval rows for the smoke run were the first three `split_test` volumes (all
|
||
ntuh). Artifacts: `results/benchmark_pipelines_smoke.json`,
|
||
`_per_row.jsonl`, `_bench.png`.
|
||
|
||
### Environment fixes made during the check
|
||
|
||
- `monai` was missing from `longitudinal-313` (pipeline C failed with
|
||
ModuleNotFoundError) — installed `pip install monai==1.6.1 --no-deps`
|
||
(torch 2.14+cu126 untouched); `import monai` verified.
|
||
|
||
## Full-dataset 10-epoch run (2026-10-04) — interim
|
||
|
||
`--train-epochs 10`, default budgets (2856 train rows = 80% of split_train,
|
||
42 eval rows = 10% of split_test, TTA on), A/C/GPU0·GPU2 finished, B still
|
||
preprocessing at time of writing:
|
||
|
||
```
|
||
python scripts/benchmark_pipelines.py --pipelines A --gpu 0 --train-epochs 10 \
|
||
--out results/benchmark_full_A.json # done
|
||
python scripts/benchmark_pipelines.py --pipelines C --gpu 2 --train-epochs 10 \
|
||
--out results/benchmark_full_C.json # done
|
||
python scripts/benchmark_pipelines.py --pipelines B --gpu 1 --train-epochs 10 \
|
||
--out results/benchmark_full_B.json # in progress (nnU-Net prep)
|
||
```
|
||
|
||
| pipeline | train s/epoch | wall | tr VRAM | s/vol (TTA) | in VRAM | params | weights | dice @10ep |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| A Unet3D(16, depth 4) | 258.8 (952 steps) | 46 min | 2.0 GB | 3.51 ± 0.93 | 0.5 GB | 1.46 M | 17.6 MB | **0.0008** |
|
||
| C MONAI UNet(16→128) | 284.3 (952 steps) | 48 min | 0.4 GB | 0.38 ± 0.06 | 0.5 GB | 1.19 M | 14.3 MB | **0.2119** |
|
||
| B nnU-Net 3d_fullres | (pending) | (pending) | — | — | — | 30.8 M | 246 MB | (pending) |
|
||
|
||
Dice distribution over 41 timed test vols: A p25/p75 = 0.0000/0.0000 (98% ≈ 0);
|
||
C p25/p75 = 0.0000/0.4048 (29% ≈ 0). Training loss (epoch 1 → 10): A
|
||
0.335 → 0.209, C 0.403 → 0.188. 10 epochs is far below convergence for these
|
||
sparse targets (median tumor 3.1 k mm³); the A/C numbers here are
|
||
intermediate, and C is already clearly ahead at this budget. Checkpoints at
|
||
`runs_bench/{A,C}/best.pt` (extend with `--resume --train-epochs <total>`).
|
||
Note: B's first `plan_and_preprocess` attempt died and the built-in 60 s retry
|
||
re-ran it successfully.
|
||
|
||
## Resuming training (`--resume`)
|
||
|
||
All three pipelines can continue from their latest benchmark checkpoint:
|
||
|
||
```
|
||
python scripts/benchmark_pipelines.py --pipelines "A B C" --resume \
|
||
--train-epochs 200 --nnu-skip-prep
|
||
```
|
||
|
||
- `--train-epochs` is the **total budget** (not epochs added). Each pipeline
|
||
trains from where its checkpoint left off up to that total.
|
||
- **A / C**: load `runs_bench/{A,C}/best.pt` (model + optimizer + scheduler
|
||
state — checkpoints written before 2026-10-04 carry model + epoch only and
|
||
fall back to fresh optimizer/scheduler) and run the remaining epochs; the
|
||
run is a no-op train (eval only) when the checkpoint already reached the
|
||
total.
|
||
- **B**: appends nnU-Net's `--c` flag (continue `fold_0` from
|
||
`checkpoint_latest.pth`, native optimizer + LR-schedule restore) and
|
||
`NNUNet_pl_EPOCHS = --train-epochs`. Pair with `--nnu-skip-prep` so the
|
||
preprocessed tree isn't `--clean`ed; the isolated Dataset221 is rebuilt
|
||
otherwise and a finished run's `checkpoint_latest.pth` is gone, so a
|
||
completed run + a bigger budget retrains B from scratch (interrupted runs
|
||
keep `checkpoint_latest.pth` and resume correctly).
|
||
- Verified 2026-10-04: A/C resumed epoch 1 → 3 (model restored, fresh
|
||
opt/sched for the old-format ckpt), re-run at 3/3 trained 0 epochs, B
|
||
continued from epoch 1 with LR restored mid-decay (0.00536 @ start of
|
||
epoch 2).
|
||
|
||
## Readiness notes for a full run
|
||
|
||
- Budget knobs: `--train-epochs` (default 100), `--max-rows` (default 80% of
|
||
split_train ≈ 2753 for A/C and for B's `--nnu-cases`, post-exclusion;
|
||
≈ 2856 pre-exclusion), `--max-eval-rows` (default 10% of split_test ≈ 40
|
||
post-exclusion), `--no-tta`.
|
||
- `--skip-train` is only usable for **A** (`runs/round0/best.pt`, trained
|
||
2026-09-25 on the *previous* split — a few patients moved between
|
||
train/test in the rebuild, so its test dice carries a small leakage
|
||
caveat). No study checkpoints exist for B (Dataset210) or C
|
||
(`runs_monai/round0/best.pt`); they must be trained.
|
||
- GPUs: 3× 80 GB available; the benchmark uses one (`--gpu`).
|
||
- B's one-time cost scales with `--nnu-cases` (plan_and_preprocess ≈ 3.5 s/case
|
||
at 24 cases with 8 preprocess procs, plus training time at the planned
|
||
batch/patch). |