Furen Xiao
e7721760f2
- Updated `05_monai_run_iterative.py` to include legacy mode and improved configuration options for training. - Introduced `UNetBAligned` class in `monai_common.py` for deep supervision and instance normalization. - Modified `train_transform` and `make_dataloader` in `monai_common.py` to support B-aligned data protocol with Z-score normalization and augmentation. - Added new loss functions `nn_dc_and_ce` and `b_dc_and_ce` in `losses.py` for improved training performance. - Enhanced `Unet3D` architecture in `unet3d.py` to support leaky ReLU and deep supervision. - Created new augmentation functions in `augment_b.py` for Pipeline B's training, including rotation, scaling, and noise. - Implemented new training scripts `run_b_100ep_stages.sh` and `run_c_bloss_100ep_after_b.sh` for structured training and evaluation. - Updated dataset handling in `dataset.py` to accommodate new augmentation and normalization strategies.
19 lines
No EOL
1 KiB
Bash
19 lines
No EOL
1 KiB
Bash
#!/bin/bash
|
|
# Wait for B's 100-ep stages (stage 2 train + stage 3 best-ckpt inference) to
|
|
# release GPU 0, then run the C b-loss-only 100-ep ablation.
|
|
source /opt/conda/etc/profile.d/conda.sh && conda activate longitudinal-313
|
|
cd /mnt/b4/xfr/git26/longitudinal
|
|
echo "waiting for GPU0 (B 100ep stages) at $(date -u)"
|
|
while true; do
|
|
mem=$(nvidia-smi --id=0 --query-gpu=memory.used --format=csv,noheader,nounits 2>/dev/null | tr -d ' ')
|
|
util=$(nvidia-smi --id=0 --query-gpu=utilization.gpu --format=csv,noheader,nounits 2>/dev/null | tr -d ' ')
|
|
if [ -n "$mem" ] && [ "$mem" -lt 1000 ] && [ -n "$util" ] && [ "$util" -lt 5 ]; then
|
|
echo "GPU0 free (mem=${mem}MiB util=${util}%) at $(date -u); launching C b-loss-only 100ep"
|
|
break
|
|
fi
|
|
sleep 120
|
|
done
|
|
python scripts/benchmark_pipelines.py --pipelines C --b-loss-only --gpu 0 --train-epochs 100 \
|
|
--eval-at "5 10 20 50" --workers 4 \
|
|
--out results/benchmark_full/benchmark_full_C_bloss_100ep.json 2>&1 | tee logs/bench_C_bloss_100ep.log
|
|
echo "C b-loss-only 100ep finished at $(date -u)" |