⚠️ ATTENTION: Fold 4 IS NOT RUNNING!
Process appears stopped/crashed. Resume command(s):
python src/run_phase5_cfcp.py --step all --folds 4 --search-seeds 42 --device cuda --resume
Pipeline Health
4/5 Active (1 STOPPED)
Folds Status
Est. Pipeline Finish
Sat 10-03 04:58
ETA: 1d 21h (Fold 4)
GPU Compute
99%
Fan: 79% | Bus: 12%
Dedicated VRAM
8.3 / 10.0 GB (83%)
RTX 3080 GDDR6X
Shared GPU RAM
0.35 GB
Clean (Zero Spillover)
GPU Thermal & Power
55°C
214.2 W Draw
CPU Usage & Temp
33.6%
Temp: --
Host RAM
20.5 / 31.9 GB (64%)
System Memory
Snapshot Time: 2026-10-01 07:17:17 [Phase 5: CAEC + CFCP | Dir: phase5_caec_cfcp]
==============================================================================================================================================================================
Fold | Phase | Status | Gen | Cand/Gen | Current Epoch | Req/Uniq | CacheHit | Guided% | Best F1 | WorstRec | Sec/Ep | ETA | Est Finish
==============================================================================================================================================================================
0 | Search | Running | 15/30 | 3/20 | 68a42fc9 (4/5) | 280/242 | 13.6%(38) | 46.4% | 0.6277 | 0.1362 | 53.5s | 21.0h | Fri 10-02 04:14
1 | Search | Running | 13/30 | 6/20 | cd794166 (1/5) | 240/225 | 6.2%(15) | 43.3% | 0.6494 | 0.1296 | 99.8s | 1d 21h | Sat 10-03 04:42
2 | Search | Running | 15/30 | 3/20 | de8d94f9 (5/5) | 280/232 | 17.1%(48) | 45.7% | 0.6433 | 0.1477 | 45.0s | 16.9h | Fri 10-02 00:13
3 | Search | Running | 10/30 | 12/20 | f5d53a0b (2/5) | 180/169 | 6.1%(11) | 39.4% | 0.6556 | 0.1587 | 57.1s | 1d 5h | Fri 10-02 13:07
4 | Search | STOPPED | 8/30 | 10/20 | 6bb217a2 (1/5) | 140/133 | 5.0%(7) | 40.7% | 0.6446 | 0.1548 | 79.5s | 1d 21h | Sat 10-03 04:58
==============================================================================================================================================================================
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[⚠️ ATTENTION: STOPPED / CRASHED WORKERS]
* Fold 4 is NOT RUNNING (Status: STOPPED)!
- Stopped at: Phase Search, Gen 8/30
- Resume command:
python src/run_phase5_cfcp.py --step all --folds 4 --search-seeds 42 --device cuda --resume
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[ACTIVE WORKER DETAILS]
* Fold 0 [RUNNING]:
- Generation: 15/30 | Active Candidate ID: 68a42fc9e04af3132bb9c72ae259b12a7831fb757a713e493d802bcb08eb39fe
- Training Progress: Epoch 4/5 (measured: 53.5s/epoch)
- Last Epoch Val: Macro-F1 = 0.5310, Loss = 0.0000, Worst-Rec = 0.0000
* Fold 1 [RUNNING]:
- Generation: 13/30 | Active Candidate ID: cd794166355b45529b151de465c166ae1d7e82ae8c476ec3dc43eefdaecc118b
- Training Progress: Epoch 1/5 (starting epoch)
* Fold 2 [RUNNING]:
- Generation: 15/30 | Active Candidate ID: de8d94f9faa0c9bc7f260a9589c7830da4f9d8075286884547e5877799b468e8
- Training Progress: Epoch 5/5 (measured: 45.0s/epoch)
- Last Epoch Val: Macro-F1 = 0.5415, Loss = 0.0000, Worst-Rec = 0.0000
* Fold 3 [RUNNING]:
- Generation: 10/30 | Active Candidate ID: f5d53a0b212e93ad135e588c6608d4df3600b46f0e5d36ed3f52ebd19f29cb93
- Training Progress: Epoch 2/5 (measured: 57.1s/epoch)
- Last Epoch Val: Macro-F1 = 0.5784, Loss = 0.0000, Worst-Rec = 0.1587
--------------------------------------------------------------------------------