⚠️ ATTENTION: Fold 4 IS NOT RUNNING!
Process appears stopped/crashed. Resume command(s):
python src/run_phase5_cfcp.py --step all --folds 4 --search-seeds 42 --device cuda --resume
Pipeline Health
4/5 Active (1 STOPPED)
Folds Status
Est. Pipeline Finish
Sat 10-03 07:10
ETA: 1d 21h (Fold 4)
GPU Compute
99%
Fan: 79% | Bus: 14%
Dedicated VRAM
8.3 / 10.0 GB (83%)
RTX 3080 GDDR6X
Shared GPU RAM
0.00 GB
Clean (Zero Spillover)
GPU Thermal & Power
55°C
204.9 W Draw
CPU Usage & Temp
29.6%
Temp: --
Host RAM
18.0 / 31.9 GB (56%)
System Memory
Snapshot Time: 2026-10-01 09:28:24 [Phase 5: CAEC + CFCP | Dir: phase5_caec_cfcp]
==============================================================================================================================================================================
Fold | Phase | Status | Gen | Cand/Gen | Current Epoch | Req/Uniq | CacheHit | Guided% | Best F1 | WorstRec | Sec/Ep | ETA | Est Finish
==============================================================================================================================================================================
0 | Search | Running | 17/30 | 5/20 | a05d1164 (4/5) | 320/273 | 14.7%(47) | 47.2% | 0.6277 | 0.1438 | 47.2s | 15.9h | Fri 10-02 01:24
1 | Search | Running | 15/30 | 4/20 | 6f7aa1a8 (5/5) | 280/260 | 7.1%(20) | 46.1% | 0.6494 | 0.1367 | 48.6s | 19.7h | Fri 10-02 05:12
2 | Search | Running | 17/30 | 7/20 | 961c8425 (2/5) | 320/263 | 17.8%(57) | 47.5% | 0.6433 | 0.1508 | 39.8s | 12.9h | Thu 10-01 22:21
3 | Search | Running | 12/30 | 11/20 | 5479a9b3 (3/5) | 220/202 | 8.2%(18) | 41.4% | 0.6556 | 0.1367 | 47.5s | 22.5h | Fri 10-02 07:55
4 | Search | STOPPED | 8/30 | 10/20 | 6bb217a2 (1/5) | 140/133 | 5.0%(7) | 40.7% | 0.6446 | 0.1548 | 79.5s | 1d 21h | Sat 10-03 07:10
==============================================================================================================================================================================
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[⚠️ ATTENTION: STOPPED / CRASHED WORKERS]
* Fold 4 is NOT RUNNING (Status: STOPPED)!
- Stopped at: Phase Search, Gen 8/30
- Resume command:
python src/run_phase5_cfcp.py --step all --folds 4 --search-seeds 42 --device cuda --resume
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[ACTIVE WORKER DETAILS]
* Fold 0 [RUNNING]:
- Generation: 17/30 | Active Candidate ID: a05d1164554c0668df546e9c08e7d9945a622f3a6909db6459d9d90df91cb642
- Training Progress: Epoch 4/5 (measured: 47.2s/epoch)
- Last Epoch Val: Macro-F1 = 0.5420, Loss = 0.0000, Worst-Rec = 0.0000
* Fold 1 [RUNNING]:
- Generation: 15/30 | Active Candidate ID: 6f7aa1a8f457d8f5a5c3de1009b5884f09cd7379379cd467243d55ca2867e5c2
- Training Progress: Epoch 5/5 (measured: 48.6s/epoch)
- Last Epoch Val: Macro-F1 = 0.6347, Loss = 0.0000, Worst-Rec = 0.0636
* Fold 2 [RUNNING]:
- Generation: 17/30 | Active Candidate ID: 961c84257c088cbb53f71ae97f4cb82bbf65d55e74b67bfd10ed5b965970d86a
- Training Progress: Epoch 2/5 (measured: 39.8s/epoch)
- Last Epoch Val: Macro-F1 = 0.5694, Loss = 0.0000, Worst-Rec = 0.1249
* Fold 3 [RUNNING]:
- Generation: 12/30 | Active Candidate ID: 5479a9b3507fcb1c4f8b94d67f47b34856738bc0cc3777b69171ff090776cb55
- Training Progress: Epoch 3/5 (measured: 47.5s/epoch)
- Last Epoch Val: Macro-F1 = 0.5957, Loss = 0.0000, Worst-Rec = 0.0259
--------------------------------------------------------------------------------