ct metrics roc
ROC curves and AUROC per monitor, each trajectory scored by its highest verdict an audit could act on, every attack trajectory a positive.
Usage
ct metrics roc [OPTIONS]
Options
| Option | Description |
|---|---|
--honest TEXT | Honest trajectories (path, dir, run ID or run URL). Mutually exclusive with --run-file. |
--attack TEXT | Attack trajectories, one curve each (repeatable). Mutually exclusive with --run-file. |
--run-file PATH | YAML run file with several honest/attack entries. |
--filter-by TEXT | Keep run-file entries whose field has this value (field=value, repeatable). |
--group-by [env|failure_type|main_task_category|side_task_category] | One curve per run and value of this field. |
--color-by TEXT | Run field mapped to curve color. |
--style-by TEXT | Run field mapped to curve linestyle. |
--min-samples INTEGER RANGE | Leave out a curve with fewer scored trajectories on either side. [default: 2; x>=1] |
--output-dir PATH | Directory for the plots (default: data/roc/<timestamp>). |
--max-workers INTEGER RANGE | Thread-pool size for loading run entries concurrently. [default: 8; x>=1] |
--help | Show this message and exit. |