major c7 cleanup #111
major c7 cleanup #111
Conversation
ottosulin
left a comment
ottosulin
left a comment
There was a problem hiding this comment.
LGTM overall, one comment about log contents.
| | **7.6.5** | **Verify that** monitoring pipelines are penetration-tested and access-controlled to avoid leakage of sensitive logs. | 3 | V | | ||
| | **7.6.1** | **Verify that** the system logs real-time metrics for safety violations (e.g., "Hallucination Detected", "PII Blocked").| 1 | D | | ||
| | **7.6.2** | **Verify that** the system triggers an alert if safety violation rates exceed a defined threshold within a specific time window. | 1 | V | | ||
| | **7.6.3** | **Verify that** logs include the specific model version and data source ID to allow tracing errors back to a specific release. | 2 | V | |
There was a problem hiding this comment.
Should do the logs include the abusing user id as well? I remember reading somewhere that some AI systems may record the type of request as part of metadata for these purposes without logging the request content itself.
Alternatively we write this in more general terms, something like logs include the specific model version and data source ID and other details necessary to investigate potential abuse.
There was a problem hiding this comment.
Alternatively we write this in more general terms, something like logs include the specific model version and data source ID and other details necessary to investigate potential abuse.
I like this idea the best.
There was a problem hiding this comment.
How about:
| # | Description | Level | Role |
|---|---|---|---|
| 7.6.3 | Verify that logs include the specific model version and other details necessary to investigate potential abuse. | 2 | V |
Updates per #111
This pull request does a major cleanup of C7 for your review :)