Define OCE problem statement and solution requirements

[p4gs]

See this LinkedIn post for background context
We need to have a well defined and understood real-world problem that OCE is meant to solve. We then need to translate this into specific but generic requirements.

Once this is done, we can then work on determining if existing frameworks or tools fit these requirements at all (e.g. OSCAL, OCSF) or if we need to devise a new OCE data model from scratch. Furthermore, we need clear MVP requirements for creating a working template/mock API to prove the real-world value and usage of an OCE API standard.

For example

Problems:

• Compliance Automation tools produce evidence in diverse and inconsistent formats, requiring auditors to vet them for completeness & accuracy each time they encounter them for the first time
• Compliance Automation tool providers and GRC practitioners do not have a shared, universal understanding of what auditors consider "gold standards" for completeness & accuracy of control evidence

Requirements:

• OCE data model must include original raw source data
• OCE data model must include data query command or method performed to retrieve raw source data
• OCE data model must include data processing commands or code that was used to analyze and transform raw source data

[honda0306]

Problem:

• Evidence of control implementation and operation is almost universally taken and provided at a point in time, severely limiting the assurance value of such evidence.
• Infrequent non-compliance (in the form of deviations/exceptions to controls) may not always pose imminent danger; however, evidence of the audited organization is aware of and reliably takes action to proactively address deficiencies that cause non-compliance.

Requirements:

• OCE data model must include a "Compliant since {DATE}" or "No deviations since {DATE}" or similar field.
• OCE data model must include an object/reference that lists the history of non-compliance (Nice to Have/NTH: set default query to all periods of non-compliance in the last 365 days)
• The OCE data model must include references to the scope, root cause analysis, duration of non-compliance, and remediation action(s) associated with each instance of non-compliance.

[JonZeolla]

I love the idea of a standard data format and/or API. I've run into this completeness and accuracy problem a lot too during audits

However, I'm not clear on how a standardized API or schema solves that problem.

My experience has been like this:

1. Auditor asks for inventory of X (say, assets)
2. Auditor then attempts to assess completeness of the inventory by walking around a building looking for tagged assets or asking employees in the various audits to share their asset tags.
3. Separately the auditor challenges the validity of the information in the inventory by cross referencing multiple fields for a given record to another system or source of truth.

I'm trying to understand if there's another angle we could use to tackle this issue

It feels to me that the completeness problem can be solved with corroboration/linking separate data sets or reports. If we can find a way for multiple different systems to corroborate an entry then that would assure an auditor that it's correct. There's a number of ways to skin that cat of course but simple items like "asset inventory says we have 52 servers in AWS" and "AWS config shows 52 servers" and "AWS cli enumeration on date in AWS account X shows 52 servers" is a good start with the ability to drill down further

Am I on the same page? Interested in discussing further and contributing

[troyfine]

I think another element to consider is the C&A of the actual command or method of collecting raw data. It’s nice to show the command that was used but how does an auditor know the command was actually complete and accurate? Would there be a way to show the data elements that could be pulled with the API v. what was actually pulled? For instance, let’s say there is an API for the HR system and the auditor wants to see all active employees. There could be several categories of active employees within the HR system. In a typical audit, an auditor would get a report from the HR system and see the parameters that were used and not used to generate the report yo confirm C&A.

[p4gs]

@glodeneye as I've told you before in the GRC Engineering Community Discord, you need to refrain from posting GRC vendor pitches/propaganda. Not only is your comment here seemingly generated by an LLM (given its extreme verbosity and the overall gestalt of the writing style, formatting, tone, etc. of it) making it come off as disingenuous and not very thoughtful, but it references multiple times a ComplianceScorecard repo and "solutions" that seemingly have nothing to do with the topic at hand except for the fact that it pertains to OSCAL and we've mentioned OSCAL in this issue. I looked through the repo and noticed that your company's open source project only pertains to generating SSP/WISP type documents, which isn't relevant to the scope of this issue.

I'm hiding your comment as spam (because that's how it reads). If you continue posting pitches or propaganda for your company, subtly or explicitly, we'll ban you from all GRC Engineering Community channels (GitHub, Discord, LinkedIn, etc.)

You are actively undermining your own, and thus your company's, trustworthiness with the community by continuing to violate this rule of ours. I am not the only one who has noticed your attempts to promote your company in ways that are not pertinent or beneficial to the vendor-agnostic community collaboration we are seeking to foster.

[oajiboye]

@JonZeolla and @troyfine raise important points. I think a model might not be able the way to solve some of problems raised so far around completeness. Auditors still have to do work to understand the environment they’re auditing.

Assessing the completeness of a population of staff, for example, (say, from Workday) would require an understanding of how Workday categorizes staff, and an understanding of what additional layers of categorization the company has created. (One or both may exist in any tool, depending on the organization). I don’t know if a model can ease that kind of process for an auditor, to the point where they can simply receive some modeled/structured data and have it address those needs.

When I used to perform audits, assessing completeness was not just about corroborating different pieces of information. It was also important that the information come from separate/independent sources, or the point of corroboration would be defeated. If information from multiple, ostensibly separate sources were presented to me in a single model, with demonstrable correlation built into the model, I would still want to probe and understand the separateness/independence of those sources. If I couldn’t, that would have some implication for the audit and the procedures to be performed. So even if there were a model, the “work” would remain.

I’ve been a little removed from audits for a minute or two now, and I’m primarily focused on controls for their risk mitigation properties, not due to any external requirements, so I’m not sure if current practice has changed. But this is my contribution.

[trumant]

Here's an example of some thinking I did recently about modeling assessments and the evidence gathered by them https://github.com/trumant/pvtr-github-repo/blob/redesign-payload-interface/architecture.md

I'm sharing it not to claim it solves the same set of problems or is a deeply thought out proposal, but to show at least one example of how I've tried to model evidence into assessment result data models.

Another example can be found in prowler-cloud/py-ocsf-models#198

What may lead to a refinement of the requirements here is looking at existing schema attempts to model evidence well and ask what can be improved or what is not supported and yet is viewed as needed or valuable to have.

[JoshDoesIT]

I'm late to the party here apparently, but I'm taking a stab at just the problems for now. Consolidating what I'm reading from all of the great feedback above and making a few tweaks based on my own personal experience, I've got:

Problems:

• Compliance automation tools produce evidence in diverse and inconsistent formats, requiring auditors to vet them for completeness & accuracy each time they encounter them for the first time.
• GRC practitioners and auditors lack a common understanding of what constitutes complete and accurate control evidence. What satisfies one auditor may be rejected by another, even for identical controls.
• Every audit engagement involves re-validating that a tool's connectors correctly retrieve data from source systems. This work is duplicated across the industry with no mechanism to certify a connector once and reuse that trust.
• Evidence is often presented as final output without visibility into how it was retrieved or transformed. Auditors are left to trust the tool blindly or manually reconstruct the collection logic.
• Knowing what data was pulled doesn't prove completeness. Auditors need to understand what could have been pulled (the full API schema, available parameters, total population) to assess whether the evidence represents the complete "picture".
• Source systems categorize data differently (e.g., what "active employee" means in Workday varies by organization). Without this context, auditors cannot properly evaluate whether the evidence scope matches the control requirement.
• When multiple data sources are presented together, auditors cannot easily confirm those sources are truly independent. Corroboration loses its value if all the data ultimately traces back to the same underlying system.
• Evidence typically reflects point-in-time snapshots without historical compliance context. Auditors need to understand compliance duration, historical deviations, root causes, and remediation actions.

May need some further refining but I'm curious what you all think.