System and Organization Controls (SOC) readiness often requires significant time from technology, compliance, and finance team members, especially when evidence collection is dependent upon elements like screenshots, manual exports, and repeated reconciliations. AI and automation offer opportunities to reduce that work, but their usefulness often depends on reliable information and an informed review. This article covers how using AI within SOC Engagements can help service organization stakeholders focus on complex risks, strengthen evidence validation, and prepare for more connected evidence collection.
Balancing Automation With Professional Judgment
Validation is a necessary element of automation. Matching records across data sets, comparing configurations against established baselines, and reviewing access listings can consume time that practitioners could otherwise use for understanding system architecture, investigating exceptions, and considering emerging risks. Automated procedures can help uncover patterns and direct attention to areas that need closer examination.
Automated change management procedures, for example, can extract changes from a source-code repository and compare them with related pull requests, change tickets, approvals, testing evidence, and deployment records to identify whether they followed the organization’s expected development lifecycle. Configuration comparisons can identify settings that deviate from a defined standard. These results can provide information for further investigation, but do not, on their own, establish whether a control operated effectively or whether compensating controls sufficiently address an identified gap.
Broader population analysis also doesn’t eliminate sampling or change the SOC Examination’s objective. A risk-based approach and professional judgment still need to be applied to provide reasonable assurance.
“We want to make sure that we’re using automation to test those baseline controls where possible so we can focus more on emerging risk.” – Ryan Boggs, “AI in SOC Examinations: Automation & Audit Judgment”
Strengthening Evidence Without Losing the Audit Trail
Automation is only as reliable as the information that supports it. Information Produced by the Entity (IPE), whether drawn from applications, spreadsheets, vendors, or manual processes, needs to be complete and accurate before it supports AI-enabled procedures. A result showing no exceptions may reflect effective controls, but it also could indicate missing records or flawed logic.
Evidence reliability can be backed up via the following three factors:
- Completeness: Reconcile record counts or control totals to source systems; confirm relevant date ranges, time zones, and cutoff periods; and evaluate filters, pagination, archived or deleted records, and other conditions that could exclude in-scope information.
- Accuracy: Trace selected fields to original records; recalculate relevant information; review report and query logic; and identify and track manual changes that could affect the information.
- Model or tool validation: Define the intended use and acceptance criteria; document the tool’s logic, assumptions, thresholds, inputs, and limitations; test it using cases with known outcomes; and investigate false positives, false negatives, errors, and systematic exclusions.
“Technology can support the work that we do … but it can’t replace accountability and professional skepticism.” – Karen Cardillo, “AI in SOC Examinations: Automation & Audit Judgment”
A qualified owner should understand the process from input through output. The automated results need to be traced back to the original tickets, logs, or configurations. Automation can weaken the audit trail rather than strengthen it without that connection.
API-Driven Evidence Collection Preparation
Application programming interface (API)-driven connections can retrieve evidence directly from cloud platforms to help reduce reliance on manual screenshot requests and support collection at an established frequency. This process gives service organizations an opportunity to reduce collection friction while simultaneously gaining more relevant visibility into configurations and control activities. It doesn’t mean the auditor is continuously examining every transaction or providing continuous assurance, though.
Preparation begins with identifying where AI is already used and which applications are relevant to the SOC Engagement. Organizations should validate connectors before relying on their outputs, including whether they retrieve the intended information and apply appropriate filters. Retaining validation evidence, queries, parameters, and outputs helps make the process repeatable and supportable. Ongoing or periodic monitoring is important as systems and processes change.
How Forvis Mazars Can Help
Our experienced SOC & HITRUST® professionals at Forvis Mazars are ready to help you prepare for what’s next. Working alongside you, we can help you assess opportunities to streamline evidence collection and identify gaps within your SOC readiness processes. If you would like to learn more, please reach out to a professional at Forvis Mazars.
Watch the on-demand webinar, “AI in SOC Examinations: Automation & Audit Judgment,” for more information.