GenAI Reliance: Establishing Thresholds to Help Reduce Risk

Middle-market organizations are increasingly using generative AI (GenAI) tools to help streamline routine tasks and improve efficiency. Outputs are initially reviewed carefully and used alongside human judgment in many cases. However, as adoption grows, AI-generated content may begin to play a larger role in business processes, increasing the importance of understanding when and how those outputs are relied upon.

This increased usage can happen gradually and without documentation. As a result, organizations may find it challenging to distinguish between AI-assisted work and decisions influenced by AI-generated outputs. Defining that threshold can help clarify accountability, strengthen monitoring, and bring GenAI use into the internal control environment.

Expanding Governance to Cover GenAI Reliance

Model reliance isn’t a new concept. Regulated institutions governed under Supervisory Letter SR 11-7 have applied these principles for years. Similar considerations inform how external auditors evaluate management’s reliance on system-generated reports under SOX 404 and how SOC 1 and SOC 2 examiners look at control reliance within service organizations.

An underlying question for internal control programs to consider: Is the reliance placed on a system or output appropriate, given what is known about its accuracy, limitations, and history?

This question can apply to revenue systems, pricing engines, vendor risk scoring, fraud detection, and now, to GenAI tools embedded in finance, operations, and customer-facing workflows.

GenAI introduces similar reliance in a less visible, more pervasive form. A traditional system usually has defined outputs and a documented audit trail. A GenAI tool used to summarize a vendor contract, draft a customer renewal email, or build a pricing recommendation, for example, won’t have structural guardrails unless the organization has deliberately built them. Without that intentional design, reliance can grow faster than oversight.

How Does Growing Reliance on AI Impact Oversight?

When GenAI tools are first introduced, oversight tends to be intentional, and teams review outputs carefully, recognizing that human judgment is needed in the process. Review practices may become compressed as pressure to scale builds, and as outputs prove useful, oversight may lessen. Over time, an organization may accumulate decisions that were shaped by a system or tool it hasn’t fully monitored or integrated into its control environment.

Data shows where this risk is concentrating. The American Institute of CPAs (AICPA) and Chartered Institute of Management Accountants’ (CIMA) “Future-Ready Finance: Technology, Productivity, and Skills Survey Report,” with insights from more than 1,400 senior finance and accounting leaders, found that 88% of respondents expected AI to be the most transformative trend in accounting and finance over the next 12 to 24 months. However, only 8% felt their organization was “very well prepared,” and only 21% felt “well prepared.”1 A companion study noted that AI-transformed organizations were nearly twice as prepared on talent, IT, and regulatory readiness, with smaller organizations the least equipped to govern AI use.2 For middle-market companies operating between those two ends, adoption may advance faster than governance.

Updated guidance from the Committee of Sponsoring Organizations of the Treadway Commission (COSO) on GenAI governance identified this dynamic as one of the most operationally significant risks in the current environment. COSO’s updated GenAI guidance highlights risks associated with rapid adoption, including transparency challenges, model drift, prompt-based manipulation, and frequent configuration changes. Without active internal controls, these risks can affect the integrity of operations, reporting, and compliance.3

A Policy Is Not the Same as Oversight

Many middle-market organizations have established AI use policies that state that GenAI outputs cannot be used as the sole basis for financial, contractual, or customer-facing decisions. They also may define acceptable use cases, restrictive uses, and prohibited activities. A written policy is an important first step, but it may not provide sufficient protection on its own. If no monitoring mechanism exists to identify when AI use moves beyond the boundaries of the policy, the organization will likely have limited visibility into how the policy is operating in practice. Outsourcing a service doesn’t mean outsourcing the accountability that comes with it. The same principle applies to outsourcing judgment to a model.

Why Do Reliance Definitions Matter for Private Companies?

For private middle-market companies, the pressure to define reliance is increasing. Private equity sponsors are asking portfolio CFOs to demonstrate how GenAI is being used in finance and operations and how outputs are being validated. M&A diligence increasingly includes AI usage walk-throughs, and weak governance can impact valuation and warranty coverage. Cyber insurance underwriters are adding GenAI-specific questions to renewal applications, and customer and vendor contracts are beginning to require AI usage disclosures and governance representations, particularly for enterprise relationships.

Establishing Clear Thresholds

Defining reliance thresholds requires an organization to make specific, documented determinations about how GenAI is used. Those determinations should be embedded in the control environment, rather than in a standalone AI governance document. A practical starting point is a tiered structure tied to decision risks:

Tier	AI Use	Risk Level	Documentation & Review
Assistance Tier	AI drafts or summarizes information; human edits, reviews, and signs off.	Low risk	Standard workflow documentation required.
Review Tier	AI output informs a financial, contractual, or customer decision and a structured human review is mandatory before the output can be acted upon.	Moderate risk	Review must be logged and attributable.
Decisioning Tier	AI output directly influences a regulated, financial, or material business determination.	High risk	Mandatory human sign-off required; full audit trail maintained, and validation standards apply.

Each tier should identify who owns the decision, what review is required before the output is used, and how reliance will be monitored over time. Since AI processes can change as models are updated, point-in-time approvals may no longer be sufficient. Ongoing monitoring should now be part of the control approach.

Extending Existing Controls

Organizations may not need to build entirely new programs to address AI reliance. Existing disciplines may already provide a foundation, including SOX 404 documentation, SOC reporting structures, vendor risk management programs, and model governance standards. What may be needed is a deliberate, documented extension of those standards to GenAI tools operating in everyday workflows.

That means inventorying GenAI tools used for finance, operations, sales, and customer functions. It also means documenting each tool’s intended use, known limitations, reliance thresholds, and validation cadence. It can be especially helpful to identify tools that were adopted as productivity accelerants but now provide input into consequential decisions. This approach treats GenAI governance as an expansion of the control environment rather than as a separate initiative.

Actions to Consider Now

Organizations evaluating GenAI reliance should consider the following:

Look over AI-assisted workflows for reliance creep, specifically finding where review steps have been shortened or eliminated since GenAI tools were introduced.
Apply model governance principles to GenAI tools used in finance, operations, sales, and customer functions, including documentation of intended use, known limitations, and reliance boundaries.
Define accountability at the threshold by documenting who owns the decision when AI output informs it, and what human review is required before that output can be acted upon.
Create reliance tiers that classify AI use by decision importance, from drafting assistance up to high-stakes decisioning with mandatory sign-off.
Incorporate GenAI-reliance threshold assessments into your next internal control review cycle, whether that cycle is SOX, SOC, or a private company governance refresh.

How Forvis Mazars Can Help

Our IT Risk & Compliance team can help middle-market organizations by assisting with documenting current practices, identifying areas for enhancement, and supporting AI governance efforts. Our skilled professionals are committed to delivering an Unmatched Client Experience^® and helping you prepare for what’s next. If you have questions or need assistance with building GenAI reliance thresholds, please connect with our team today.