An AI vendor can pass a conventional security review and still create an unacceptable exposure path. The gap is often not a missing SOC 2 report or an incomplete contract clause. It is the difference between what the vendor says its platform does and how employees, developers, and integrated systems actually use it. This AI vendor risk assessment guide is designed for security teams that need to close that gap before sensitive data, intellectual property, or regulated information reaches a model they cannot adequately observe or control.
AI vendor assessments should not be treated as a procurement checkpoint. They are a security decision that needs to account for data flows, model behavior, integration privileges, vendor operations, and real-world enterprise usage. Visibility comes first. A policy cannot protect interactions your team cannot see, classify, or investigate.
Start with Usage Evidence, Not the Vendor Questionnaire
Vendor documentation has value, but it is a statement of intended operation. It rarely answers the questions that matter after deployment: Which teams are using the service? What are they submitting? Which connected repositories can the tool access? Are developers calling the model through an approved API, a personal account, or an embedded feature in another application?
Before reviewing controls, establish the proposed use case in operational terms. Identify the business owner, expected users, access method, data categories, model functions, integrations, and environments involved. A customer-support summarization tool with tightly scoped inputs presents a different risk profile than an enterprise coding assistant connected to source control, tickets, and internal documentation.
This distinction prevents a common failure mode: approving a vendor based on a narrow use case, then discovering months later that its capabilities have spread across departments and data sources. The assessment should define what is being approved, for whom, and under what conditions. Anything beyond that boundary requires review.
AI Vendor Risk Assessment Guide: Define the Risk Boundary
The first phase is scoping. Security teams should be able to describe the risk boundary without relying on vendor marketing language. Ask what enters the system, what the system can retrieve, what it generates, where information is retained, and who can act on its output.
Classify the data and the actions
Data classification remains central, but AI changes how data moves. Prompts, uploads, retrieval context, conversation histories, feedback signals, generated files, and API metadata may all contain sensitive information. Review each of these paths separately.
Also distinguish between an AI tool that advises and one that acts. A model that drafts an internal memo can create confidentiality and accuracy concerns. An AI agent that can create accounts, modify records, send messages, or execute code adds authorization and change-control risk. The second category needs stronger identity controls, narrower permissions, logging, and human approval at meaningful decision points.
Identify the model and hosting arrangement
The name of the application is not enough. Determine whether the vendor operates its own model, calls a third-party foundation model, uses several models depending on task or region, or supports customer-selected models. Each arrangement changes the chain of responsibility and the organizations that may process enterprise data.
Hosting matters as well. Assess where inference occurs, where data is stored, whether processing can cross geographic boundaries, and whether customer content can be used for training, evaluation, or product improvement. Contract language should be precise about these conditions. A broad statement that data is handled securely does not answer whether prompts are retained for 30 days, whether support staff can access them, or whether a subprocesser receives them.
Map integrations and identity paths
Integrations often carry more risk than the chat interface. An AI vendor with read access to document repositories may expose far more data than a vendor receiving manual text input. Review OAuth scopes, service accounts, API keys, delegated permissions, webhook destinations, and administrative roles.
Apply least privilege to the integration, not just the user account. If the tool only needs access to a defined knowledge base, it should not receive access to every shared drive. If a service account is necessary, its activity should be attributable, monitored, and removable without disrupting unrelated systems.
Evaluate the Controls That Matter in Practice
A useful review separates declared controls from controls your team can validate. Certifications, architecture diagrams, and security questionnaires are inputs. They should lead to evidence-based follow-up, not replace it.
Request evidence for at least these areas:
- Data retention, deletion, training, and subprocesser practices, including the treatment of prompts, files, and generated content.
- Identity and access controls, including SSO, MFA, SCIM support, administrative roles, session controls, and tenant isolation.
- Logging and auditability, including what events are recorded, how long logs are retained, and whether security teams can export them.
- Application and infrastructure security, such as encryption boundaries, vulnerability management, penetration testing, incident response, and secure development practices.
- AI-specific safeguards, including prompt injection defenses, retrieval restrictions, output filtering, model evaluation, abuse detection, and red-team testing.
The depth of review should match the exposure. A low-volume, public-data use case may not justify the same assessment burden as an agent with access to financial systems. But lower risk does not mean no risk. It means the required safeguards, approval path, and monitoring level can be proportionate.
Security teams should also test how the vendor handles difficult questions. Can it explain how a user interaction is logged? Can it identify the subprocesser that handles a specific service component? Can it show how an administrator revokes access during an incident? Clear answers and supporting artifacts are stronger signals than broad claims about enterprise readiness.
Score Risk by Exposure and Detectability
A traditional vendor score often emphasizes the vendor’s control posture. That is necessary, but incomplete for AI. Add two dimensions: exposure and detectability.
Exposure measures the sensitivity and volume of information the service can receive or retrieve, plus the consequences of erroneous output or autonomous action. Detectability measures whether your organization can see the relevant activity in time to investigate and respond. A vendor may have sound internal controls while remaining a high operational risk if security teams cannot identify who used it, what data was shared, or which model processed the request.
This produces more useful decisions than a single pass-fail rating. A tool may be approved for public and internal data but prohibited from regulated data. It may be permitted for a pilot group with SSO and no connected repositories, while broader deployment waits for audit-log access. It may be rejected because its data-use terms conflict with company requirements. The outcome should state the rationale, scope, compensating controls, owner, and review date.
Treat Approval as the Beginning of Monitoring
AI capabilities, vendor terms, model providers, and employee behavior change quickly. A point-in-time assessment becomes stale when the vendor adds connectors, changes retention defaults, introduces agentic features, or shifts workloads to a new model provider.
Build reassessment triggers into the approval record. Material changes should include new integrations, expanded data classes, new autonomous actions, changes to training or retention terms, security incidents, and major model updates. Procurement renewal is not an adequate trigger on its own.
Continuous visibility is equally important inside the enterprise. Security teams need to discover approved and unapproved AI usage, observe interaction patterns, identify risky behavior, and preserve context for investigation. That includes browser-based tools, direct API use, embedded AI features, and developer workflows that may bypass the original procurement process.
This is where an operational AI security program differs from a paper governance program. If an employee pastes source code into an unapproved tool, or an approved vendor begins receiving sensitive records through a newly enabled connector, the security team needs evidence that supports action. ModelSight is built around that visibility-first requirement: understand actual AI usage before attempting to control it.
Make the Assessment Defensible
A defensible assessment leaves a record that another security leader, auditor, or incident responder can understand. Preserve the approved use case, data classification, architecture assumptions, vendor evidence, identified gaps, risk acceptance decisions, required controls, and reassessment triggers. Assign an accountable business owner and a security owner.
Avoid treating exceptions as informal permissions. If business urgency requires a limited exception, document its expiry date and the controls that make it tolerable. Temporary access has a habit of becoming permanent unless someone owns its review.
The goal is not to eliminate AI adoption. It is to prevent unknown adoption from becoming an incident-response problem. When teams can see the vendor, the model, the data path, and the behavior around it, they can make faster decisions with evidence instead of assumptions.