Evaluating AI procurement software: a buyer's checklist (2026)

Why you need a checklist

Every procurement-software vendor claims AI now, and the volume is hard to evaluate. Nexcade, a logistics AI company, tracks the public AI announcements of 40 to 50 leading forwarders and logistics providers, and its CEO described the pace of the past six to nine months as overwhelming, according to a FreightWaves interview. Meanwhile the work is growing: a Hackett Group projection cited in Arkestro’s October launch release says procurement workloads will increase 8% in 2026 alone while operating budgets and headcounts decline.

A checklist keeps the buying conversation on evidence. The framework below is ours; every example in it comes from a vendor announcement or survey we verified as of late October 2026.

Step 1: define the job and the autonomy level

Start from the process, not the product. Decide the lowest level of autonomy that solves it, because each level has different requirements:

Generative AI automates routine processes and supports decisions; agentic AI takes on tasks and interacts with other business systems, as the CPOstrategy article on procurement talent puts it. Most “AI” features in a demo sit in the first two rows. Scoring a fourth-row tool with first-row controls is the most common evaluation failure. See our AI skills gap analysis for who should review what.

Step 2: separate agent capability from AI-washing

AI-washing is dressing ordinary automation, or a generic chatbot bolted onto a product, in proprietary-AI language. Five questions cut through it:

• What does the feature do with live data? A model that answers questions from static documents is different from one that reads your open contracts and shipment records. Flexport’s MCP server, for example, exposes six named operations — track a shipment, browse active shipments and exceptions, search contract and spot rates, browse quotes, create a booking, look up account details — each callable by an outside assistant. Specific tool lists are testable; “AI-powered insights” are not.
• Can it act, or only advise? Arkestro says its MCP connectivity lets agents monitor sourcing activity, summarize what needs attention, draft supplier communications and prepare new competitive bids, with the enterprise controlling what information an agent can access and what actions it can take. Ask for that same specificity from anyone.
• How was the number measured? SAP claims 95% accuracy for the automated UNSPSC spend classification in its new Ariba Spend Analysis and Insights, a figure published in SAP’s own launch materials with no independent benchmark. Ask any vendor who measured the accuracy, on whose data, weighted by line count or by spend.
• Who will talk to us in production? Arkestro’s release quotes Crown Holdings’ director of indirect sourcing saying the company cut the time spent reviewing and coordinating supplier price updates by roughly 50%. That is a vendor-reported customer statement in a vendor press release — a legitimate example of the genre, and still a claim to verify by calling the customer.
• What breaks if the model changes? Ask whether the “proprietary engine” is a wrapped third-party model, and what happens to your workflows and pricing when the vendor swaps it.

Step 3: audit your data before buying intelligence

The same launch materials that claim 95% classification accuracy also name the problem that claim exists to solve: SAP writes, citing a 2026 Economist Enterprise report it sponsored, that one of the greatest barriers to harnessing AI is the lack of high-quality data, and that executive confidence in procurement’s ability to collaborate with the rest of the business fell from 90% in 2025 to 74% in 2026. Vendors are building the data fix into the product — Ariba Spend Analysis and Insights, generally available since Sept. 16, pulls spend continuously from Ariba, Fieldglass, Concur, SAP Cloud ERP and customer-managed sources, and enriches supplier records with Dun & Bradstreet hierarchy data weekly, moving to daily.

Before any demo, run your own checks: duplicate suppliers merged on tax IDs, the share of spend sitting in fallback category codes, and a few thousand invoice lines already classified by your team as a blind test set for the vendor. The same messy master data that slows your analysts slows every model. Our seven-step supplier master cleanup covers the routine.

Step 4: treat integration standards as a test of seriousness

Model Context Protocol is an open standard for connecting AI applications to external systems — data sources, tools and workflows — with support across major AI assistants and developer tools. Agent-to-agent (A2A) covers the machine-to-machine side. In 2026 these are useful dividing lines:

• SAP says Joule Work plans and executes multistep workflows across SAP and non-SAP systems using A2A and MCP, with enforced access control so third-party agents can invoke Joule agents through a governed agent gateway.
• Flexport’s server runs independently from its public API with separate rate limits and works with Claude, ChatGPT and Codex, Microsoft Copilot, Meta’s Muse and internal tools.
• Ask a vendor with no agent interface what its roadmap is and how it will control what an outside assistant can do to your data. “Connect it to ChatGPT” is a plan; “paste into ChatGPT” is shadow AI, which FreightWaves’ interviewees confirm is already happening in their industry.

Step 5: demand governance and auditability in writing

Pathlock’s 2026 AI Governance Gap Report, a May 2026 survey of 286 IT, compliance and security decision-makers summarized by SAPinsider, found 38% of organizations already allow agents to create or modify business records such as vendor and material master data, and 28% allow agents to approve transactions. Yet 52% cannot fully verify every action agents execute, only 19% report complete real-time visibility into agent actions, and just 17% have a team or officer dedicated to AI governance. Financial risk from incorrect transactions ranked third among concerns, behind security and compliance.

The governance features shipping now show what to demand from any vendor, not just SAP:

• Behavior visibility. Joule Studio gained observability and administrative controls over agent behavior, model usage and spend.
• A central control plane. The new SAP Enterprise AI Gateway consolidates authentication, policy enforcement, routing, monitoring, usage tracking and auditing for AI traffic (MCPs, APIs, A2As and LLMs); SAP plans general availability for Q1 2027.
• Guardrails inside existing permissions. SAP’s TechEd guide lists audit trails, role-based permissions and human-in-the-loop checkpoints as built-in for Joule Work’s agentic capabilities.
• Real revocation. On Flexport’s server, an administrator enables the connection org-wide, every assistant acts only within the permissions of the individual whose login it uses, and the admin can revoke any user or switch the whole organization off, ending active sessions immediately.

The RFP question that summarizes it: “Reconstruct for me, on demand, every action your agent took last month, with inputs, outputs and the human who approved each write.”

Step 6: build the ROI case on your baseline, not their slide

Vendor metrics exist — Arkestro’s ~50% price-review time reduction, Nexcade’s reported doubling of files handled per person, staged quotes replacing four or five emails each — but they are all vendor-reported. Structure your own measurement before signing:

Label every vendor percentage in your business case as vendor-reported, exactly as we do in our coverage of the Arkestro launch.

Step 7: write contract terms for the AI features

Standard SaaS terms were not drafted for software that acts. Negotiate six additions with legal:

1. Permissions. A written list of the systems and data an agent may read and change, with advance notice before that scope changes.
2. Data use. Whether your data trains models — and require that it not train models serving your competitors. SAP states plainly that it does not share customer data with anyone to train or improve large language models; treat that as the ask for everyone.
3. Logging. Every agent action logged, retained for a defined period, exportable, and available for audit on request.
4. Incident notice. Prompt notification if the system is manipulated (for example, through content injected into supplier emails or documents) or acts outside its permissions.
5. Liability. Who pays when an agent books the wrong freight, issues a wrong PO or releases a wrong price. Flexport’s account model makes each agent inherit one person’s permissions — the contract must say what happens when that person’s agent errs; our Flexport MCP story covers how its terms allocate that responsibility.
6. Pricing durability. What happens to fees if the AI feature is repriced, renamed or folded into a higher tier after launch.

Putting it together

Score each shortlisted platform on six areas — autonomy fit, evidence behind claims, data readiness, agent interfaces, governance and auditability, and AI contract terms — with pass signals and red flags per row. A tool can fail Step 1 and still be the right purchase for a drafting job; the failure mode the checklist prevents is buying fourth-row autonomy and discovering, at audit time, that nobody can show what it did.