Share:

AI Is Good at Picking Qualified Suppliers. It Still Struggles to Pick the Best One.

Companies are rapidly integrating generative AI into procurement and sourcing decisions. The promise is obvious. AI can read thousands of pages of supplier proposals faster than any human team, summarize technical requirements in seconds, and create the appearance of consistency and objectivity in evaluation.  But there is an important distinction managers are starting to overlook.

The same AI system that performs extremely well at identifying whether a supplier meets minimum requirements may perform much less reliably when judging which supplier is truly better.

Recent research published in the Journal of Business Logistics examined this issue by comparing how large language models evaluated supplier bids against evaluations completed by experienced procurement professionals. It analyzed 123 supplier proposals tied to 31 public procurement projects conducted by the State of Ohio between 2023 and 2024. The projects involved complex IT services contracts, many containing large, text-heavy bid packages requiring evaluative judgment rather than simple arithmetic comparisons.

The researchers tested three reasoning-oriented AI models: OpenAI o3, Grok-3-Mini, and DeepSeek R1. They then compared their evaluations against human procurement scores.  The findings revealed a surprisingly clear pattern.

AI performed well when evaluating compliance signals. These are signals tied to baseline qualifications and technical requirements. Does the supplier meet the required certifications? Do they satisfy the mandatory specifications? Did they include the required documentation? Are the implementation requirements addressed?

On these types of tasks, the AI models showed relatively high agreement with human evaluators and relatively stable scoring behavior across repeated evaluations.  But the results changed once proposals shifted from compliance to differentiation.

When suppliers attempted to distinguish themselves through strategic capabilities, innovation claims, implementation approaches, past experience, or value-added propositions, AI scoring became far more volatile. The same proposal could receive meaningfully different evaluations across repeated AI runs even when the prompt and underlying content remained unchanged.

That volatility matters.

In procurement, the most important decisions often occur after baseline qualification has already been established. Most serious bidders can satisfy the minimum requirements. Competitive advantage comes from identifying which supplier will create superior long-term value, adapt better during uncertainty, collaborate more effectively, or reduce implementation risk in ways that are difficult to fully codify.

Those judgments require interpretation, contextual reasoning, and tradeoff assessment. Humans are imperfect at this too, but experienced procurement professionals rely on domain expertise and pattern recognition developed over years of evaluating suppliers and managing outcomes.

Large language models work differently. They generate probabilistic outputs based on statistical relationships in language rather than genuine understanding of supplier quality or operational fit. That distinction becomes especially important in ambiguous or strategically nuanced evaluations.

Many executives currently frame AI adoption as a replacement question: “Can AI evaluate suppliers as well as humans?” That is the wrong question.  A better question is: “Which parts of supplier evaluation are structured enough for AI to handle reliably, and which parts still require human judgment?”

The answer emerging from the research suggests a hybrid approach.

In the first stage, AI can handle qualification screening. It can rapidly process proposals, verify compliance requirements, identify missing information, summarize technical content, and flag inconsistencies. This reduces administrative burden and allows procurement professionals to focus their attention where it matters most.

In the second stage, humans should take the lead in evaluating differentiation. This is where procurement teams assess strategic fit, implementation realism, innovation potential, relationship quality, operational flexibility, and long-term value creation. These decisions are often embedded in subtle contextual cues that AI systems do not evaluate consistently.

One of the most interesting findings from the study is that AI volatility itself may become a useful management signal.  When repeated AI evaluations produce highly inconsistent scores, managers should interpret that inconsistency as a warning sign rather than a nuisance. In many cases, volatility may indicate that the proposal contains ambiguous, subjective, or strategically complex content requiring deeper human review.

In other words, AI uncertainty may serve as a diagnostic tool for identifying where human expertise is most valuable.  This has implications beyond procurement.

Many organizations are currently deploying generative AI into judgment-heavy workflows involving hiring, performance evaluations, contract review, lending decisions, and strategic analysis. In many of these contexts, AI may excel at standardized screening tasks while struggling with contextual differentiation and nuanced tradeoffs.  Managers should resist the temptation to confuse speed with understanding.

The real opportunity is not eliminating humans from decision processes. It is reallocating human attention more effectively.

The organizations that benefit most from generative AI will likely be those that understand where automation creates leverage and where human expertise still creates advantage.

Supplier selection sits directly at that intersection.

 

Based on research published in the Journal of Business Logistics: 

Finnegan A. McKinley, Anne E. Dohmen, and Vincent E. Castillo, “Do Humans and GAI See Eye to Eye? Implications of LLM Scoring Volatility in Supplier Evaluations,” Journal of Business Logistics, 2026, 47. https://doi.org/10.1111/jbl.70072.

More Blogs

supply chain AI
Blogs
June 16, 2026

SAP Sapphire 2026: Key Takeaways & What’s Next for Supply Chain

Scott Luton: “SAP Sapphire 2026 featured a major emphasis on the Autonomous Enterprise and AI-powered business operations. From your perspective, what was one session (or prevalent theme amongst attendees) that you found to be particularly intriguing? And why?” Richard Howells: “Great session by Dominik Mark Metzger on the Future of Supply Chain – SAP Sapphire Virtual – Session Catalog. At SAP Sapphire, we’re introducing Autonomous Supply Chain Management, enabled by new Joule Assistants and Industry AI scenarios that apply this model to daily operations across planning, manufacturing, logistics, engineering, and asset management. General availability will be phased throughout 2026, starting now. By introducing a new set of AI-driven assistants and agents, we’re moving orchestration toward an autonomous operating model, where planning, manufacturing, logistics, and asset operations increasingly anticipate, coordinate, and resolve without manual intervention at every step. Highlight for me was the simple definition of the Autonomous Supply Chain where “People direct, AI Assistants orchestrate, and Agents execute”. Also, a callout to Rebecca Kaufmann from Takeda Pharmaceuticals for a great overview of how her company are, and plan to leverage AI to “augment human expertise and reduce the technical burden”.” Scott Luton: “There’s a growing sense across the industry that…
logistics
Blogs
July 10, 2025

Five Questions I Wished People Asked Me About Public Sector Supply Chains

Special Guest Blog Post written by Gary Smith, CPIM-F, CSCP-F, CLTD-F, Author of “The Bridge,” Supply Chain Engineer, and Educator   1. Do Supply Chains exist in the public sector? The short answer is “YES!” I spent the first 25 years of my career in the private sector where I worked in warehouse operations, industrial engineering, and consulting, all in the logistics andsupply chain area. I cut my teeth on retail, automotive, chemical, food and beverage, publishing, and manufacturing. During that time, I also completed projects for the public sector in education and for the Department of Defense. In 2005 I was named Director of Supply Chain Operations for the New York City Housing Authority (NYCHA). We operated a 200,000 sq. ft. warehouse that shipped repair material to 335 developments in New York’s five boroughs. In 2013, I was named Vice President of Supply Logistics for New York City Transit, the largest public transportation organization in North America. We operated over 1.5 million sq. ft. of warehouse space covering all of New York City. Several years ago, I was on a national committee for the Association for Supply Chain Management (ASCM). We created this elevator speech to describe Supply Chain…