Share:

AI Is Good at Picking Qualified Suppliers. It Still Struggles to Pick the Best One.

Companies are rapidly integrating generative AI into procurement and sourcing decisions. The promise is obvious. AI can read thousands of pages of supplier proposals faster than any human team, summarize technical requirements in seconds, and create the appearance of consistency and objectivity in evaluation.  But there is an important distinction managers are starting to overlook.

The same AI system that performs extremely well at identifying whether a supplier meets minimum requirements may perform much less reliably when judging which supplier is truly better.

Recent research published in the Journal of Business Logistics examined this issue by comparing how large language models evaluated supplier bids against evaluations completed by experienced procurement professionals. It analyzed 123 supplier proposals tied to 31 public procurement projects conducted by the State of Ohio between 2023 and 2024. The projects involved complex IT services contracts, many containing large, text-heavy bid packages requiring evaluative judgment rather than simple arithmetic comparisons.

The researchers tested three reasoning-oriented AI models: OpenAI o3, Grok-3-Mini, and DeepSeek R1. They then compared their evaluations against human procurement scores.  The findings revealed a surprisingly clear pattern.

AI performed well when evaluating compliance signals. These are signals tied to baseline qualifications and technical requirements. Does the supplier meet the required certifications? Do they satisfy the mandatory specifications? Did they include the required documentation? Are the implementation requirements addressed?

On these types of tasks, the AI models showed relatively high agreement with human evaluators and relatively stable scoring behavior across repeated evaluations.  But the results changed once proposals shifted from compliance to differentiation.

When suppliers attempted to distinguish themselves through strategic capabilities, innovation claims, implementation approaches, past experience, or value-added propositions, AI scoring became far more volatile. The same proposal could receive meaningfully different evaluations across repeated AI runs even when the prompt and underlying content remained unchanged.

That volatility matters.

In procurement, the most important decisions often occur after baseline qualification has already been established. Most serious bidders can satisfy the minimum requirements. Competitive advantage comes from identifying which supplier will create superior long-term value, adapt better during uncertainty, collaborate more effectively, or reduce implementation risk in ways that are difficult to fully codify.

Those judgments require interpretation, contextual reasoning, and tradeoff assessment. Humans are imperfect at this too, but experienced procurement professionals rely on domain expertise and pattern recognition developed over years of evaluating suppliers and managing outcomes.

Large language models work differently. They generate probabilistic outputs based on statistical relationships in language rather than genuine understanding of supplier quality or operational fit. That distinction becomes especially important in ambiguous or strategically nuanced evaluations.

Many executives currently frame AI adoption as a replacement question: “Can AI evaluate suppliers as well as humans?” That is the wrong question.  A better question is: “Which parts of supplier evaluation are structured enough for AI to handle reliably, and which parts still require human judgment?”

The answer emerging from the research suggests a hybrid approach.

In the first stage, AI can handle qualification screening. It can rapidly process proposals, verify compliance requirements, identify missing information, summarize technical content, and flag inconsistencies. This reduces administrative burden and allows procurement professionals to focus their attention where it matters most.

In the second stage, humans should take the lead in evaluating differentiation. This is where procurement teams assess strategic fit, implementation realism, innovation potential, relationship quality, operational flexibility, and long-term value creation. These decisions are often embedded in subtle contextual cues that AI systems do not evaluate consistently.

One of the most interesting findings from the study is that AI volatility itself may become a useful management signal.  When repeated AI evaluations produce highly inconsistent scores, managers should interpret that inconsistency as a warning sign rather than a nuisance. In many cases, volatility may indicate that the proposal contains ambiguous, subjective, or strategically complex content requiring deeper human review.

In other words, AI uncertainty may serve as a diagnostic tool for identifying where human expertise is most valuable.  This has implications beyond procurement.

Many organizations are currently deploying generative AI into judgment-heavy workflows involving hiring, performance evaluations, contract review, lending decisions, and strategic analysis. In many of these contexts, AI may excel at standardized screening tasks while struggling with contextual differentiation and nuanced tradeoffs.  Managers should resist the temptation to confuse speed with understanding.

The real opportunity is not eliminating humans from decision processes. It is reallocating human attention more effectively.

The organizations that benefit most from generative AI will likely be those that understand where automation creates leverage and where human expertise still creates advantage.

Supplier selection sits directly at that intersection.

 

Based on research published in the Journal of Business Logistics: 

Finnegan A. McKinley, Anne E. Dohmen, and Vincent E. Castillo, “Do Humans and GAI See Eye to Eye? Implications of LLM Scoring Volatility in Supplier Evaluations,” Journal of Business Logistics, 2026, 47. https://doi.org/10.1111/jbl.70072.

More Blogs

human factor in supply chain
Blogs
March 4, 2026

Adapt or Be Left Behind: Jorge Morales on Technology, Personal Growth and the Human Core of Supply Chain

At Manifest 2026, Scott Luton spent time with his friend, Jorge Morales, Global Chief Operating Officer of the International Supply Chain Education Alliance (ISCEA), for a conversation that centered not on hype, but on growth. ISCEA is a global certification body serving supply chain professionals around the world. But as Morales put it, “We’re in the knowledge business; but most of all, we’re in the personal growth business.” Through certifications, membership programs, advisory boards, and global events, ISCEA empowers professionals to stay current, expand their networks, and advance their careers. In a volatile era, that mission has never been more relevant.   Beyond the AI Hype Cycle When asked about dominant themes from 2025 to 2026, Morales pointed to a shift in how the industry views technology. “Last year we were still in the bubble of the hype of AI,” he said. Many professionals feared being replaced or assumed AI would solve everything overnight. That perspective has matured. “Technology by itself is not what determines your level of success,” Morales explained. “Understanding the basics, knowing how things work; THAT is key.” AI and data tools remain critical across procurement, logistics, manufacturing, planning and more. But success depends on how organizations…
inventory control
Blogs
August 13, 2026

Blame Follows the Messenger

Customers blame retailers less when the delay notification comes from someone else   When something goes wrong, most companies reach for the same playbook: communicate early, communicate often, and keep customers informed. It is difficult to argue with that logic. Customers hate surprises, and transparency has become a cornerstone of good service. Whether the issue is a delayed shipment, a production disruption, or a missed commitment, managers generally assume that more communication demonstrates accountability and builds trust. New research published in the Journal of Business Logistics suggests the real lesson is narrower and more useful than “communicate more.” The study shows that communication does more than transfer information. It also shapes how customers assign responsibility. The volume of communication was not what mattered. The identity of the messenger was. Researchers presented consumers with identical delivery delays following an online purchase. In the study, the same standard emails were sent to every participant: an order confirmation, a shipment notice, and a delivery confirmation. What varied was only how customers learned about the delay itself. Some received a delay notification from the retailer, some from the carrier, and others discovered the delay on their own through package tracking. The delivery outcome was…