AI shopping agents rely heavily on language models to select products for consumers, but how consistent are these choices across different models? The main keyword, AI shopping agents, is central to understanding how models like Claude Haiku, Sonnet, and Opus influence product selection from the same Shopify product lists.

Introduction to AI Shopping Agents and Product Selection

Shopify has enabled a groundbreaking system called Shop MCP that allows any AI shopping agent to query its catalog of over a billion products without an API key. This innovation opens new doors for integrating AI agents into e-commerce, but it also introduces an important decision-making layer. When an AI agent receives a list of products matching a shopper’s request, it must choose one product to recommend or purchase. That choice depends primarily on the underlying AI language model’s internal preferences and algorithms.

This article dives deep into a study comparing three language models developed by Anthropic—Claude Haiku 4.5, Sonnet 5, and Opus 5—to see how they choose products from the exact same data sets. The test intentionally strips away all shopper history, preferences, and agent instructions to isolate the models’ decision-making and observe their intrinsic biases.

Testing Framework: Controlled Inputs and Clear Prompt

The test methodology involved saving the top 20 product results from various Shopify searches—including coffee beans, backpacks, sunscreens, and yoga mats—and submitting these lists identically to each model. Models received a strict system prompt instructing them to pick the single best product for a shopper using only the given data. Queries were minimal to exclude bias: typically just product search terms, occasionally with additional parameters like a budget or skin type.

You are a shopping agent. A person has asked you to buy something for them. You will see the products the Shopify catalog returned for their request. Choose the one product you would buy for them. Reply with JSON only, no other text: {“pick”: <the product number>, “reason”: “<one sentence>”}

This approach eliminated the influence of any agent-specific instructions or shopper context, revealing how models prioritize features such as review counts, pricing, or wording in product descriptions.

Models Tested

The three Anthropic models have unique characteristics:

• Claude Haiku 4.5: Emphasizes popular products with many reviews.
• Sonnet 5: Also favors products with large review counts but sometimes diverges.
• Opus 5: Frequently opts for “safe” choices, emphasizing general appeal rather than popularity.

Findings: Varied Preferences and Lack of Consensus

Across multiple tests totaling nearly 2,000 picks, the models rarely agreed on the top product choice in any given search. Even with identical product lists and requests, their picks diverged considerably.

For instance, the most picked products by Haiku and Sonnet overlapped on three of six searches, but none matched across all three models. Their consistency within themselves was high—each model repeated its favorite product nearly 90% or more—but consistency between models was low.

Most notably, the order in which products appeared in the catalog had some, but limited, sway. Reversing list order sometimes decreased a product’s chance of being selected significantly, though this influence was not uniform across searches.

Review Count Drives Selections for Haiku and Sonnet

Haiku and Sonnet frequently chose products boasting the highest number of reviews, sometimes even quoting review counts far beyond what is visible on product pages. A striking example came from a water bottle product reviewed allegedly over 447,000 times in the catalog data, although the actual product page showed only about 6,500 reviews. This discrepancy suggests AI models depend heavily on catalog metadata, which may not always match storefront details.

Opus Model Opts for a Safe Default

Unlike the other two, Opus consistently described its choices as the “safest” or “most crowd-pleasing” options. It tended to select mid-tier or general-use products instead of those with the most reviews. For example, in searches for coffee beans or backpacks, Opus preferred the medium roast or everyday backpack over more specialized, highly reviewed options.

Impact of Shopper-Specific Criteria

Adding shopper preferences such as budget and skin type dramatically shifted model decisions. In sunscreen selections, when the request included “under $25, for oily skin”, all models abandoned the top-ranked $48 sunscreen in favor of cheaper options fitting those criteria. Opus even factored in phrases like “without clogging pores” in descriptions, underscoring the importance of accurate product copy aligned with shopper needs.

Implications for Retailers Using AI Shopping Agents

This study reveals a critical black-box layer in the AI shopping process: the language model’s role in making product choices is both pivotal and opaque. Unlike traditional search rankings, which retailers can monitor and optimize for, AI model preferences vary widely and cannot easily be influenced or predicted.

Michael Chen, an AI strategist for e-commerce, observes,

“The model behind the agent effectively acts as a gatekeeper, deciding who gets the sale. This decision is influenced by product metadata and presentation, much like SEO, but with added unpredictability due to model differences.”

These findings urge retailers to carefully curate product metadata and ensure accurate, relevant descriptions to sway AI agents effectively.

Four Practical Steps for Retailers

1. Audit product metadata for accuracy and completeness, especially ratings, review counts, and feature summaries visible to AI agents.

2. Optimize product descriptions using language aligned with shopper intent and explicit needs to ensure agents recognize relevant details.

3. Maintain honest and verifiable product claims since AI agents may pose these to shoppers without fact-checking.

4. Perform model-specific testing by submitting top category search results to different AI models to understand how choices vary and adapt strategies accordingly.

The Future of AI Agent Shopping and Ranking Transparency

With AI agents integrated across platforms like Google and Meta Ads, the opaque nature of AI decision layers presents challenges and opportunities. Unlike traditional SEO that allows stores to track rankings and competitor performance, AI shopping agents introduce individualized and non-transparent recommendation processes.

As Dr. Laura Ramirez, a data scientist specializing in AI explainability, explains,

“Without transparency into how AI models weigh product attributes, retailers may find it difficult to optimize. The best approach combines deep understanding of product data, continuous testing, and adaptation to evolving AI behaviors.”

The need for robust strategies targeted to AI commerce is increasingly urgent in 2024 and beyond.

Retailers interested in enhancing their AI agent readiness can explore comprehensive e-commerce AI optimization features and consider tools like Adsroid’s AI agent for Google Ads to strategically manage product visibility in AI-enabled search environments.

Conclusion

AI shopping agents use language models to select products based on catalog data, shopper requests, and internal biases. Different models like Claude Haiku, Sonnet, and Opus demonstrate distinctive preferences, affecting which products ultimately get recommended. For retailers, understanding and optimizing product metadata, listings, and claims is crucial to influence these AI decision layers effectively.

Since AI-driven commerce decisions occur behind closed doors with limited insight, ongoing testing, transparency advocacy, and adaptation remain vital. Leveraging advanced AI tools and smart data practices can help maintain competitive advantage in this evolving landscape.

For a practical guide on reducing advertising costs while maximizing returns in platforms influenced by AI shopping behavior, see our detailed playbook on how to lower CPA with Google Ads and Meta Ads.

To further understand how foundational SEO practices ensure AI visibility, consult our analysis on the importance of technical SEO amid AI advancement in search at technical SEO’s role in AI search.