Test shows Claude models pick different Shopify products

Haiku and Sonnet consistently favored products with the highest review counts, while Opus 5 preferred "safe," general-purpose options, citing safety in 222 of 300 reasons. A notable discrepancy emerged with a water bottle listed in Shopify’s catalog as having 447,221 reviews, a figure not visible on the retailer’s product page, which showed only 6,537 reviews. Both Haiku and Sonnet relied on this inflated catalog number to make their selections, whereas Opus ignored it. Additionally, moving a product from the first to the last position in the list significantly reduced its selection rate for all models, despite none mentioning position in their reasoning.

The study highlights that agentic commerce outcomes depend heavily on the underlying language model’s biases and the data quality of the catalog it accesses. As companies build shopping agents, the choice of model and the accuracy of product metadata will determine which stores receive customer orders.