Three AI Models Shopped Shopify\'s Catalog and Each Picked a Different Winner

Three AI models shopped Shopify's catalog and each picked a different winner, Search Engine Journal reported on November 2, 2026. The same products, the same catalog, and three different recommendations. The experiment is a small story with a large implication: as shopping moves inside AI assistants, the store that "wins" depends on which model the customer happens to ask, and nobody outside the model maker can see why.

The point the coverage makes

The point made in the coverage is blunt. Google rankings are a black box that a store can still watch: you can see where you rank, track movement over time, and infer what changed. The AI shopping decision layer is a black box with nothing to monitor. There is no rank tracker for an assistant's recommendation, no Search Console for which products a model chose, and no report explaining why one product was picked over an identical competitor.

That asymmetry is the story. Merchants spent two decades learning to measure search visibility, and the tools for that measurement are mature. AI shopping has no equivalent instrumentation, so a store can lose the recommendation entirely and never see a dashboard move. Sales simply soften, and the cause is invisible.

Why three models disagree

Different models weighing the same catalog differently is not a bug; it is what different training, different retrieval, and different ranking signals produce. One model may weight reviews more heavily, another may favour price, another may prefer detailed descriptions or strong product data. The customer asking the question gets one answer, presented with confidence, and has no reason to suspect that a different assistant would have recommended something else entirely.

For merchants, that means there is no single "AI optimisation" target. Optimising for one model's preferences may hurt with another, and the preferences themselves are undocumented and shifting. The practical response is not to chase one model but to make the product data excellent across every dimension a model could plausibly weigh: complete descriptions, accurate specifications, honest reviews, clear pricing, and availability kept current.

What merchants can actually do

First, treat product data as the new ranking factor. In classic search, the page did the selling; in AI shopping, the structured product record does much of the work before any page is ever shown. Incomplete or sloppy product data is not just a conversion problem anymore, it is a visibility problem, because a model cannot recommend what it cannot understand.

Second, test your own products the way customers do: ask several AI assistants for recommendations in your category and see what comes back. It is manual, it is slow, and it is currently the only monitoring available. Keep a simple log of which products get recommended, by which assistant, and for which queries, and watch for patterns over time. Third, keep every other channel healthy. If AI recommendations are unmeasurable, the channels you can measure, classic search, email, direct, become your early-warning system for demand shifts. Manual testing also reveals how your products are described, which matters because a model that misunderstands a product will recommend a competitor it understands better. Treat every product field as a sales argument the model can actually read, and keep it current.

The measurement gap the industry must close

The longer-term question is whether anyone will build the monitoring layer AI shopping needs. Rank trackers emerged because search engines exposed results that could be observed; AI assistants expose a single answer per question, which is far harder to sample at scale. Until the model makers offer reporting, or third parties build reliable sampling tools, merchants are flying blind on a growing share of purchase decisions.

That gap will not last forever, but it may last long enough to redistribute market share. The stores that win in the meantime will be the ones whose product data is complete, whose reviews are genuine, and whose offers are easy for any model to understand. The experiment's three different winners are a warning: in AI shopping, there is no longer one race. There is a separate race inside every assistant, and right now nobody is keeping score.

Sources

Sources: Search Engine Journal, November 2, 2026.