84% of top US retailers publish no AI-agent rules
Refusal rate by category
Bars: all sites checked. Right: share refused. 5–6 refusals per category; directional only, not a ranking.
Refused Error Not refused
Source: NRF Top 100 Retailers 2026; checks 3 dates, Oct 31 to Nov 2, 2026; n=83
Two categories with fewer than 5 refusals (7 refusals) are counted in totals only. Select a category for detail.
Beyond retail
80% of the largest North American industrial distributors and 67% of top private goods marketplaces publish no AI-agent rules.
15 sites each, drawn from the 2026 Industrial Distribution Big 50 and the a16z Marketplace 100 rankings, same labeled agent and robots.txt rules, checked Nov 1–2, 2026; refusal rates hidden below 5 refusals.
Site-level detail
83 sites · weekly refresh · status history
Notes
Definitions
- Refused: the site returned an HTTP 403 or 429, or a bot-challenge page, to our labeled agent; cause not established.
- Published rules: robots.txt names AI agents or crawlers (allow or disallow), or has a Disallow group for
*. - Error: timeout or network error, counted as an error, not a refusal.
- Not refused: the checked files and homepage loaded normally.
Method
Sites = NRF Top 100 Retailers 2026 (ranked by US retail sales), grouped by primary category; niches under 10 merged; one company excluded. A logged-out check of public policy files (robots.txt, llms.txt, /.well-known) and the homepage response, aggregated by category. n=83, 3 dates, Oct 31 to Nov 2, 2026. Dated observations, not statements of intent.
What we tested and its limits
- One labeled bot. Every request used one honest user agent that names this project and links to it. No browser disguise, logins, cookies or retries around a refusal.
- One run per check date, one request per endpoint. Each site got at most one request each to robots.txt, llms.txt, a few /.well-known files and the homepage (plus at most one retry after a timeout), capped at 8 requests, 2 seconds apart.
- robots.txt respected. Any path robots.txt disallowed for our bot was skipped. On the first refusal we stopped checking that site.
- One vantage point. All checks ran from a single cloud network. Sites may answer other networks, regions or real shoppers differently.
- Coverage. 83 of the NRF Top 100. The rest were not covered, including one company excluded over a potential conflict of interest.
- What "refused" covers. An HTTP 403 or 429 response, or a bot-challenge or CAPTCHA page, on any checked file. Timeouts and network errors count as errors, not refusals. We did not establish why a site refused.
- Changes between check dates. 3 sites changed status since the previous published check (3 dates, Oct 31 to Nov 2, 2026), from one vantage point.
- ACP not measured. ACP has no settled public discovery file; ACP checkout runs through agent-platform integrations not visible externally, so it was not measured. 0 sites served the proposed acp.json.
- No ranking. Results are aggregated by category. We do not rank or name retailers.
Disclaimer
Independent, non-commercial research. Not affiliated with any company studied or any current or former employer of the author. No logins, CAPTCHAs, rate limits or refusals bypassed. robots.txt is a voluntary standard (RFC 9309), not a contract. Not legal advice. Brand names identify sites only. Corrections or opt-out: use the request form above.