How website access controls can undermine spending on AI visibility

AI-visibility spending can conflict with website access policy

An organization committed £350,000 to becoming easier to find and reference in AI answers while relevant AI tools could not fully access its websites and other digital properties. The unnamed investigator who observed the commitment described two decisions working against each other: paying for greater visibility while restricting access to the information needed to support it. The case shows a coordination failure between spending decisions and access controls.

The conflict matters as consumers increasingly use ChatGPT and Claude to search the web. For a business, access means whether those tools can retrieve its current, authoritative information. The investigator describes weak preparation even among major brands that maintain their websites and invest in visibility. When those businesses block AI crawlers, automated systems that retrieve website content, they can reduce their influence over AI answers while those answers continue to appear.

Because access affects which information AI tools can retrieve, website restrictions have commercial consequences as well as a technical purpose. Marketing may want current information available, while engineering and security apply controls that limit automated access. These decisions need a shared policy so spending on visibility and website protection serve an agreed business objective.

Available information shapes how AI describes a brand

An AI service can discuss a business even when it cannot retrieve the business’s own pages. A deliberate restriction, configuration mistake or problem making page content readable to automated tools can leave current information unavailable, so the service may turn to material it can reach elsewhere. That material can include media coverage, accessible historic pages, archived documents republished on another website and stale copies of company information.

These alternative accounts may describe an earlier product, contain superseded facts or present the business differently from its current positioning. An answer based on them can therefore get facts wrong or misrepresent the brand. Restricting company-owned content leaves those external accounts available while removing an opportunity for the business to supply its current account.

External pages already play a substantial role in brand mentions. Research by AirOps found that 85% of brand mentions in AI responses originated from third-party pages rather than domains owned by the brands. AirOps has a commercial stake in demand for AI-visibility work. Its figure describes where mentions originated; it measures neither misinformation nor the share of third-party reliance caused by crawler restrictions.

The role of external pages also limits what opening access can achieve. Media reporting and other third-party material can be useful and current, and an AI tool may select those pages or combine several accounts even when a company’s website is available. A business can improve the information under its control, but the AI service decides how to select, interpret and present it.

Improving that information requires content maintenance as well as access. Outdated, contradictory or erroneous website content can remain available and then be cited confidently by an AI system. An improperly prepared PDF creates another problem: the document may be accessible, yet the system can misinterpret it and produce incorrect information. Access controls, content accuracy and document preparation therefore need separate checks.

Businesses already make a related commercial assessment through search engine optimization, or SEO, which aims to improve visibility in search results. Preventing Google from indexing company pages, adding them to its searchable collection, would risk reduced search visibility, lower traffic and possible effects on sales and revenue. Those consequences are hypothetical here, and AI access needs its own assessment because search-engine returns do not transfer directly to AI answers.

Human website checks can miss AI-access failures

An access assessment needs to examine the rules applied to automated visitors. Research by US Tech Automations found that 19.1% of 100,000 top-ranked websites blocked at least one of 21 leading AI crawlers through robots.txt, the file used to state crawler-access rules. OpenAI’s crawlers faced restrictions most frequently. Nearly one in five websites in the sample therefore published at least one explicit exclusion through those rules.

The company’s research also draws commercial attention to AI-access problems. Its measurement covers explicit robots.txt restrictions, but security controls and technical barriers can cause additional retrieval failures. The finding also leaves the intent behind each exclusion open: a published rule may reflect an approved decision or an unintended setting.

To examine failures beyond published rules, the unnamed investigator used 24 agents to assess access involving 15 key AI tools. The tests checked whether tools could reach information an organization considered current and authoritative. Human access was tested alongside AI-related routes, the paths through which automated tools tried to retrieve information, so the results could reveal differences between the two. These observations show failure patterns rather than their prevalence across websites.

The first pattern showed how those differences can escape an ordinary website check. Every tested human route worked, while several AI-related routes were unavailable. A person visiting the website could therefore see working pages even though automated tools could not retrieve the same information. A successful human visit establishes only that the tested human route works.

The second pattern showed a wider gap between human and automated access. All human access worked, but a majority of the AI-related tools were refused access, leaving much of the tested AI access unavailable while customer-facing checks could still pass. Across the reported observations, some websites excluded nearly all the AI tools examined.

The third pattern required a different diagnosis because 14 AI-related routes received no reply. Only one AI route fell outside that group, and one human route was also blocked. A missing reply and an explicit refusal are different test outcomes, so treating both as deliberate blocking would assign an intent the observation does not establish.

The fourth pattern showed why familiar services provide an incomplete check. Google and Microsoft routes remained available while other AI routes were refused access. A team checking only those familiar services could therefore miss restrictions elsewhere. The useful question is which tools can retrieve which information, with human access and each relevant automated route assessed separately.

AI restrictions need a deliberate business rationale

Finding an unavailable route starts a policy review because some restrictions serve legitimate purposes. Bots, scrapers and crawlers were estimated to account for 53% of internet traffic in 2025, giving organizations a large volume of automated activity to manage. AI crawlers account for part of that traffic. The estimate explains the pressure behind protective controls, although each restriction still needs a business rationale.

One rationale concerns how content may be used after retrieval. A business may restrict access to prevent model training where copyright exposure or ethical concerns make that use unacceptable. It may also protect the commercial value of paid-for or licensed material. These concerns can justify accepting reduced access to information that might otherwise improve how the business appears in AI answers.

Operational and legal obligations can also justify restrictions. Excessive requests can consume bandwidth or disrupt website operation, while security and compliance requirements can demand controls over information, particularly personal data that must not be supplied for model training. A broader security policy may exclude automated agents whose access creates an unacceptable risk. Visibility decisions therefore need input from the people responsible for those protections.

An unintended barrier calls for a different response because its commercial effect may never have been approved. Older anti-bot and anti-scraping filters can continue to catch AI crawlers, while overly restrictive settings in a content delivery network, or CDN, a service that distributes website content, can exclude AI agents. Other security services can impose similar restrictions. Teams should trace the failed request through these controls to find which setting caused the failure.

A request can also reach a website without yielding readable content. Some sites depend on JavaScript, code used to generate or change page content, that certain crawlers cannot turn into readable pages; the investigator describes this limitation as common. Errors in robots.txt or related settings can separately exclude intended access. These mechanisms need different fixes because a rendering failure concerns what the crawler can read, while an access rule determines whether it may retrieve the page.

Other barriers can enter through infrastructure and installed software. Internet Protocol, or IP, address rules and geographic restrictions may inadvertently exclude AI tools, while third-party plug-ins can add restrictions on automated visitors. Unnoticed meta tags, instructions embedded in page code, can impose AI restrictions across a website. Authentication, paywalls, CAPTCHA checks that ask visitors to demonstrate they are human, and request-rate limits can also prevent retrieval or reject legitimate AI-agent activity.

The presence of any one mechanism does not establish whether the result is accidental. A paywall may deliberately protect licensed content, while an unnoticed rule may obstruct information intended for public discovery. Diagnosis must therefore establish both what happened and whether it matches an approved policy.

Give AI access a shared owner and a repeatable review

The first step is to diagnose which AI-related routes cannot retrieve current, authoritative information and investigate why. Automated testing, including remote AI-powered assessment platforms, can help identify affected tools and distinguish intended restrictions from unintended failures. Those tests should examine actual retrieval alongside human access checks. The investigator’s route patterns show why working pages in a browser provide an incomplete assessment.

Once the cause is understood, the second step is to correct accidental restrictions and address what produced them. Depending on the diagnosis, this may mean changing old bot rules, security settings, plug-ins, page instructions, request limits or rendering behavior. The aim is to restore the access the organization intended and reduce the chance of recurrence. Availability creates an opportunity for authoritative information to be used; citation and commercial return remain separate outcomes.

With accidental failures addressed, the third step is to review deliberate restrictions with the stakeholders responsible for them. Establish why each restriction exists, what it protects and whether that rationale still holds as more people use AI-based information search. Some policies may warrant reversal as the commercial consequences change. Others should remain where copyright, licensing, bandwidth, security or compliance obligations justify reduced access.

Those decisions need someone accountable for bringing commercial and technical interests together, which is the fourth step. Accountability should sit with a stakeholder responsible for visibility and revenue, supported by engineering and security expertise. Engineering and security remain essential because they manage many of the controls and understand the protections they provide. For an organization funding AI visibility, this ownership connects the spending decision to access policy while preserving authority over legitimate safeguards.

Where access problems have been identified, that coordination deserves priority as concerns about AI visibility and misinformation grow. Remediation should distinguish unintended failures from approved restrictions so teams correct the former and review the rationale for the latter. The accountable owner must also check that the implemented settings reflect those decisions. Otherwise, a policy review can leave the original technical conflict in place.

Keeping implementation aligned with policy requires the fifth step: repeat testing and review restrictions periodically. AI tools and crawler behavior change, while website teams modify software, settings and security controls, so access that works after a correction can fail later. Continuing checks should detect those changes and return consequential failures or restrictions to the responsible stakeholders. A previously justified exclusion may then need a fresh decision about the protection it provides and the authoritative information it makes unavailable.

Key executive takeaways

• Diagnose AI access failures: Engineering and security teams need to test whether relevant AI tools can retrieve current, authoritative information. Compare automated retrieval with human access and trace failures to access rules, security controls or unreadable content.
• Correct unintended barriers: Website owners need to fix accidental restrictions in bot filters, security settings, plug-ins and page rendering. Verify that each change restores the intended access; availability alone does not guarantee citation or commercial return.
• Review deliberate restrictions: Marketing, legal and security stakeholders need to weigh visibility benefits against copyright, licensing, bandwidth and compliance requirements. Retain restrictions with a clear business rationale and reconsider those whose costs outweigh their protection.
• Assign shared accountability: Give a stakeholder responsible for visibility and revenue ownership of AI access policy, supported by engineering and security expertise. This connects visibility spending to the controls that determine whether AI tools can retrieve company information.
• Repeat access testing: Website and security teams need recurring retrieval checks, particularly after software or configuration changes. Escalate new failures and revisit approved restrictions as AI tools, crawler behavior and business priorities change.