How AI traffic is giving edge infrastructure a second job

The edge is acquiring a second job

AI is changing a basic infrastructure decision on the web. Delivering content increasingly requires deciding which machines may retrieve it and on what terms. Content delivery networks and edge platforms put data close to users to reduce delay. When controls sit on the delivery path, the same infrastructure can identify callers, block abuse, apply access rules and enforce commercial permissions.

Simon Wistow, co-founder of Fastly, says automated activity now ranges from ordinary API calls and Answer Engines querying websites to crawlers designed to evade controls. Fastly sells edge infrastructure and benefits commercially if companies need more controls at the edge, so that characterization carries a commercial interest. For infrastructure leaders, the practical question extends beyond performance. Identity and intended use can also determine whether a request should be served.

Why speed still anchors the edge

The original case for edge infrastructure remains relevant. Delays accumulate as traffic moves between systems, and large files take time to cross finite bandwidth. Caching reduces part of that cost by keeping copies of frequently requested content geographically closer to demand. A user or application can then retrieve nearby content instead of repeatedly reaching an origin system farther away.

Wistow cites Google studies as showing that every additional 100 milliseconds of latency cost Google roughly 5% of its customers. That figure should be treated specifically as his characterization of those studies unless the underlying Google research is checked. The engineering point is simpler: network and processing delays accumulate, while caching removes some of that delay by placing content closer to the requester.

Wistow says Fastly itself emerged from a change in website workloads. He says co-founder Artur Bergman was running the pop culture wiki that eventually became Fandom and found that existing CDN providers struggled with increasingly dynamic content. Websites were generating and changing more material, creating demand for caching systems that could support those workloads efficiently.

Hardware economics also shaped Fastly’s approach, according to Wistow. He recalls that SSDs were becoming more available as Fastly was being developed and could provide the performance required for large, fast caches. Fastly viewed SSDs as expensive in dollars per gigabyte but cheap in dollars per I/O, a measure more closely tied to how quickly a cache could handle requests.

That history provides a useful frame for AI traffic. Dynamic web applications expanded the work performed by delivery infrastructure while caching and proximity remained important. Machine-generated demand can expand that workload again by adding identity and access decisions to the delivery path. Infrastructure may now need to know more about a machine before serving it.

AI traffic changes what infrastructure has to distinguish

Wistow estimates that bots already account for around 50% of all web traffic and says bot traffic is growing faster than human traffic. This is a Fastly co-founder’s estimate rather than a neutral benchmark in this draft. At that scale, his argument is that machine-generated requests become a routine infrastructure workload, making classification relevant to architecture, security and access policy.

“Bot” covers requests with very different purposes. Wistow’s examples include applications making API calls, Answer Engines querying sites for information, and automated crawlers repeatedly requesting or probing content. He also says Fastly customers distinguish identified bots associated with OpenAI, Anthropic, Google and Meta from malicious crawlers using techniques such as headless browsers or EC2 instances to hammer sites.

A headless browser is software that operates a browser without its normal visual interface, allowing automated activity to resemble ordinary browsing. EC2 can also originate automated requests from cloud infrastructure used for many legitimate workloads. Wistow’s examples show why a simple “bot” label provides limited information about intent and why edge operators may seek additional identity and behavioral signals.

That distinction can change access policy. A company could permit an API integration, accept an identified AI crawler, rate-limit another automated client and block behavior associated with abuse. Those choices affect which requests travel onward to applications and origin systems. Applying the controls at the edge enforces the policy earlier in the request path.

The commercial question follows from the technical one. Publishers may want to distinguish among AI companies, types of machine use, geographic access and licensing arrangements. Businesses can also value reach, discovery, AI services and licensing differently. Infrastructure providers have a commercial incentive to offer finer controls when customers want those distinctions.

Access policy is becoming a technical and commercial problem

Once an automated client can be identified, a content owner can connect that identity to an access rule. A publisher could permit an identified AI crawler, block another requester, apply geographic conditions or seek payment for a particular form of access. The infrastructure handling the request then becomes an enforcement point for the business rule, connecting decisions by commercial, legal and infrastructure teams to the same request path.

Wistow says some Fastly customers want AI access blocked and that Fastly will help customers block access, including geographically where required. He also says Fastly puts customers in contact with AI companies and helps broker licensing deals between content owners and content retailers/streaming service providers. These are Fastly’s descriptions of customer demand and its own commercial role, and the company can benefit if such demand increases the value of its services.

The same enforcement mechanism can support defensive controls. Identifying unwanted automated behavior at the edge can let an organization reject or constrain requests before they reach deeper application systems. Classification quality is central because errors can block desired traffic or allow abusive traffic through. The business value depends on both the policy being enforced and the reliability of the signals used to enforce it.

Different operators can choose different policies. A publisher that values broad distribution may treat AI access differently from one whose premium material supports a licensing business. A platform may prioritize API availability, while an operator facing aggressive crawling may prioritize resource protection. Wistow captures Fastly’s view with the statement: “There is no one size fits all.”

The edge starts bundling performance, security and policy

Fastly describes its approach to AI traffic as including semantic caching, content labeling, protection against data exfiltration and malicious prompts, blocking and licensing assistance. Semantic caching organizes reuse around similarity in meaning rather than requiring an exactly repeated request. Fastly places this capability alongside controls governing how AI-related requests and content are handled. These are vendor claims about Fastly’s own offering.

Fastly also describes content labeling and security controls as parts of this broader edge role. In its framing, labels provide information that can help determine how content should be handled, while protections against malicious prompts and data exfiltration address attempts to manipulate AI systems or extract protected information. Blocking provides an enforcement mechanism. Together, these functions make Fastly’s case for combining delivery with AI-related security policy at the edge.

Resilience remains part of that operational role. Wistow describes Fastly’s ambition as helping customers remain protected from cyber-attacks, bots and what the company calls “internet weather,” including random cable breaks. Fastly has a commercial incentive to frame these problems as suitable for edge infrastructure. The architectural point is that delivery, traffic filtering and resilience can all act on requests before they reach an origin system.

Main highlights

• The edge is acquiring a second job: Edge infrastructure is moving beyond performance to identify machine traffic, enforce access rules and apply commercial permissions. Leaders should treat AI access policy as part of infrastructure planning.
• Speed still anchors the edge: Caching and proximity remain essential as AI adds new demands to delivery infrastructure. Organizations should preserve performance while adding machine identity and policy controls to the request path.
• AI traffic requires better classification: Bot traffic includes legitimate APIs and identified AI crawlers as well as abusive automation. Leaders should use identity and behavioral signals to distinguish traffic rather than applying one policy to every bot.
• Access policy is both technical and commercial: AI access decisions can reflect licensing, geography, security and business priorities. Infrastructure, legal and commercial teams should align on which machines can access content and under what terms.
• The edge is bundling performance, security and policy: Edge platforms increasingly combine caching, traffic filtering, AI security controls and access enforcement. Leaders should evaluate these capabilities together while scrutinizing vendor claims and classification accuracy.