The internet is being rebuilt for machines

Source: TechCrunch

Cloud infrastructure has long been designed around humans who search, click, scroll, and stream in a steady and predictable fashion. AI agents behave differently. They can unleash a swell of activity, spinning up multiple sub‑agents that query hundreds of databases, search documents, and call APIs in seconds and then disappear as quickly as they arrived.

Under that premise, Amazon is redesigning a core piece of its cloud infrastructure. On Thursday, AWS launched its next generation of OpenSearch Serverless — a fully managed search and vector database (essentially a system for storing and retrieving information at scale) that’s designed specifically for agentic workloads. AWS says the new system can instantly scale up when agents trigger tasks and scale back down to zero when idle.

The launch reflects a growing realization across the tech industry: infrastructure originally designed for a human‑driven internet doesn’t work as well in a world increasingly populated by agents.

The rise of machine‑generated traffic

• While AI agents still represent a relatively small portion of internet activity, machine‑generated traffic is already significant and poised to grow. Cloudflare reports that bots accounted for 31 % of overall HTTP traffic over the last six months. AI crawlers, search engines, and assistants made up roughly a quarter of all bot requests during that period.
• “Non‑human traffic will exceed human traffic sometime in the first half of 2027,” said Li Yi Ohlsen, senior product manager at Cloudflare, to TechCrunch (source).

At Google’s I/O developer conference, the company announced that users will be able to start delegating tasks to AI systems—researching purchases, booking travel, browsing the web, and interacting with apps (TechCrunch article). Enterprises are also deploying agents internally and for their customers, creating new kinds of machine‑generated traffic behind the scenes.

AWS’s next‑gen OpenSearch Serverless

“The timing is straightforward. Agents are moving from experimentation into production, and they create traffic patterns that previous infrastructure simply wasn’t designed for,”
— Tia White, General Manager for Amazon OpenSearch Service, TechCrunch

Key technical changes

• Decoupled compute and storage – compute can scale up in seconds to accommodate traffic bursts and scale down to zero, so customers pay $0 when agents are idle.
• In the prior Serverless version, storage and compute were coupled, requiring at least one instance to remain operational, leading to idle compute costs.
• The new model is analogous to paying for a metered parking spot rather than a permanently reserved one.

At launch, OpenSearch Serverless integrates natively with AI development platforms like Vercel and Kiro, allowing developers to deploy production‑ready search and vector backends for agents without managing infrastructure.

Industry response

• Databricks and Snowflake are repositioning themselves as AI memory and retrieval systems for enterprise data (TechCrunch podcast).
• Microsoft has rolled out updates to Azure designed to handle AI agent bursts and share memory between agents (Microsoft Dev Blog).
• Cloudflare introduced infrastructure aimed at giving agents persistent environments and instant scalability (Cloudflare press release).

Implications

The more companies deploy AI agents, the greater the pressure to redesign infrastructure around machine‑generated workloads. This shift could make agents cheaper and easier to deploy at larger scales, accelerating the transition to an internet increasingly powered by autonomous software.