Publishers and content creators have historically relied on traditional keyword-based search to help users navigate their website’s content. However, traditional search is built on outdated assumptions: users type in keywords to indicate intent, and the site returns a list of links for the most relevant results. It’s up to the visitor to click around, skim pages, and piece together the answer they’re looking for.
AI has reset expectations and that paradigm is breaking: how information is searched has fundamentally changed.
New type of visitors
Users no longer want to search websites the old way. They’re used to interacting with AI systems like Copilot, Claude, and ChatGPT, where they can simply ask a question and get an answer. We’ve moved from search engines to answer engines.
At the same time, websites now have a new class of visitors, AI agents. Agents face the same pain with keyword search: they have to issue keyword queries, click through links, and scrape pages to piece together answers. But they also need more: a structured way to ask questions and get reliable answers across websites. This means that websites need a way to give the agents they trust controlled access, so that information is retrieved accurately.
Website owners need a way to participate in this shift.
A new search model for the Agentic Web
If AI has reset expectations, what comes next? To meet both people and agents where they are, websites need more than incremental upgrades to keyword search. They need a model that makes conversational access to content a first-class part of the web itself.
Today, Cloudflare, the connectivity cloud company powering ~20% of the web, announced a collaboration with Microsoft which makes it simple for any website to offer conversational search for both human users and AI agents.
This combines Microsoft’s open standard NLWeb with Cloudflare’s AutoRAG infrastructure, allowing website owners to make their content instantly searchable through natural language queries (not just keywords). With one click, websites can now answer questions directly, rather than just returning lists of links. This can let any site function like an AI app, so users and agents alike can query its contents naturally.
Key capabilities include:
- Conversational search for humans: Visitors can ask questions in plain English and get direct answers, just like they expect from ChatGPT or Claude
- Structured access for AI agents: The new MCP endpoint gives trusted AI agents a reliable way to query website content
- Automatic indexing and updates: AutoRAG continuously crawls, indexes, and keeps content fresh without any manual work
- Think SEO for the agentic web: As AI agents become the new traffic source, this ensures websites stay discoverable and relevant
Discussion about this post