We use cookies to enhance user experience, personalize content, and analyze traffic. Cookie Policy

← Back to all articles

AI Research Agents: Deep Research and Where Fetches Fail

How an AI research agent plans, searches, reads, and cites, and where its page fetches fail, from 12 deep research runs and 264 source URLs we tested.

by Unknown Proxies

15 min read

November 12, 2026

AI Research Agents: Deep Research and Where Fetches Fail

An AI research agent is a language model running a loop: it breaks a question into searches, opens the most promising results, pulls out the facts, and writes a report with citations. ChatGPT deep research, Gemini Deep Research, Claude's Research mode, and Perplexity are hosted versions. You can build your own on Claude Code, the Claude Agent SDK, LangGraph, or a plain script that calls a model and a search API.

The weak step is reading. We ran 12 research tasks with Claude Code and logged every request. With a plain "cite your sources" prompt, the agent opened 1 page across 6 reports, and 38 of its 39 citations pointed at pages it never opened. When we told it to read before citing, it made 32 fetches. Only 15 of them returned usable text, and only 2 of the 17 that didn't came back flagged as errors.

This guide covers how a deep research workflow runs, where its fetches fail and why the failures are easy to miss, and a small fetch tool that makes them visible. If you're choosing a search layer, see Tavily vs Exa vs Serper. This post is about what happens after the search.

How an AI research agent works

Most deep research systems run the same six steps, whatever the vendor:

  1. Plan. The model splits the question into sub-questions and search queries. "Four-day work week productivity" becomes queries for trial results, the main paper, and critiques.
  2. Search. A search tool returns titles, URLs, and usually a short summary for each result. Anthropic's WebSearch, Tavily, Exa, Brave, and Serper all do this.
  3. Triage. The model picks which results to open. This is where it decides whether a snippet is enough.
  4. Fetch and read. A fetch tool downloads each page, converts it to text, and often runs a second, cheaper model over it to extract only what was asked.
  5. Synthesize. The agent merges what it read, notes conflicts, and often runs more searches to fill gaps.
  6. Cite. The report links each claim to a source URL.

Hosted products loop through steps 2–5 many times and may split the work across parallel sub-agents. Anthropic's write-up of its multi-agent research system describes a lead agent that sends sub-agents to search in parallel and a separate pass that attaches citations at the end. The steps stay the same, and so does the gap between step 3 and step 6: nothing in the loop checks that a cited page was actually read.

AI research agent loop from plan to search, triage, fetch and read, synthesize, and cite, with fetch failures returning to the model as ordinary text

Hosted deep research vs a research agent you build

The workflow is the same in both cases. What differs is who runs the fetches and which IP the sites see.

Setup Who fetches pages IP the site sees What you control
Gemini Deep Research, ChatGPT deep research, Claude Research, Perplexity The vendor The vendor's own IP ranges The prompt and the sources you upload
Claude Code or the Claude Agent SDK with WebFetch Your machine Your server's IP Tools, prompts, timeouts, proxy
LangGraph, CrewAI, or your own loop with a search API Your fetch code Your server's IP, or a proxy Everything, including how failures are reported

The hosted products are the right call when you need a report and don't need to control the sources. Build your own when the agent has to read sources a hosted product can't reach, run on a schedule, feed another system, or meet an audit requirement about what was actually read. Once you build it, its fetch failures are yours to handle. AI agent examples shows how research traffic compares with other agent jobs: many domains, one to three requests each.

Claude Code's WebFetch runs on the machine running the agent and sends Claude-User (claude-code/2.1.310; +https://support.anthropic.com/) as its user agent. Our ClaudeBot and GPTBot guide explains why that request fails Anthropic's IP verification when it comes from your own server.

What our deep research test found

We ran Claude Code 2.1.310 with claude-sonnet-5-5, limited to the WebSearch and WebFetch tools, on a Hetzner server in Nuremberg. All traffic went through a local logging proxy, so every page fetch showed up as a connection. We used six research questions with very different source types:

Each question ran twice. Prompt A asked for "a short report (about 300 words) with sources cited as URLs." Prompt B added one sentence: "Read the source pages with WebFetch before citing them; don't rely on search snippets alone."

Prompt A (cite sources) Prompt B (read before citing)
Searches per run 1–3 0–3
Page fetches, all 6 runs 1 32
Fetches that returned usable text 1 15
Cited URLs 39 22
Cited URLs never successfully read 38 8
Wall time per run 14–20 s 20–152 s
Cost per run $0.047–0.092 $0.035–0.114

With prompt A, five of the six reports were written entirely from search-result summaries. The sixth fetched one page, the PgBouncer config docs. Four of the reports said so plainly, for example: "The search tool returned summaries, and I didn't open the underlying pages to verify them." The AI Act report only said it hadn't read the Official Journal text, and the microplastics report didn't mention it at all. The links looked like normal citations either way.

With prompt B, the agent was honest about what it couldn't read. All 8 citations without a successful read were labeled in the text as unverified, "candidate sources (not read)", or "couldn't open". The microplastics run got usable text from none of its 8 fetches, and it listed all four of its sources as unverified candidates. That's the right behavior. But a pipeline that extracts the links from the reports still gets 8 URLs that nothing read. The mortgage-rate report quoted 2026 rate figures next to a Freddie Mac link that had returned a 403 and a Nasdaq link that had timed out.

Failed fetches cost time as well as accuracy. The six read-first runs took 315 seconds together, and 120 of those were two 60-second timeouts on one site in the mortgage-rate run.

Where research agent fetches fail

Here is what the 32 read-first fetches actually returned.

Outcomes of 32 WebFetch calls in six read-first research runs, with 15 usable pages and 17 failures, only 2 of which were flagged as errors

Outcome Fetches Flagged as an error?
Usable page text 15 —
HTTP 403, returned as text 6 No
Timeout after 60 s 2 Yes
PDF passed through as raw bytes 2 No
Bot-check page (AWS WAF, PMC verification) 2 No
Landing page with no findings 2 No
Cookie notice instead of the article (PubMed) 1 No
Redirect to another host, not followed 1 No
404 on a URL the model made up 1 No

Only the two timeouts came back with is_error: true. Everything else arrived as an ordinary tool result, usually a short sentence written by WebFetch's extraction model, and the agent had to read that sentence to work out that the fetch had failed. Each row has a different cause and a different fix.

Blocks that come back as text

A 403 reached the agent as "The server returned HTTP 403 Forbidden. The response body was not retrieved." Sonnet understood it every time. A smaller model or a framework that only checks for exceptions may not. The six 403s came from Freddie Mac's investor-relations site, Barchart, two Cloudflare-protected news sites, and an association of state drinking-water regulators. None of those pages was readable from our datacenter IP under any user agent we tried (more on that below).

Bot checks that look like a successful fetch

The worst case was technologynetworks.com. Its AWS WAF answered with HTTP 202, an empty body, and x-amzn-waf-action: challenge. AWS documents this as the challenge action's response: a 202 for a client that hasn't solved the JavaScript challenge. A 202 Accepted is a success code, so WebFetch passed the empty page to its extraction model. The model replied that the page "contains only a title" and quoted the agent's own extraction prompt back as that title. A model skimming that reply could easily conclude the article was just short.

PMC returned a "checking your browser" page, and PubMed returned a notice that cookies must be enabled. In both cases the extraction model correctly said the page had no review in it, but the tool still reported a success.

PDFs

Papers, regulator reports, and investor releases are often PDFs, and they're usually the primary sources a research report should rest on. Claude Code 2.1.310's WebFetch passed both PDFs in our runs to its extraction model as raw bytes. The model saw "compressed streams (FlateDecode)" and font tables, not text. A direct test on the arXiv PDF of "Attention Is All You Need" gave the same result. WebFetch does save the binary to a local file, so an agent that also has the Read tool can open it. Our research agent only had WebSearch and WebFetch, so it couldn't. With pypdf, all four PDFs the servers actually returned, including both that failed in the agent runs, gave between 25,000 and 116,000 characters of clean text.

Redirects across hosts

Nature article URLs answer with a 303 to idp.nature.com, which sets a cookie and redirects back. Browsers and httpx follow it and get the full article: 16,506 characters of text in our test. Claude Code's WebFetch documentation says that when a URL redirects to a different host, it returns a notice with the target instead of following it. The agent saw a redirect to a login page and gave up on the Nature paper. In the URL survey below, 11 of the 264 URLs redirected through a different host, including Yahoo News to consent.yahoo.com.

Timeouts that stall the run

Both nasdaq.com fetches hung for 60 seconds and failed with "timeout of 60000ms exceeded". With curl, the same URL and the Claude-User user agent hung over HTTP/1.1 until our 30-second limit. A Chrome user agent got a 200 in 1.3 seconds, and curl's default user agent hung too. Depending on the client, the site's Akamai edge either refused the request at once or held the connection open until the client gave up. The agent's run sat idle for two minutes. A 15-second read timeout would have cost 30 seconds.

Dead and invented URLs

The WHO 404 wasn't in any search result. The model built the URL from memory, and it was wrong. The search index has dead links too: 14 of the 258 search-result URLs we surveyed answered 404 or 410, six of them on a single site. Treat any URL the model typed itself as a guess until it returns a 200.

How often research sources can't be read

The agent runs only fetched 33 pages. To get a broader rate, we pooled 264 URLs on 181 sites from the search results and agent fetches across all runs, plus one pilot run, and fetched each one from the same Hetzner IP with three user agents: Chrome 155, Claude-User, and a descriptive bot string (acme-research-agent/1.0). The same classifier, the script in the next section, judged every response.

Readability of 264 research source URLs fetched from a datacenter IP with three user agents, about two thirds readable and about a quarter blocked under each

User agent Read 403 Challenge page 404/410 Other
Chrome 155 175 (66%) 38 23 15 13
Descriptive bot UA 174 (66%) 41 21 14 14
Claude-User 168 (64%) 41 28 15 12

"Other" is mostly pages under 600 characters of text, connection errors, and a few 400 and 503 responses.

Three things stand out:

Per site, 132 of the 181 sites gave us at least one readable page under the descriptive user agent, and 33 answered every URL we tried with a 403, a challenge page, a 429, or a timeout. If you run a research agent from a cloud server, plan on about a quarter of the sources it finds being blocked from there, and a third being unreadable for one reason or another.

Make fetch failures visible to the agent

The fix doesn't need a new framework. Give the agent a fetch tool that returns a plain verdict, read: true or read: false with a reason, and keep a ledger of what was read. Then check the report's citations against that ledger before you accept it.

This version uses httpx 0.28.1, trafilatura 2.3.1, and pypdf 7.0.1 on Python 3.12. It's the script that classified the 264 URLs above.

# research_fetch.py: fetch sources for a research agent and record whether each one was actually read.
import io
import json
import os
import re
import sys
from urllib.parse import urldefrag

import httpx
import trafilatura
from pypdf import PdfReader

HEADERS = {
    # Identify your agent honestly and give site owners a way to reach you.
    "User-Agent": "acme-research-agent/1.0 (+https://acme.example/bot)",
    "Accept": "text/markdown, text/html;q=0.9, application/pdf;q=0.8, */*;q=0.1",
}
CHALLENGE_MARKERS = ("Just a moment...", "_cf_chl", "cf-turnstile", "captcha-delivery", "px-captcha")
MIN_CHARS = 600  # less than this is a JS shell, a landing page, or a wall, not a source


def fetch_source(client: httpx.Client, url: str) -> dict:
    out = {"url": url, "read": False}
    try:
        r = client.get(url)
    except httpx.TimeoutException:
        return out | {"reason": "timeout"}
    except httpx.HTTPError as e:
        return out | {"reason": f"network error: {type(e).__name__}"}

    ctype = r.headers.get("content-type", "").split(";")[0].strip()
    out |= {"status": r.status_code, "final_url": str(r.url), "content_type": ctype}
    if r.headers.get("cf-mitigated") or r.headers.get("x-amzn-waf-action"):
        return out | {"reason": "challenge"}
    if r.status_code == 202 and not r.content:
        return out | {"reason": "empty 202"}
    if r.status_code >= 400:
        return out | {"reason": f"http {r.status_code}"}

    if ctype == "application/pdf" or r.content[:5] == b"%PDF-":
        try:
            pages = PdfReader(io.BytesIO(r.content)).pages[:40]
            text = "\n".join(page.extract_text() or "" for page in pages)
        except Exception as e:  # truncated, encrypted, or mislabeled PDFs
            return out | {"reason": f"unreadable pdf: {type(e).__name__}"}
    elif ctype in ("text/markdown", "text/plain"):
        text = r.text
    else:
        text = trafilatura.extract(r.text, favor_recall=True) or ""

    if len(text) < MIN_CHARS:
        # Only a short page counts as a challenge; a long article may quote these markers.
        if any(marker in r.text[:200_000] for marker in CHALLENGE_MARKERS):
            return out | {"reason": "challenge"}
        return out | {"reason": f"only {len(text)} chars of text"}
    return out | {"read": True, "chars": len(text), "text": text}


def unread_citations(report: str, ledger: dict) -> list[str]:
    """Return URLs cited in the report that were never successfully read."""
    read = {urldefrag(u).url for e in ledger.values() if e["read"] for u in (e["url"], e["final_url"])}
    # Allow one level of balanced parentheses, as in DOI links, without eating a Markdown link's ")".
    found = re.findall(r"https?://(?:[^\s()<>\]\"']|\([^\s()<>]*\))+", report)
    cited = {u.rstrip(".,;") for u in found}
    return sorted(u for u in cited if urldefrag(u).url not in read)


if __name__ == "__main__":
    ledger = {}
    with httpx.Client(
        headers=HEADERS,
        follow_redirects=True,
        timeout=httpx.Timeout(15.0, connect=5.0),
        proxy=os.environ.get("RESEARCH_PROXY"),  # e.g. http://user:pass@host:port
    ) as client:
        for url in sys.argv[1:]:
            entry = fetch_source(client, url)
            ledger[url] = entry
            print(json.dumps({k: v for k, v in entry.items() if k != "text"}))

Run it on a few of the URLs from our test:

python research_fetch.py \
  https://www.nature.com/articles/s41562-025-02259-6 \
  https://research.chalmers.se/publication/553740/file/553740_Fulltext.pdf \
  https://www.nasdaq.com/press-release/mortgage-rates-average-667-2026-08-13
{"url": "https://www.nature.com/articles/s41562-025-02259-6", "read": true, "status": 200, "final_url": "https://www.nature.com/articles/s41562-025-02259-6", "content_type": "text/html", "chars": 16506}
{"url": "https://research.chalmers.se/publication/553740/file/553740_Fulltext.pdf", "read": true, "status": 200, "final_url": "https://research.chalmers.se/publication/553740/file/553740_Fulltext.pdf", "content_type": "application/pdf", "chars": 66039}
{"url": "https://www.nasdaq.com/press-release/mortgage-rates-average-667-2026-08-13", "read": false, "status": 403, "final_url": "https://www.nasdaq.com/press-release/mortgage-rates-average-667-2026-08-13", "content_type": "text/html", "reason": "http 403"}

The Nature paper and the Chalmers PDF, both failures in the agent runs, now read cleanly. Nasdaq still refuses this client, but the request fails in under a second with a reason the agent can act on, instead of hanging for 60 seconds.

How to wire it in:

When a proxy helps a research agent

Proxies only help with one row of those tables: blocks that depend on where the request comes from. A research agent on a cloud server looks like a datacenter client to every site it visits. The 59 URLs that refused all three user agents fit that pattern, though some may be reacting to the HTTP client rather than the IP. Routing fetches through residential proxies is a reasonable next step for public pages like those. Set RESEARCH_PROXY and the script above sends everything through it. Research traffic is broad and shallow, a few requests per domain, so per-request rotation works, and a sticky session only matters when the agent paginates through one site. We didn't test residential exits in this run, so measure your own block rate before and after rather than assuming it drops to zero. Proxies for AI agents covers how to tell an IP block from a fingerprint block, and datacenter vs residential proxies covers the trade-offs.

A proxy does nothing for:

Pace the fetches too. A research agent that fans out to 30 URLs at once on one domain looks like a crawler, whatever IP it uses.

FAQ

What is an AI research agent?

It's a language model that answers a question by planning searches, opening the results, extracting facts, and writing a cited report, repeating those steps until it has enough. Deep research products from OpenAI, Google, Anthropic, and Perplexity are hosted AI research agents. Frameworks such as the Claude Agent SDK and LangGraph let you build your own.

AI search runs one query and answers from the top results, often from the snippets alone. Deep research plans several queries, reads many pages, follows up on gaps, and takes minutes instead of seconds. Our runs show that the reading part isn't automatic. Unless it's prompted or forced to, an agent can produce a deep-research-style report from summaries alone.

Why does my research agent cite pages it never read?

Search tools return a summary with each URL, and the model can write a plausible claim and attach the URL without fetching it. In our test that happened for 38 of 39 citations with a plain prompt. Require reads in the prompt, record every successful fetch, and check the report's URLs against that record before you use it.

Can AI research agents read PDFs?

Only if the fetch tool extracts the text. Claude Code 2.1.310's WebFetch handed both PDFs in our runs to the model as raw bytes, which it couldn't read. A fetch tool that runs pypdf on application/pdf responses read all four PDFs the servers actually returned in our survey.

Do AI research agents need proxies?

Not to start. Most sources were readable from a plain cloud server, and many failures had nothing to do with IPs. Add a proxy when your logs show a lasting block rate on public pages that fail for every user agent. Then measure whether it changes.

Conclusion

An AI research agent is only as reliable as the pages it actually read, and nothing in the standard deep research loop checks that. In our tests, a plain prompt produced reports cited almost entirely from search snippets, and even a read-first prompt left a third of the citations unread, though each one was flagged in the text. Most fetch failures arrived as ordinary tool results, not errors.

Make the fetch tool say plainly whether it read the page. Extract PDFs, keep timeouts short, and check citations against a ledger before you trust the report. Then look at what's left. If public sources keep refusing your server's IP under every user agent, that's when residential proxies are worth testing. Compare plans on the pricing page.

About the Author

Unknown Proxies

Proxy Infrastructure Team

Stay Unknown

High-performance dedicated proxies optimized for speed and reliability. Get uncompromising quality, 99.9% uptime, and unmatched support. Stay Unknown.

Explore Plans