We use cookies to enhance user experience, personalize content, and analyze traffic. Cookie Policy

← Back to all articles

AI Agent Observability: Tracing Requests, Blocks, and Retries

AI agent observability for web tools: why agent traces mark blocked pages as successful, where retries disappear, and how to trace each request. Tested.

by Unknown Proxies

14 min read

November 9, 2026

AI Agent Observability: Tracing Requests, Blocks, and Retries

AI agent observability means recording enough about each run to rebuild what the agent did afterwards: which model calls it made, which tools it called, and, for agents that use the web, every HTTP request, its status code, every retry, and the exit it went out through. Agent frameworks cover the first two well. The web layer is usually missing, and that's where blocks, rate limits, and retries happen.

We measured the gap. A Claude agent read five product pages through a fetch tool while Claude Code exported its own OpenTelemetry traces. Two pages were blocked: one with a 403 challenge and one with a block page served as 200 OK. Claude Code recorded all 5 tool calls as success=true in all 4 runs. A page that got 429, 429, then 200 showed up as a single 4-second tool span with no attempt count. The model's answer was correct every time. The telemetry wasn't.

There are two fixes. The tool tells the agent runtime when a page was blocked, and the tool emits its own spans, one per HTTP attempt, keyed by the tool call ID. Both are below, with the test, a tested Python snippet, and the dashboards worth building on top.

What AI agent observability covers

A web agent's run passes through four layers, and each one answers different questions:

Layer What it records Questions it answers Who usually emits it
Model calls Prompt, output, tokens, latency, cost Why did the agent choose this step? What did the run cost? Agent framework, LLM observability platform
Tool calls Tool name, arguments, result size, duration, success Which tools ran, in what order, and how long did they take? Agent framework
HTTP attempts Method, URL, status, retry number, response headers Was the page served, blocked, or rate limited? How many requests did the site actually see? Your tool code or HTTP client instrumentation
Network egress Proxy session, exit IP, CONNECT tunnels, proxy errors Which exit did the site see? Did the proxy or the site fail? Your tool code, proxy logs

LLM observability platforms such as Langfuse, Arize Phoenix, and LangSmith accept OpenTelemetry traces over OTLP, so all four layers can end up in one trace view. The first two layers come almost for free. The last two only exist if your tool code produces them.

OpenTelemetry already has names for this. The GenAI semantic conventions define an execute_tool span with gen_ai.tool.name and gen_ai.tool.call.id, still marked Development. The HTTP client span conventions say instrumentations "SHOULD create an HTTP span for each attempt to send an HTTP request over the wire," and they define http.request.resend_count for retries. In our tests, the agent runtime and the common Python HTTP instrumentations both stopped short of that.

Test: what an agent's own telemetry records for blocked pages

Setup

We built a local mock store with five product pages, each with a different failure:

The agent was Claude Code 2.1.310 in claude -p mode on claude-haiku-5-5. Its only tool was a small MCP stdio server with a fetch_page tool that owns its retries: it waits for Retry-After on 429 and 503, up to 3 attempts. That's the design AI agent architecture recommends, because the model can't wait between calls.

We turned on Claude Code's built-in OpenTelemetry export, including the beta tracing:

export CLAUDE_CODE_ENABLE_TELEMETRY=1
export CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1
export OTEL_TRACES_EXPORTER=otlp OTEL_LOGS_EXPORTER=otlp OTEL_METRICS_EXPORTER=none
export OTEL_EXPORTER_OTLP_PROTOCOL=http/json
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:19411
export OTEL_LOG_TOOL_DETAILS=1 OTEL_LOG_TOOL_CONTENT=1

A small collector on that port saved every OTLP payload. We ran the task 4 times with a plain fetch tool that returns HTTP <status> plus the page text and never sets isError. Then we ran it 3 times with a flagged version that returns isError: true when the page was blocked.

Results

Each run took 6 turns, 8.3 to 8.8 seconds, and $0.0004 to $0.0006. The store logged 8 requests per run for 5 tool calls. All 7 runs gave the same telemetry pattern:

Table comparing the HTTP responses each mock product page sent with the success value Claude Code's telemetry recorded, for a plain fetch tool and one that sets isError on blocked pages

Where trace context stops

The tool's own spans need to connect to the agent's spans somehow. SEP-414 documents the MCP convention for that: a W3C traceparent in the request's _meta. Claude Code's docs say it passes TRACEPARENT to Bash subprocesses and a traceparent header on outbound HTTP MCP requests. Our stdio server got neither. Its environment had no TRACEPARENT or OTEL_* variables, and every tools/call carried only this in _meta:

{"claudecode/toolUseId": "toolu_01XUkQQiBXSPJ4aXm6B4x3FH", "progressToken": 2}

That ID is enough. It matched the tool_use_id and gen_ai.tool.call.id attributes on Claude Code's spans and events in every run. In -p mode, each claude_code.tool span also arrived as the root of its own trace, so the tool call ID was the most reliable join key anyway. Record it on your tool spans, and you can put the agent's view and the wire's view side by side.

Why retries disappear under HTTP instrumentation

The usual fix for missing HTTP data is auto-instrumentation. It helps, but with one catch: retries that happen below the instrumented call don't get their own spans. We tested opentelemetry-instrumentation 0.66b2 against the same mock pages, with requests 2.34.2 mounting a urllib3.Retry(total=3, backoff_factor=0.5, status_forcelist=[429, 503], respect_retry_after_header=True) adapter, and with httpx 0.28.1 in a hand-written retry loop:

Setup Spans for /product/3 (429, 429, 200) What the span says
requests instrumentation, urllib3 Retry 1 200, 4,005 ms, status UNSET
urllib3 instrumentation only, urllib3 Retry 1 200, 4,004 ms, status UNSET
requests + urllib3 instrumentation 1 Only the requests span appears
httpx instrumentation, manual retry loop 3 429 ERROR, 429 ERROR, 200

The mock store logged all three requests in every case. When Retry handles the retries, they happen inside a single instrumented call, so you get one span with the final status. The /product/5 result shows something else: urllib3 sent its first retry after the 503 with no delay at all, because its backoff only starts from the second consecutive error. Nobody set http.request.resend_count on any span.

Two smaller details. The /product/4 challenge and the /product/2 block page each got one span, but only the 403 was marked ERROR. A block page served with 200 looks healthy at the HTTP layer too. And the requests and urllib3 instrumentations emit the old http.status_code attribute until you set OTEL_SEMCONV_STABILITY_OPT_IN=http, which switches them to http.response.status_code, as the HTTP migration guide describes. Check which name your dashboards query.

How to trace web requests from an agent tool

The fix is to emit the spans yourself in the tool's network code: one execute_tool span per tool call, one CLIENT span per HTTP attempt, and a block classification on both. This version uses httpx and the OpenTelemetry SDK (tested with httpx 0.28.1 and opentelemetry-sdk 1.45.2 on Python 3.14):

import os
import sys
import time
from datetime import timezone
from email.utils import parsedate_to_datetime
from urllib.parse import urlsplit, urlunsplit

import httpx
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
from opentelemetry.trace import SpanKind, Status, StatusCode
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

# Console output for the demo. In production, use BatchSpanProcessor(OTLPSpanExporter()).
provider = TracerProvider(resource=Resource.create({"service.name": "agent-web-tools"}))
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter(
    formatter=lambda s: f"{s.name:28} {s.status.status_code.name:5} {dict(s.attributes)}\n")))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("agent.web")

RETRY_STATUSES = {429, 503}
MAX_ATTEMPTS = 3
MAX_WAIT_S = 30  # longest Retry-After the tool will wait out
PROXY_URL = os.environ.get("PROXY_URL")              # credentials stay here, never in spans
PROXY_SESSION = os.environ.get("PROXY_SESSION", "")  # a label you choose, e.g. the sticky session ID
UA = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36"  # Keep the browser version current.

client = httpx.Client(proxy=PROXY_URL, headers={"user-agent": UA}, timeout=20)


def classify(r: httpx.Response) -> str:
    if r.headers.get("cf-mitigated") == "challenge":
        return "challenge"
    if r.status_code == 429:
        return "rate_limited"
    if r.status_code == 407:
        return "proxy_auth"
    if r.status_code in (401, 403):
        return "forbidden"
    if r.status_code >= 500:
        return "server_error"
    if r.status_code >= 400:
        return "http_error"
    if r.status_code >= 300:  # httpx doesn't follow redirects unless told to
        return "redirect"
    text = r.text[:20000].lower()
    if "verify you are a human" in text or "captcha" in text:
        return "soft_block"
    return "ok"


def retry_after_seconds(r: httpx.Response) -> float | None:
    value = r.headers.get("retry-after", "")
    if value.isdigit():
        return float(value)
    try:  # HTTP-date form
        when = parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return None
    if when.tzinfo is None:  # HTTP dates are always UTC
        when = when.replace(tzinfo=timezone.utc)
    return max(0.0, when.timestamp() - time.time())


def span_url(url: str) -> str:
    """The URL without userinfo, query string, or fragment, which can carry credentials and tokens."""
    parts = urlsplit(url)
    host = parts.netloc.rpartition("@")[2]
    return urlunsplit((parts.scheme, host, parts.path, "", ""))


def fetch_page(url: str, call_id: str, meta: dict | None = None) -> tuple[str, httpx.Response]:
    """One execute_tool span per tool call, one CLIENT span per HTTP attempt."""
    parent = TraceContextTextMapPropagator().extract(meta or {})  # uses _meta.traceparent if the client sent one
    with tracer.start_as_current_span("execute_tool fetch_page", context=parent, attributes={
        "gen_ai.operation.name": "execute_tool",
        "gen_ai.tool.name": "fetch_page",
        "gen_ai.tool.call.id": call_id,
        "server.address": urlsplit(url).hostname,
        "app.proxy.session": PROXY_SESSION,
    }) as tool_span:
        for attempt in range(MAX_ATTEMPTS):
            with tracer.start_as_current_span("GET", kind=SpanKind.CLIENT, attributes={
                "http.request.method": "GET",
                "url.full": span_url(url),
                "server.address": urlsplit(url).hostname,
                "app.proxy.session": PROXY_SESSION,
            }) as span:
                if attempt:
                    span.set_attribute("http.request.resend_count", attempt)
                r = client.get(url)
                outcome = classify(r)
                span.set_attribute("http.response.status_code", r.status_code)
                span.set_attribute("app.fetch.outcome", outcome)
                for h in ("retry-after", "cf-mitigated"):
                    if h in r.headers:
                        span.set_attribute(f"http.response.header.{h}", [r.headers[h]])
                if r.status_code >= 400:
                    span.set_attribute("error.type", str(r.status_code))
                    span.set_status(Status(StatusCode.ERROR))
            if r.status_code not in RETRY_STATUSES or attempt == MAX_ATTEMPTS - 1:
                break
            wait = retry_after_seconds(r)
            if wait is None:
                wait = 2 ** attempt  # no usable Retry-After: short backoff
            if wait > MAX_WAIT_S:
                break  # a longer pause than this call can wait: report the outcome
            time.sleep(wait)

        tool_span.set_attribute("app.fetch.attempts", attempt + 1)
        tool_span.set_attribute("app.fetch.outcome", outcome)
        if outcome != "ok":
            tool_span.set_attribute("error.type", outcome)
            tool_span.set_status(Status(StatusCode.ERROR, outcome))
        return outcome, r


if __name__ == "__main__":
    for i, url in enumerate(sys.argv[1:], start=1):
        fetch_page(url, call_id=f"demo-{i}")

Install httpx and opentelemetry-sdk, then pass URLs on the command line. Run against the mock store's /product/3 and /product/4 through a local authenticating proxy, it printed:

GET                          ERROR {'http.request.method': 'GET', 'url.full': 'http://127.0.0.1:18980/product/3', 'server.address': '127.0.0.1', 'app.proxy.session': 'task-42', 'http.response.status_code': 429, 'app.fetch.outcome': 'rate_limited', 'http.response.header.retry-after': ('2',), 'error.type': '429'}
GET                          ERROR {'http.request.method': 'GET', 'url.full': 'http://127.0.0.1:18980/product/3', 'server.address': '127.0.0.1', 'app.proxy.session': 'task-42', 'http.request.resend_count': 1, 'http.response.status_code': 429, 'app.fetch.outcome': 'rate_limited', 'http.response.header.retry-after': ('2',), 'error.type': '429'}
GET                          UNSET {'http.request.method': 'GET', 'url.full': 'http://127.0.0.1:18980/product/3', 'server.address': '127.0.0.1', 'app.proxy.session': 'task-42', 'http.request.resend_count': 2, 'http.response.status_code': 200, 'app.fetch.outcome': 'ok'}
execute_tool fetch_page      UNSET {'gen_ai.operation.name': 'execute_tool', 'gen_ai.tool.name': 'fetch_page', 'gen_ai.tool.call.id': 'demo-1', 'server.address': '127.0.0.1', 'app.proxy.session': 'task-42', 'app.fetch.attempts': 3, 'app.fetch.outcome': 'ok'}
GET                          ERROR {'http.request.method': 'GET', 'url.full': 'http://127.0.0.1:18980/product/4', 'server.address': '127.0.0.1', 'app.proxy.session': 'task-42', 'http.response.status_code': 403, 'app.fetch.outcome': 'challenge', 'http.response.header.cf-mitigated': ('challenge',), 'error.type': '403'}
execute_tool fetch_page      ERROR {'gen_ai.operation.name': 'execute_tool', 'gen_ai.tool.name': 'fetch_page', 'gen_ai.tool.call.id': 'demo-2', 'server.address': '127.0.0.1', 'app.proxy.session': 'task-42', 'app.fetch.attempts': 1, 'app.fetch.outcome': 'challenge', 'error.type': 'challenge'}

The proxy credentials don't appear anywhere in the output. Credentials or tokens in the target URL don't either: span_url drops the userinfo, query string, and fragment from url.full, and the request still goes out with the full URL. The GET spans also show what the agent's trace couldn't: three attempts, two waits, and why the call took 4 seconds.

Trace waterfall joining Claude Code's tool span to the fetch tool's execute_tool span and three GET attempts by tool call ID, plus a blocked call marked as an error in both

A few details worth knowing:

For browser-based agents, the HTTP layer lives inside the browser, so per-request spans are harder to get. Record a HAR per task instead. In Playwright, browser.new_context(record_har_path="task-42.har") writes one when the context closes. The HAR analyzer shows which requests came back 403 or 429.

Block signals to record on every request

The classifier only works if the attributes behind it get recorded. These are the signals that separated the outcomes in our test, plus the proxy-side ones:

Signal Where it shows Record as
cf-mitigated: challenge Response header, any status http.response.header.cf-mitigated, outcome challenge
429 plus Retry-After Status and header Status, http.response.header.retry-after, outcome rate_limited
401 / 403 with no challenge header Status Outcome forbidden
200 with block-page markers Body only Outcome soft_block; HTTP metrics alone will miss it
407, or ProxyError on HTTPS Proxy response or exception Outcome proxy_auth; the site never saw the request
Retry number Your retry loop http.request.resend_count
Proxy session label Your config A custom attribute such as app.proxy.session

The proxy session label is what lets you tell "this site blocks us" apart from "this exit is burned." If one session label has most of the challenges for a domain and the others don't, that points to IP reputation. If every session gets them, look at your request rate, headers, or the site's policy. How websites detect AI agents covers the signals sites check besides the IP.

Dashboards and alerts for web agents

Once the spans exist, a few queries cover most incidents:

If 429s dominate, HTTP 429 Too Many Requests covers the pacing side. If 403s dominate, start with HTTP 403 Forbidden.

What to keep out of agent traces

Traces often go to third-party backends and stay there for weeks, so decide what goes in:

Where proxies fit

AI agent observability is how you find out whether a proxy would help at all. If the blocked share is spread across every session label and lines up with your request rate, slow down first. A new IP won't fix pacing. If blocks cluster on a cloud server's IP range or on one exit, the exit is the problem.

For agents that read many independent pages, rotating residential proxies spread requests across exits. A rotating line can get a new exit on every request, so record the session label you can see rather than guessing the exit afterwards. For multi-step tasks, give each task one sticky residential session and put its label on every span. Unknown Proxies sticky sessions hold an IP for up to 2 hours, so a task's whole trace maps to one exit. Agents that log in to the same account every day are better served by a static ISP proxy, and its IP can be the label.

A proxy doesn't change a site's terms, robots rules, or rate limits. Use the traces to keep the agent within them. A challenge page is the site telling you something, and the right response is often to stop.

FAQ

What is AI agent observability?

It's the practice of recording what an AI agent did during a run (model calls, tool calls, HTTP requests, and network egress) so you can debug failures, measure cost, and check whether the agent's answer matches what happened. For web agents, the part most often missing is the HTTP layer: status codes, retries, and blocks.

What is the difference between LLM observability and AI agent observability?

LLM observability tracks individual model calls: prompts, outputs, tokens, latency. Agent observability adds the loop around them: which tools ran, in what order, what they returned, and what the tools did in the outside world. A trace can show a perfect model call that acted on a blocked page.

Does OpenTelemetry support AI agents?

Yes. The GenAI semantic conventions define spans such as execute_tool, with attributes like gen_ai.tool.name and gen_ai.tool.call.id, though they're still marked Development. Claude Code exports OpenTelemetry metrics, events, and beta traces, and many LLM observability platforms accept OTLP. The HTTP side uses the standard HTTP client conventions.

Why does my agent trace show success when the page was blocked?

Because the tool returned the block page as normal content. In our test, Claude Code recorded success=true for a 403 challenge and for a 200 block page until the tool returned isError: true. Classify the response in the tool, and report blocks as tool errors.

How do I trace retries in an AI agent?

Retry in your own loop and create one HTTP client span per attempt, with http.request.resend_count on each retry. With urllib3's Retry adapter, the requests and urllib3 instrumentations recorded a 3-attempt fetch as a single span in our test.

Should I log fetched page content in traces?

Only while you're investigating. Page text helps you recognize soft blocks, but it can contain personal data and makes traces much larger. Record the outcome classification and the few headers that explain it on every request, and keep full content for targeted debugging.

Final thoughts

AI agent observability for web agents depends on what the tool reports. Agent runtimes trace the model and the tool call, and they trust the tool's word on whether the call worked. If the tool returns a challenge page as text, the trace says success. If the retries happen inside the HTTP library, the trace shows one long span.

Make the fetch tool the source of truth. Classify every response, return blocks as tool errors, emit one span per HTTP attempt with the tool call ID and proxy session label, and keep credentials and page content out. Then the trace shows what the website saw. When the data says the exit is the problem, compare residential and ISP plans.

About the Author

Unknown Proxies

Proxy Infrastructure Team

Stay Unknown

High-performance dedicated proxies optimized for speed and reliability. Get uncompromising quality, 99.9% uptime, and unmatched support. Stay Unknown.

Explore Plans