Multi-agent systems split one job across several AI agents. Each agent has its own instructions, tools, and context window, and an orchestrator, a handoff rule, or plain code decides who works on what. For agents that browse the web, the hard part usually isn't how they message each other. It's the state they share without meaning to: one cookie jar, one browser profile, one exit IP, and one target site's patience.
The rule that follows is simple. Give every agent task its own session, meaning its own cookies, connection pool, and sticky exit IP. Then make all agents draw from one shared request budget per target domain. Separate sessions keep agents from corrupting each other's state. The shared budget keeps the system as a whole from hitting a site harder than a single client should.
We tested this with Claude Code: an orchestrator sent three sub-agents in parallel to read regional prices from a mock storefront. When the sub-agents shared one session and the page didn't show which region was active, the final table was wrong in 3 of 3 runs, and the orchestrator blamed the storefront each time. With a session per agent it was right in all 8 runs. Giving each sub-agent its own MCP server didn't help until each server's config differed. The details and the fixes are below.
What multi-agent systems are
In the older research sense, a multi-agent system is any set of autonomous agents acting in a shared environment, such as robots, trading bots, or simulated traffic. Most people searching the term today mean LLM agents, and that's the sense used here: several model-driven agents, each running its own tool loop, working toward one result.
Splitting the work buys two things. First, each agent keeps a smaller, cleaner context, because a sub-agent reading three product pages doesn't need the orchestrator's whole conversation. Second, independent subtasks run in parallel. Anthropic's write-up of its multi-agent research system reports that a Claude Opus 4 lead with Claude Sonnet 4 sub-agents "outperformed single-agent Claude Opus 4 by 90.2%" on its internal research eval, and that parallel sub-agents "cut research time by up to 90% for complex queries."
The cost is real too. The same post says multi-agent systems "use about 15× more tokens than chats," and that without detailed task descriptions "agents duplicate work, leave gaps, or fail to find necessary information." On the web, duplicated work means duplicated requests to the same site.
If you're deciding whether you need agents at all, AI agent vs chatbot vs automation covers that first. If you want the parts of one agent, start with AI agent architecture.
Multi-agent architectures and their web traffic
The frameworks use different names for a small set of shapes. The OpenAI Agents SDK separates agents as tools from handoffs. LangChain's multi-agent docs list subagents, handoffs, skills, routers, and custom workflows. Claude Code has subagents. What matters for the network is who opens connections to the target site, how many do it at once, and whether they're meant to be the same visitor.
| Architecture | Who talks to the website | Concurrency on one host | Session rule |
|---|---|---|---|
| Orchestrator and workers (subagents, agents as tools) | Each worker, in parallel | As many as the workers you launch | One session per worker |
| Router | The one or more specialists the router picks | Usually low | One session per specialist call |
| Handoffs (triage, then specialist) | Whichever agent holds the conversation | One at a time | Pass the session along with the handoff |
| Sequential pipeline (researcher, then writer) | Usually only the first stage | One | One session for the fetching stage |
| Hierarchical (supervisors of supervisors) | Leaf workers | Multiplies at each level | One session per leaf, one budget for the tree |

The orchestrator-workers row is where browsing systems get into trouble, because parallel workers are the reason to build one. CrewAI and LangGraph build the same shapes in their own vocabulary. LangGraph vs CrewAI covers how each one handles a blocked request.
Where browsing agents end up sharing one session
Nobody sets out to put three agents in one cookie jar. It happens because the web tool lives at the process level and the agents don't:
- One MCP server for every sub-agent. In Claude Code, a server configured for the session, in
.mcp.jsonor with--mcp-config, runs as one process that every sub-agent calls. In our runs, every sub-agent's calls went to the same process ID, and eachtools/callcarried only aprogressTokenand aclaudecode/toolUseIdin_meta, with nothing naming the sub-agent. The server can't tell callers apart, so whatever it keeps (cookies, a logged-in browser, an open tab) is shared. - One browser. Browser MCP servers such as Playwright MCP drive one browser per server process by default. Playwright MCP's README warns that "concurrent MCP clients sharing the same workspace will conflict" and tells you to start extra clients "with
--isolatedor point it at a distinct--user-data-dir." Sub-agents behind one stdio server are one client. - One HTTP client. In your own code, a module-level
requests.Sessionorhttpx.Clientthat every agent's tool imports is one cookie jar. - One proxy setting.
HTTPS_PROXYexported for the agent process gives every agent the same exit. Behind a rotating gateway with no session ID, the opposite happens: every request can leave from a different IP, which breaks each agent's own session. Sticky vs rotating proxies covers that trade-off for single clients.
Sharing an exit IP isn't automatically wrong. Sharing mutable session state between agents with different goals is. Some of it is easy to spot, like a cart that suddenly holds another agent's items. Some of it is silent, like a region, currency, or language cookie that quietly changes what every later page says.
Test: three shopper sub-agents and one storefront
We wanted to know what that sharing does to real agent output, and whether the model notices.
Setup
- Storefront: a local mock with six monitors over two list pages. The region is a cookie set by
/region/us,/region/de, or/region/uk(a302back to the homepage), and prices differ by region. Each response takes 400–800 ms. Every request is logged with its source address and the region cookie it arrived with. - Agents: Claude Code 2.1.310 in
claude -p, orchestrator and sub-agents on Claude Haiku. The orchestrator had only theAgenttool and was told to launch threeshoppersub-agents in parallel, one per region, and merge their reports into one table. Each shopper had only afetch_pagetool and was told to switch to its region, then read both pages. - Fetch tool: a small MCP server with a cookie jar. To stand in for separate proxy exits, it can bind outgoing connections to different loopback addresses (
127.0.0.2,127.0.0.3, and so on), so the storefront sees different client IPs. - Two storefront variants: "visible" pages show "Prices shown for: Germany (EUR)" and currency symbols. "Silent" pages show bare numbers and no region marker, the way many B2B portals and data tables do.
Results
| Setup | Runs | Server processes | Client IPs the site saw | Requests | Correct tables |
|---|---|---|---|---|---|
| Shared session, visible region | 3 | 1 | 1 | 25–28 | 3 of 3, after re-reads |
Session per agent (session argument), visible region |
3 | 1 | 3 | 15 | 3 of 3 |
| Shared session, silent | 3 | 1 | 1 | 15 | 0 of 3 |
Session per agent (session argument), silent |
3 | 1 | 3 | 15 | 3 of 3 |
| Inline MCP server in one shopper definition, silent | 2 | 1–2 | 1–2 | 15–16 | 0 of 2 |
| Three definitions, identical inline configs, silent | 1 | 1 | 1 | 15 | 0 of 1 |
| Three definitions, inline configs differing in one env var, silent | 2 | 3 | 3 | 15 | 2 of 2 |
On the visible storefront, the shared session still produced right answers, because the shoppers could see the region flip under them and redo their reads. That recovery cost something. Compared with separate sessions, the shared runs made 25–28 requests instead of 15, 16 sub-agent model calls instead of 9–10, used 49.9k–52.0k input tokens instead of 31.7k–34.0k, and took 19.8–23.6 s instead of 12.0–14.1 s.
On the silent storefront, there was nothing to notice. The three shoppers sent their region switches within 0.8–1.3 s of each other, and the last one to land won. It was the UK switch in all three runs. In two of them, all six list-page reads were served UK prices, so the US, DE, and UK columns were identical. In the third, the US shopper read page 1 before the other switches landed, so its column mixed US prices from page 1 with UK prices from page 2.

The orchestrator flagged doubt in all three silent runs, and to its credit it never presented the table as clean. But it blamed the storefront every time ("the region switch never took effect") and never named the shared session. A person reading that report would go debug the website.
What the inline-server runs showed
Claude Code lets a sub-agent definition declare its own MCP servers in mcpServers. The docs say inline servers "are connected when the subagent starts" and "disconnected when it finishes." That sounds like a server per sub-agent, so we put the fetch server inline in the shopper definition and launched three shoppers.
In 2.1.310 that is not what happened. Three parallel instances of one definition shared one server process. Three separate definitions whose inline configs were identical also shared one process. In one run, the first shopper to finish took the server down with it. The UK shopper's in-flight call came back Connection closed, its retry started a fresh process with an empty cookie jar, and it read US prices for page 2 into the UK column. The fix was to make each definition's config different. With a different environment variable per definition, each sub-agent got its own process, its own jar, and its own exit, and both runs came back correct.

Two more numbers apply across setups. In 16 of the 17 runs, three requests were in flight against the one host at the peak (two in the other), because three agents in parallel send three clients' worth of traffic. And the session-argument approach worked in all 6 runs: every one of the 72 tool calls carried the right session name. That's still a model choice, though, not a guarantee, which is why the next section binds the session in code where it can.
How to give each agent its own session and exit IP
The pattern is the same in every framework: create the session when you create the agent task, close it when the task ends, and don't leave the choice of session to the model when you can avoid it.
In your own code: a session per agent, a budget per domain
This Python layer gives each agent task its own httpx.AsyncClient (its own cookie jar and connections) and its own sticky proxy session. Every agent shares one DomainBudget, which caps requests in flight and spacing per host, and pauses the host for everyone when any agent gets a 429 or 503. AgentSession follows redirects itself, so each hop waits its turn on its own host. It's GET-only on purpose. A layer that retries POSTs can submit an action twice.
import asyncio
import os
import time
import uuid
from datetime import timezone
from email.utils import parsedate_to_datetime
from urllib.parse import urlsplit
import httpx
def retry_after_seconds(value: str, default: float = 30) -> float:
"""Retry-After is either a number of seconds or an HTTP date."""
if value.isdigit():
return int(value)
try:
when = parsedate_to_datetime(value)
except (TypeError, ValueError):
return default
if when.tzinfo is None: # HTTP dates are always UTC
when = when.replace(tzinfo=timezone.utc)
return max(0.0, when.timestamp() - time.time())
class DomainBudget:
"""Shared by every agent: caps in-flight requests and spacing per host,
and pauses the host for all agents when any of them gets a 429 or 503."""
def __init__(self, max_in_flight: int = 2, min_interval: float = 1.0):
self.max_in_flight = max_in_flight
self.min_interval = min_interval
self._slots: dict[str, asyncio.Semaphore] = {}
self._next_start: dict[str, float] = {}
self._paused_until: dict[str, float] = {}
self._lock = asyncio.Lock()
async def _wait_turn(self, host: str) -> None:
while True:
async with self._lock:
now = time.monotonic()
start = max(now, self._next_start.get(host, now), self._paused_until.get(host, now))
self._next_start[host] = start + self.min_interval
await asyncio.sleep(start - now)
if self._paused_until.get(host, 0) <= time.monotonic():
return
async def get(self, client: httpx.AsyncClient, url: str, retries: int = 1) -> httpx.Response:
host = urlsplit(url).hostname
slots = self._slots.setdefault(host, asyncio.Semaphore(self.max_in_flight))
for _ in range(retries + 1):
async with slots:
await self._wait_turn(host)
response = await client.get(url)
if response.status_code not in (429, 503):
return response
seconds = retry_after_seconds(response.headers.get("retry-after", ""))
self._paused_until[host] = max(self._paused_until.get(host, 0), time.monotonic() + seconds)
return response
class AgentSession:
"""One per agent task: its own cookie jar, connection pool, and sticky exit IP."""
def __init__(self, name: str, budget: DomainBudget):
self.session_id = f"{name}{uuid.uuid4().hex[:8]}"
# Session syntax in the username is provider-specific; copy yours from the dashboard.
user = f"{os.environ['PROXY_USER']}-session-{self.session_id}"
proxy = f"http://{user}:{os.environ['PROXY_PASS']}@{os.environ['PROXY_HOST']}"
self.client = httpx.AsyncClient(proxy=proxy, timeout=30)
self.budget = budget
async def get(self, url: str, max_redirects: int = 5) -> httpx.Response:
# Follow redirects here, not in httpx, so every hop waits for its own host's budget.
for _ in range(max_redirects + 1):
response = await self.budget.get(self.client, url)
if response.next_request is None: # not a redirect, or a 3xx with no Location
return response
url = str(response.next_request.url)
raise httpx.TooManyRedirects("Too many redirects", request=response.request)
async def aclose(self) -> None:
await self.client.aclose()
Each agent's fetch tool calls its own AgentSession.get(). A minimal driver for the three shoppers looks like this:
import asyncio
import re
from agents_net import AgentSession, DomainBudget
SHOP = "http://127.0.0.1:18970"
async def shopper(region: str, budget: DomainBudget) -> tuple[str, list[str]]:
session = AgentSession(region, budget)
try:
response = await session.get(f"{SHOP}/region/{region}")
response.raise_for_status() # never read default prices under this region's label
prices = []
for page in (1, 2):
response = await session.get(f"{SHOP}/monitors?page={page}")
response.raise_for_status()
prices += re.findall(r"<li>([^<]+)</li>", response.text)
return region, prices
finally:
await session.aclose()
async def main() -> None:
budget = DomainBudget(max_in_flight=2, min_interval=0.5)
for region, prices in await asyncio.gather(*(shopper(r, budget) for r in ("us", "de", "uk"))):
print(region, prices)
asyncio.run(main())
We ran it with httpx 0.28.1 on Python 3.12 against the silent storefront, through a local authenticating proxy that maps each session ID to its own source address. The storefront was set to return one 429 with Retry-After: 2. All three regions came back correct from three different client IPs, with at most 2 requests in flight. After the US session hit the 429, no agent sent a new request to the host for 2 s, and the queued requests then resumed 0.5 s apart.
In Claude Code: one inline server config per agent
For Claude Code sub-agents, define one agent per identity and give each an inline server whose config differs, for example by a session variable the server appends to the proxy username:
---
name: shopper-de
description: Reads monitor prices from the German storefront
tools: mcp__web__fetch_page
model: haiku
mcpServers:
- web:
type: stdio
command: node
args: ["/opt/agents/fetch-server.mjs"]
env:
PROXY_SESSION: "shopper-de"
---
Switch the storefront to the DE region, then read every price on every page.
Our runs passed the same fields as JSON through --agents. Agent files in a project's .claude/agents/ folder load inline servers only after you trust that folder. Claude Code passes its own environment to stdio servers, so shared values like PROXY_USER and PROXY_PASS can come from there, and only the session variable needs to differ. Don't launch several copies of one definition in parallel and expect separate servers. In our runs they shared one.
If you already use the Claude Agent SDK proxy setup, the same rule applies there. Everything one MCP server process holds is shared by every agent that calls it.
When a session argument is enough
The cheapest change is the one from our test: the fetch tool requires a session parameter, and the server keeps one cookie jar and exit per value. It needs no framework support and worked in all 72 calls we made. Use it for read-only research where a wrong session costs a re-run. For logins, carts, and anything that writes, bind the session in code or config, because the model can still pass the wrong value.
Share the request budget, not the session
Separate sessions fix correctness. They also make the site see more clients, and that's a responsibility, not a loophole. Three parallel sub-agents put three requests in flight on one host in almost every run of our test. Anthropic's research system launches 3–5 sub-agents that each run 3 or more tool calls in parallel, so one task there can put 15 or more requests in flight.
Size the budget by target domain, not by agent:
- Start at 1–2 requests in flight per domain for the whole system, with a delay between request starts, and raise it only while
429s stay at zero. - When any agent gets a
429or503withRetry-After, pause that domain for every agent, as theDomainBudgetabove does. Sending another agent's request during the pause ignores the site's signal and can earn a longer block. RFC 6585 defines429 Too Many Requestsand allowsRetry-Afteron it. - Deduplicate fetches across agents only for pages that read the same for every identity. A shared cache keyed by URL saves repeat fetches of public docs or articles, but a page that varies by cookies, account, region, or exit IP needs a cache scoped to the agent task. Otherwise the DE shopper gets the US shopper's cached prices.
- Give separate exit IPs only to identities that really are separate, such as different regions, different accounts you own, or independent research tasks. Don't use them to multiply your request rate past what a site allows. Check the site's terms and
robots.txtthe same way you would for a single scraper.
When agents should share one IP
Handoffs are the exception. If a triage agent logs into an account and hands the conversation to a checkout agent, the checkout agent is continuing the same visit. Pass the whole session along: the cookie jar, the browser context or storage state, and the same sticky exit. Switching IP mid-session can trigger a re-login, a step-up challenge, or a fraud check, especially on retail and banking sites.
The same goes for several agents working one account in sequence. Keep them on that account's IP, ideally a static one, and run them one at a time. Two agents driving one logged-in account in parallel is the cookie race from our test, with real money attached.
Which proxies fit a multi-agent system
Count concurrent identities, not agents. That number is how many separate sessions you need open at once.
- Parallel research workers on public pages: sticky residential sessions, one session ID per worker task, discarded when the task ends. Residential proxies let you open many sessions from one plan, with regional exits for geo-specific checks like the price comparison above.
- Long-lived agents tied to accounts: one static ISP IP per account, kept across days. ISP plans fit agents that should look like the same customer every time.
- Mixed systems: residential for the fan-out research layer, ISP for the agents that log in.
Whatever you buy, the egress rules from proxies for AI agents still apply: check the exit before a task starts, cap bandwidth per task, and send model API traffic direct rather than through the proxy. AI agent frameworks compared lists which framework tools actually follow your proxy settings.
FAQ
What is a multi-agent system?
A multi-agent system is a setup where several agents, each with its own instructions, tools, and context, work on parts of one task and an orchestrator, router, handoff rule, or code combines the results. In AI today, that usually means several LLM agents, such as a research lead with parallel sub-agents.
What is the difference between a multi-agent system and a single agent with tools?
A single agent keeps one context and decides every step itself. A multi-agent system splits the work so each agent sees only its part, and independent parts can run in parallel. It's often faster and more thorough on broad research. It also uses far more tokens, and on the web it multiplies sessions and concurrent requests.
Do multi-agent systems need more proxies?
They need one session per concurrent identity, not one IP per agent. Five workers reading public pages can share a residential plan with five sticky session IDs. Agents that log in need the account's own stable IP, and agents that hand off a visit should keep the same one.
Can sub-agents share one browser?
Only if they run one after another on the same task. In parallel, they share tabs, cookies, and local storage. Give each concurrent agent its own browser context, or its own browser MCP server with an isolated profile and its own proxy.
Why did my sub-agents return each other's data?
Most often they share a cookie jar or browser profile through one tool process, and a setting one agent changed, like region, currency, language, or the logged-in account, applied to the others. Check whether all of them call the same MCP server or HTTP client, and give each its own session.
Are multi-agent systems better than single agents?
For broad questions that split into independent parts, often yes. Anthropic's research system outperformed a single agent by 90.2% on its internal research eval. For narrow tasks, or tasks where every step depends on the last, a single agent is cheaper and easier to debug.
Final thoughts
Multi-agent systems make browsing faster by running agents side by side, and that is exactly what exposes shared state. In our test, three sub-agents sharing one session produced a wrong price table in every run where the page didn't show the region, and the orchestrator blamed the website. One session per agent fixed it in every run. In the Python version, one shared budget per domain held three sessions to two requests in flight and paused all of them when one got a 429.
Build it in that order: decide which agents are separate identities, give each one its own cookies and sticky exit, and put every agent behind one budget per site. For the exits, use residential proxies for parallel research workers and ISP proxies for agents that hold an account.
