Rotating Proxies with Python Requests: Code and Retries

Q: How do I rotate proxies in Python requests?

Keep a list of proxy URLs, choose one per request with random.choice() or itertools.cycle(), and pass it as proxies={"http": proxy, "https": proxy}. Retry proxy errors on a different proxy.

Q: How do I retry a failed request with a different proxy?

Catch requests.exceptions.ProxyError and timeouts, mark that proxy as unhealthy for a short time, pick another proxy, and send the request again. Adapter-level Retry cannot do this, because it keeps the same proxy.

Q: How do I rotate proxies with asyncio?

With aiohttp, pass proxy= on each session.get() call and cycle through your list. With httpx, create one AsyncClient per proxy and cycle through the clients. Cap concurrency with asyncio.Semaphore.

Q: Why does my rotating gateway return the same IP with requests.Session?

Session reuses TCP connections. A gateway that rotates per connection then sees one connection and keeps the same IP. Use separate requests.get() calls, or close the session between requests.

To rotate proxies with Python requests, keep a list of proxy URLs, pick the next one for each request, and pass it as proxies={"http": url, "https": url}. When a request fails at the proxy level, retry it on a different proxy and put the failed one on a short cooldown. That is the whole pattern. The rest of this guide is the code and the rules that keep it from turning into a retry storm.

Rotate per request only when each URL stands alone. If the workflow logs in, keeps cookies, or walks through a cart, keep one sticky proxy for the whole flow. Changing the IP while reusing the same cookies causes avoidable challenges, logouts, and inconsistent responses.

Python scraper flow for rotating proxies, sessions, retries, and backoff

Rotating proxy Python requests: quick setup

Save your proxies one per line in proxies.txt, in http://user:pass@host:port format. If your provider exports host:port:user:pass, the proxy formatter converts the list. Then:

import random
import time
import requests

def load_proxies(path: str) -> list[str]:
    with open(path) as f:
        return [line.strip() for line in f if line.strip()]

PROXIES = load_proxies("proxies.txt")
COOLDOWN_SECONDS = 60
cooldown_until: dict[str, float] = {}

def pick_proxy() -> str:
    now = time.monotonic()
    healthy = [p for p in PROXIES if cooldown_until.get(p, 0) <= now]
    return random.choice(healthy or PROXIES)

def fetch(url: str, attempts: int = 3) -> requests.Response | None:
    for attempt in range(1, attempts + 1):
        proxy = pick_proxy()
        try:
            response = requests.get(
                url,
                proxies={"http": proxy, "https": proxy},
                timeout=(5, 20),
            )
        except requests.exceptions.ProxyError as error:
            if "407" in str(error):
                # Bad credentials affect every proxy that shares them. Stop.
                raise RuntimeError("Proxy credentials rejected (407)") from error
            # Proxy-level failure: cool this proxy down and try another one.
            cooldown_until[proxy] = time.monotonic() + COOLDOWN_SECONDS
            continue
        except (requests.exceptions.ConnectTimeout,
                requests.exceptions.ReadTimeout):
            cooldown_until[proxy] = time.monotonic() + COOLDOWN_SECONDS
            continue

        if response.status_code == 429:
            # Target rate limit: wait before the next attempt.
            retry_after = response.headers.get("Retry-After", "")
            delay = int(retry_after) if retry_after.isdigit() else 2 ** attempt
            time.sleep(delay + random.uniform(0, 1))
            continue

        return response  # 2xx, 3xx, 403, 404, and others go back to the caller

    return None

for url in ["https://example.com/a", "https://example.com/b"]:
    r = fetch(url)
    print(url, r.status_code if r is not None else "failed")

We ran this loop against local stand-in proxies (one dead, two working) with requests 2.34.2 in October 2026. The dead proxy raised ProxyError, went on cooldown, and every later request went to a working proxy. A 429 with Retry-After: 1 was retried after about 1.7 seconds. What the code does:

Picks a random healthy proxy for every request.
Retries proxy errors and timeouts on a different proxy, and stops at once on a 407 because the credentials are wrong.
Puts a failing proxy on a 60-second cooldown instead of deleting it.
Waits on 429, using Retry-After when the target sends seconds.
Returns 403 to the caller. Repeating a 403 on new IPs rarely helps.

Never log the full proxy URL. It contains the password. Log the host or an index.

Verify that rotation works

Before you point the loop at a real target, confirm that the exit IP changes:

for _ in range(5):
    proxy = pick_proxy()
    ip = requests.get(
        "https://ipv4.unknownproxies.com/ip",
        proxies={"http": proxy, "https": proxy},
        timeout=(5, 20),
    ).text.strip()
    print(proxy.split("@")[-1], "->", ip)

If every line shows the same IP, either the list has one real exit, or a rotating gateway is in sticky mode. If you see your own IP, the https key is missing or an environment variable is interfering.

Python requests proxy format

The proxies dict maps the target scheme to a proxy URL:

proxy_url = "http://username:password@host:port"
proxies = {"http": proxy_url, "https": proxy_url}

The https key means "use this proxy for HTTPS targets." The proxy URL itself still usually starts with http://, because HTTP proxies tunnel HTTPS with CONNECT. Our Python requests proxies dictionary reference covers per-host keys, environment variables, and SOCKS. If passwords contain @, :, or #, URL-encode them as shown in the Python requests proxy authentication guide.

Pass proxies= on each call when you rotate. Do not rotate by changing session.proxies, because environment proxies such as HTTPS_PROXY override session.proxies. If you need a session that ignores the environment, see Session.trust_env and NO_PROXY in Requests.

Round-robin vs random rotation

random.choice() spreads load without state. Round-robin gives each proxy an even share:

from itertools import cycle

proxy_cycle = cycle(PROXIES)

def next_proxies() -> dict[str, str]:
    proxy = next(proxy_cycle)
    return {"http": proxy, "https": proxy}

Method	Good for	Weak point
Round-robin (`itertools.cycle`)	Even load, predictable logs	Keeps hitting a dead proxy unless you skip it
Random (`random.choice`)	Simple, no shared state	Uneven in short runs
Health-weighted (cooldowns, as above)	Real pools with failures	More code
Rotating gateway	One endpoint, provider rotates	Less control over which IP you get

Check proxies before you rotate

Dead proxies waste your first requests. Test the list once at startup, in parallel, and keep only the ones that answer:

from concurrent.futures import ThreadPoolExecutor

import requests

TEST_URL = "https://ipv4.unknownproxies.com/ip"


def is_alive(proxy: str) -> bool:
    try:
        response = requests.get(
            TEST_URL,
            proxies={"http": proxy, "https": proxy},
            timeout=(3, 8),
        )
        return response.ok
    except requests.RequestException:
        return False


def filter_alive(proxies: list[str], workers: int = 10) -> list[str]:
    with ThreadPoolExecutor(max_workers=workers) as pool:
        flags = list(pool.map(is_alive, proxies))
    return [proxy for proxy, ok in zip(proxies, flags) if ok]


PROXIES = filter_alive(load_proxies("proxies.txt"))
print(f"{len(PROXIES)} proxies passed the check")

Use a neutral IP endpoint for this check, not your real target. A startup check does not replace cooldowns, because a proxy that passes now can fail ten minutes later.

Rotate proxies asynchronously with aiohttp and httpx

For hundreds of URLs, threads get heavy. Async clients handle more in-flight requests with less overhead. With aiohttp, pass the proxy on each request, which makes rotation a one-liner:

import asyncio
from itertools import cycle

import aiohttp

PROXIES = [
    "http://username:password@proxy1.example.com:8000",
    "http://username:password@proxy2.example.com:8000",
]
URLS = [f"https://example.com/page/{i}" for i in range(20)]


async def fetch(session, url, proxy, limit):
    async with limit:
        try:
            async with session.get(url, proxy=proxy) as response:
                return url, response.status
        except (aiohttp.ClientError, asyncio.TimeoutError) as error:
            return url, type(error).__name__


async def main() -> None:
    pool = cycle(PROXIES)
    limit = asyncio.Semaphore(5)
    timeout = aiohttp.ClientTimeout(total=20, connect=5)
    async with aiohttp.ClientSession(timeout=timeout) as session:
        results = await asyncio.gather(
            *(fetch(session, url, next(pool), limit) for url in URLS)
        )
    for url, result in results:
        print(url, result)


asyncio.run(main())

The semaphore caps concurrency, which matters more than the number of proxies. aiohttp ignores proxy environment variables unless you create the session with trust_env=True, so the environment override that affects Requests does not apply here. Credentials in the proxy URL work in current aiohttp. The separate proxy_auth argument raises a DeprecationWarning in aiohttp 3.14 (checked October 2026).

httpx sets the proxy on the client, not the request, per the HTTPX proxy docs. To rotate, build one AsyncClient per proxy and cycle through the clients:

import asyncio
from itertools import cycle

import httpx

PROXIES = [
    "http://username:password@proxy1.example.com:8000",
    "http://username:password@proxy2.example.com:8000",
]
URLS = [f"https://example.com/page/{i}" for i in range(20)]


async def main() -> None:
    timeout = httpx.Timeout(20.0, connect=5.0)
    clients = [httpx.AsyncClient(proxy=p, timeout=timeout) for p in PROXIES]
    pool = cycle(clients)
    limit = asyncio.Semaphore(5)

    async def fetch(url: str):
        client = next(pool)
        async with limit:
            try:
                response = await client.get(url)
                return url, response.status_code
            except httpx.TransportError as error:
                return url, type(error).__name__

    try:
        for url, result in await asyncio.gather(*(fetch(u) for u in URLS)):
            print(url, result)
    finally:
        for client in clients:
            await client.aclose()


asyncio.run(main())

In our test against stand-in proxies, both versions alternated between two proxies, and a dead proxy surfaced as ClientProxyConnectionError in aiohttp and ConnectError (a TransportError) in httpx. Add the same cooldown logic as the sync version if you run long jobs.

Rotate proxies with threads

For more throughput, run a thread pool. Give each thread its own Session, because Requests does not promise that one Session is thread-safe:

import threading
from concurrent.futures import ThreadPoolExecutor

local = threading.local()

def get_session() -> requests.Session:
    if not hasattr(local, "session"):
        local.session = requests.Session()
    return local.session

def fetch_one(url: str) -> tuple[str, int | str]:
    proxy = pick_proxy()
    try:
        r = get_session().get(url, proxies={"http": proxy, "https": proxy}, timeout=(5, 20))
        return url, r.status_code
    except requests.RequestException as e:
        cooldown_until[proxy] = time.monotonic() + COOLDOWN_SECONDS
        return url, type(e).__name__

urls = [f"https://example.com/page/{i}" for i in range(50)]
with ThreadPoolExecutor(max_workers=5) as pool:
    for url, result in pool.map(fetch_one, urls):
        print(url, result)

Keep max_workers well below your proxy count at first. Five workers on twenty proxies means each IP gets a fraction of the traffic. Five workers on one proxy is just a faster way to get rate limited.

Sticky sessions for multi-step flows

requests.Session keeps cookies and connection pooling. For a login or account flow, pin one proxy to one session for the whole task:

def run_account_task(proxy: str) -> None:
    with requests.Session() as session:
        proxies = {"http": proxy, "https": proxy}
        session.get("https://example.com/login", proxies=proxies, timeout=(5, 20))
        session.get("https://example.com/account", proxies=proxies, timeout=(5, 20))

Use one session per account, task, or profile. Do not rotate the proxy inside it unless you also reset cookies and state. For retail and ticketing flows, where login, queue, cart, and checkout must come from one stable address, ISP proxies give you a static dedicated IP to pin to each task. The sticky vs rotating proxies guide explains when that continuity matters.

Session mapping for Python requests with sticky and rotating proxy choices

Backconnect rotating proxies in Python

With a backconnect or residential gateway, your code uses one endpoint and the provider rotates the exit IP behind it:

gateway = "http://username:password@gateway.example:8000"
proxies = {"http": gateway, "https": gateway}

for url in urls:
    response = requests.get(url, proxies=proxies, timeout=(5, 20))
    print(url, response.status_code)

Whether this rotates on every request depends on the provider. Some gateways rotate per connection. Others hold an IP for a session ID, a username parameter, or a port. One catch with Requests: a Session reuses connections, so a per-connection gateway may give you the same IP for several requests in a row. Use plain requests.get() (a new connection each time) when you want a fresh IP on every call.

Our guide to backconnect rotating proxies explains how gateways assign IPs. The Unknown Proxies dashboard generates residential proxy lines as rotating or sticky. Each sticky line has its own session ID, so you can load a list of sticky lines into PROXIES and get one stable IP per line.

SOCKS5 proxies in a rotation

If your plan uses SOCKS5, install the extra and use the same rotation code with SOCKS URLs:

pip install "requests[socks]"

PROXIES = [
    "socks5h://username:password@host1:1080",
    "socks5h://username:password@host2:1080",
]

socks5h:// resolves DNS through the proxy. socks5:// resolves it locally. For the protocol tradeoff, read SOCKS5 vs HTTP proxy.

Adapter retries vs rotation retries

Requests can retry inside one connection pool with urllib3.util.Retry:

from requests import Session
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retry = Retry(
    total=3,
    backoff_factor=1,
    status_forcelist=[429, 502, 503, 504],
    allowed_methods={"GET", "HEAD"},
    respect_retry_after_header=True,
)

session = Session()
session.mount("http://", HTTPAdapter(max_retries=retry))
session.mount("https://", HTTPAdapter(max_retries=retry))

These retries reuse the same proxy, because the proxy is fixed for the call. That is fine for a brief 503 from the target. For proxy failures, retry at the application level on a different proxy, as in the fetch() function above.

Watch for a second retry layer when this code runs inside an agent framework. LangGraph's default node RetryPolicy retries every requests.HTTPError, 403 included, and never retries a requests.ConnectionError. CrewAI vs LangGraph for web agents shows the request counts and a retry_on that fixes it.

Rules that prevent retry storms:

Retry timeouts and 502, 503, and 504.
Back off on 429. Do not retry right away.
Treat repeated 403s as an access signal, not a retry target.
Never retry 407. It means the credentials are wrong for every proxy that shares them.
Do not retry POST, checkout, or account-changing actions blindly.
Cap retries per URL and add jitter.

Retry and backoff loop for Python scraping with rotating proxies

Preventing rate limits

More proxies do not fix a request rate the target does not allow. Before you scale:

Limit concurrent workers per proxy.
Add jitter around repeated tasks.
Respect Retry-After.
Cache pages that have not changed.
Track request rate per proxy, endpoint, and session.

Common mistakes

Rotating session.proxies while HTTPS_PROXY is set in the environment. The environment wins.
Setting only the http key, so HTTPS requests skip the proxy.
Retrying a failed request on the same proxy.
Rotating proxies while reusing the same cookies.
Retrying immediately after 429.
Loading SOCKS URLs without requests[socks].
Logging proxy passwords.

Change one variable at a time. If success improves only when rotation changes, session design was probably the issue. If it improves only when delay increases, the issue was pacing.

FAQ

How do I rotate proxies in Python requests?

Keep a list of proxy URLs, choose one per request with random.choice() or itertools.cycle(), and pass it as proxies={"http": proxy, "https": proxy}. Retry proxy errors on a different proxy.

How do I retry a failed request with a different proxy?

Catch requests.exceptions.ProxyError and timeouts, mark that proxy as unhealthy for a short time, pick another proxy, and send the request again. Adapter-level Retry cannot do this, because it keeps the same proxy.

Should I rotate proxies on every request?

Only when each request is independent. Do not rotate during login, account checks, carts, checkout, or multi-step browsing.

How do I check whether a proxy works in Python?

Send one request through it to a neutral IP endpoint with a short timeout, such as timeout=(3, 8). Treat a RequestException or a non-2xx response as failure. Run the check in a thread pool and keep only the proxies that pass.

How do I rotate proxies with asyncio?

With aiohttp, pass proxy= on each session.get() call and cycle through your list. With httpx, create one AsyncClient per proxy and cycle through the clients. Cap concurrency with asyncio.Semaphore.

Why does my rotating gateway return the same IP with requests.Session?

Session reuses TCP connections. A gateway that rotates per connection then sees one connection and keeps the same IP. Use separate requests.get() calls, or close the session between requests.

Why do all my rotating proxies get 429?

Your request rate or concurrency is too high for the target. Reduce per-proxy pressure, add jitter, and respect Retry-After. Adding proxies at the same rate usually just delays the block.

Final thoughts

Rotate independent requests, retry proxy failures on a different proxy, and cool down IPs that fail. Keep sticky sessions for accounts and multi-step flows. Add timeouts and conservative backoff before you add workers. For choosing a proxy type, the best proxy for web scraping guide compares residential, ISP, and datacenter options.

Technical references: Requests advanced usage: proxies, urllib3 Retry API, HTTPX proxies, and MDN HTTP 429 Too Many Requests.