Python Requests Proxies: Dict Format, Sessions, Env Vars

To use a proxy with Python requests, pass a proxies dict that maps the target URL scheme to a proxy URL, then set a timeout:

proxies = {
    "http": "http://user:pass@host:port",
    "https": "http://user:pass@host:port",
}
requests.get("https://example.com", proxies=proxies, timeout=15)

Three rules cover most setups:

• The key (http, https) is the scheme of the target URL. The value is the proxy URL.
• The proxy URL usually starts with http://, even for HTTPS targets. The proxy tunnels HTTPS with CONNECT.
• Credentials go inside the URL as user:pass@. URL-encode special characters in the password.

The important part is separating proxy connection problems from target-site responses. A 407 means proxy authentication failed. A timeout may mean the proxy, route, or target is slow. A 403 or 429 often means the target received your request and rejected it.

This page is the reference for the proxies dict: keys, format, sessions, environment variables, SOCKS5, httpx and aiohttp equivalents, and quick debugging. If you need the surrounding fetch, parse, pagination, and storage pipeline first, start with the Python scraping guide. If your main problem is rotating many proxies across a scraper, read the rotating proxy Python guide after the basic format works.

Python requests proxy quick setup

Install requests if your environment does not already have it:

pip install requests

Then pass a proxy dictionary into the request:

import requests

proxy_url = "http://username:password@proxy.example.com:8000"

proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

response = requests.get(
    "https://ipv4.unknownproxies.com/ip",
    proxies=proxies,
    timeout=15,
)

print(response.status_code)
print(response.text)

Use a neutral IP echo endpoint first. That tells you whether Python is reaching the proxy and whether the proxy is the exit path before you test a real target.

Do not skip timeout. Without a timeout, a bad proxy route can leave a worker hanging much longer than intended.

Check that traffic really uses the proxy

A 200 response alone does not prove the proxy was used. Compare your direct exit IP with the IP seen through the proxy. This script runs as written once PROXY_URL is set:

import os
import requests

IP_URL = "https://ipv4.unknownproxies.com/ip"
proxy_url = os.environ["PROXY_URL"]
proxies = {"http": proxy_url, "https": proxy_url}

try:
    direct_ip = requests.get(IP_URL, timeout=(5, 15)).text.strip()
    proxy_ip = requests.get(IP_URL, proxies=proxies, timeout=(5, 15)).text.strip()
except requests.exceptions.ProxyError as error:
    raise SystemExit(f"Could not use the proxy (bad host, port, or credentials): {error}")
except requests.exceptions.Timeout:
    raise SystemExit("Timed out: check the proxy host and port, or the route to it")

print("direct:", direct_ip)
print("proxy: ", proxy_ip)
print("proxy applied" if direct_ip != proxy_ip else "same IP: the proxy was not applied")

If both lines show the same address, the usual causes are a missing https key, an environment proxy that wins over your session settings, or a redirect that left the proxied scheme. Run the same test from the machine and container that will run the job, not only from your laptop.

Proxies dictionary keys

The Requests proxy docs and the select_proxy() function in requests/utils.py support these keys:

For each request, Requests picks the first match in this order: scheme://hostname, scheme, all://hostname, all. Host keys match the exact hostname, not subdomains. Proxy URLs must include the scheme.

If you only set "http", HTTPS requests go direct (or through an environment proxy). That is the most common reason a proxy "does not work" in Requests. It also applies to redirects: a request that starts on http:// and redirects to https:// leaves the proxy if you only set the http key.

HTTP proxy format in Requests

For a plain HTTP target, use the http key:

import requests

proxies = {
    "http": "http://username:password@proxy.example.com:8000",
}

response = requests.get(
    "http://example.com",
    proxies=proxies,
    timeout=15,
)

print(response.status_code)

The value is the proxy URL, not the target URL. The general format is:

http://username:password@host:port

If your proxy provider gives you a compact line like this:

host:port:username:password

convert it before passing it to requests:

http://username:password@host:port

The proxy converter is useful when you need to reformat larger proxy lists without editing every line by hand.

HTTPS proxy requests in Python

For HTTPS target pages, set the https key too:

import requests

proxy_url = "http://username:password@proxy.example.com:8000"

proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

response = requests.get(
    "https://example.com",
    proxies=proxies,
    timeout=15,
)

print(response.status_code)

The https key means "use this proxy for HTTPS destinations." It does not always mean the proxy URL itself should start with https://.

Many HTTP proxies support HTTPS websites by using the HTTP CONNECT method to create a tunnel to the destination host. In that common setup, the proxy URL still starts with http://, while the destination URL starts with https://.

Only use an https:// proxy URL when your proxy provider explicitly gives you an HTTPS proxy endpoint. Requests hands the URL to urllib3, which then opens TLS to the proxy itself and a second TLS session to the destination (the urllib3 docs call this a TLS-in-TLS tunnel, configurable with proxy_ssl_context). Test that route with the IP check above before you rely on it.

Proxy authentication in the URL

Most paid proxy services use username/password authentication in the proxy URL:

proxy_url = "http://customer-123:secret-pass@proxy.example.com:8000"

proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

If your username or password contains reserved URL characters such as @, :, /, ?, #, or spaces, encode those values before building the proxy URL:

from urllib.parse import quote

username = quote("customer@example.com", safe="")
password = quote("p@ss:word/with symbols", safe="")

proxy_url = f"http://{username}:{password}@proxy.example.com:8000"

That prevents Python from reading part of the password as the host, port, or URL separator.

If the proxy returns 407 Proxy Authentication Required, focus on credentials first. Our Python requests proxy authentication guide covers the full 407 checklist, why HTTPS targets raise ProxyError instead of a 407 response, and the limits of HTTPProxyAuth.

Keep sessions stable when state matters

Use requests.Session() when one workflow should keep headers, cookies, and connection pooling together:

import requests

session = requests.Session()

session.proxies.update({
    "http": "http://username:password@proxy.example.com:8000",
    "https": "http://username:password@proxy.example.com:8000",
})

session.headers.update({
    "User-Agent": "Mozilla/5.0 (compatible; ExampleBot/1.0)",
})

home = session.get("https://example.com", timeout=15)
account = session.get("https://example.com/account", timeout=15)

print(home.status_code, account.status_code)

Keep one session tied to one logical identity. If the workflow logs in, carries cookies, checks account state, or follows a multi-step path, do not rotate the proxy inside the same session unless you also reset cookies and state.

For stateless scraping, per-request rotation may be fine. For account or cart flows, stable proxy sessions are usually safer. The sticky vs rotating proxies guide explains that tradeoff.

For choosing a minimal User-Agent, Accept, locale, cookie, and cache-validator policy, use the web scraping headers guide. Avoid copying a complete browser request into requests, because browser-only metadata may contradict the actual client.

Proxy environment variables (HTTP_PROXY, HTTPS_PROXY, ALL_PROXY)

Requests reads the standard proxy variables when you do not pass a proxy for that scheme. Uppercase and lowercase names both work. If both exist, the lowercase one wins.

export HTTP_PROXY="http://username:password@proxy.example.com:8000"
export HTTPS_PROXY="http://username:password@proxy.example.com:8000"
export ALL_PROXY="socks5h://username:password@proxy.example.com:1080"
export NO_PROXY="localhost,127.0.0.1,.internal.example.com"

With those set, a plain requests.get(url) uses the proxy without any proxies argument. Precedence works like this:

So if a session proxy seems ignored, check for an environment proxy first. To turn environment proxies off for one session, see our guide to Session.trust_env and NO_PROXY in Requests.

Keep credentials out of code

Do not hard-code real proxy credentials in source files. Use your own environment variable so credentials stay out of Git history and shared snippets.

export PROXY_URL="http://username:password@proxy.example.com:8000"

Then read the value in Python:

import os
import requests

proxy_url = os.environ["PROXY_URL"]

proxies = {
    "http": proxy_url,
    "https": proxy_url,
}

response = requests.get(
    "https://ipv4.unknownproxies.com/ip",
    proxies=proxies,
    timeout=15,
)

print(response.text)

For production jobs, pass secrets through your host, CI system, container runtime, or secret manager instead of committing .env files.

SOCKS5 proxies with Requests

If your proxy endpoint is SOCKS5, install the SOCKS extra:

pip install "requests[socks]"

Then use a SOCKS URL:

import requests

proxies = {
    "http": "socks5://username:password@proxy.example.com:1080",
    "https": "socks5://username:password@proxy.example.com:1080",
}

response = requests.get(
    "https://example.com",
    proxies=proxies,
    timeout=15,
)

print(response.status_code)

The Requests docs describe SOCKS support as available since version 2.10.0 through the optional extra. The scheme decides where DNS happens:

Use socks5h:// when DNS resolution should happen through the proxy:

proxies = {
    "http": "socks5h://username:password@proxy.example.com:1080",
    "https": "socks5h://username:password@proxy.example.com:1080",
}

For normal web scraping with requests, HTTP(S) proxy endpoints are usually simpler. Use SOCKS5 when your provider, network route, or tool specifically requires it. The SOCKS5 vs HTTP proxy guide covers the protocol choice.

The same proxy in httpx and aiohttp

If you move off requests, the proxy URL stays the same. Only the argument name changes. The HTTPX proxy docs use a proxy argument on the client or a top-level call, and credentials go in the URL:

import os
import httpx

proxy_url = os.environ["PROXY_URL"]

with httpx.Client(proxy=proxy_url, timeout=15.0) as client:
    response = client.get("https://ipv4.unknownproxies.com/ip")
    print(response.text.strip())

For SOCKS in httpx, install httpx[socks]. For different proxies per scheme, httpx uses mounts with httpx.HTTPTransport(proxy=...). The older proxies= argument is gone from current httpx, so snippets that use it fail on a fresh install (checked against httpx 0.28.1).

aiohttp takes proxy= on each request or session call, and it also accepts credentials in the URL:

import asyncio
import os

import aiohttp


async def main() -> None:
    proxy_url = os.environ["PROXY_URL"]
    timeout = aiohttp.ClientTimeout(total=15)
    async with aiohttp.ClientSession(timeout=timeout) as session:
        async with session.get("https://ipv4.unknownproxies.com/ip", proxy=proxy_url) as response:
            print((await response.text()).strip())


asyncio.run(main())

Two differences matter when you compare clients. aiohttp ignores proxy environment variables unless you create the session with trust_env=True, while requests and httpx read them by default. And aiohttp 3.14 emits a DeprecationWarning for the older proxy_auth argument, so keep credentials in the URL. Our Requests no-proxy guide covers the environment behavior in more detail.

Add timeouts, status handling, and safe retries

Proxy code should fail clearly. Start with a timeout and explicit status handling:

import requests

proxy_url = "http://username:password@proxy.example.com:8000"
proxies = {"http": proxy_url, "https": proxy_url}

try:
    response = requests.get(
        "https://example.com",
        proxies=proxies,
        timeout=15,
    )
    print(response.status_code)
except requests.ProxyError as error:
    print("Proxy connection failed:", error)
except requests.Timeout:
    print("Request timed out")
except requests.RequestException as error:
    print("Request failed:", error)

For light retry behavior, mount an HTTPAdapter with urllib3.util.Retry:

from requests import Session
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retry = Retry(
    total=3,
    connect=2,
    read=2,
    status=2,
    backoff_factor=1,
    status_forcelist=[429, 502, 503, 504],
    allowed_methods={"GET", "HEAD"},
    respect_retry_after_header=True,
)

session = Session()
adapter = HTTPAdapter(max_retries=retry)
session.mount("http://", adapter)
session.mount("https://", adapter)

session.proxies.update({
    "http": "http://username:password@proxy.example.com:8000",
    "https": "http://username:password@proxy.example.com:8000",
})

response = session.get("https://example.com", timeout=15)
print(response.status_code)

Retries are not a substitute for pacing. If every worker gets HTTP 429 Too Many Requests, reduce concurrency, add jitter, respect Retry-After, and lower requests per proxy before expanding the pool.

Common Python requests proxy errors

If the exception does not make the failing layer clear, work through the general proxy error checklist before changing Python code, credentials, and the proxy pool at the same time.

Log the proxy endpoint or proxy ID, target URL, status code, elapsed time, retry count, and exception class. Without those fields, proxy debugging turns into guesswork.

Choosing residential or ISP proxies for Requests

The right proxy type depends on the request pattern:

Use ISP proxies for flows that must keep one IP from login to checkout, and residential proxies when you need broad rotation or location diversity. See residential plans for bandwidth pricing. Once a single proxy works, the rotating proxies with Python Requests guide shows how to spread traffic across many.

FAQ

How do I use a proxy with Python requests?

Create a proxies dictionary and pass it to requests.get(), requests.post(), or a requests.Session. Use {"http": proxy_url, "https": proxy_url} when you want both HTTP and HTTPS targets to use the proxy.

Should the HTTPS proxy value start with http or https?

Usually http://. The https dictionary key means the rule applies to HTTPS destinations. Many HTTP proxies tunnel HTTPS traffic with CONNECT, so the proxy URL itself still starts with http://.

What is the proxies dictionary format in Python requests?

Keys are target schemes ("http", "https", "all") or scheme plus host ("https://api.example.com"). Values are proxy URLs with a scheme, such as "http://user:pass@host:port" or "socks5h://host:1080".

How do I authenticate a proxy in Python requests?

Put the username and password in the proxy URL: http://username:password@host:port. URL-encode special characters in credentials before building the URL.

How do I set a proxy for every request in a session?

Call session.proxies.update({"http": proxy_url, "https": proxy_url}) once, then make all calls through that session. If a proxy from HTTP_PROXY or HTTPS_PROXY is set in the environment, it overrides session.proxies, so pass proxies= per call or set session.trust_env = False.

Why is requests showing my local IP instead of the proxy IP?

The proxy may not be applied to the URL scheme you are testing. Set both http and https keys, verify the proxy URL format, and test against a neutral IP endpoint before testing your target site.

Final thoughts

Get one proxied request working before you add anything else: the right proxy URL, both http and https keys, encoded credentials, a timeout, and an exit IP check. Then separate proxy failures (ProxyError, 407) from target responses (403, 429). Keep each logical identity on one session, and choose ISP or residential proxies based on whether the flow needs a stable IP or a rotating pool.

Technical references: Requests advanced usage, urllib3 advanced usage, HTTPX proxies, and MDN CONNECT.