Web scraping headers are the HTTP metadata your scraper sends with each request: User-Agent, Accept, Accept-Language, cookies, and, in browsers, client hints and Sec-Fetch-* fields. The rule that matters most is consistency. Send a small set that matches the client that really sends it, keep one profile per session, and capture a real browser request before you copy anything.
Short version:
- Know what your client sends by default. Python
requestsannounces itself aspython-requests/2.xwithAccept: */*. Many sites treat that differently from a browser. - Pick one profile and keep it whole. Either an honest bot profile (your own
User-Agent, simpleAccept) or a complete, current browser profile. Do not mix parts of both. - Match the details. A Chrome on Windows
User-Agentgoes withSec-CH-UA-Platform: "Windows". A FirefoxUser-Agentsends noSec-CH-UAat all. - Keep the order and casing the client would use. Browsers send headers in a fixed order, and HTTP/2 headers are lowercase.
- Headers are only one layer. TLS and HTTP/2 fingerprints, cookies, and request rate still identify the client.
Headers do not grant permission or guarantee access. Check the site's terms, robots.txt, and applicable rules before collecting data.
This guide shows real browser header sets, explains header order and client hints, gives Python examples, and shows how to isolate header problems from authentication, rate limits, page state, and proxy reputation. For the complete fetch, classify, parse, paginate, and store sequence, use the Python scraping workflow.
Web scraping headers: a practical baseline
Start with the fewest headers your target and parser require.
| Header | What it communicates | Practical rule |
|---|---|---|
User-Agent |
Client or application identity | Use a stable, truthful value that matches the client you operate |
Accept |
Response media types the client can parse | Ask for HTML when the job parses HTML |
Accept-Language |
Preferred response languages | Set it only when locale affects the data you need |
Authorization |
Credentials for an approved API or protected resource | Use only credentials issued for that resource; never log the value |
Cookie |
Server-managed session state | Let a session or browser cookie jar manage it |
Referer |
The page from which a navigation originated | Send it only when the workflow actually has that navigation context |
Origin |
Origin of certain cross-origin or state-changing requests | Let a browser set it; do not add it to ordinary GET requests without reason |
If-None-Match / If-Modified-Since |
Cached version already held by the client | Reuse server validators to avoid downloading unchanged pages |
Your HTTP client should normally calculate Host, Content-Length, connection behavior, and compression details. Manually forcing transport headers can create incorrect lengths, unsupported encodings, or connection reuse problems.
The MDN HTTP headers reference separates request, response, representation, authentication, and other field types. Use the definition for a specific header instead of assuming that every field visible in browser developer tools belongs in a scraper.

Realistic browser header sets: Chrome vs Firefox
This is what a desktop browser sends for a top-level page load (typing a URL) over HTTPS and HTTP/2, in the order the browser sends it. The values come from an older capture (Chrome 140 and Firefox 143), so current browsers report higher versions. The version numbers and the Sec-CH-UA brand string change with every release. Capture your own from DevTools (Network tab, select the document request, view request headers) before you copy anything.
| Header | Chrome on Windows (example capture) | Firefox on Windows (example capture) |
|---|---|---|
sec-ch-ua |
"Chromium";v="140", "Not=A?Brand";v="24", "Google Chrome";v="140" |
Not sent |
sec-ch-ua-mobile |
?0 |
Not sent |
sec-ch-ua-platform |
"Windows" |
Not sent |
upgrade-insecure-requests |
1 |
1 |
user-agent |
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 |
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:143.0) Gecko/20100101 Firefox/143.0 |
accept |
text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 |
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 |
sec-fetch-site |
none |
none |
sec-fetch-mode |
navigate |
navigate |
sec-fetch-user |
?1 |
?1 |
sec-fetch-dest |
document |
document |
accept-encoding |
gzip, deflate, br, zstd |
gzip, deflate, br, zstd |
accept-language |
en-US,en;q=0.9 |
en-US,en;q=0.5 |
priority |
u=0, i |
u=0, i |
A few things stand out:
- Chrome's
User-Agentis reduced. The OS part is frozen atWindows NT 10.0(even on Windows 11), and the minor version parts are always zero (140.0.0.0). A UA string with a full build number likeChrome/140.0.7339.80does not match current Chrome. - Firefox sends no
Sec-CH-UAheaders. A FirefoxUser-AgentplusSec-CH-UAheaders is a contradiction no real browser produces. Sec-Fetch-*values depend on context. A link click inside a site sendssec-fetch-site: same-originand usually areferer. Afetch()API call sendssec-fetch-mode: corsandsec-fetch-dest: empty, notnavigateanddocument.Accept-Encoding: br, zstdonly makes sense if your client can decode Brotli and Zstandard.
To see the exact User-Agent and client hints your own browser sends, open the user agent checker. It breaks the string into browser, version, OS, and device, and shows the Sec-CH-UA values next to it.
Current user agent formats (October 2026)
The captures above are from Chrome 140 and Firefox 143. As of 15 October 2026, the stable Chrome build on Windows is 155.0.8059.85 according to Chromium Dashboard, and Mozilla's product details feed lists Firefox 158.0 as the latest release. The formats did not change, only the numbers:
| Browser | Desktop user agent format (Windows) |
|---|---|
| Chrome 155 | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36 |
| Firefox 158 | Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:158.0) Gecko/20100101 Firefox/158.0 |
Chrome freezes the minor version at 0.0.0 and the Windows version at 10.0 under user-agent reduction. MDN documents that Firefox's rv: value matches its version and the desktop Gecko trail is always 20100101 (User-Agent reference). Browsers update every few weeks, so a string you hard-code today is stale by next quarter. Read the version from a browser you actually installed.
How to capture the headers your browser sends
Do this once per target instead of trusting any list, including this one:
- Open DevTools, go to the Network tab, and reload the page.
- Click the document request (the first row, type
document) and open the Headers panel. - Read the request headers. If your browser offers a raw view, use it to see the order the headers were sent in.
- Right-click the request, choose Copy, then "Copy as cURL" to get a command you can run and change one header at a time.
- Send the same request from your scraper to an echo endpoint such as
https://httpbin.org/headersand compare. Anything your client adds or drops shows up immediately.
Remove the cookie header from a copied cURL command before you reuse it. Cookies are session state, not part of the header profile.
What Python requests sends by default
A plain requests.get() sends four headers, from requests.utils.default_headers():
| Header | Default value |
|---|---|
User-Agent |
python-requests/2.x.y |
Accept-Encoding |
gzip, deflate (plus br or zstd if those decoders are installed) |
Accept |
*/* |
Connection |
keep-alive |
Check what was actually sent with response.request.headers:
import requests
r = requests.get("https://example.com", timeout=(5, 20))
print(r.request.headers)
The python-requests user agent is honest, and some sites block it on sight. Replacing it is reasonable. Replacing it with a Chrome string while keeping Accept: */* and no client hints creates a profile that matches no browser. Choose one of the two profiles below and set it completely.
Header order and consistency with the User-Agent
Browsers send headers in a fixed order for each request type. Chrome, for example, sends the sec-ch-ua group before user-agent, and accept-language near the end. Over HTTP/2, all header names are lowercase, and the pseudo-headers (:method, :authority, :scheme, :path) come first in an order that also differs between browsers.
Most HTTP clients do not reproduce this:
requestssends its default headers first, then yours. The Requests docs on header ordering say the default order "will be preferred" over the order you pass per request. To control the order, setsession.headersto your own ordered dict.requestsspeaks HTTP/1.1 only. A browser speaks HTTP/2 or HTTP/3 to most large sites.
Use this table to check a profile before you send it:
If the User-Agent says |
Then also |
|---|---|
| Chrome or Edge, desktop | sec-ch-ua-mobile: ?0, sec-ch-ua-platform matching the OS, brand list with the same major version |
| Chrome on Android | sec-ch-ua-mobile: ?1, sec-ch-ua-platform: "Android" |
| Firefox | No sec-ch-ua* headers, Firefox Accept value |
| Safari | No sec-ch-ua* headers |
| Any browser, HTTPS page load | Sec-Fetch-* values that match the request type |
| Any profile | Accept-Language that fits the proxy location you chose |
One profile per session. Do not rotate the User-Agent on every request while the cookies and IP stay the same. That pattern looks less like a person than a fixed profile does.
Client hints: Sec-CH-UA and Accept-CH
User-Agent client hints are Chromium headers that carry the browser identity in structured form. Chrome, Edge, and other Chromium browsers send three low-entropy hints by default on HTTPS requests: Sec-CH-UA, Sec-CH-UA-Mobile, and Sec-CH-UA-Platform.
High-entropy hints, such as Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform-Version, Sec-CH-UA-Arch, and Sec-CH-UA-Model, are sent only after the server asks for them with an Accept-CH response header. If a site sends Accept-CH and your scraper never adds those hints on later requests, the site can see that.
The brand list in Sec-CH-UA includes a deliberately odd "GREASE" entry, such as "Not=A?Brand";v="24". Its text changes between Chrome versions. Copy the whole value from the same browser version as your User-Agent, not from an old capture.
The same values are visible to JavaScript as navigator.userAgentData. If you change the User-Agent in a real browser (for example in Selenium) without changing the client hints, the two disagree. MDN documents each hint in its Sec-CH-UA reference.
Headers are not the whole fingerprint
A complete Chrome header set sent from Python requests still arrives over Python's TLS stack and HTTP/1.1. Sites can read the TLS handshake (often summarized as a JA3 or JA4 fingerprint) and HTTP/2 settings, and compare them with what the headers claim.
Your options, from simplest:
- Honest bot profile. Keep your own
User-Agent. Nothing contradicts anything. Works where the site allows declared bots. - HTTP client with browser impersonation. Libraries such as
curl_cffisend Chrome- or Safari-like TLS and HTTP/2 settings with matching headers, for examplecurl_cffi.get(url, impersonate="chrome"). - A real browser. Playwright, Puppeteer, or Selenium produce every header, hint, and handshake themselves. The cost is CPU, memory, and bandwidth.
Two short examples. httpx speaks HTTP/2 once you install the extra (pip install httpx[http2]) and set http2=True on the client, per the HTTPX HTTP/2 docs:
import httpx
with httpx.Client(http2=True) as client:
r = client.get("https://example.com", timeout=20)
print(r.http_version, r.status_code)
curl_cffi copies a browser's TLS and HTTP/2 settings. The chrome target is an alias for the latest Chrome profile the library ships, listed in the curl_cffi impersonation docs:
from curl_cffi import requests
r = requests.get("https://example.com", impersonate="chrome", timeout=20)
print(r.status_code)
Echo the result through https://httpbin.org/headers once to confirm what your client actually sends. None of these options overrides a site's access rules, rate limits, or IP reputation checks.
What each important header does
Headers are inputs to content negotiation, caching, authentication, and session handling. Treat them as protocol controls, not a bag of anti-block tricks.
User-Agent
User-Agent identifies the client software making the request. A transparent collector can use an application-specific value with a version and contact page:
ExampleResearchBot/1.2 (+https://example.org/crawler-info)
That format gives a site owner a way to understand and contact the operator. If you are using a browser for authorized QA or rendering, keep the browser's own user agent rather than overriding it with a value from a different browser or operating system.
Changing only the user agent does not turn an HTTP library into a browser. JavaScript behavior, supported content encodings, cookies, TLS, and browser-generated metadata still differ. MDN documents the syntax and common directives in its User-Agent reference.
If your collector uses Scrapy, the Scrapy user agent guide shows how to set the project default, add a spider or request override, and verify which value the downloader sent. For a real browser workflow, the Selenium user agent guide covers Chrome launch options, browser-side verification, and profile consistency.
Accept
Accept tells the server which media types the client can process. For an HTML collector, a simple value is usually enough:
Accept: text/html,application/xhtml+xml
For an approved JSON API, prefer application/json. Do not request images, signed exchanges, or other formats that your client cannot parse. Always verify the response Content-Type before sending the body to an HTML or JSON parser. If image binaries are the intended result, the guide to scraping images from website pages separates HTML discovery from validated asset downloads.
Accept-Language
Accept-Language can change translated text, currency presentation, and regional content:
Accept-Language: en-US,en;q=0.8
Set it when language is part of the dataset, then record the chosen locale with each observation. A language header does not guarantee geographic content. Sites may also use the URL, account settings, cookies, or network location.
Cookies
Cookies often carry consent, locale, authentication, and experiment state. Use a cookie jar instead of copying a Cookie header from developer tools. The server may update cookies on any response, and a static copied value quickly becomes stale.
Some sites set consent or locale cookies on the first page and expect them on the next. If a product page returns a consent wall when you request it cold, load the homepage or category page first in the same session and let the jar collect what the server sets. Do not copy tracking cookies from your own browser.
Keep one cookie jar with one logical session. If you rotate the proxy or user identity between requests but retain the same cookies, the server sees contradictory session signals. If a workflow does not need state, avoiding unnecessary cookies makes it easier to reproduce.
Referer and Origin
Referer describes a previous page in a real navigation. Origin is used by browsers in CORS and some state-changing request contexts. Neither is a universal requirement for an ordinary document fetch.
Do not invent a referring page or add Origin everywhere. If a permitted workflow genuinely follows page A to page B, a browser or session-aware client can preserve that context. If an endpoint requires CSRF state, use the supported authentication flow rather than fabricating only its headers.
Authorization and custom X- headers
Some sites load data through a JSON endpoint that the page calls with fetch(). Those calls often carry Authorization, x-csrf-token, x-api-key, or similar custom fields. Find them in the Network tab by filtering to Fetch/XHR, then check where the value comes from. A CSRF token usually sits in the HTML or a cookie and changes per session, so read it at runtime instead of hard-coding it. Only call endpoints you are permitted to use, and never log the token.
Conditional request headers
Caching headers reduce traffic and make recurring monitoring more efficient. Store an earlier response's ETag or Last-Modified, then send the corresponding validator:
If-None-Match: "saved-etag-value"
If-Modified-Since: Tue, 30 Jun 2026 08:00:00 GMT
A 304 Not Modified response means the cached representation is still current. It saves bandwidth and parsing work. Follow the site's cache policy, because not every resource supplies validators and an ETag may vary by representation.
Python requests example with stable headers
Use requests.Session() to keep headers, cookies, and connection pooling together:
import requests
session = requests.Session()
session.headers.update({
"User-Agent": "ExampleResearchBot/1.2 (+https://example.org/crawler-info)",
"Accept": "text/html,application/xhtml+xml",
"Accept-Language": "en-US,en;q=0.8",
})
response = session.get(
"https://example.com/public-page",
timeout=(5, 20),
)
response.raise_for_status()
content_type = response.headers.get("Content-Type", "")
if "text/html" not in content_type:
raise ValueError(f"Expected HTML, received {content_type!r}")
html = response.text
This honest-bot baseline deliberately omits Host, Connection, Content-Length, Accept-Encoding, Cookie, and browser-only metadata. requests and its underlying libraries manage the transport details; the session manages cookies received from the server.
If you need a browser profile in requests instead, set the full set in order on the session, and keep it in sync with one real browser version:
from collections import OrderedDict
session.headers = OrderedDict([
("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:143.0) Gecko/20100101 Firefox/143.0"),
("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"),
("Accept-Language", "en-US,en;q=0.5"),
("Accept-Encoding", "gzip, deflate"),
("Upgrade-Insecure-Requests", "1"),
("Sec-Fetch-Dest", "document"),
("Sec-Fetch-Mode", "navigate"),
("Sec-Fetch-Site", "none"),
("Sec-Fetch-User", "?1"),
])
This example uses a Firefox profile because Firefox sends no client hints, so there are fewer values to keep in sync. Accept-Encoding lists only what requests can decode without extra packages. The TLS fingerprint is still Python's, as explained above.
For proxy routing, add a proxy configuration without changing the target headers:
proxy_url = "http://username:password@proxy.example.com:8000"
session.proxies.update({
"http": proxy_url,
"https": proxy_url,
})
Keep proxy credentials in environment variables or a secret manager in real projects. The Python requests proxies guide covers URL formatting, authentication, timeouts, and route verification.
Browser headers differ from HTTP client headers
Chrome, Firefox, and other browsers generate request metadata from the actual navigation, browser version, policy, and page context. That includes fields such as Sec-Fetch-Site, Sec-Fetch-Mode, and Sec-Fetch-Dest.
These Fetch Metadata request headers help servers understand where a request came from and how it will be used. In browser automation, let the browser produce them. In a normal HTTP client, omitting browser-only fields is more coherent than hard-coding a set captured from an unrelated navigation.
The same rule applies to compression and client hints. Do not claim an encoding your client cannot decode, and do not pair a mobile user agent with desktop client hints.
If the required public data exists only after permitted browser rendering, use an actual browser. The Playwright proxy setup guide explains how to keep browser contexts and network sessions isolated.
Keep headers, cookies, and proxy identity together
A stable request profile has more than one component:
- One client implementation and version.
- One deliberate header policy.
- One cookie jar for each logical session.
- One locale for data that depends on language or region.
- One proxy route for each stateful session.
- One bounded request schedule with caching and backoff.
For independent public pages, each task can start a clean session. For a multi-page workflow, keep the same cookies and route until that workflow ends. Rotating the IP midway through a login, cart, or consent flow can break server-side state even if every header stays the same.
If you need many independent routes, residential proxies support location targeting and rotation. For a stateful job such as a retail checkout or a ticketing queue session, keep one route for the whole flow: a sticky residential session, or a static address from the ISP plans. Proxies address network route and IP concentration; they do not repair invalid headers, missing authorization, or a disallowed use case.
Diagnose a suspected header block
Do not respond to every 403 by adding more headers. Change one variable at a time:
- Save the status, final URL, redirect history,
Content-Type, and a small redacted body sample. - Confirm that the requested resource and collection method are permitted.
- Test the same URL at low frequency without a proxy.
- Compare the HTTP library's actual outgoing headers (
response.request.headers) with what a browser sends. Export a HAR file from DevTools for the same page and open it in the HAR analyzer to see the browser's headers, redirects, and blocked requests side by side. - Start a new session without copied cookies.
- Add only the one header the application demonstrably requires.
- If direct and proxied results differ, keep headers constant and test the network route separately.
- Stop or cool down on access-denied and rate-limit responses.
Never log authorization values, complete cookies, proxy passwords, or personal data. Redact them at collection time rather than relying on a later cleanup.

Common header mistakes
Copying a complete browser request blindly
A copied request includes temporary cookies, navigation-specific metadata, experiment values, and browser capabilities. It may work once and then fail unpredictably. Use a capture as a reference for header names, values, and order, but drop the cookies and session tokens and let your client manage them.
Randomizing headers on every request
Random combinations can contradict one another and make failures impossible to reproduce. Keep a versioned header policy per client. Change it deliberately, test it on a small URL set, and record the result.
Treating 403 and 429 as the same problem
An HTTP 403 Forbidden response is an access denial. An HTTP 429 Too Many Requests response is a rate-limit signal. A correct header set will not fix excessive request volume; honor Retry-After, lower concurrency, cache responses, and add backoff.
Cloudflare Error 1020 is another distinct case: a site owner's firewall rule denied the request at Cloudflare's edge. Use the Cloudflare Error 1020 guide to separate that rule match from a generic header bug.
Ignoring response headers
Response metadata is part of the protocol. Inspect Content-Type, Retry-After, Location, Set-Cookie, cache validators, and rate-limit fields where provided. A 200 response containing a consent page or login page is not successful data extraction.
Production checklist
Before scaling a collector, verify:
- The source and collection method are allowed for your use case.
- The user agent is stable and appropriate for the actual client.
Acceptmatches formats the parser supports.- Locale is explicit when it changes the data.
- Cookies live in a per-session jar and secrets are redacted from logs.
- Conditional requests and caching avoid repeat downloads.
- Stateful sessions keep one network route.
- Timeouts, bounded retries, jitter, and
Retry-Afterhandling are enabled. - Metrics separate status codes, page classifications, parser failures, and proxy failures.
- A small canary run passes before concurrency increases.
For recurring jobs, track the header-policy version with each run. When a target changes, you can compare failure rates before and after a controlled update instead of guessing which random field helped. The broader guide to crawling a website without getting blocked shows how to place that stable request profile inside a bounded queue, per-host scheduler, cache, and response-classification loop.
FAQ
What headers should I use for web scraping?
Use the minimum set the permitted resource needs: a stable User-Agent, an Accept value that matches your parser, and Accept-Language when locale changes the data. Let your HTTP library set transport headers and let a session manage cookies.
How do I find the headers my browser sends?
Open DevTools, go to the Network tab, reload the page, and click the document request. The Headers panel lists every request header. Right-click the request and choose Copy as cURL to replay it, then remove the cookie header before you reuse it.
Do web scraping headers prevent 403 errors?
They can fix a request that was denied because required metadata, authentication, or session context was missing. They cannot override permissions, site policy, firewall rules, IP reputation, or excessive traffic. Find the source of the 403 before you change the request.
Should I rotate the User-Agent header?
Not by default. A stable client identity is easier to operate and debug. For authorized cross-browser testing, use real browser versions and treat each browser context as one coherent profile instead of rotating a single header.
Should I copy Chrome headers into Python requests?
Not blindly. Chrome headers describe client hints, compression, and navigation context that requests does not reproduce, and the TLS handshake still looks like Python. Use an honest minimal profile, a complete and current Firefox-style profile, an impersonating client, or a real browser.
Does header order matter for web scraping?
It can. Browsers send headers in a fixed order, and over HTTP/2 the names are lowercase. Some bot detection compares that order with the claimed browser. In requests, set session.headers to an ordered dict to control it.
Are headers or proxies more important for scraping?
They solve different layers. Headers describe the request and session context. Proxies provide the network route. Make the request correct and compliant at a conservative rate first, then test proxies when IP location, reputation, or concentration is the measured constraint.
Final thoughts
Good web scraping headers are minimal, accurate, and consistent with the client, cookies, locale, and network session that send them. Capture a real request from the browser version you are matching, drop the cookies, and compare it with your scraper's output through an echo endpoint. Fix the request when metadata is wrong, slow down when rate is the problem, and look at proxy plans only when the network route is the verified constraint.
