We use cookies to enhance user experience, personalize content, and analyze traffic. Cookie Policy

← Back to all articles

Web Scraping Headers: Realistic Header Sets and Order

Web scraping headers that stay consistent: real Chrome and Firefox header sets, header order, Sec-CH-UA client hints, Python requests defaults, and 403 checks.

by Unknown Proxies

19 min read

July 2, 2026

Updated October 17, 2026

Web Scraping Headers: Realistic Header Sets and Order

Web scraping headers are the HTTP metadata your scraper sends with each request: User-Agent, Accept, Accept-Language, cookies, and, in browsers, client hints and Sec-Fetch-* fields. The rule that matters most is consistency. Send a small set that matches the client that really sends it, keep one profile per session, and capture a real browser request before you copy anything.

Short version:

Headers do not grant permission or guarantee access. Check the site's terms, robots.txt, and applicable rules before collecting data.

This guide shows real browser header sets, explains header order and client hints, gives Python examples, and shows how to isolate header problems from authentication, rate limits, page state, and proxy reputation. For the complete fetch, classify, parse, paginate, and store sequence, use the Python scraping workflow.

Web scraping headers: a practical baseline

Start with the fewest headers your target and parser require.

Header What it communicates Practical rule
User-Agent Client or application identity Use a stable, truthful value that matches the client you operate
Accept Response media types the client can parse Ask for HTML when the job parses HTML
Accept-Language Preferred response languages Set it only when locale affects the data you need
Authorization Credentials for an approved API or protected resource Use only credentials issued for that resource; never log the value
Cookie Server-managed session state Let a session or browser cookie jar manage it
Referer The page from which a navigation originated Send it only when the workflow actually has that navigation context
Origin Origin of certain cross-origin or state-changing requests Let a browser set it; do not add it to ordinary GET requests without reason
If-None-Match / If-Modified-Since Cached version already held by the client Reuse server validators to avoid downloading unchanged pages

Your HTTP client should normally calculate Host, Content-Length, connection behavior, and compression details. Manually forcing transport headers can create incorrect lengths, unsupported encodings, or connection reuse problems.

The MDN HTTP headers reference separates request, response, representation, authentication, and other field types. Use the definition for a specific header instead of assuming that every field visible in browser developer tools belongs in a scraper.

Consistent scraping request profile connecting headers, cookies, locale, client capabilities, and one network session

Realistic browser header sets: Chrome vs Firefox

This is what a desktop browser sends for a top-level page load (typing a URL) over HTTPS and HTTP/2, in the order the browser sends it. The values come from an older capture (Chrome 140 and Firefox 143), so current browsers report higher versions. The version numbers and the Sec-CH-UA brand string change with every release. Capture your own from DevTools (Network tab, select the document request, view request headers) before you copy anything.

Header Chrome on Windows (example capture) Firefox on Windows (example capture)
sec-ch-ua "Chromium";v="140", "Not=A?Brand";v="24", "Google Chrome";v="140" Not sent
sec-ch-ua-mobile ?0 Not sent
sec-ch-ua-platform "Windows" Not sent
upgrade-insecure-requests 1 1
user-agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:143.0) Gecko/20100101 Firefox/143.0
accept text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
sec-fetch-site none none
sec-fetch-mode navigate navigate
sec-fetch-user ?1 ?1
sec-fetch-dest document document
accept-encoding gzip, deflate, br, zstd gzip, deflate, br, zstd
accept-language en-US,en;q=0.9 en-US,en;q=0.5
priority u=0, i u=0, i

A few things stand out:

To see the exact User-Agent and client hints your own browser sends, open the user agent checker. It breaks the string into browser, version, OS, and device, and shows the Sec-CH-UA values next to it.

Current user agent formats (October 2026)

The captures above are from Chrome 140 and Firefox 143. As of 15 October 2026, the stable Chrome build on Windows is 155.0.8059.85 according to Chromium Dashboard, and Mozilla's product details feed lists Firefox 158.0 as the latest release. The formats did not change, only the numbers:

Browser Desktop user agent format (Windows)
Chrome 155 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36
Firefox 158 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:158.0) Gecko/20100101 Firefox/158.0

Chrome freezes the minor version at 0.0.0 and the Windows version at 10.0 under user-agent reduction. MDN documents that Firefox's rv: value matches its version and the desktop Gecko trail is always 20100101 (User-Agent reference). Browsers update every few weeks, so a string you hard-code today is stale by next quarter. Read the version from a browser you actually installed.

How to capture the headers your browser sends

Do this once per target instead of trusting any list, including this one:

  1. Open DevTools, go to the Network tab, and reload the page.
  2. Click the document request (the first row, type document) and open the Headers panel.
  3. Read the request headers. If your browser offers a raw view, use it to see the order the headers were sent in.
  4. Right-click the request, choose Copy, then "Copy as cURL" to get a command you can run and change one header at a time.
  5. Send the same request from your scraper to an echo endpoint such as https://httpbin.org/headers and compare. Anything your client adds or drops shows up immediately.

Remove the cookie header from a copied cURL command before you reuse it. Cookies are session state, not part of the header profile.

What Python requests sends by default

A plain requests.get() sends four headers, from requests.utils.default_headers():

Header Default value
User-Agent python-requests/2.x.y
Accept-Encoding gzip, deflate (plus br or zstd if those decoders are installed)
Accept */*
Connection keep-alive

Check what was actually sent with response.request.headers:

import requests

r = requests.get("https://example.com", timeout=(5, 20))
print(r.request.headers)

The python-requests user agent is honest, and some sites block it on sight. Replacing it is reasonable. Replacing it with a Chrome string while keeping Accept: */* and no client hints creates a profile that matches no browser. Choose one of the two profiles below and set it completely.

Header order and consistency with the User-Agent

Browsers send headers in a fixed order for each request type. Chrome, for example, sends the sec-ch-ua group before user-agent, and accept-language near the end. Over HTTP/2, all header names are lowercase, and the pseudo-headers (:method, :authority, :scheme, :path) come first in an order that also differs between browsers.

Most HTTP clients do not reproduce this:

Use this table to check a profile before you send it:

If the User-Agent says Then also
Chrome or Edge, desktop sec-ch-ua-mobile: ?0, sec-ch-ua-platform matching the OS, brand list with the same major version
Chrome on Android sec-ch-ua-mobile: ?1, sec-ch-ua-platform: "Android"
Firefox No sec-ch-ua* headers, Firefox Accept value
Safari No sec-ch-ua* headers
Any browser, HTTPS page load Sec-Fetch-* values that match the request type
Any profile Accept-Language that fits the proxy location you chose

One profile per session. Do not rotate the User-Agent on every request while the cookies and IP stay the same. That pattern looks less like a person than a fixed profile does.

Client hints: Sec-CH-UA and Accept-CH

User-Agent client hints are Chromium headers that carry the browser identity in structured form. Chrome, Edge, and other Chromium browsers send three low-entropy hints by default on HTTPS requests: Sec-CH-UA, Sec-CH-UA-Mobile, and Sec-CH-UA-Platform.

High-entropy hints, such as Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform-Version, Sec-CH-UA-Arch, and Sec-CH-UA-Model, are sent only after the server asks for them with an Accept-CH response header. If a site sends Accept-CH and your scraper never adds those hints on later requests, the site can see that.

The brand list in Sec-CH-UA includes a deliberately odd "GREASE" entry, such as "Not=A?Brand";v="24". Its text changes between Chrome versions. Copy the whole value from the same browser version as your User-Agent, not from an old capture.

The same values are visible to JavaScript as navigator.userAgentData. If you change the User-Agent in a real browser (for example in Selenium) without changing the client hints, the two disagree. MDN documents each hint in its Sec-CH-UA reference.

Headers are not the whole fingerprint

A complete Chrome header set sent from Python requests still arrives over Python's TLS stack and HTTP/1.1. Sites can read the TLS handshake (often summarized as a JA3 or JA4 fingerprint) and HTTP/2 settings, and compare them with what the headers claim.

Your options, from simplest:

  1. Honest bot profile. Keep your own User-Agent. Nothing contradicts anything. Works where the site allows declared bots.
  2. HTTP client with browser impersonation. Libraries such as curl_cffi send Chrome- or Safari-like TLS and HTTP/2 settings with matching headers, for example curl_cffi.get(url, impersonate="chrome").
  3. A real browser. Playwright, Puppeteer, or Selenium produce every header, hint, and handshake themselves. The cost is CPU, memory, and bandwidth.

Two short examples. httpx speaks HTTP/2 once you install the extra (pip install httpx[http2]) and set http2=True on the client, per the HTTPX HTTP/2 docs:

import httpx

with httpx.Client(http2=True) as client:
    r = client.get("https://example.com", timeout=20)
    print(r.http_version, r.status_code)

curl_cffi copies a browser's TLS and HTTP/2 settings. The chrome target is an alias for the latest Chrome profile the library ships, listed in the curl_cffi impersonation docs:

from curl_cffi import requests

r = requests.get("https://example.com", impersonate="chrome", timeout=20)
print(r.status_code)

Echo the result through https://httpbin.org/headers once to confirm what your client actually sends. None of these options overrides a site's access rules, rate limits, or IP reputation checks.

What each important header does

Headers are inputs to content negotiation, caching, authentication, and session handling. Treat them as protocol controls, not a bag of anti-block tricks.

User-Agent

User-Agent identifies the client software making the request. A transparent collector can use an application-specific value with a version and contact page:

ExampleResearchBot/1.2 (+https://example.org/crawler-info)

That format gives a site owner a way to understand and contact the operator. If you are using a browser for authorized QA or rendering, keep the browser's own user agent rather than overriding it with a value from a different browser or operating system.

Changing only the user agent does not turn an HTTP library into a browser. JavaScript behavior, supported content encodings, cookies, TLS, and browser-generated metadata still differ. MDN documents the syntax and common directives in its User-Agent reference.

If your collector uses Scrapy, the Scrapy user agent guide shows how to set the project default, add a spider or request override, and verify which value the downloader sent. For a real browser workflow, the Selenium user agent guide covers Chrome launch options, browser-side verification, and profile consistency.

Accept

Accept tells the server which media types the client can process. For an HTML collector, a simple value is usually enough:

Accept: text/html,application/xhtml+xml

For an approved JSON API, prefer application/json. Do not request images, signed exchanges, or other formats that your client cannot parse. Always verify the response Content-Type before sending the body to an HTML or JSON parser. If image binaries are the intended result, the guide to scraping images from website pages separates HTML discovery from validated asset downloads.

Accept-Language

Accept-Language can change translated text, currency presentation, and regional content:

Accept-Language: en-US,en;q=0.8

Set it when language is part of the dataset, then record the chosen locale with each observation. A language header does not guarantee geographic content. Sites may also use the URL, account settings, cookies, or network location.

Cookies

Cookies often carry consent, locale, authentication, and experiment state. Use a cookie jar instead of copying a Cookie header from developer tools. The server may update cookies on any response, and a static copied value quickly becomes stale.

Some sites set consent or locale cookies on the first page and expect them on the next. If a product page returns a consent wall when you request it cold, load the homepage or category page first in the same session and let the jar collect what the server sets. Do not copy tracking cookies from your own browser.

Keep one cookie jar with one logical session. If you rotate the proxy or user identity between requests but retain the same cookies, the server sees contradictory session signals. If a workflow does not need state, avoiding unnecessary cookies makes it easier to reproduce.

Referer and Origin

Referer describes a previous page in a real navigation. Origin is used by browsers in CORS and some state-changing request contexts. Neither is a universal requirement for an ordinary document fetch.

Do not invent a referring page or add Origin everywhere. If a permitted workflow genuinely follows page A to page B, a browser or session-aware client can preserve that context. If an endpoint requires CSRF state, use the supported authentication flow rather than fabricating only its headers.

Authorization and custom X- headers

Some sites load data through a JSON endpoint that the page calls with fetch(). Those calls often carry Authorization, x-csrf-token, x-api-key, or similar custom fields. Find them in the Network tab by filtering to Fetch/XHR, then check where the value comes from. A CSRF token usually sits in the HTML or a cookie and changes per session, so read it at runtime instead of hard-coding it. Only call endpoints you are permitted to use, and never log the token.

Conditional request headers

Caching headers reduce traffic and make recurring monitoring more efficient. Store an earlier response's ETag or Last-Modified, then send the corresponding validator:

If-None-Match: "saved-etag-value"
If-Modified-Since: Tue, 30 Jun 2026 08:00:00 GMT

A 304 Not Modified response means the cached representation is still current. It saves bandwidth and parsing work. Follow the site's cache policy, because not every resource supplies validators and an ETag may vary by representation.

Python requests example with stable headers

Use requests.Session() to keep headers, cookies, and connection pooling together:

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "ExampleResearchBot/1.2 (+https://example.org/crawler-info)",
    "Accept": "text/html,application/xhtml+xml",
    "Accept-Language": "en-US,en;q=0.8",
})

response = session.get(
    "https://example.com/public-page",
    timeout=(5, 20),
)
response.raise_for_status()

content_type = response.headers.get("Content-Type", "")
if "text/html" not in content_type:
    raise ValueError(f"Expected HTML, received {content_type!r}")

html = response.text

This honest-bot baseline deliberately omits Host, Connection, Content-Length, Accept-Encoding, Cookie, and browser-only metadata. requests and its underlying libraries manage the transport details; the session manages cookies received from the server.

If you need a browser profile in requests instead, set the full set in order on the session, and keep it in sync with one real browser version:

from collections import OrderedDict

session.headers = OrderedDict([
    ("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:143.0) Gecko/20100101 Firefox/143.0"),
    ("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"),
    ("Accept-Language", "en-US,en;q=0.5"),
    ("Accept-Encoding", "gzip, deflate"),
    ("Upgrade-Insecure-Requests", "1"),
    ("Sec-Fetch-Dest", "document"),
    ("Sec-Fetch-Mode", "navigate"),
    ("Sec-Fetch-Site", "none"),
    ("Sec-Fetch-User", "?1"),
])

This example uses a Firefox profile because Firefox sends no client hints, so there are fewer values to keep in sync. Accept-Encoding lists only what requests can decode without extra packages. The TLS fingerprint is still Python's, as explained above.

For proxy routing, add a proxy configuration without changing the target headers:

proxy_url = "http://username:password@proxy.example.com:8000"
session.proxies.update({
    "http": proxy_url,
    "https": proxy_url,
})

Keep proxy credentials in environment variables or a secret manager in real projects. The Python requests proxies guide covers URL formatting, authentication, timeouts, and route verification.

Browser headers differ from HTTP client headers

Chrome, Firefox, and other browsers generate request metadata from the actual navigation, browser version, policy, and page context. That includes fields such as Sec-Fetch-Site, Sec-Fetch-Mode, and Sec-Fetch-Dest.

These Fetch Metadata request headers help servers understand where a request came from and how it will be used. In browser automation, let the browser produce them. In a normal HTTP client, omitting browser-only fields is more coherent than hard-coding a set captured from an unrelated navigation.

The same rule applies to compression and client hints. Do not claim an encoding your client cannot decode, and do not pair a mobile user agent with desktop client hints.

If the required public data exists only after permitted browser rendering, use an actual browser. The Playwright proxy setup guide explains how to keep browser contexts and network sessions isolated.

Keep headers, cookies, and proxy identity together

A stable request profile has more than one component:

  1. One client implementation and version.
  2. One deliberate header policy.
  3. One cookie jar for each logical session.
  4. One locale for data that depends on language or region.
  5. One proxy route for each stateful session.
  6. One bounded request schedule with caching and backoff.

For independent public pages, each task can start a clean session. For a multi-page workflow, keep the same cookies and route until that workflow ends. Rotating the IP midway through a login, cart, or consent flow can break server-side state even if every header stays the same.

If you need many independent routes, residential proxies support location targeting and rotation. For a stateful job such as a retail checkout or a ticketing queue session, keep one route for the whole flow: a sticky residential session, or a static address from the ISP plans. Proxies address network route and IP concentration; they do not repair invalid headers, missing authorization, or a disallowed use case.

Diagnose a suspected header block

Do not respond to every 403 by adding more headers. Change one variable at a time:

  1. Save the status, final URL, redirect history, Content-Type, and a small redacted body sample.
  2. Confirm that the requested resource and collection method are permitted.
  3. Test the same URL at low frequency without a proxy.
  4. Compare the HTTP library's actual outgoing headers (response.request.headers) with what a browser sends. Export a HAR file from DevTools for the same page and open it in the HAR analyzer to see the browser's headers, redirects, and blocked requests side by side.
  5. Start a new session without copied cookies.
  6. Add only the one header the application demonstrably requires.
  7. If direct and proxied results differ, keep headers constant and test the network route separately.
  8. Stop or cool down on access-denied and rate-limit responses.

Never log authorization values, complete cookies, proxy passwords, or personal data. Redact them at collection time rather than relying on a later cleanup.

Troubleshooting tree separating header mismatches from session, rate-limit, authorization, and proxy-route problems

Common header mistakes

Copying a complete browser request blindly

A copied request includes temporary cookies, navigation-specific metadata, experiment values, and browser capabilities. It may work once and then fail unpredictably. Use a capture as a reference for header names, values, and order, but drop the cookies and session tokens and let your client manage them.

Randomizing headers on every request

Random combinations can contradict one another and make failures impossible to reproduce. Keep a versioned header policy per client. Change it deliberately, test it on a small URL set, and record the result.

Treating 403 and 429 as the same problem

An HTTP 403 Forbidden response is an access denial. An HTTP 429 Too Many Requests response is a rate-limit signal. A correct header set will not fix excessive request volume; honor Retry-After, lower concurrency, cache responses, and add backoff.

Cloudflare Error 1020 is another distinct case: a site owner's firewall rule denied the request at Cloudflare's edge. Use the Cloudflare Error 1020 guide to separate that rule match from a generic header bug.

Ignoring response headers

Response metadata is part of the protocol. Inspect Content-Type, Retry-After, Location, Set-Cookie, cache validators, and rate-limit fields where provided. A 200 response containing a consent page or login page is not successful data extraction.

Production checklist

Before scaling a collector, verify:

For recurring jobs, track the header-policy version with each run. When a target changes, you can compare failure rates before and after a controlled update instead of guessing which random field helped. The broader guide to crawling a website without getting blocked shows how to place that stable request profile inside a bounded queue, per-host scheduler, cache, and response-classification loop.

FAQ

What headers should I use for web scraping?

Use the minimum set the permitted resource needs: a stable User-Agent, an Accept value that matches your parser, and Accept-Language when locale changes the data. Let your HTTP library set transport headers and let a session manage cookies.

How do I find the headers my browser sends?

Open DevTools, go to the Network tab, reload the page, and click the document request. The Headers panel lists every request header. Right-click the request and choose Copy as cURL to replay it, then remove the cookie header before you reuse it.

Do web scraping headers prevent 403 errors?

They can fix a request that was denied because required metadata, authentication, or session context was missing. They cannot override permissions, site policy, firewall rules, IP reputation, or excessive traffic. Find the source of the 403 before you change the request.

Should I rotate the User-Agent header?

Not by default. A stable client identity is easier to operate and debug. For authorized cross-browser testing, use real browser versions and treat each browser context as one coherent profile instead of rotating a single header.

Should I copy Chrome headers into Python requests?

Not blindly. Chrome headers describe client hints, compression, and navigation context that requests does not reproduce, and the TLS handshake still looks like Python. Use an honest minimal profile, a complete and current Firefox-style profile, an impersonating client, or a real browser.

Does header order matter for web scraping?

It can. Browsers send headers in a fixed order, and over HTTP/2 the names are lowercase. Some bot detection compares that order with the claimed browser. In requests, set session.headers to an ordered dict to control it.

Are headers or proxies more important for scraping?

They solve different layers. Headers describe the request and session context. Proxies provide the network route. Make the request correct and compliant at a conservative rate first, then test proxies when IP location, reputation, or concentration is the measured constraint.

Final thoughts

Good web scraping headers are minimal, accurate, and consistent with the client, cookies, locale, and network session that send them. Capture a real request from the browser version you are matching, drop the cookies, and compare it with your scraper's output through an echo endpoint. Fix the request when metadata is wrong, slow down when rate is the problem, and look at proxy plans only when the network route is the verified constraint.

About the Author

Unknown Proxies

Proxy Infrastructure Team

Stay Unknown

High-performance dedicated proxies optimized for speed and reliability. Get uncompromising quality, 99.9% uptime, and unmatched support. Stay Unknown.

Explore Plans