A Scrapy user agent is the value Scrapy sends in the HTTP User-Agent request header. Set USER_AGENT in settings.py when the whole project should use one identity, use a spider's custom_settings for one crawler, or pass headers to scrapy.Request when a specific request needs an override.

For most permitted crawls, a stable application-specific value is easier to operate than a random browser string. Changing the header alone does not turn Scrapy into Chrome, grant access, or fix excessive request volume. Review the source's terms and robots.txt, prefer an API or feed when one meets the need, and keep the crawl rate within the site's rules.

This guide implements each Scrapy user agent scope, explains which value wins, shows controlled rotation for legitimate compatibility testing, and separates user-agent problems from session, proxy, 403, and 429 failures.

Scrapy User Agent Quick Setup

Add a project-wide value to your Scrapy project's settings.py:

USER_AGENT = "ExampleResearchBot/1.0 (+https://example.org/crawler-info)"

Scrapy's documented default is Scrapy/VERSION (+https://scrapy.org). The built-in UserAgentMiddleware uses the USER_AGENT setting unless that request already has a User-Agent header.

An application-specific value is useful when you operate a crawler and can publish a page that identifies its purpose and contact method. If you run authorized browser-compatibility tests, use the identity of the browser you actually operate rather than combining a browser user agent with Scrapy's non-browser behavior.

Keep the other controls explicit too:

ROBOTSTXT_OBEY = True

DOWNLOAD_DELAY = 2.0
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

The project template enables ROBOTSTXT_OBEY, while Scrapy retains a false fallback for historical compatibility. Confirm the effective setting instead of assuming an older project has it enabled. Scrapy's AutoThrottle documentation also notes that AutoThrottle still respects the normal per-domain concurrency and minimum-delay settings.

Choose the Right Configuration Scope

Use the narrowest scope that represents the real behavior you want.

Do not use DEFAULT_REQUEST_HEADERS merely to set the user agent. Scrapy provides USER_AGENT for that job, while DEFAULT_REQUEST_HEADERS is intended for other project-wide defaults such as Accept and Accept-Language.

Set a User Agent for One Spider

Use custom_settings when one spider needs a different default without changing the rest of the project:

import scrapy


class CatalogSpider(scrapy.Spider):
    name = "catalog"
    allowed_domains = ["example.com"]
    start_urls = ["https://example.com/catalog"]

    custom_settings = {
        "USER_AGENT": (
            "ExampleCatalogMonitor/2.1 "
            "(+https://example.org/crawler-info)"
        ),
    }

    def parse(self, response):
        for product in response.css("article.product"):
            yield {
                "name": product.css("h2::text").get(),
                "url": response.urljoin(product.css("a::attr(href)").get()),
            }

Scrapy applies spider settings above project settings, so this spider uses its own value while other spiders continue to use settings.py. Keep configuration at class level; do not mutate the global settings object after crawling has started.

Override the Header for One Request

Pass headers when one request genuinely needs a different value:

import scrapy


class CompatibilitySpider(scrapy.Spider):
    name = "compatibility"

    async def start(self):
        yield scrapy.Request(
            "https://example.com/public-page",
            headers={
                "User-Agent": (
                    "ExampleCompatibilityCheck/1.0 "
                    "(+https://example.org/crawler-info)"
                ),
            },
        )

    def parse(self, response):
        sent_user_agent = response.request.headers.get(
            "User-Agent", b""
        ).decode()
        self.logger.info("Sent User-Agent: %s", sent_user_agent)

The request-specific header wins because UserAgentMiddleware only supplies its configured value when the header is absent. The Request.headers documentation describes the case-insensitive dictionary-like header object Scrapy uses.

This example uses async def start(), the current initial-request API. Scrapy 2.17 removed start_requests() after deprecating it in 2.13, so a spider that targets current Scrapy should not rely on the old synchronous hook.

If every request from the spider needs the same value, prefer custom_settings. Repeating the same header at every call site makes later changes easy to miss.

How Scrapy Applies the User Agent

The built-in flow is straightforward:

1. Your spider creates a Request.
2. Downloader middleware processes that request in configured order.
3. UserAgentMiddleware checks for an existing User-Agent header.
4. If none exists, it adds the value from USER_AGENT.
5. The downloader sends the resulting request.

This means a request override wins over the spider or project default. A spider's custom_settings value wins over the same project setting. Command-line settings have still higher settings priority, so a value passed with scrapy crawl ... -s USER_AGENT=... can override project or spider settings for that run.

Use the command-line form for a temporary diagnostic, not as hidden production configuration:

scrapy crawl catalog \
  -s USER_AGENT="ExampleResearchBot/1.1 (+https://example.org/crawler-info)"

Record the active value with each deployment. Otherwise, a crawler can behave differently in a worker than it does on a developer machine even though the spider code is identical.

User-Agent and robots.txt Matching

ROBOTSTXT_USER_AGENT is related but separate. It selects the user-agent token Scrapy uses when matching robots.txt groups. When that setting is None, Scrapy uses a request-specific User-Agent header or the USER_AGENT setting, in that order.

The Scrapy settings reference documents that lookup. The Robots Exclusion Protocol defines how crawlers match product tokens and access rules, but robots directives do not replace permission, site terms, privacy review, or applicable law.

Should You Rotate Scrapy User Agents?

Do not rotate the Scrapy user agent on every request by default. Random changes make logs harder to compare and can conflict with cookies, language, client capabilities, or a stable proxy session.

Rotation is reasonable when you own or have authorization to test multiple supported client profiles. In that case, select a profile deliberately and keep it stable for the logical session. A custom downloader middleware can enforce that policy:

from hashlib import sha256
from urllib.parse import urlsplit


class StableUserAgentMiddleware:
    def __init__(self, user_agents):
        self.user_agents = user_agents

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("USER_AGENT_PROFILES"))

    def process_request(self, request, spider):
        if not self.user_agents or "User-Agent" in request.headers:
            return None

        profile_key = request.meta.get("session_id")
        if profile_key is None:
            profile_key = urlsplit(request.url).hostname or "default"

        digest = sha256(str(profile_key).encode()).digest()
        index = int.from_bytes(digest[:4], "big") % len(self.user_agents)
        request.headers["User-Agent"] = self.user_agents[index]
        return None

Register it before the built-in user-agent middleware and provide the approved profiles:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.StableUserAgentMiddleware": 490,
}

USER_AGENT_PROFILES = [
    "ExampleCompatibilityCheck/1.0 (+https://example.org/crawler-info)",
    "ExampleCompatibilityCheck/1.1 (+https://example.org/crawler-info)",
]

The middleware respects an explicit request header, then chooses the same configured profile for the same session_id. Without a session ID, it stays stable per target hostname. Because the built-in middleware uses setdefault, it will not replace the value this middleware has already chosen.

Use real, maintained profiles that correspond to your test plan. A stale list copied from the internet creates the appearance of variety without reproducing the advertised clients' TLS behavior, JavaScript, cookies, content encodings, or browser-generated headers. The broader web scraping headers guide explains why a coherent minimal request is safer than a copied browser header bundle.

Keep User Agent, Cookies, and Proxy Sessions Aligned

Headers and proxies operate at different layers. The user agent describes the client; the proxy changes the network route. Neither substitutes for correct authentication, request pacing, or permission.

If a workflow keeps cookies across multiple pages, keep its user-agent profile, cookie jar, and proxy session stable too. Use the same identifier for the middleware's session_id and Scrapy's cookiejar, then pass all session metadata to follow-up requests:

import os
import scrapy


class RegionalCatalogSpider(scrapy.Spider):
    name = "regional_catalog"

    async def start(self):
        session_key = "catalog-us-1"
        yield scrapy.Request(
            "https://example.com/catalog",
            callback=self.parse_catalog,
            meta={
                "proxy": os.environ["PROXY_URL"],
                "session_id": session_key,
                "cookiejar": session_key,
            },
        )

    def parse_catalog(self, response):
        next_page = response.css("a.next::attr(href)").get()
        if next_page is None:
            return

        yield response.follow(
            next_page,
            callback=self.parse_catalog,
            meta={
                "proxy": response.meta["proxy"],
                "session_id": response.meta["session_id"],
                "cookiejar": response.meta["cookiejar"],
            },
        )

Scrapy's HttpProxyMiddleware reads the proxy request metadata. CookiesMiddleware uses one cookie jar by default, but the cookiejar request key lets one spider maintain separate logical cookie sessions. That key is not sticky: if you omit it from the follow-up, Scrapy falls back to the default jar instead of continuing the selected session.

Keep credentials in an environment variable or secret manager, never in the spider or logs. Use a sticky proxy route for a multi-page session; per-request rotation is better reserved for independent requests that do not share cookies or state.

If routing is the verified constraint, compare residential proxies for location and rotation with stable options in the best proxy for web scraping guide. Changing IPs will not repair a wrong selector, a login requirement, or an excessive crawl rate.

Debug a Scrapy User Agent That Is Not Working

First verify what Scrapy actually sent. Log response.request.headers, use a controlled endpoint you are authorized to inspect, or capture the request on a test server you own. Redact cookies, authorization, and proxy credentials before saving logs.

Then work through the failure by layer:

The Header Does Not Match USER_AGENT

Search for every place that can set it:

rg -n "USER_AGENT|User-Agent|UserAgentMiddleware" .

Check custom_settings, request headers, downloader middleware, command-line -s options, and deployment-specific settings modules. Scrapy settings have priorities; editing the lowest-priority file will not change a higher-priority value.

Also inspect middleware that replaces headers rather than using setdefault. A custom middleware running later can overwrite the built-in value even when settings.py is correct.

The Site Still Returns 403

A correct Scrapy user agent can fix only a user-agent-specific configuration problem. It cannot resolve missing authorization, disallowed automation, expired cookies, a CSRF flow, browser-only rendering, IP reputation, or a firewall rule.

Run a controlled comparison at low rate. Keep the URL, cookies, proxy, and delay fixed; change only the user-agent configuration. If both attempts return the same response classification, restore the honest stable value and investigate the other layers. Use the HTTP 403 guide to separate access denial from proxy connection errors.

Do not keep cycling strings against an access denial. If the site requires an approved API, account, or browser workflow, use that supported path.

The Site Returns 429

429 Too Many Requests is a pacing problem, not evidence that the user-agent string is wrong. Honor Retry-After when present, lower per-domain concurrency, add delay and bounded backoff, cache unchanged responses, and stop immediate retries.

AutoThrottle can adjust delay from observed latency, but it does not grant more capacity. Start conservatively and measure responses by endpoint. The HTTP 429 guide covers retry and monitoring decisions in more detail.

The Direct Request Works but the Proxy Request Fails

Keep the user agent constant and test the proxy separately. A 407 points to proxy authentication. A timeout points to the proxy route, target, DNS, or network path. A target-generated 403 or 429 means the proxy likely connected but the target rejected or limited the request.

Verify the exit IP with a neutral endpoint, then compare one direct and one proxied request at the same low rate. Do not rotate headers and proxies simultaneously; changing two variables hides the cause.

Production Checklist

Before deploying the spider:

• Use one clear USER_AGENT default for the project or spider.
• Publish a crawler information/contact page when an application-specific identity is appropriate.
• Confirm the effective ROBOTSTXT_OBEY and ROBOTSTXT_USER_AGENT settings.
• Override individual requests only when the endpoint or test case requires it.
• Keep user agent, cookies, locale, and proxy route stable within a stateful session.
• Set timeouts, conservative concurrency, bounded retries, and backoff.
• Classify 403, 407, 429, timeout, and parse failures separately.
• Log the header-policy version, not secrets or full cookie values.
• Run a small canary crawl before increasing scope or concurrency.

The goal is reproducibility. When the response changes, you should be able to identify whether the cause was code, configuration, rate, session state, proxy route, or target behavior.

Frequently Asked Questions

What is Scrapy's default user agent?

Scrapy documents the default as Scrapy/VERSION (+https://scrapy.org), where the installed version is inserted at runtime. Set USER_AGENT if your crawler should use a specific application identity.

How do I set a user agent in Scrapy?

Set USER_AGENT in settings.py for the project, add it to a spider's custom_settings for one spider, or pass headers={"User-Agent": "..."} to scrapy.Request for one request.

Does a request header override Scrapy's USER_AGENT?

Yes. The built-in UserAgentMiddleware adds the configured value only when the request does not already contain a User-Agent header.

Can Scrapy rotate user agents automatically?

The built-in middleware supplies one configured value. Controlled rotation requires custom or third-party middleware. Prefer a stable profile per logical session, and use rotation only for authorized tests that genuinely require multiple maintained profiles.

Will changing the Scrapy user agent fix 403 errors?

Only when the denial is caused by that header. A 403 may instead come from authorization, policy, cookies, browser requirements, request behavior, firewall rules, or network reputation. Change one variable at a time.

Do I need a proxy with a custom user agent?

No. A custom user agent and a proxy solve different problems. Add a proxy only when location, IP reputation, stable routing, or per-IP concentration is a demonstrated requirement for an allowed workflow.

Conclusion

A reliable Scrapy user agent setup starts with one explicit USER_AGENT, a narrower spider or request override only where needed, and logs that prove which header was sent. If you rotate profiles for authorized compatibility tests, keep each choice aligned with its cookies, locale, and proxy session instead of randomizing every request.

Treat the header as one part of the request—not as a universal block fix. Respect site rules, control concurrency and retries, and diagnose 403, 407, 429, session, and routing failures separately. That makes the Scrapy implementation predictable enough to test, audit, and maintain.