Is Data Scraping Legal? A Practical Compliance Guide

Is data scraping legal? Often, for public and factual data, but never automatically. As of October 2026, US courts have not treated logged-out scraping of public pages as hacking under the Computer Fraud and Abuse Act (CFAA). Contract, copyright, privacy, and anti-circumvention claims can still apply, depending on what you access, how, and what you do with it.

A low-risk project starts with an authorized API, a licensed feed, your own data, or narrowly selected public facts. Risk rises when a scraper crosses a login or technical barrier, ignores binding terms or a stop notice, copies protected expression, collects personal data, burdens a service, or feeds a regulated decision.

If you need the engineering side first, the data scraping guide covers retrieval, extraction, validation, and monitoring. This article covers the legal review around that workflow. It states facts and cites sources, but it is not legal advice. Laws and platform rules change, and only qualified counsel can evaluate a specific project.

Six questions to ask before you scrape

Do not reduce the review to "public or private." Run each source through six separate questions:

No single green cell settles the answer. A project can have permission to fetch a page but lack a lawful basis to process the personal data on it. It can collect uncopyrightable facts and still breach a contract or a database right. Legality is a conclusion about the whole project, not a property of the tool, so treat the review as several gates.

Public data is not unrestricted data

Public visibility matters for computer-access law, but it is not a license. A price, a profile, an article, and a court record can all be public while raising different contract, copyright, privacy, and reuse questions.

What US courts have decided so far

The table lists the decisions people cite most. Each one is narrow, and none settles scraping as a whole.

The pattern is consistent. Courts have been skeptical of using the CFAA against logged-out public scraping, but hiQ shows that contract terms accepted through an account can still produce a judgment and a deletion order. Meta's win turned on its own wording, and platforms have since rewritten theirs. Instagram's current terms, for example, say the automated-collection rule applies whether or not the user is logged in.

These rulings do not make all public-page scraping lawful. The CFAA is one law, procedural posture matters, and contract, privacy, intellectual-property, and tort claims can still apply. Accessing an account, private API, paywall, or restricted system without permission is a different analysis from fetching a page anyone can open.

Use these access rules in the crawler design:

• Never use credentials, session cookies, tokens, or accounts you are not authorized to use.
• Do not bypass a login, paywall, CAPTCHA, IP block, or other technical control to reach data outside the approved scope.
• Stop when access is revoked or a source sends a legal notice; preserve the notice and escalate it.
• Keep public and authenticated collection in separate jobs, credentials, logs, and approvals.
• Record the exact URLs and interfaces that were approved instead of granting a crawler an entire domain.

Terms, permission, and robots.txt are different controls

Website terms can create contractual risk independently of computer-access law. Review the terms that apply to the exact interface, account, and intended reuse. Whether online terms form an enforceable agreement can depend on notice, assent, jurisdiction, and the parties' conduct, so do not ask an engineer to make that legal conclusion from a checkbox alone.

An API key or account also does not imply permission for every endpoint or purpose. Check the API agreement, field license, rate limits, retention rules, display requirements, and restrictions on combining or reselling data. Written permission should name the source, fields, frequency, purpose, recipients, retention period, and expiration date.

What major sites' terms say about scraping

Many large platforms prohibit automated collection in their terms of service. This table summarizes the relevant clauses as of October 2026. Always read the current official text, because platforms update it often.

A terms violation is mainly a contract question. Whether the terms bind you depends on how you agreed to them (for example, by creating an account), and courts treat logged-in and logged-out access differently. A breach can still lead to account bans, IP blocks, cease-and-desist letters, and lawsuits, so treat a prohibition in the terms as a stop signal unless you have written permission.

robots.txt is another input, not a substitute for this review. The Robots Exclusion Protocol standardizes how crawlers discover rules, but it is not access authorization or a security control. Respect applicable crawler directives and site policies while separately documenting permission, contracts, privacy, and legal requirements.

Copyright protects expression, not every fact

Data scraping often mixes facts with protected expression. In the United States, the Copyright Office explains that copyright does not protect facts, ideas, systems, or methods of operation, although it can protect the way they are expressed.

That distinction supports collecting a narrowly defined price, SKU, event date, or measurement more readily than copying a product description, photograph, article, review, page layout, or other creative material. A database's original selection or arrangement may also carry protection even when individual facts do not. Fair use and licensing questions are fact-specific; they are not automatic exemptions for training, research, or analytics.

Reduce copyright risk at the data-model level:

• Define factual fields before collection and reject unapproved page content.
• Store source URLs, timestamps, field provenance, and license information.
• Do not save full HTML, images, or long text "just in case" when structured facts are enough.
• Use short-lived, access-controlled fixtures for parser testing only when policy permits them.
• Review any plan to republish, syndicate, sell, or train on collected material.

Projects involving EU sources need an additional database-right check. The EU's Database Directive can protect substantial investment in obtaining, verifying, or presenting database contents, and it addresses extraction or reuse of substantial parts as well as some repeated, systematic extraction of insubstantial parts. A field being factual does not end that separate analysis.

The EU's Copyright in the Digital Single Market Directive adds text and data mining exceptions. Article 3 covers research organizations and cultural heritage institutions. Article 4 allows text and data mining by anyone with lawful access, unless the rightsholder has reserved its rights "in an appropriate manner, such as machine-readable means". For AI training and large-scale analysis, check for that reservation (in terms, robots.txt, or metadata) before you rely on the exception.

Personal data creates obligations after collection

Names, emails, profile URLs, device identifiers, precise locations, employment histories, and values that can be linked back to a person may be personal data. Removing one obvious name may not anonymize a record if it can still be reidentified or combined with other data.

Under the EU GDPR, "personal data" and "processing" are broad concepts, and collection itself is processing. The official GDPR text sets out principles and requirements including lawfulness, purpose limitation, data minimization, accuracy, retention limits, security, transparency, and data-subject rights. A business may need a documented lawful basis and, when it obtains data indirectly, may have notice obligations subject to specific exceptions. Regulators do enforce this against scrapers. In September 2024 the Dutch data protection authority fined Clearview AI 30.5 million euros for building a facial-recognition database from scraped web images without consent.

In California, the CCPA gives covered consumers rights over personal information and imposes duties on covered businesses. The California Attorney General's CCPA guidance explains applicability, consumer rights, notices, and what counts as personal information. Do not assume every profile or publicly viewable page falls outside privacy law; definitions, exemptions, business thresholds, and intended use all matter.

Escalate privacy review when the project includes:

• Sensitive data, children, health, biometrics, precise location, finances, or government identifiers.
• Profiles at scale, identity matching, inferred attributes, or monitoring of individual behavior.
• Hiring, housing, credit, insurance, education, healthcare, or another high-impact decision.
• Sale, sharing, advertising, enrichment, or disclosure to customers and vendors.
• Cross-border transfers or people in jurisdictions the team has not mapped.
• A source that makes deletion, correction, objection, or opt-out requests hard to honor.

Privacy compliance is an operating capability, not a paragraph in a policy. You need a data inventory, retention schedule, access controls, request-handling process, vendor terms, and a way to locate and delete a person's records across raw, normalized, backup, and exported datasets.

Build a compliance preflight before the first request

A useful preflight produces an auditable decision, not a vague note saying "public data." The owner should complete it before scheduling or scaling the collector.

1. Define the purpose. Write one specific use, the users of the output, and decisions the data will influence.
2. Choose the least risky source. Prefer your own export, an authorized API, a licensed feed, an open-data portal, or written permission before HTML collection.
3. Inventory fields. Label each field as factual, creative, personal, sensitive, confidential, or unknown. Reject unknown fields by default.
4. Map access. Record whether the interface is public, authenticated, paid, rate-limited, or technically restricted, plus who owns any credentials.
5. Review rules and rights. Save the applicable terms, license, robots.txt, privacy notice, and permission record with a review date.
6. Map jurisdictions. Consider the operator, source, people represented in the data, storage location, customers, and downstream recipients.
7. Set collection limits. Approve URL patterns, fields, frequency, concurrency, retry ceiling, retention, and a kill switch.
8. Plan data rights. Document notice, lawful basis where required, access, correction, deletion, objection, opt-out, and vendor workflows.
9. Get legal review. Escalate uncertainty, sensitive or personal data, technical restrictions, adverse notices, large-scale republication, and high-impact uses.
10. Re-review changes. New fields, sources, countries, customers, purposes, or collection volume require a fresh decision.

Turn the legal review into technical controls

Approval should become configuration that the scraper can enforce. A compliance memo that never reaches the job queue will not control scope drift.

At minimum, implement:

• A domain and URL-pattern allowlist, with redirects rejected when they leave scope.
• A response-field allowlist so parsers cannot silently begin storing new data.
• Source-specific concurrency, delay, cache, and retry limits.
• Automatic pause rules for repeated 401, 403, 429, CAPTCHA, or policy-page responses.
• Provenance on every record: source, retrieval time, purpose, parser version, and permission or license reference.
• Collection-time redaction for cookies, authorization headers, proxy credentials, and unapproved personal data.
• Retention and deletion jobs that cover raw responses, derived tables, logs, fixtures, backups, and exports.
• Role-based access, encryption, audit logs, incident response, and vendor controls appropriate to the data.

Monitor policy signals as carefully as parser errors. A new login requirement, changed terms, modified robots.txt, different response class, or request from a data subject should pause the affected path for review. Do not automatically route around the change.

For recurring collection, the price monitoring pipeline guide shows how to combine scoped queues, validation, retention, and review gates.

Proxies change routing, not legality

Proxies have legitimate uses in approved scraping: measuring public regional content, assigning stable egress to a customer job, separating environments, and avoiding accidental concentration from shared cloud infrastructure. They do not create permission, a privacy basis, a copyright license, or a right to ignore a denial.

Treat rotation as an engineering choice that must stay inside the approved request budget. Rotating addresses to defeat an explicit block, conceal prohibited volume, or continue after access is revoked is a stop signal, not a scaling strategy. If the same 403 or challenge appears across routes, debug authorization, scope, terms, session state, and request behavior instead of increasing the pool.

When geographic measurement is approved and genuinely affects the data, residential proxies can provide country, state, or city routing. The best proxy for web scraping guide explains the operational tradeoffs among residential, ISP, and datacenter routes. Complete the legal and policy review before choosing any of them.

Data scraping legality examples

These examples are triage signals, not legal conclusions:

The source-specific guides linked in the terms table add detail after this broad review. For retail data, the competitor price scraping guide focuses on factual offer fields, source scope, and validation, and the Zillow guide separates prohibited consumer-page automation from approved datasets.

FAQ

Is scraping data legal if it is publicly available?

Sometimes, but public availability is only one factor. You still need to consider the access method, terms, copyright or database rights, privacy law, jurisdiction, purpose, and load on the site. Public visibility does not grant unrestricted reuse.

Is data scraping legal in the United States?

There is no blanket US rule. Courts have been reluctant to apply the CFAA to logged-out scraping of public pages, as in the 2022 hiQ decision and the 2024 Meta v. Bright Data ruling. Contract, copyright, privacy, and state-law claims can still apply, so the source, access controls, data, agreement, and use decide the result.

Is data scraping legal in the EU?

It depends. Scraping personal data triggers the GDPR even when the data is public, and extracting a substantial part of a database can raise database-right questions. The text and data mining exception in the Copyright in the Digital Single Market Directive does not cover content where the rightsholder has reserved its rights in a machine-readable way.

Can you get sued for web scraping?

Yes. The usual claims are breach of contract, copyright infringement, trespass to chattels, and anti-circumvention or computer-access statutes. In hiQ v. LinkedIn the scraper ended up with a $500,000 judgment and an order to delete its data, so a prohibition in the terms plus a stop notice is a real risk even when the data is public.

Does robots.txt decide whether scraping is legal?

No. robots.txt communicates crawler preferences under a technical protocol. It is not a contract, a permission document, or a legal ruling. Respect it where it applies, and run the access, terms, privacy, and intellectual-property review separately.

What compliance steps should a company take before scraping competitor data?

Run the preflight above. Write down the purpose, check for an API or licensed feed first, read the site's terms and robots.txt, limit collection to factual fields such as price and availability, avoid personal data, set rate limits and a stop rule, keep provenance on every record, and have counsel review the plan. Re-review when you add sources, fields, or countries. The data mining vs web scraping comparison explains how collection differs from downstream analysis.

Is it legal to use proxies for web scraping?

Using a proxy is legal. The risk comes from what it is used for. Rotating IPs to get around an explicit block, a revoked permission, or a cease-and-desist letter can strengthen contract, CFAA, or state-law claims, because it shows you knew access was refused. Approved regional testing or stable egress does not create that problem.

Final thoughts

So, is data scraping legal? The defensible answer comes from a documented review of authorization, terms, data rights, privacy, jurisdiction, purpose, and scraper behavior. Start with the least risky source, collect only approved fields, turn the decision into enforced limits, and stop for counsel when the scope changes or a source objects. Every source should have an owner, every field a purpose, every request a budget, and every record a deletion path.