Python scraping is the process of fetching permitted website pages, extracting defined fields from their HTML, validating those fields, and saving structured records. For a static site, the simplest reliable stack is requests for HTTP and Beautiful Soup for parsing.

The code is only one part of the job. A dependable scraper also checks the response before parsing, limits pagination, separates network failures from selector failures, and stops when a site returns a rate limit or access denial. Check the site's terms, access rules, and applicable law before collecting data; robots.txt is a useful machine-readable signal, but it is not permission by itself.

This tutorial builds a small scraper against Books to Scrape, a practice catalog made for scraping exercises. It collects book titles, prices, stock messages, and URLs, follows pagination, and writes the result to CSV.

Python Scraping Workflow at a Glance

A maintainable collector has five distinct stages:

1. Discover: define the allowed pages, required fields, and stopping rule.
2. Fetch: request one page with an honest client identity and explicit timeout.
3. Classify: check status, final URL, and content type before parsing.
4. Parse and validate: extract fields, normalize URLs, and reject incomplete records.
5. Store and monitor: save observations with source metadata and track failures separately.

Keeping these stages separate matters. If parse_products() accepts stored HTML instead of making its own request, you can test a selector change without repeatedly contacting the website.

Choose Requests, Beautiful Soup, or a Browser

Start with the least complex client that returns the data you are allowed to collect.

Before adding a browser, save the response body and search it for the expected field. Some pages contain the data in embedded JSON even when the visible interface uses JavaScript. If browser rendering is genuinely required, the official Playwright Python documentation covers installation and browser setup.

Do not use a browser merely because an HTML selector returned nothing. The response may be an error page, consent page, login page, or changed template. Classify it first.

Install the Python Scraping Packages

Create a virtual environment, then install the two packages used in the example:

python -m venv .venv
source .venv/bin/activate
python -m pip install requests beautifulsoup4

On Windows PowerShell, activate the environment with .venv\Scripts\Activate.ps1.

requests handles the HTTP session, redirects, status codes, and timeouts. Beautiful Soup turns HTML into a searchable tree. Its official documentation describes select() and select_one(), which use familiar CSS selectors.

Inspect the Page Before Writing Selectors

Open one representative page in developer tools and identify the smallest container that repeats once per record. On the practice catalog, each book is inside article.product_pod.

Inside that container, the example needs:

• h3 a for the title and relative detail URL
• .price_color for the displayed price
• .availability for the stock message
• li.next a outside the cards for the next-page URL

Prefer selectors tied to semantic elements or stable attributes. Deep paths such as body > div:nth-child(2) > div > ... break when an unrelated wrapper changes. Keep every target-specific selector inside the parser so future markup changes have one repair point.

Build a Safe Fetch Function

Fetching should return HTML only after it has confirmed that the response is usable:

import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "ExampleCatalogResearch/1.0 (+https://example.org/collector-info)",
    "Accept": "text/html,application/xhtml+xml",
})

def fetch_html(url: str) -> str:
    response = session.get(url, timeout=(5, 20))
    response.raise_for_status()

    content_type = response.headers.get("Content-Type", "")
    if "text/html" not in content_type.lower():
        raise ValueError(f"Expected HTML, received {content_type!r}")

    return response.text

Use both a connection and read timeout. The Requests quickstart notes that Requests does not time out unless you set one, and raise_for_status() turns unsuccessful HTTP responses into visible exceptions.

The client identity above is deliberately descriptive rather than copied from Chrome. Replace the example contact page with real operator information when appropriate. For a deeper header policy, use the web scraping headers guide.

Parse and Validate Each Record

The parser should receive HTML and its source URL. urljoin() converts each relative link into a complete URL, while field checks prevent broken cards from entering the dataset.

from bs4 import BeautifulSoup
from urllib.parse import urljoin

def parse_books(html: str, page_url: str) -> tuple[list[dict], str | None]:
    document = BeautifulSoup(html, "html.parser")
    records = []

    for card in document.select("article.product_pod"):
        link = card.select_one("h3 a")
        price = card.select_one(".price_color")
        availability = card.select_one(".availability")

        if not link or not price or not availability:
            continue

        title = link.get("title", "").strip()
        href = link.get("href", "").strip()

        if not title or not href:
            continue

        records.append({
            "title": title,
            "price": price.get_text(strip=True),
            "availability": availability.get_text(" ", strip=True),
            "url": urljoin(page_url, href),
            "source_page": page_url,
        })

    next_link = document.select_one("li.next a")
    next_url = urljoin(page_url, next_link["href"]) if next_link else None

    return records, next_url

Skipping an incomplete card is acceptable only if you also count and monitor rejected records. Otherwise, a selector change can quietly turn a successful HTTP response into bad data. In production, return a rejection count or raise an alert when the accepted-record count falls outside its normal range.

Follow Pagination Without an Infinite Crawl

A pagination loop needs more than a “next” selector. Bound it with an explicit page limit and a set of previously seen URLs:

import time

START_URL = "https://books.toscrape.com/"
MAX_PAGES = 5

all_books = []
seen_pages = set()
next_url = START_URL

while next_url and len(seen_pages) < MAX_PAGES:
    if next_url in seen_pages:
        raise RuntimeError(f"Pagination loop detected at {next_url}")

    seen_pages.add(next_url)
    html = fetch_html(next_url)
    books, next_url = parse_books(html, next_url)

    if not books:
        raise RuntimeError("No books parsed; inspect the saved response")

    all_books.extend(books)

    if next_url:
        time.sleep(1.5)

print(f"Collected {len(all_books)} books from {len(seen_pages)} pages")

The fixed delay is suitable for this bounded tutorial, not a universal crawl rate. Follow the target's documented limits and slow down or stop on 429. For larger schedules, use the request delay calculator to model worker count and pacing rather than guessing.

Store URLs in seen_pages after redirects have been resolved if the target frequently redirects. Also enforce an allowed-host list before following links so a malformed or external “next” URL cannot move the collector outside its intended scope.

Save the Results to CSV

The standard library can write the validated dictionaries without another dependency:

import csv

fieldnames = ["title", "price", "availability", "url", "source_page"]

with open("books.csv", "w", newline="", encoding="utf-8") as output:
    writer = csv.DictWriter(output, fieldnames=fieldnames)
    writer.writeheader()
    writer.writerows(all_books)

For recurring runs, add observed_at, a parser version, and a stable record key. Keep the raw displayed price as collected, then normalize currency and numeric value in a separate transformation step. That preserves evidence if a parser or locale rule later changes.

CSV works for a small export. Use a database when multiple workers must deduplicate records, update observations transactionally, or query change history.

Check Robots.txt and Site Rules

Python includes urllib.robotparser for reading and querying robots.txt rules:

from urllib.robotparser import RobotFileParser

robots = RobotFileParser("https://example.com/robots.txt")
robots.read()

allowed = robots.can_fetch("ExampleCatalogResearch/1.0", "https://example.com/products")
print("Allowed by robots.txt:", allowed)

The Python robotparser documentation also covers crawl_delay(), request_rate(), and sitemap entries. The standardized behavior of the protocol is defined in RFC 9309.

Treat this check as one input to your access decision. A missing or permissive file does not override terms, authentication, copyright, privacy obligations, or instructions from the site owner. Conversely, a disallowed path is a clear reason not to send the collector there.

Test the Parser Without Repeated Live Requests

Save a permitted sample page as a test fixture and call parse_books() with the stored HTML. At minimum, test:

• a normal page with several records
• the last page, which has no next link
• a card with a missing price or URL
• an empty page or unexpected template
• a login, consent, access-denied, or rate-limit response

Assert field values, absolute URLs, record count, and the next-page result. Network tests can verify one small canary request separately; parser tests should be deterministic and fast.

A 200 OK does not prove that parsing succeeded. Monitor HTTP status, final URL, content type, response size, accepted records, rejected records, duplicates, and parser-empty rate. Quarantine unexpected pages for review rather than overwriting a previously good dataset with zero rows.

When Proxies Fit a Python Scraping Project

Many Python website scraping jobs do not need a proxy. Start with direct, low-rate requests and make the parser reliable first. A proxy can be appropriate for authorized geo-specific testing, independent regional observations, or distributing an approved workload that would otherwise depend on one network route.

Proxies do not repair broken selectors, missing permission, invalid authentication, or excessive request volume. If direct and proxied requests behave differently, keep the URL, headers, cookies, and timing constant while testing the route. The best proxy for web scraping guide explains the tradeoffs among residential, ISP, and datacenter routes.

For stateless public pages that require legitimate location coverage, residential proxies offer rotation and geographic targeting. For stateful workflows, keep cookies and one sticky route together. The Python proxy requests guide covers authentication, timeouts, and route verification without mixing those details into the parser.

Common Python Scraping Failures

Change one variable at a time. Rotating IPs, randomizing headers, replacing the parser, and raising concurrency together makes the original failure impossible to diagnose.

Frequently Asked Questions

Is Python good for website scraping?

Yes. Python has mature HTTP, HTML parsing, browser automation, validation, and storage libraries. Its readable syntax also makes it practical to keep fetching, parsing, and data cleanup in separate testable functions.

Do I need Beautiful Soup for web scraping?

No. Beautiful Soup is convenient for irregular HTML and CSS selectors, but structured APIs can be parsed with the standard json module. lxml, Scrapy, and browser locators are alternatives for other performance and workflow needs.

Can requests scrape JavaScript websites?

requests downloads the HTTP response but does not execute JavaScript. It works if the required data is already in the HTML, embedded structured data, or an approved API response. Use a browser only when permitted data truly requires rendering or interaction.

How often should a Python scraper send requests?

There is no universal safe interval. Use the site's published limits, Retry-After, robots.txt signals, and written permission where applicable. Begin with a small canary run, cap concurrency, cache unchanged pages, and stop on denial or sustained rate limiting.

Should I rotate proxies on every page?

Not by default. Per-request rotation can break stateful sessions and make debugging harder. Use it only for independent, authorized requests where the proxy plan and target workflow support rotation; otherwise keep one session and route together.

Conclusion

Reliable Python scraping is a data pipeline, not a single CSS selector. Fetch with explicit timeouts, classify each response, parse stored HTML, validate records, cap pagination, test against fixtures, and monitor empty or rejected results before increasing scale.

Start with one permitted page and the simplest client that exposes the required data. Once the direct workflow produces trustworthy records, add scheduling, storage, browser rendering, or proxy infrastructure only when the project's requirements justify the extra complexity.