We use cookies to enhance user experience, personalize content, and analyze traffic. Cookie Policy

← Back to all articles

How to Scrape Google Search Results: Python and APIs

How to scrape Google search results in 2026: Playwright code for organic results, People Also Ask, and related searches, plus SERP APIs and Google's terms.

by Unknown Proxies

16 min read

July 4, 2026

Updated October 17, 2026

How to Scrape Google Search Results: Python and APIs

To scrape Google search results, you collect the titles, URLs, snippets, positions, and SERP features (People Also Ask, related searches) that Google shows for a query. As of October 2026, the reliable ways are a SERP API for volume and a JavaScript-capable browser such as Playwright for small checks. Plain requests plus Beautiful Soup no longer works, because Google Search has required JavaScript since January 2025.

Short answer: use Google Search Console for your own site's data, a SERP API for rank tracking and People Also Ask research at volume, and Playwright only for a handful of pages you check yourself. Google's terms and robots.txt do not allow automated access to /search, so read the terms section before you automate the HTML page.

This guide gives working Python code for a SERP API and for Playwright, shows how to extract Google related questions and AI Overviews, and covers localization, failures, and validation. It is technical guidance, not legal advice.

Can you scrape Google search results? What Google's terms say

Google's Terms of Service list, under "Don't abuse our services", this example of abuse:

"using automated means to access content from any of our services in violation of the machine-readable instructions on our web pages (for example, robots.txt files that disallow crawling, training, or other activities)"

Google's robots.txt contains Disallow: /search for all user agents. When you read the two together, automated collection of the Google results page goes against Google's terms. Google also enforces this with technology. In January 2025 it made JavaScript mandatory for Search and launched an anti-bot system that it calls SearchGuard.

Google has also gone to court. On December 19, 2025, Google sued SerpApi in the US District Court for the Northern District of California, claiming SerpApi broke the DMCA by circumventing SearchGuard. On July 20, 2026, the judge dismissed those claims. Claims about search results with no copyrighted content were dismissed without leave to amend. A narrower claim about licensed content in Knowledge Panels was dismissed with leave to amend, and reports say Google filed an amended complaint on August 10, 2026, which SerpApi has moved to dismiss. We could not confirm a ruling on that motion as of October 2026.

The ruling dealt with the DMCA claim only. It did not change Google's terms of service or robots.txt.

What this means in practice:

Whether a specific project is lawful depends on your jurisdiction, contracts, and use of the data. Our data scraping legality guide lists the questions to take to a lawyer.

Choose a method for scraping Google results

Goal Best method Notes
Clicks, impressions, and queries for your own site Google Search Console (UI or API) First-party data. It does not show competitors' results.
Rank tracking or PAA research at volume Third-party SERP API Structured JSON, location parameters, per-request pricing
Search inside a small set of your own domains Vertex AI Search or Custom Search JSON API (existing customers only) Not a full copy of google.com results
Check a few SERPs as a real browser shows them Playwright or another headless browser Markup changes often. Highest terms risk.
Plain requests + Beautiful Soup Does not work reliably since January 2025 Google returns a JavaScript-required page or a challenge

Search Console is not a general SERP feed. Google's Search Console documentation describes it as the source of truth for how your own site performs in Google Search.

Google's Custom Search JSON API is closed to new customers. Existing customers have until January 1, 2027 to move to another solution. Google suggests Vertex AI Search for search across up to 50 domains. For full web search it asks you to contact Google. Microsoft retired the Bing Search APIs on August 11, 2025, so Bing is not a simple fallback either.

Decision flow for choosing a source before scraping Google search results

Define the result schema before writing code

A SERP changes by query, time, language, country, city, device, and sometimes session state. A title and URL without that context is not a reproducible observation.

Use a record like this:

{
  "query": "example running shoes",
  "source": "serp-api",
  "result_type": "organic",
  "position": 1,
  "title": "Example result",
  "url": "https://example.com/running-shoes",
  "display_domain": "example.com",
  "snippet": "Example description shown in the result.",
  "country": "US",
  "language": "en",
  "device": "desktop",
  "observed_at": "2026-10-04T12:00:00Z",
  "collector_version": "serp-v2"
}

Keep organic results separate from ads, local packs, shopping blocks, featured snippets, video carousels, top stories, People Also Ask, and related searches. Each module has a different shape and ranking logic. Do not flatten them into fake organic positions.

Store the original URL and a normalized URL. Remove known tracking parameters for deduplication, but keep the original for audits. Do not use the visible title as a unique key, because titles change between observations.

Plan for 10 organic results per page. In September 2025 Google stopped honoring the num=100 URL parameter that many rank trackers used to get 100 results in one request. To go deeper, you need one request per page.

Option 1: scrape Google search results with a SERP API

A SERP API runs the browsers, locations, and parsing for you and returns JSON. Most providers give you organic results, related questions, related searches, and other modules as separate arrays. That makes this the most practical method for "google related questions scraping" at any volume.

This example uses SerpApi's documented response fields. SerpApi is the company Google sued, which is a good reminder to review any vendor's terms. Other providers return the same data under different names. For example, DataForSEO uses item types such as organic, people_also_ask, and related_searches.

import os
from datetime import datetime, timezone

import requests


def search_serp_api(query: str, country: str = "us", language: str = "en") -> dict:
    response = requests.get(
        "https://serpapi.com/search.json",
        params={
            "engine": "google",
            "q": query,
            "gl": country,
            "hl": language,
            "api_key": os.environ["SERPAPI_KEY"],
        },
        timeout=60,
    )
    response.raise_for_status()
    data = response.json()
    observed_at = datetime.now(timezone.utc).isoformat()

    return {
        "query": query,
        "country": country.upper(),
        "language": language,
        "observed_at": observed_at,
        "organic": [
            {
                "position": item.get("position"),
                "title": item.get("title"),
                "url": item.get("link"),
                "snippet": item.get("snippet"),
            }
            for item in data.get("organic_results", [])
        ],
        "related_questions": [
            item.get("question") for item in data.get("related_questions", [])
        ],
        "related_searches": [
            item.get("query") for item in data.get("related_searches", [])
        ],
    }

Treat an empty array as a valid answer. Many queries have no People Also Ask box. Log quota and authentication errors separately from "no results", and cache each query and locale pair so a retry does not pay for the same SERP twice.

If you already use the Custom Search JSON API

Existing customers can keep using the API until January 1, 2027. Each request needs an API key, a Programmable Search Engine ID (cx), and a query. The REST documentation describes the endpoint. Google's overview lists 100 free queries per day and $5 per 1,000 extra queries, up to 10,000 per day.

import os

import requests


def custom_search(query: str) -> list[dict]:
    response = requests.get(
        "https://www.googleapis.com/customsearch/v1",
        params={
            "key": os.environ["GOOGLE_API_KEY"],
            "cx": os.environ["GOOGLE_SEARCH_ENGINE_ID"],
            "q": query,
            "num": 10,
        },
        timeout=20,
    )
    response.raise_for_status()
    return [
        {"position": i, "title": item.get("title"), "url": item.get("link")}
        for i, item in enumerate(response.json().get("items", []), start=1)
    ]

This API does not return People Also Ask or related searches, and its results can differ from google.com. Use queries.nextPage in the response for pagination. Do not guess offsets.

Option 2: scrape Google results with Playwright (Python)

Use this method only for small, internal checks where you accept the terms risk above. Google requires JavaScript, so you need a real browser engine. The script below opens one SERP, stops on a challenge or consent page, and extracts organic results, People Also Ask questions, and related searches.

Install it first:

pip install playwright
playwright install chromium

Then run:

import os
from datetime import datetime, timezone
from urllib.parse import parse_qs, urlencode, urlparse

from playwright.sync_api import sync_playwright

# Google markup is not an API. These selectors were chosen to depend on
# structure (a link that wraps an <h3>, accordion items, a visible heading)
# rather than on generated class names. Check them against a saved page
# before each project, and expect to update them.
EXTRACT_JS = """
() => {
  const findHeading = (labels) =>
    [...document.querySelectorAll('div, span, h2, h3')]
      .find((el) => labels.includes(el.textContent.trim()));

  // Walk up from a heading until a parent holds at least `min` matches.
  const containerWith = (start, selector, min) => {
    let node = start;
    while (node && node !== document.body) {
      if (node.querySelectorAll(selector).length >= min) return node;
      node = node.parentElement;
    }
    return null;
  };

  const unique = (items) => [...new Set(items.filter(Boolean))];

  const organic = [...document.querySelectorAll('#search a:has(h3)')].map((a) => ({
    title: a.querySelector('h3').textContent.trim(),
    url: a.getAttribute('href'),
  }));

  const paaSelector = '[data-q], [aria-expanded]';
  const paaHeading = findHeading(['People also ask']);
  const paaBox = paaHeading && containerWith(paaHeading, paaSelector, 2);
  const questions = paaBox
    ? [...paaBox.querySelectorAll(paaSelector)]
        .map((el) => (el.getAttribute('data-q') || el.textContent).trim())
        .filter((text) => text.length > 8)
    : [];

  const relatedSelector = 'a[href*="/search?"]';
  const relatedHeading = findHeading(['People also search for', 'Related searches']);
  const relatedBox = relatedHeading && containerWith(relatedHeading, relatedSelector, 2);
  const related = relatedBox
    ? [...relatedBox.querySelectorAll(relatedSelector)].map((a) => a.textContent.trim())
    : [];

  return { organic, questions: unique(questions), related: unique(related) };
}
"""


def clean_url(href: str) -> str:
    # Some layouts wrap targets in /url?q=...; recover the destination.
    if href and href.startswith("/url?"):
        return parse_qs(urlparse(href).query).get("q", [href])[0]
    return href


def scrape_serp(query: str, country: str = "us", language: str = "en") -> dict:
    url = "https://www.google.com/search?" + urlencode(
        {"q": query, "hl": language, "gl": country}
    )
    launch_args = {"headless": True}
    if os.environ.get("PROXY_SERVER"):
        proxy = {"server": os.environ["PROXY_SERVER"]}  # e.g. http://host:port
        if os.environ.get("PROXY_USERNAME"):
            proxy["username"] = os.environ["PROXY_USERNAME"]
            proxy["password"] = os.environ.get("PROXY_PASSWORD", "")
        launch_args["proxy"] = proxy

    with sync_playwright() as p:
        browser = p.chromium.launch(**launch_args)
        try:
            context = browser.new_context(locale=f"{language}-{country.upper()}")
            page = context.new_page()
            page.goto(url, wait_until="domcontentloaded", timeout=30_000)

            if "/sorry/" in page.url or page.locator("#captcha-form").count():
                raise RuntimeError("Challenge page returned. Stop this run.")
            if urlparse(page.url).hostname == "consent.google.com":
                raise RuntimeError("Consent page returned. Skip or review this locale.")

            page.wait_for_selector("#search", timeout=15_000)
            data = page.evaluate(EXTRACT_JS)
        finally:
            browser.close()

    organic = []
    for item in data["organic"]:
        target = clean_url(item["url"])
        if target and target.startswith("http") and target not in {r["url"] for r in organic}:
            organic.append({"position": len(organic) + 1, "title": item["title"], "url": target})

    return {
        "query": query,
        "country": country.upper(),
        "language": language,
        "observed_at": datetime.now(timezone.utc).isoformat(),
        "organic": organic,
        "related_questions": data["questions"],
        "related_searches": data["related"],
    }


if __name__ == "__main__":
    print(scrape_serp("residential proxies"))

Notes on what this code does and does not do:

The Playwright proxy guide explains browser-context isolation and proxy authentication in more detail.

People Also Ask (PAA) is the box of expandable questions that Google shows for many informational queries. Marketers collect it to find content gaps and FAQ topics. There are three ways to get it:

  1. SERP API. The simplest option. SerpApi returns PAA as related_questions, and each item has the question, the answer snippet, and the source link. DataForSEO returns people_also_ask items. This is the usual answer to "google related questions scraping API".
  2. Headless browser. The EXTRACT_JS function above reads the questions shown on the first load, usually four. Each question is an accordion. Google loads the answer when you click it and usually adds more questions below. To collect a deeper tree, click a question with locator.click(), wait for the new items, and run the extraction again. Every click is more traffic, so cap the depth (for example, two levels).
  3. Manual export. For a few dozen keywords, copying the questions by hand or with a browser extension is often faster than building a collector.

Store PAA data as its own record type with the query, locale, question text, its order in the box, the answer source URL if you collect it, and the observation time. PAA questions change often and differ by location, so one snapshot is a sample, not a fixed list.

Related searches ("People also search for", usually at the bottom of the page) work the same way. Store them as a separate list per query. Do not merge them with PAA.

Other SERP features: AI Overviews, video results, carousels, and "About this result"

Preserve location and language deliberately

Search results for the same query differ by country, city, language, and device. Record the market you intended separately from the network exit you used, because the two do not always match.

For each observation, keep:

Do not infer location only from a country-code domain. With a SERP API, use its documented location parameters. With a browser, keep the language, timezone, and proxy exit in the same region.

Run a control query whose regional behavior you already know. If a US job suddenly returns another market's language or domains, quarantine the batch instead of storing misleading rankings.

When proxies fit Google search data collection

A SERP API already handles routing, so you do not need your own proxies for it. Adding proxies to an API quota does not raise the quota.

For small, permitted browser checks, a proxy controls which location Google sees. Use a residential proxy when you need a consumer IP in a specific country, state, or city. Keep one sticky session for one page load and its subrequests. Rotate only between independent queries, never halfway through a page.

A proxy does not grant permission, and it does not make high-volume collection of Google's results page acceptable under Google's terms. If localized checks need residential routing, residential proxies support country, state, and city targeting with sticky or rotating sessions. The best proxy for web scraping guide compares residential, ISP, and datacenter options.

Google search result collection pipeline with localization, classification, parsing, and validation

Handle rate limits, challenges, and failures

Classify the response before any retry:

Result Meaning Action
API quota error Key or plan limit reached Stop, cache, and wait for the documented reset
HTTP 429 Request rate is too high Honor Retry-After, reduce concurrency, and cool down
HTTP 403 Access or policy denial Stop retries and review the source and method
JavaScript-required page A plain HTTP client was used Switch to an API or a browser. Do not retry.
Consent page Expected content was not returned Handle consent manually or exclude that locale
/sorry/ or CAPTCHA Google challenged the automation Stop the run. Do not automate challenge solving.
HTTP 5xx or timeout Temporary network or service failure Retry a few times with exponential backoff
Page loads, zero parsed results No results, or the markup changed Save a sample and send it to review

The HTTP 429 guide covers bounded exponential backoff and Retry-After.

Do not retry every failure through a fresh IP. That turns a clear stop signal into more traffic and hides the evidence you need to find the real cause.

Validate SERP data before using it

Validation catches records that look plausible but are wrong:

Track fetch success separately from data success. A collector with 99% successful page loads can still be broken if half the pages produce no valid results.

FAQ

Can I scrape Google search results legally?

Google's terms prohibit automated access that violates its robots.txt, and robots.txt disallows /search. A July 2026 court ruling dismissed Google's DMCA claims against SerpApi, but it did not change those terms. Whether a specific project creates legal liability depends on jurisdiction, contracts, method, and use, so get legal advice. Search Console and approved APIs are the low-risk options.

How do I scrape Google search results with Python?

Use a SERP API with requests for structured results, or Playwright for a small number of rendered pages. Since January 2025, Google requires JavaScript, so requests with Beautiful Soup against google.com usually returns a JavaScript-required page or a challenge.

Is there an official Google search results API?

Not for new customers. The Custom Search JSON API is closed to new customers and ends for existing customers on January 1, 2027. Google points to Vertex AI Search for searching up to 50 domains. Search Console APIs cover only properties you own.

The easiest method is a SERP API that returns related questions as a field. In a headless browser, find the "People also ask" heading, read the accordion questions below it, and click questions to load more. Cap the click depth and expect the markup to change.

How do I scrape the URLs from Google search results?

In Playwright, read the href of each link that wraps an <h3> inside #search, then unwrap any /url?q= redirect. A SERP API returns the URL directly, for example in the link field. Expect 10 organic results per page.

Can I scrape Google AI Overviews?

Some SERP APIs return AI Overviews and AI Mode answers as separate modules with their cited sources. Parsing them from raw HTML is fragile, and an overview does not appear for every query, so store it as its own record type and treat a missing one as a valid result.

Do I need proxies to scrape Google?

Not with a SERP API. For small browser checks, a proxy controls which location Google sees, but it does not change Google's terms or prevent challenges.

Final thoughts

To scrape Google search results in 2026, start with the source. Use Search Console for your own site, a SERP API for volume and SERP features such as People Also Ask, and a headless browser only for small checks where you accept the terms risk.

Whatever the method, record the query context, classify every response, validate the records, and stop on quotas or challenges. If you would rather buy a tool than write a parser, the data scraping tools comparison lists no-code and code options, and the AI web scraping guide covers LLM extraction.

About the Author

Unknown Proxies

Proxy Infrastructure Team

Stay Unknown

High-performance dedicated proxies optimized for speed and reliability. Get uncompromising quality, 99.9% uptime, and unmatched support. Stay Unknown.

Explore Plans