How to Scrape Google Search Results With Python (and Why It Breaks After 50 Searches)

Every developer tries it once. Send a request to google.com/search, parse the HTML, pull out the links. It feels like it should take an hour.

This guide shows what actually happens when you do that in September 2026, with the real script and the real response. Then it shows what it takes to keep going, and the shortcut most teams end up using.

The simplest possible scraper

Here is the script everyone writes first. It uses the requests library and pretends to be Chrome.

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/130.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

resp = requests.get(
    "https://www.google.com/search",
    params={"q": "best running shoes", "hl": "en"},
    headers=headers,
    timeout=20,
)
print(resp.status_code, len(resp.content), "bytes")

soup = BeautifulSoup(resp.text, "html.parser")
for block in soup.select("div.g"):
    link = block.select_one("a")
    title = block.select_one("h3")
    if link and title:
        print(title.get_text(), link["href"])

What Google actually sent back

We ran that exact script on September 6, 2026 from a normal home connection. The output:

200 92632 bytes

Then nothing. No titles, no links. The page was a 200 OK with about 90 KB of HTML, and it contained none of the result blocks the selector looks for. We checked for the usual markers, div.g and data-hveid, and neither was present.

That is the modern Google response to a request that does not run JavaScript. It sends a shell. The results are loaded by scripts that a plain HTTP client never executes. The script did not fail loudly. It just found an empty page.

Ten years ago this script printed ten results. Today it prints nothing, and the change happened quietly.

What it takes to actually scrape Google

To get real results at any volume, you need four things. Each one is a project, not a setting.

1. A real browser

Because the page is built with JavaScript, you need a browser engine. Playwright or Selenium with headless Chrome works. It renders the page like a person’s browser, and then you can read the results out of the DOM.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://www.google.com/search?q=best+running+shoes&hl=en")
    page.wait_for_selector("h3", timeout=10000)
    for h in page.query_selector_all("h3"):
        print(h.inner_text())
    browser.close()

This works for the first few searches. A headless browser uses about 200 to 500 MB of memory per instance and takes a few seconds per page. A thousand searches is a real machine and a real wait.

2. Proxies, and the right kind

Google counts requests per IP address. After a few dozen from one address in a short time, it shows a captcha or answers with HTTP 429. Datacenter IPs get flagged fastest. You need residential or mobile proxies that rotate, and you need them in the country and city you are searching from, because location changes the results.

Residential proxy bandwidth is sold by the gigabyte. A rendered Google results page is several hundred kilobytes with all its scripts, so bandwidth becomes the largest line in the budget.

3. Captcha and block handling

Even with good proxies, a share of requests get a captcha or a block. Your code must detect it, drop that proxy session, wait, and retry. At scale this is most of the code. Providers that fetch millions of pages a day treat a high retry rate as normal and only count the page as done when a real result comes back.

4. A parser you keep updating

Google changes its HTML often and without notice. Class names are random strings that rotate. A selector that works today breaks next month. Then you have to tell an organic result from an ad, a People Also Ask box, an AI Overview, a local pack, and a video row. There are forty other block types after that, each with its own structure. Our guide on SERP features lists them. Every one is a parser to write and maintain.

What this costs you

Put it together for a modest job: 5,000 searches a week.

For a lot of teams the honest answer is this. A developer spends a few days a month keeping a scraper alive, and the proxy bill is still larger than a SERP API bill would have been.

The same job with a SERP API

A SERP API runs the browsers, the proxies, the retries, and the parsers on its side. You send one HTTPS request and get JSON back. Here is the same search in Python:

import os
import requests

resp = requests.post(
    "https://api.serplify.io/v1/serp/search",
    headers={"Authorization": f"Bearer {os.environ['SERPLIFY_KEY']}"},
    json={
        "keyword": "best running shoes",
        "location": {"code": 2840},
        "language": {"code": "en"},
        "device": "desktop",
        "format": "advanced",
    },
    timeout=60,
)
resp.raise_for_status()
data = resp.json()["data"]

print(data["feature_types"])
for item in data["items"]:
    if item["type"] == "organic":
        print(item["rank_absolute"], item["rank_group"], item["domain"], item["url"])

Run on the same day, that printed the page as it really was. An AI Overview at position 1. Runner’s World at position 2. Reddit at 3. A People Also Ask block at 4. And so on down the page, each item with its type and both position numbers.

The request took about a second. It cost $0.005. If Google had blocked the fetch, it would have cost nothing, because only successful pages are billed. Location is a code, 2840 for the United States, and any city has one. Depth goes up to 100 results in the same request.

That is the trade. You lose control over the browser and proxy layer. You gain a stable, typed response and a bill that scales with successful pages, not with attempts.

When you might still scrape it yourself

There are honest reasons to build your own scraper.

• You need something no API returns. A rare block type, a screenshot, or a raw HTML archive. Some APIs offer raw HTML too, so check before you build.
• You are learning. Rendering a page with Playwright and reading the DOM is a good exercise.
• Volume is tiny and infrequent. Ten searches a week from a browser you drive by hand is fine.

For anything ongoing, or anything a business depends on, the math points the other way.

A note on terms and law

Search results are public pages, and fetching public pages is common practice across the web. Google’s terms of service restrict automated access to its services, and how that applies depends on where you are and what you do with the data. SERP API providers operate at scale under those conditions, and many businesses rely on them. If this matters for your company, read the terms and get advice. We are not lawyers.

If you want to try the API route

Create an account, take the $1 starting balance, and run the Python example above. Our step-by-step tutorial walks through the API key, the Playground, and reading the response. If you already have a DataForSEO integration, the drop-in endpoints accept the same request shape.

For what to do with the data once you have it, start with how to track rankings in bulk.

Quick recap

• A plain request to Google returns a page with no results. The results are built by JavaScript.
• Real scraping needs a browser, rotating residential proxies, captcha handling, and a parser you maintain forever.
• Google blocks repeated requests from one address within dozens of searches.
• A SERP API returns the same page as typed JSON in one call, for a fraction of a cent, and bills only successful pages.