How to Find All Pages on a Website: 5 Methods, From Quick to Complete

To find all pages on a website, start with its sitemap. Open the site’s robots.txt file, follow the Sitemap line, and you get the list of pages the owner wants search engines to see. Then fill the gaps with a site: search, Search Console, a crawler, and the Wayback Machine.

No single method finds every page. Each one sees the site from a different angle. Below are all five, from the quickest to the most complete. The examples use our own site, serplify.io, checked on October 9, 2026.

The five methods at a glance

Method 1: Check the sitemap (start with robots.txt)

A sitemap is a file that lists the pages a site wants search engines to find. Google allows up to 50,000 URLs or 50 MB in one sitemap. Bigger sites split their pages across many sitemaps and list them in a sitemap index file.

1. Open https://example.com/robots.txt in your browser.
2. Look for a line that starts with Sitemap:.
3. Open that address. If it is a sitemap index, open each sitemap it lists.

Here is the robots.txt file for serplify.io on October 9, 2026:

User-agent: *
Allow: /

Sitemap: https://serplify.io/sitemap-index.xml

Most people guess /sitemap.xml first. On our site, that address now redirects to /sitemap-index.xml, which is also what the robots.txt line points to. Without a redirect like that, a guess can end in a 404. That index lists one file, /sitemap-0.xml, which held 53 URLs: 37 blog posts, 6 topic pages, the blog home page, and 9 other pages.

To count the URLs in a sitemap, you can run one line in a terminal:

curl -s https://serplify.io/sitemap-0.xml | grep -o "<loc>" | wc -l

If robots.txt has no Sitemap line, try the common addresses. Many sites use /sitemap.xml. WordPress sites often use /wp-sitemap.xml, or /sitemap_index.xml with the Yoast SEO plugin.

Limits. A sitemap only shows what the owner chose to list. It can be out of date. Pages the owner keeps out of search, such as thank-you pages, are often missing.

Some sites also publish a file at /llms.txt. It is a short list of the pages the owner thinks matter most, written for AI tools. It is never a full list, but it tells you which pages the owner cares about. Our guide to what llms.txt is explains the format.

Method 2: Search Google with site:

Type site:example.com into Google to see pages Google has indexed from that domain. Add a folder to narrow it, such as site:example.com/blog/. Our guide to Google search operators covers more ways to filter.

The site: search is quick, but it does not give you a real count. We ran two site: searches for US desktop results on October 9, 2026, and saved the raw result pages:

At the same time, our sitemap had 44 blog URLs. So the folder search estimated more pages than exist and showed fewer than a third of them. The whole-domain search estimated fewer pages than the blog alone.

Google’s own docs warn that the site: operator does not always return every indexed URL. Use it for a quick look, or to spot pages that should not be in Google at all. If you need site: checks across hundreds of domains, a SERP API returns each results page as data you can store and compare.

Method 3: Use the Search Console Pages report (your own site)

If you own the site, Google Search Console shows what Google knows about it.

1. Open Search Console and pick your property.
2. Go to Indexing, then Pages.
3. Read the two totals: indexed and not indexed.
4. Scroll to Why pages aren’t indexed and click a reason to see example URLs.

Google says the totals are complete from its point of view. The example lists are not. Each one stops at 1,000 rows, and Google says it may not show every URL even below that. If a page you expected is missing or not indexed, our guide on why a website is not showing up on Google walks through the usual causes.

The Performance report adds one more list. Its Pages tab shows every page that earned an impression in Google. To pull longer lists, use the Search Console API.

Method 4: Run a crawler

A crawler is a program that opens a page, follows every link on it, and repeats. Googlebot works this way, and you can run a small one yourself.

Screaming Frog’s SEO Spider is a common choice. The free version crawls up to 500 URLs. A licence costs £199 a year and removes the limit (checked October 2026).

1. Enter the home page address and press Start.
2. When the crawl ends, open the Internal tab and filter to HTML.
3. Export the list.

A crawler finds only pages that something links to. It misses orphan pages, which are pages no other page links to. To find them, compare your crawl with the sitemap. Any URL in the sitemap that the crawl never reached is an orphan.

Be polite on sites you do not own. Keep the crawl speed low and let the crawler obey robots.txt.

Method 5: Find old and deleted pages in the Wayback Machine

The Internet Archive’s Wayback Machine saves copies of web pages over time. Its CDX API lists every address it has stored for a domain. Paste this into your browser and swap in any domain:

https://web.archive.org/cdx/search/cdx?url=serplify.io/*&fl=original,timestamp,statuscode&collapse=urlkey

Each line shows one URL, the first time it was saved, and the status code it returned then.

On October 9, 2026, that query returned 149 unique URLs for serplify.io. The oldest was saved in 2016. Most came from 2017, years before this site launched, including pages like /academy and /affiliate. Many others were image and style files.

That is the value of the archive. It finds pages that no sitemap or crawl will show today. If old pages still get links from other sites, redirect each one to its closest live page. That way the referral visits from those links are not lost.

Limits. The archive only knows pages it happened to save. Pages published recently may not be there yet.

How to find hidden pages on a website

“Hidden” usually means a page with no link in the menu or anywhere else. Three checks catch most of them:

1. Compare the sitemap with a crawl. Sitemap URLs the crawl missed are orphans.
2. Read the Wayback list for old paths that still load.
3. Read the robots.txt file. Its Disallow lines name folders the owner asked crawlers to skip.

Stay on public pages. Admin panels, account areas, and anything behind a login are private, and you should not try to get into them.

Which method to use

For a quick count, use the sitemap. For your own site, add the Search Console Pages report. For a full audit, run a crawler and compare it with the sitemap. For a site’s history, or a competitor’s old pages, use the Wayback Machine. A site: search is best kept for spot checks.