Before anything is tested, discovery builds the list of pages you choose from. The portal shows this stage as Crawling Site. This article explains how that list is built, so you can predict what it will contain.
Sitemap first
We look for a sitemap at the usual addresses on your site. If we find one, we use the pages it lists, including pages listed in any sitemaps it points to. A sitemap is the most reliable way to make sure we find your pages, because it does not depend on your links.
Following links, if there is no sitemap
If there is no sitemap, we start at your home page, load it in a browser, collect its links and follow them, page by page, until there are no new pages to find.
What is included and what is not
- Your home page is always included.
- Only pages on the same address. Pages on a different domain or subdomain are not followed. If your blog lives on a separate subdomain, add it as its own website.
- Addresses that differ only after a question mark count as one page. For example /products?page=2 and /products?page=3 are treated as /products. This stops filtered and paginated listings from flooding the list.
- Links to email addresses, phone numbers and page anchors are ignored.
- There is an upper limit of 3,000 pages per discovery. This is a safety limit, separate from how many pages your plan lets you scan at once.
Excluding parts of your site
When you add a site, or start a new full site scan, you can enter Exclude paths, separated by commas. Anything under each path is skipped entirely. For example, excluding /news also skips /news/2026/article. This is the easiest way to keep a large archive or a store's thousands of product pages out of your list.
Grouping
Found pages are grouped by the first part of their address, so all of /blog/... sits together. This makes choosing one example of each template much quicker.
Cancelling
You can cancel while discovery is running. Cancelling removes that scan, including the pages found so far. Your website stays in your list and you can start again at any time.
When pages are missing
The usual causes are pages behind a login, pages only reachable through forms or scripted menus, a firewall or bot protection turning us away, or a sitemap that is out of date. Adding or fixing a sitemap solves most of these. For more, see What happens during discovery in Getting Started.