Field guide · technical workflow
When browsing filters create more URLs than useful pages
A category with colour, material, size and sorting options can generate far more combinations than a retailer can meaningfully explain. Those controls may be excellent for shoppers but poor standalone search destinations. Start by identifying observed URL classes, not by blocking every query string.
Who this is for: Help a catalogue engineering lead separate shopper filter functionality from low-value URL combinations that drain crawling.
Editorial preview — facts, worked examples and publication approval are still being checked.
The practical problem
A practical when browsing filters create more urls than useful pages exercise: Classify actual facet URL combinations by shopper intent and technical behaviour. The outcome is a documented decision backed by record crawl, canonical and indexing treatments separately. The final check is to confirm real product URLs remain discoverable after filter controls change.
What you will be able to check
- Classify actual facet URL combinations by shopper intent and technical behaviour.
- Record crawl, canonical and indexing treatments separately.
- Confirm real product URLs remain discoverable after filter controls change.
Reproducible scenario
Worked example: When browsing filters create more URLs than useful pages
Imagine a furniture catalogue whose chair category has a colour filter, a material filter, a sort control and an in-stock toggle. A crawler can request many combinations even though a human usually uses only a few. An operations director wants to reduce wasted crawl paths without hiding useful categories or product links. The example has no real request-log counts and must not be presented as a measured crawl-budget saving.
Starting assumptions
- The unfiltered chair category remains available through normal HTML navigation and links to individual products.
- A small crawl and access-log sample can distinguish human-useful filtered views from arbitrary parameter permutations.
Decisions and checks
Identify combinations the storefront actually publishes
Do not multiply theoretical option counts and assume those routes exist. The reviewer inventories actual filter links, server responses, parameters and product-listing differences. Logged crawler requests show which patterns are repeatedly discovered rather than merely mathematically possible.
What to verify: A verified URL-class inventory separates stable curated paths from sorting, stock, pagination and uncontrolled combinations.
Choose a crawl and index rule for each URL class
A stable filtered collection may have genuine search value if it offers unique buying guidance; an arbitrary sort state usually does not. The team records whether robots disallow, canonical or noindex is suitable and remembers that blocked URLs cannot reveal their page-level noindex directive.
What to verify: A class-by-class decision register explains the intended indexation, crawler access and any exceptions requiring platform-specific validation.
Prove product discoverability after limiting unnecessary routes
Before and after the change, follow actual HTML links from the category to products on later pages. Re-run filter controls on mobile and inspect any fallback navigation. Avoid a global block that leaves important products discoverable only through disallowed filters.
What to verify: A post-release crawl confirms the approved product paths remain reachable, while filtered URLs produce the documented response and metadata.
Expected output: The useful deliverable is a filter-class policy with real source URLs, preserved product discovery paths and a tested change record. It is not a claim of improved rankings or a blanket rule that every facet must be hidden. The system should stay easy to shop, while technical owners can explain why each crawlable combination exists.
This faceted navigation crawl budget review scenario is a fictional technical example, not a completed inspection or measured result for a merchant.
Inspect the discovered URL patterns
Use a representative crawl and available server logs to group colour, size, sorting and availability parameters. Record which classes are linked in HTML and which are invented by crawling combinations. A high URL count indicates where to investigate; it is not proof that all filters are unhelpful.
Reference: official · 2026-11-07
Choose a rule for each filter class
Google's faceted navigation guide discusses robots rules for unnecessary crawling. Distinguish crawl prevention from page-level noindex, which must be fetched to be seen. Keep genuinely useful filter destinations only where they have a stable customer job, distinct content and a coherent canonical URL.
Reference: official · 2026-11-07
Verify product discovery afterwards
Test mobile filter controls, pagination and reachability of products from indexable categories before and after changes. Treat uncertainty as a documented exception rather than applying a broad disallow that might hide useful inventory. Compare the same URL samples and record the chosen release owner.
Reference: official · 2026-11-07
Decision aid
Decision table: Faceted URLs: diagnose a catalogue crawl trap
| Condition to investigate | Recommended decision | Proof to gather |
|---|---|---|
| Sort=price reorders the same products | Normally treat as a presentation choice rather than a separate indexable collection. | Inspect the emitted links, canonical and actual sorted content without creating another landing page. |
| Colour=blue changes the displayed subset | Require a buyer and content case before promoting this to a stable category. | Check actual demand, product assortment and editorial value compared with its parent. |
| Stock=available fluctuates by inventory state | Make a deliberate transient-state policy and test when products enter and leave stock. | Capture response, canonical and product discovery under both stocked and unavailable conditions. |
What a passing test looks like
These checks describe the evidence that should exist after a real implementation. They are not reports of tests already completed on a customer website.
Identify combinations the storefront actually publishes
Expected result: A verified URL-class inventory separates stable curated paths from sorting, stock, pagination and uncontrolled combinations.
Keep as evidence: Keep a dated sample of real URLs, final HTTP and HTML output, and the reviewer decision for the identify combinations the storefront actually publishes.
Choose a crawl and index rule for each URL class
Expected result: A class-by-class decision register explains the intended indexation, crawler access and any exceptions requiring platform-specific validation.
Keep as evidence: Keep a dated sample of real URLs, final HTTP and HTML output, and the reviewer decision for the choose a crawl and index rule for each url class.
Prove product discoverability after limiting unnecessary routes
Expected result: A post-release crawl confirms the approved product paths remain reachable, while filtered URLs produce the documented response and metadata.
Keep as evidence: Keep a dated sample of real URLs, final HTTP and HTML output, and the reviewer decision for the prove product discoverability after limiting unnecessary routes.
Scope and limitations
This illustration cannot tell you how often Google crawls a real catalogue, which facet has search demand or whether a platform supports a particular directive. Server logs, linked HTML and the actual faceted implementation must be inspected. The formula in the separate calculator models possibility, not live route volume.
Sources and verification
These references explain documented technical behaviour. The worked decisions are illustrative until checked on your own installation.
- official reference — Google's faceted navigation documentation explains how parameter combinations can generate huge crawl spaces and outlines URL and crawling controls. Recorded: 2026-11-07