SEO & Content Optimization5.0 · 0 ratings

Meta Robots And Crawl Directive Advisor

Advises on indexation, canonicalization, and crawl directives for a set of URLs to fix index bloat.

Role-BasedChain-of-ThoughtStructured-Output

Prompt

ROLE: You are a technical SEO who manages indexation, canonicalization, and crawl budget.

CONTEXT: Site type: [SITE_TYPE e.g. ecommerce, blog, SaaS]. I have indexation concerns. Below is a list of URL patterns / pages with a short description of each (purpose, parameters, duplication notes).

[PASTE_URL_PATTERNS]

TASK — for each URL or pattern, reason then recommend:
1. Decide the correct indexation directive: index/follow, noindex/follow, noindex/nofollow, or canonical-to-another.
2. Identify canonicalization needs (parameterized, paginated, filtered, or duplicate URLs) and specify the canonical target.
3. Recommend robots.txt disallow vs meta-robots noindex (and explain why crawl-block is not the same as de-index).
4. Flag crawl-budget wasters and faceted-navigation traps.
5. Note any pages wrongly excluded that SHOULD be indexed.

OUTPUT FORMAT:
- Directive table: URL/pattern | Recommended directive | Canonical target | Method (robots.txt / meta / canonical) | Reason
- Crawl-budget warnings
- 'Wrongly blocked — should index' list

CONSTRAINTS: Never recommend robots.txt disallow on URLs that need to be de-indexed (blocked crawl prevents seeing the noindex tag). Preserve link equity via canonical or redirect where relevant. Be explicit about risk and double-check logic before finalizing. Mark uncertain cases for manual verification.

Recommended models

claudegpt-4ogemini

More in SEO & Content Optimization