Technical Seo

Technical SEO Audit — Phase 2

Executive Summary

Technical SEO is the infrastructure layer that determines whether all other optimization work is visible to Google. A technically broken site cannot rank regardless of content quality or link authority. In 2025, technical SEO has expanded to include three new critical areas: (1) INP (Interaction to Next Paint), which replaced FID as a Core Web Vital in March 2024 — target 500ms Poor

• LCP: 4.0s Poor
• CLS: 0.25 Poor
• TTFB: 1,800ms Poor
• Crawl budget (large sites): indexed pages should be within 20% of sitemap count
• HTTPS: required — any HTTP-accessible URL is a trust and ranking signal issue
• Canonical coverage: every page must have a canonical (self-referencing or pointing to preferred)
• Structured data: Organization + LocalBusiness schema on homepage; FAQPage on all service pages; HowTo where applicable
• Hreflang: required for any multi-language or multi-region site — errors cause ranking in wrong country
• Site speed: Google uses mobile-first indexing — test mobile CWV separately from desktop

Numbered Action Plan:

Immediate (Week 1, No Dev Needed)

1. Run site crawler — python3 scripts/site_crawler.py --url [URL] --max-pages 100 --output {DATA_DIR}/crawl/ --csv. Captures all pages, status codes, titles, meta descriptions, H1s, word counts, internal links. Effort: 30 min setup + crawl time.
2. Check GSC Coverage report — identify Excluded, Crawled Not Indexed, and Discovered Not Indexed pages. Flag any page type systematically excluded. Effort: 20 min.
3. Validate robots.txt — curl https://domain.com/robots.txt. Check for accidental Disallow: / rule. Verify sitemap URL is listed. Effort: 5 min. Priority: 25 (5×5).
4. Validate XML sitemap — check all sitemap URLs return 200, are in canonical form, and match GSC indexation count. Flag if sitemap includes redirect or noindex URLs. Effort: 15 min.
5. Test HTTPS enforcement — navigate to http://domain.com (no https) — should 301 redirect to HTTPS. Check SSL expiry date. Effort: 5 min.

Short-Term (Week 2–4, Dev Involvement)

1. Fix redirect chains — use curl -IL [URL] to trace chains. Any 301→301→200 chain = wasted PageRank. Flatten to direct 301. Effort: 30 min identification + 1–2 hrs dev. Priority: 20.
2. Audit canonical tags — check for: missing canonicals, self-conflicting canonicals, canonicals pointing to redirect URLs, paginated pages with wrong canonical. Effort: 1–2 hrs.
3. Fix duplicate content — www vs non-www, http vs https, trailing slash vs no trailing slash — all should 301 to one canonical version. Check: site:domain.com -www. Effort: 1 hr dev.
4. Implement missing structured data — add Organization + LocalBusiness schema to homepage, FAQPage to all service pages, HowTo to process pages. Use Google's Rich Results Test to validate. Effort: 2–4 hrs.
5. Test JavaScript rendering — check key content (services, contact info) via Google Search Console → URL Inspection → "View Tested Page" → rendered HTML. Any missing content = JS rendering issue. Effort: 30 min.

---

Step 1: Read Project Context & Run Crawl

Read {AUDIT_DIR}/intake-data.md for business name, URL, and project paths.

# Run site crawler for comprehensive technical data
python3 scripts/site_crawler.py \
  --url [WEBSITE_URL] \
  --max-pages 100 \
  --output {DATA_DIR}/crawl/ \
  --csv

# Validate URL + basic signals
python3 scripts/check_url.py --url [WEBSITE_URL] --full

Use crawler output (crawl-issues-*.json) to populate findings. Supplement with:

Tools for this phase: | Tool | Purpose | Cost | |------|---------|------| | Screaming Frog SEO Spider | Full site crawl — redirects, canonicals, status codes, schema, headings | Paid/Free (≤500 URLs) | | Google Search Console | Index coverage, manual actions, Core Web Vitals report, URL inspection | Free (requires access) | | Ahrefs Site Audit | Crawl health score, broken links, orphan pages, crawl depth issues | Paid | | SEMrush Site Audit | Technical health score, crawlability, HTTPS, Core Web Vitals | Paid | | Google Rich Results Test | Schema validation — confirms rich result eligibility per page | Free | | Schema Markup Validator (validator.schema.org) | Schema syntax validation without rich result eligibility check | Free | | Lighthouse | CWV, best practices, PWA check — per-page technical baseline | Free | | Wayback Machine | Historical snapshots — detect major content/structure changes | Free |

---

Section 1: Crawlability & Indexation

Robots.txt Analysis

Fetch [domain]/robots.txt → check:

• File exists and returns HTTP 200?
• Blocking critical resources (CSS, JS files needed for rendering)? → Use Google URL Inspection → View Rendered Page to verify
• Correct User-agent directives (Googlebot, GPTBot, PerplexityBot if desired)?
• Sitemap(s) declared (Sitemap: https://domain.com/sitemap.xml)?
• No accidental Disallow: / on production site?
• Rate limiting: Crawl-delay directive set? (optional — Google mostly ignores it)

AI Crawler Access Check (14 Crawlers)

AI crawlers are now as important as Googlebot for visibility. Over 35% of top 1,000 websites inadvertently block at least one major AI crawler (Originality.ai 2025). Check robots.txt for all 14 AI crawlers:

Tier 1 — AI Search (MUST ALLOW for AI visibility):

| Crawler | User-Agent | Operator | Check Result | |---------|-----------|----------|-------------| | GPTBot | GPTBot | OpenAI (ChatGPT Search, 900M+ users) | Allowed / Blocked / Not Mentioned | | OAI-SearchBot | OAI-SearchBot | OpenAI (search-only, no training) | | | ChatGPT-User | ChatGPT-User | OpenAI (user-initiated browsing) | | | ClaudeBot | ClaudeBot | Anthropic (Claude web search) | | | PerplexityBot | PerplexityBot | Perplexity (best AI referral traffic) | |

Tier 2 — AI Ecosystem (RECOMMEND ALLOW):

| Crawler | User-Agent | Operator | Notes | |---------|-----------|----------|-------| | Google-Extended | Google-Extended | Google — Gemini training. Blocking does NOT affect search ranking or AIO appearance. | | GoogleOther | GoogleOther | Google — research, experimental features | | Applebot-Extended | Applebot-Extended | Apple — Apple Intelligence (2B+ devices) | | Amazonbot | Amazonbot | Amazon — Alexa, Amazon AI | | FacebookBot | FacebookBot | Meta — Meta AI (3B+ app users) |

Tier 3 — Training Only (strategy-dependent):

| Crawler | User-Agent | Recommendation | |---------|-----------|---------------| | CCBot | CCBot | Context-dependent (Common Crawl training data) | | anthropic-ai | anthropic-ai | Context-dependent (Claude training, separate from ClaudeBot) | | Bytespider | Bytespider | BLOCK for most Western businesses (aggressive, low value) | | cohere-ai | cohere-ai | Context-dependent (enterprise AI training) |

Also check for:

• `` on key pages (blocks all AI use)
• X-Robots-Tag: noai HTTP headers
• Bot-specific meta tags: ``
• JavaScript-rendered content (AI crawlers do NOT execute JS — content invisible to them)

AI Crawler Access Score: Tier 1 all allowed = 50 pts; Tier 2 all allowed = 25 pts; no blanket AI blocks = 15 pts; llms.txt present = 10 pts. Total: /100.

IndexNow Protocol Check

IndexNow enables near-instant indexing on Bing/Copilot when content is published or updated. ChatGPT Search also uses Bing's index, so IndexNow benefits both platforms.

| Check | Status | |-------|--------| | IndexNow key file at /.well-known/indexnow-key.txt or /[key].txt | Present / Absent | | IndexNow API integration (pings on publish/update) | Active / Not implemented | | Bing Webmaster Tools verified | Yes / No | | Sitemap submitted to Bing | Yes / No |

Implementation: Add IndexNow key file + CMS plugin (WordPress: IndexNow plugin; custom: POST to https://api.indexnow.org/indexnow on content change). Effort: 30 min. Impact: near-instant Bing/Copilot indexation vs. waiting days for natural crawl.

llms.txt Check

Check for /llms.txt (emerging standard — tells AI systems what content is most useful):

• File exists at [domain]/llms.txt? → Present / Absent
• If present: does it list key service pages with descriptions?
• Extended version at /llms-full.txt? → Present / Absent
• If absent: flag as GEO opportunity — fewer than 5% of sites have this (early adopter advantage). See Phase 14 (AI SEO) for generation protocol.

XML Sitemap Audit

Fetch /sitemap.xml and /sitemap_index.xml: | Check | Pass/Fail | Notes | |-------|----------|-------| | Returns HTTP 200 | | | | Submitted to GSC | | GSC → Sitemaps | | Contains all canonical, indexable URLs | | | | Excludes 404s, 301s, noindex pages | | Run Screaming Frog → filter sitemap vs. crawl | | lastmod dates accurate (not all identical) | | Identical = auto-generated = Google ignores | | File size 10)

**Healthy indexed:sitemap ratio:** Within 20% (e.g., 50 page sitemap → 40–60 indexed is healthy)

Over-indexed (>120%): orphan pages, parameter variants, staging pages leaking
Under-indexed (800ms waste significant crawl budget.

### Orphan Pages & Crawl Depth
- Orphan pages (0 internal links): list from crawler output → are any important pages orphaned?
- Crawl depth: key service/location pages within ≤3 clicks of homepage?
- Use Screaming Frog → Crawl Analysis → Pages by Click Depth → flag depth >4

---

## Section 2: Site Architecture & URL Structure

### URL Quality Assessment
| Check | Pass/Fail | Example Issue |
|-------|----------|--------------|
| Lowercase URLs only | | `/Services/` instead of `/services/` |
| Hyphens as separators (not underscores) | | `/drain_cleaning/` instead of `/drain-cleaning/` |
| Descriptive, keyword-inclusive paths | | `/page-1/` instead of `/drain-cleaning-chicago/` |
| No dynamic parameters on canonical pages | | `/service.php?id=4` instead of `/drain-cleaning/` |
| Consistent trailing slash (all or none) | | Mix of `/about/` and `/about` → duplicate |
| Max 3-click depth for all critical pages | | Service pages at depth 5+ |

### Internal Link Architecture
- Logical parent-child structure (`/services/[service-name]/`)?
- Service area pages nested correctly (`/locations/[city-state]/`)?
- Breadcrumb structure matches URL hierarchy?
- BreadcrumbList schema on all inner pages?

---

## Section 3: HTTPS & Security Headers

| Check | Pass/Fail | Details |
|-------|----------|---------|
| SSL certificate valid + not expiring within 30 days | | Expiry: [date] |
| All HTTP pages 301 redirect to HTTPS | | Test: `curl -I http://domain.com` |
| www to non-www (or vice versa) 301 canonical redirect | | One canonical version only |
| No mixed content (HTTP resources on HTTPS pages) | | Chrome DevTools → Console → Mixed Content warnings |
| HSTS header (`Strict-Transport-Security`) | | `max-age=31536000; includeSubDomains` |
| `X-Frame-Options` or `CSP frame-ancestors` (clickjacking) | | |
| `X-Content-Type-Options: nosniff` | | |
| `Referrer-Policy` set | | |

---

## Section 4: Redirect Management

From Screaming Frog → Response Codes → filter 3xx:

| Issue | Count | Impact | Fix |
|-------|-------|--------|-----|
| Redirect chains (A→B→C) | | High | Update to direct A→C |
| Redirect loops (A→B→A) | | Critical | Fix immediately |
| 302 used where 301 needed (permanent moves) | | Medium | Change to 301 |
| Old migration redirects returning 200 at destination? | | Low | Verify integrity |
| Redirects losing URL slug (redirecting to homepage) | | High | Fix to correct canonical |

**301 vs. 302 rule:** 301 = permanent (passes ~99% link equity); 302 = temporary (withholds equity — misuse = ranking loss). All site restructuring must use 301.

---

## Section 5: Canonical Tags

From Screaming Frog → Directives → filter Canonical:

| Check | Pass/Fail |
|-------|----------|
| Self-referencing canonicals on all pages | |
| No conflicting canonical + noindex on same page (contradictory directives) | |
| Parameter variants canonicalize to clean URL | |
| Canonical in `` not body | |
| Canonical matches exactly (HTTPS, www preference, trailing slash) | |
| Cross-domain canonicals (for syndicated content) | |
| Pagination: self-canonical OR canonical to page 1 (depending on uniqueness) | |

---

## Section 6: Structured Data / Schema

**Validate with:** Google Rich Results Test + Schema Markup Validator per page type.

**Required for local businesses:**
| Schema Type | Present? | Validates? | Rich Result Eligible? | Priority |
|------------|---------|-----------|----------------------|---------|
| LocalBusiness | ✅/❌ | ✅/❌ | ✅/❌ | Critical |
| Organization (homepage) | ✅/❌ | ✅/❌ | N/A | Critical |
| BreadcrumbList (all inner pages) | ✅/❌ | ✅/❌ | ✅/❌ | High |
| Service (per service page) | ✅/❌ | ✅/❌ | N/A | High |
| FAQPage (service + location pages) | ✅/❌ | ✅/❌ | ✅/❌ | High — AIO trigger |
| AggregateRating (if reviews displayed) | ✅/❌ | ✅/❌ | ✅/❌ | High |
| HowTo (instructional pages) | ✅/❌ | ✅/❌ | ✅/❌ | Medium |
| Article / BlogPosting | ✅/❌ | ✅/❌ | ✅/❌ | Medium |
| VideoObject (if video content) | ✅/❌ | ✅/❌ | ✅/❌ | Medium |
| Person (author pages) | ✅/❌ | ✅/❌ | N/A | Medium |
| Event (if applicable) | ✅/❌ | ✅/❌ | ✅/❌ | Low |

**LocalBusiness schema must include:** `@type`, `@id` (unique URL with fragment), `name`, `url`, `telephone`, `address` (with all PostalAddress sub-properties), `geo` (GeoCoordinates), `openingHoursSpecification`, `image`, `priceRange`, `sameAs` (array), `description`, `hasMap`, `parentOrganization` (not `branchOf` — deprecated 2025).

**FAQPage schema (AIO optimization):** Highest-impact schema for AI Overview inclusion — add 3–5 FAQ entries per service and location page targeting "People Also Ask" questions.

---

## Section 7: JavaScript Rendering

| Check | Pass/Fail | Tool |
|-------|----------|------|
| Meaningful content visible in View Source (no JS required) | | View Source |
| Google renders page correctly | | GSC → URL Inspection → View Rendered Page |
| Navigation links crawlable without JS | | Screaming Frog + JS disabled crawl |
| Lazy-loaded images use `loading="lazy"` with valid `src` | | Screaming Frog → Images |
| SPA framework: SSR or SSG used? | | Check Next.js/Nuxt/Gatsby config |
| No critical content only in AJAX calls Googlebot can't access | | Compare View Source vs. Rendered |

---

## Section 8: Core Web Vitals — Technical Assessment

**2025 CWV Thresholds (INP replaced FID — March 2024):**
| Metric | Good | Needs Improvement | Poor | Target |
|--------|------|------------------|------|--------|
| **LCP** | 2.5s | 200ms | 0.1 | 800ms | 200ms (CWV failing) | 4 | 3 | 12 | 4–16 hrs |
| Redirect chains (A→B→C) | 3 | 4 | 12 | 1–2 hrs |
| Mixed content (HTTP on HTTPS) | 3 | 4 | 12 | 1–4 hrs |
| Canonical conflicts (canonical + noindex) | 4 | 4 | 16 | 1–2 hrs |
| JavaScript-only nav (Googlebot can't crawl) | 5 | 3 | 15 | 4–16 hrs |
| Tier 1 AI crawlers blocked in robots.txt | 4 | 5 | 20 | 15 min |
| IndexNow not implemented (Bing/Copilot delay) | 3 | 5 | 15 | 30 min |
| llms.txt missing (AI discoverability gap) | 3 | 4 | 12 | 1–2 hrs |
| JS-rendered content invisible to AI crawlers | 4 | 3 | 12 | 4–8 hrs |

---

## Per-Issue Output Format

Status: ✅ Pass | ⚠️ Warning | ❌ Fail Issue: [Specific description with URL examples] Impact: Critical | High | Medium | Low Priority Score: [Impact 1-5] × [Feasibility 1-5] = [X] Fix: [Exact steps — no vague advice] Effort: Quick Fix (500ms (Core Web Vital) | 🔴 Critical | 4–8 hrs dev | 20 | PageSpeed Insights | | Duplicate content (www/non-www/http/https) | 🟠 High | 1 hr dev | 20 | Screaming Frog | | Canonical tag missing or wrong | 🟠 High | 1–2 hrs | 20 | Screaming Frog | | Redirect chains (3+ hops) | 🟠 High | 2 hrs dev | 16 | curl -IL | | JavaScript content not rendering | 🟠 High | 4–8 hrs dev | 16 | GSC URL Inspection | | Missing structured data (LocalBusiness/FAQPage) | 🟠 High | 2–4 hrs | 16 | Rich Results Test | | Tier 1 AI crawlers blocked (GPTBot/ClaudeBo

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mshahiddigital
• Source: mshahiddigital/agentic-local-seo-audit
• License: MIT

Install and usage instructions live in the source repository linked above.