Technical SEO is the least glamorous part of SEO, and also the part that quietly breaks everything else when it’s wrong. You can write the best content in your industry and build a perfect backlink profile — but if Google can’t crawl, render, or index your pages properly, none of it matters.

Crawling vs. Indexing: The Distinction Most People Miss

Crawling is Google’s bots visiting and reading a page. Indexing is Google deciding that page is worth storing and potentially showing in search results. A page can be crawled but not indexed (Google saw it, decided it wasn’t worth including), or blocked from crawling entirely (Google never even looked). Fixing the wrong one wastes time — you need to know which problem you actually have, and Search Console’s URL Inspection tool tells you exactly that.

XML Sitemaps: Your Map for Google

An XML sitemap is a file listing every URL you want indexed, submitted directly to Google Search Console. It doesn’t guarantee indexing, but it dramatically speeds up discovery — especially for new sites or new pages that don’t yet have many internal links pointing to them. The format itself is defined by the independent Sitemaps.org protocol, which every major search engine — not just Google — reads from.

Robots.txt: Small File, Big Consequences

Your robots.txt file sits at yourdomain.com/robots.txt and tells search engines which parts of your site they’re allowed to crawl. It’s a plain text file, editable in seconds — which is exactly why it’s so easy to break by accident. A single misplaced Disallow: / line blocks your entire site from Google, and I’ve seen this exact mistake take a site’s traffic to zero overnight. The full spec — now a formal internet standard as of RFC 9309 — is worth a skim if you want to know exactly what every directive does.

Canonical Tags: Telling Google Which Version Is Real

Duplicate or near-duplicate content is more common than most site owners realize — a product page accessible via three different URL parameters, an HTTPS and HTTP version both live, a “www” and non-“www” version both resolving. A canonical tag (<link rel="canonical" href="...">) tells Google which version is the “real” one to index and rank, consolidating ranking signals that would otherwise be split across duplicates. Google’s own canonicalization documentation covers exactly how it chooses a canonical when you don’t specify one — and it doesn’t always pick the one you’d want.

Mobile-First Indexing: The Default, Not the Exception

Google confirmed in its October 2023 announcement that mobile-first indexing had landed for essentially the entire web — meaning Google predominantly uses the mobile version of your site’s content for indexing and ranking, not desktop. This means whatever your mobile version is missing — content hidden behind accordions that don’t load, images without alt text, a stripped-down navigation — is what Google actually sees, even if your desktop version is flawless. Test your actual mobile rendering in Search Console’s URL Inspection tool, not just how it looks on your own phone.

Common Technical SEO Mistakes That Quietly Kill Rankings

• Noindex tags left on from a staging environment, forgotten after launch
• Pagination without rel=”next”/”prev” or proper canonical handling, splitting authority across paginated pages
• Redirect chains (A → B → C instead of A → C directly), which waste crawl budget and dilute link equity
• Mixed content warnings after an HTTP-to-HTTPS migration that was never fully cleaned up
• JavaScript-rendered content that Googlebot can crawl but renders differently or slower than what users see

Site Architecture: Make It Shallow

Every important page should be reachable within 3 clicks from your homepage. Deep, buried pages get crawled less often and rank worse — not because of some arbitrary penalty, but because fewer internal links point to them and Google allocates less attention to pages that seem less important within your own site structure.

• Keep important pages within 3 clicks of the homepage
• Use a logical URL structure — /category/subcategory/page, not random IDs
• Fix orphan pages (pages with zero internal links pointing to them)
• Implement breadcrumb navigation, and mark it up with Breadcrumb schema
• Set canonical tags on any page with duplicate or near-duplicate content

Crawl Budget: When It Actually Matters

Crawl budget — how many pages Googlebot will crawl on your site in a given period — is a real constraint for large sites, generally 10,000+ pages, or sites that publish content extremely frequently. For a small business site with 50 pages, it’s rarely the bottleneck. Google’s own crawl budget documentation confirms this directly — don’t spend hours optimizing crawl budget on a site where it was never the limiting factor.

Migrations: Where Technical SEO Gets Dangerous

Site migrations — domain changes, HTTPS moves, replatforming — are where technical SEO mistakes cause the most damage, because they can affect every single page at once. The non-negotiables: map every old URL to its new equivalent with 301 redirects, update your sitemap before launch, and monitor Search Console daily for the first two weeks after going live.

Technical SEO pairs directly with on-page fundamentals — see the full Strategy Guide for how they work together.