Screaming Frog SEO Spider is a desktop crawler that fetches your site the way a search engine would, then lays out every URL, status code, title, heading, canonical, redirect and directive in a spreadsheet you can interrogate. For technical SEO it is close to indispensable, because it turns vague suspicions about a site into a precise list of problems. The free version crawls up to five hundred URLs, which is enough for small sites and for learning, while the paid licence removes that limit and unlocks the configuration and integration features that make serious audits possible.

How We Use Crawl Data For Clients

A crawl produces thousands of rows, and the skill is deciding which fifty matter. At AAMAX.CO we run crawls as the first step of every technical engagement, then convert the output into a prioritised roadmap ordered by revenue impact rather than issue count, and implement the fixes ourselves. Our SEO services combine crawl-driven technical work with content and authority strategy, and because we deliver web development in-house we can resolve template-level problems properly instead of documenting them. We support clients worldwide and we focus on the issues that actually move rankings.

Configure Before You Crawl

The most common mistake is hitting start immediately. Configuration determines whether your crawl reflects reality. First, decide on rendering: if the site relies on JavaScript to display content or links, enable JavaScript rendering under the spider configuration, otherwise the crawler sees an empty shell and reports nonsense.

Set a sensible crawl speed. Default threading can overwhelm modest hosting, so throttle requests if you are crawling a live production site, and crawl outside peak hours where possible. Decide whether to respect robots.txt: respecting it shows what search engines see, while ignoring it reveals what you are blocking, and both views are useful at different moments.

Choose your user agent deliberately, since some sites serve different responses to different agents, and configure include or exclude rules to keep the crawl focused. Excluding infinite faceted navigation parameters on a large ecommerce site is often the difference between a usable crawl and one that never finishes.

Crawl Modes And When To Use Them

Spider mode starts from a URL and follows links, which mirrors organic discovery and is the default for audits. List mode takes a supplied set of URLs and checks only those, which is ideal for validating a redirect map during a migration, checking a list of pages from Search Console, or auditing a sitemap.

Sitemap mode crawls the URLs in your XML sitemap, which is how you verify that everything you have declared as important actually resolves correctly. Comparing a spider crawl against a sitemap crawl is one of the fastest ways to find orphaned pages and sitemap entries that no longer exist.

The Reports That Matter Most

Start with response codes. Group by status: server errors demand immediate attention, client errors reveal broken internal links, and redirects need checking for chains and loops. Every internal link should point to a final destination, not through two hops.

Move to page titles and meta descriptions, filtering for missing, duplicate, too long and too short. Duplicates usually indicate a template problem rather than an editorial one, which makes them cheap to fix at scale. Check H1 tags for missing and multiple instances, then review canonicals for pages canonicalised to unexpected URLs, which is a frequent cause of pages silently dropping out of the index.

Inspect directives to find noindex and nofollow tags, particularly any left behind from development. Review the images tab for oversized files and missing alt text. Then look at internal link counts to identify pages with very few inbound internal links, which is where your orphan and depth problems live.

Crawl Depth, Structure And Orphans

The crawl depth column tells you how many clicks each page sits from your starting point. Important pages buried at depth five or six are crawled rarely and rank poorly, and the fix is almost always internal linking rather than content. Sort by depth and check whether your priority pages are where you assume they are.

To find orphans properly, connect Search Console and Analytics through the API access configuration before crawling. Screaming Frog will then flag URLs that receive traffic or impressions but were not reachable by internal links, which is the most reliable orphan detection available in the tool.

Advanced Uses Worth Learning

Custom extraction is the feature that separates casual users from confident ones. Using CSS path or XPath selectors, you can pull any element from every page in a crawl: publication dates, author names, product prices, stock status, schema properties or the presence of a specific tracking script. This turns the crawler into a site-wide auditing engine for whatever matters to your project.

Custom search lets you find pages containing or missing a given string in the source, which is how you verify analytics implementation or locate leftover staging references. Structured data validation checks schema across the whole site rather than one URL at a time. And connecting PageSpeed Insights via API brings performance data alongside the crawl, so you can see which slow pages also receive traffic.

Turning Findings Into A Roadmap

A raw export is not an audit. Prioritise by impact and effort. Anything preventing indexing of revenue pages comes first: server errors, accidental noindex tags, wrong canonicals, blocked resources. Next come issues affecting many pages through templates, since one fix resolves thousands of rows.

Then handle redirect chains, broken internal links and orphaned important pages. On-page issues such as duplicate titles follow, prioritised by which pages already receive impressions. Cosmetic issues on pages with no traffic and no strategic value belong at the bottom, and it is entirely reasonable to leave some of them undone.

Making Crawls A Habit

Crawl regularly rather than once. Sites drift: plugins add parameters, redesigns break links, content teams publish orphans, and a single deployment can introduce a sitewide directive error. Monthly crawls of smaller sites and scheduled crawls of large ones catch regressions while they are cheap to fix.

Save your configuration so every crawl is comparable, export the key tabs each time, and track issue counts over time. That trend line is a better measure of technical health than any single audit. Learn Screaming Frog properly and you gain the ability to diagnose almost any technical SEO problem in an afternoon. If you would rather have that diagnosis and the fixes delivered together, our team is ready to help.