Screaming Frog SEO Spider: The Guide to Technical Website Audits

What Is the Screaming Frog SEO Spider?

The Screaming Frog SEO Spider is a desktop program that crawls your website to make technical SEO problems visible. You install it locally on Windows, macOS, or Ubuntu. It works like a search engine crawler: it follows links, collecting data on every URL as it goes – from status codes to meta tags to headings and links – and turns it into reports you can evaluate.

It is exactly this ability to quickly analyze large volumes of URLs that has made the SEO Spider the standard tool for technical SEO. Whether you are hunting broken links, checking redirects, or optimizing title tags: Screaming Frog delivers the data foundation for well-founded decisions.

Getting Started

Installation and License

Getting started is straightforward. You download the SEO Spider from the official Screaming Frog website and follow the installer. The free version crawls up to 500 URLs per website – enough for small projects or a first impression.

Anyone who wants to crawl more URLs, save and reopen crawls, or use advanced configurations needs a paid annual license. You will find the current prices and the full feature comparison directly on the vendor's site – check them there, they change from year to year. After purchase, you enter your license key under Licence > Enter Licence Key and restart the application.

Choosing a Storage Mode

For larger crawls, the storage mode determines performance:

• Database Storage: recommended if your machine has an SSD. It crawls more URLs, saves automatically, and lets you quickly access earlier crawls. Found under File > Settings > Storage Mode.
• Memory Storage: useful without an SSD. Crawls can only be saved as long as enough memory is free.

Starting and Configuring a Crawl

There are two main ways to start a crawl.

In standard Spider mode, you enter the start URL and click "Start". The Spider crawls this URL and all internal links on the same subdomain. Progress runs in real time, and you can pause and resume at any point.

In List mode (Mode > List), you upload or paste a fixed list of URLs – ideal for purposefully checking an XML sitemap, a migration table, or a backlink list.

Important Configurations

By default, Screaming Frog crawls much like Google. Under Configuration, you adjust the behavior:

• JavaScript rendering: Many modern websites load content via JavaScript. Under Configuration > Spider > Rendering, you enable rendering – the Spider then uses a built-in Chromium to render pages like a browser. For JS-heavy pages this is mandatory, otherwise you will be missing content and links.
• Crawl subdomains: Configuration > Spider > Crawl all Subdomains captures all subdomains of the main domain.
• Include and exclude: With "Include" and "Exclude", you use regular expressions to control which URL paths get crawled.
• Large websites: If needed, increase the memory under Configuration > System > Memory Allocation and disable crawling of images, CSS, or JavaScript when you only need the HTML structure.
• Speed: Under Configuration > Speed, you limit threads and requests per second so as not to overload weaker servers.
• User-Agent: Under Configuration > User-Agent, you simulate, for example, the Googlebot or Bingbot.

Understanding Crawl Data

As soon as the crawl is running, the tabs of the interface fill up in real time. The view is split in two: the top window lists all crawled URLs with columns such as status code, content type, page title, and word count. This data is grouped by topic:

• Internal / External: all internal and external URLs, respectively.
• Response Codes: URLs by HTTP status (2xx, 3xx, 4xx, 5xx).
• Page Titles / Meta Description / H1 / H2: the respective on-page elements.
• Images, Directives, Canonicals, Hreflang: images, robots directives, canonicals, and multilingual markup.

When you click a URL at the top, the bottom window shows details: which pages link to it (Inlinks), where it links to (Outlinks), embedded images, and loaded resources.

On the right, two areas help you get a quick overview. The Overview summarizes the most important data points across all tabs – a click on a value jumps straight to the matching filter. The Issues tab proactively highlights problems, warnings, and opportunities, classified by type (Issue, Warning, Opportunity) and priority (High, Medium, Low), each with an explanation and a suggested fix. This way you find critical points without going through every tab one by one.

Concrete Analyses

Finding broken links: In the Response Codes tab, you filter by client errors (4xx) and server errors (5xx). Via the Inlinks view, you see which pages link to the broken URLs – so you can fix them in a targeted way.

Checking redirects: The Redirect Chains report uncovers redirect chains and loops. Make sure internal links point directly to the final destination instead of running through unnecessary redirects.

Titles and meta descriptions: In the Page Titles and Meta Description tabs, you filter by elements that are missing, too long, too short, or duplicated.

Duplicate content: Screaming Frog detects exact duplicates via hash values and near-duplicates via similarity algorithms (to be enabled under Configuration > Content > Duplicates).

Images and alt text: In the Images tab, you filter by images above a certain size or without alt text.

Directives and canonicals: The Directives tab shows URLs with noindex or nofollow, the Canonicals tab checks the canonical implementation.

Creating and Checking XML Sitemaps

You check an existing XML sitemap in List mode: upload the file or paste the URLs, crawl, and then uncover errors, redirects, or non-indexable URLs within the sitemap. You generate a new sitemap after a complete crawl under Sitemaps > XML Sitemap – including options for last-modified date, priority, and a separate image sitemap.

Advanced Features

For special cases, Screaming Frog offers powerful tools:

• Custom Extraction: With XPath, CSS Path, or regex, you pull any data out of the HTML – such as product prices, author names, or social meta tags. Configured under Configuration > Custom > Extraction. An XPath example that extracts the text of the first H1:

• Custom Search: Under Configuration > Custom > Search, you search for text strings or code snippets in the source code – for example, to check whether a tracking code is present on all pages.
• API integrations: Connect the Spider with Google Analytics 4, Google Search Console, and PageSpeed Insights. This way you pull user, performance, and Core Web Vitals data directly into the tool for every crawled URL.
• Scheduling and comparing crawls: Licensed users schedule automatic crawls and compare two crawls with each other – ideal for checking changes over time or staging against live.
• AI prompts during the crawl: Newer versions let you apply ChatGPT or LLM prompts to page elements during the crawl – for example, to generate alt-text suggestions or to analyze text content. This automates tasks that used to be purely manual.

Exporting Data and Saving Crawls

You export all data from the top window via "Export" as CSV, Excel, or Google Sheet. For large volumes of data – such as all inlinks to 4xx pages – you use the Bulk Export in the main menu. In Database Storage mode, crawls are saved automatically and can be reopened, renamed, or duplicated under File > Crawls.

Screaming Frog for Competitive Analysis

Even though it is primarily a technical tool: you can crawl a competitor's website and sort by the "Inlinks" column in the Internal tab. Pages with an especially high number of internal links are often the central hub or pillar pages – a hint at the competitor's content strategy. Via Bulk Export > Links > All Anchor Text, you additionally export all anchor texts and see which keywords are internally emphasized.

Conclusion

The Screaming Frog SEO Spider is a versatile workhorse for technical SEO. From quick troubleshooting through the analysis of individual page elements to API integrations and AI-supported prompts, it covers the entire range of an audit. Whether regular maintenance, a one-off audit, or competitive analysis – it delivers the data on which good optimization decisions are built.