Screaming Frog SEO Spider: The Guide to Technical Website Audits

What is the Screaming Frog SEO Spider?

The Screaming Frog SEO Spider is a desktop program that crawls your website like a search engine bot and makes technical errors visible in analyzable tables. Imagine an inspector walking through a large office building. He opens every door, presses every light switch, and notes down what is broken. The SEO Spider does exactly that for your website. It follows the links, collects data for each URL from status codes to headings, and stores everything clearly.

The program runs locally on Windows, macOS, or Linux. The crawl uses your own bandwidth and RAM. That distinguishes it from cloud crawlers on third-party infrastructure. The great benefit lies in the speed. A crawl across a few thousand pages takes only minutes. After that, you answer questions about broken links with two clicks. This data is the basis for any reliable SEO audit.

When you need it and when you do not

You need the SEO Spider as soon as your website has more than a few dozen pages. It shows you what a crawler actually finds there. For a tiny business card website, a look into the Search Console is enough. For backlink research, search volumes, or rankings, it is the wrong tool. It only ever sees the website itself and no external signals.

What the SEO Spider costs

The pricing question has two answers. Up to 500 URLs per crawl, the tool costs nothing. Beyond that, an annual license per user applies. You can find the current amount on the manufacturer's product page. We do not mention it here because license prices change. An outdated number when budgeting does more harm than good.

Free version: 500 URLs

The free version crawls completely, but stops at 500 URLs and leaves out some features. For a small website or a quick cross-check, it is perfectly sufficient. As soon as you want to save crawls or secure your own configurations, you hit the limit.

License: duration, what is included

The license is valid for one year and is tied to the user, not the device. The same person may install it on several of their own computers. If it expires, the program falls back to the 500 URL limit. You cannot open saved crawls after that. Anyone working in a team needs one license per person. From five licenses upwards, the manufacturer scales the unit price down. Always weigh the license against the cost of a large cloud suite.

Log File Analyser: second license, different task

Screaming Frog sells a second program called Log File Analyser. It reads server log files and shows which URLs the Googlebot actually requested. The SEO Spider, on the other hand, shows what a crawler might find. Both programs have separate licenses. A license for the SEO Spider therefore does not unlock the Log File Analyser. How you set up the evaluation is explained in our article on log file analysis.

First steps

Before you start, you need the right hardware and the correct storage mode.

System requirements: which hardware for which project size

The system requirements state a 64-bit system and at least 4 GB of RAM. That is the lower limit for a small crawl and not a recommendation for project work. RAM, SSD, and storage mode are the deciding factors. As a rule of thumb from our audits, this scaling applies.

Installation and license key

You download the SEO Spider from the manufacturer's website and follow the installer. You enter the license key under Licence > Enter Licence Key and restart the program. Only then are rendering, Custom Extraction, and scheduling available.

Choosing the storage mode

For larger crawls, the storage mode determines the performance. You find it under File > Settings > Storage Mode.

• Database storage mode. Recommended as soon as your computer has an SSD. It handles significantly more URLs and saves continuously.
• RAM storage mode. Useful without an SSD. Crawls can only be saved as long as enough RAM is free. A crash costs the entire run.

Starting and configuring a crawl

You can search a complete website or specifically check a list of URLs.

Spider mode and list mode

In the standard spider mode, you enter the start URL and click Start. The spider crawls this URL and all internal links of the same subdomain. The progress runs in real time, and you can pause at any time.

In the list mode under Mode > List, you upload a fixed URL list. This is the mode for an XML sitemap or a backlink list. In a relaunch project, we checked the complete old URL list this way. We found redirect chains across three stations that would have remained invisible in the normal crawl of the new website.

Important configurations

By default, the SEO Spider crawls similarly to Google. Under Configuration, you adapt the behavior to your needs.

• JavaScript rendering. Many websites load content via JavaScript. Under Configuration > Spider > Rendering, you activate the rendering. The spider then uses an integrated Chromium. For JS-heavy pages, this is mandatory, otherwise you will miss content and links.
• Subdomains. Configuration > Spider > Crawl > Crawl All Subdomains captures all subdomains of the main domain.
• Include and exclude. Via Include and Exclude, you control which paths are crawled using regular expressions. For shops, excluded filter parameters often save half the time.
• Large websites. Increase the memory under File > Settings > Memory Allocation. Disable images, CSS, and JavaScript if you only need the HTML structure.
• Speed. Under Configuration > Speed, you limit threads and requests per second so that weaker servers do not collapse.
• User-Agent. Under Configuration > User-Agent, you simulate Googlebot, Bingbot, or a mobile browser.

Saving and reusing configurations

The spider saves configurations as a file. Create three profiles, then you will never configure from scratch again. We work with exactly these three profiles.

1. Technical audit fast. Leaves out images, CSS, and JavaScript and delivers the overview of status codes in minutes.
2. Content inventory. Activates word count, headings, and Custom Extraction for author and date.
3. Migration in list mode. Checks an old URL list for redirect targets.

Understanding crawl data

The interface is divided into two parts and offers a quick overview on the right. The upper window lists all crawled URLs with columns like status code, content type, and title. This data is grouped into tabs.

• Internal and External. All internal or external URLs respectively.
• Response Codes. URLs by HTTP status, meaning 2xx, 3xx, 4xx, and 5xx.
• Page Titles, Meta Description, H1, H2. The respective on-page elements with length and duplicate filter.
• Images, Directives, Canonicals, Hreflang. Images, robots instructions, canonicals, and multilingual markup.

If you click on a URL at the top, the lower window shows the details. Inlinks names the pages that link to it. Outlinks shows the targets. On the right are two areas for a quick overview. The Overview summarizes the key figures of all tabs. A click on a value jumps directly into the matching filter. The Issues tab highlights problems and warnings, classified by priority and with a suggested solution.

Concrete analyses

Here you learn how to uncover the most common SEO errors with the spider.

• Finding broken links. In the Response Codes tab, you filter for client errors (4xx) and server errors (5xx). The inlinks view shows which pages link to the broken URL. That is exactly where you correct the link. You check external hits with 403 or 429 in the browser beforehand: Many websites block crawlers, the page itself is then accessible.
• Checking redirects. The Redirect Chains report uncovers chains and loops. Internal links should point directly to the final target. How you resolve chains is explained in our article on redirects and redirect chains.
• Titles and meta descriptions. In the Page Titles and Meta Description tabs, you filter for missing, too long, too short, or duplicate.
• Duplicate content. The spider recognizes exact duplicates via hash values. It finds near-duplicates via a similarity value under Configuration > Content > Duplicates.
• Images and alt texts. In the Images tab, you filter for images above a size threshold or without alt text.
• Directives and canonicals. The Directives tab shows URLs with noindex or nofollow. The Canonicals tab shows contradictory canonical specifications.

Creating and checking XML sitemaps

You check an existing sitemap in list mode. You upload the file, start the crawl, and then read out errors or non-indexable URLs. You generate a new sitemap after a complete crawl under Sitemaps > XML Sitemap. You have options there for the date of the last modification, the priority, and a separate image sitemap. Which URLs belong in there at all is clarified in our article on XML sitemaps.

Advanced features

For special cases, Screaming Frog offers powerful tools that take a lot of manual work off your hands.

Custom Extraction

With XPath, CSS Path, or Regex, you pull any data from the HTML. This is ideal for product prices, author names, or publication dates. You configure this under Configuration > Custom > Extraction. With Configuration > Custom > Search, you search for text in the source code. This way you check, for example, whether a tracking code is on all pages.

API integrations: GA4, Search Console, PageSpeed

You connect the spider directly to Google Analytics 4, the Google Search Console, and PageSpeed Insights. After that, user data, clicks, and Core Web Vitals per crawled URL are in the same table as the technical fields. With this, you find the pages that get impressions but are technically not indexable in one step.

Comparing crawls and segmentation

Licensed users compare two crawls of the same domain and immediately see what has changed. This is the fastest way to check a new release against last week. With segmentation, you divide a crawl into areas beforehand, such as shop, magazine, and help section. You then read error rates per segment instead of as an inaccurate total number.

AI prompts in the crawl

Newer versions apply prompts against models from OpenAI, Google Gemini, or a local Ollama during the crawl. This helps enormously to automatically generate alt text suggestions. The prompts are located under Configuration > Custom > Custom JavaScript. You store the access key under Configuration > API Access > AI. You bear the costs of the API calls yourself. For large websites, it is better to apply the prompts to a segment, not to the entire domain.

Accessibility check in the crawl

The spider now also checks rules for accessibility and lists violations per URL. Behind this is the open rule set Axe by Deque. You can activate this under Configuration > Spider > Extraction. Only a part of the criteria can be checked by machine this way. The rest is only covered by a manual check with a screen reader and keyboard, meaning a separate accessibility audit.

Automating crawls: scheduling and command line

If you only start the Spider now and then, you do not need this section. But as soon as you run the same crawl every week or want to know after every release whether something broke, automation pays off. There are two ways to do it.

The first is the built-in scheduler. Under File > Scheduling you set up a recurring crawl: start URL, time, a saved configuration and a destination folder to which the results are exported as CSV. At the scheduled time, Screaming Frog then starts on its own. The only requirement is that the computer is running at that moment. A scheduled crawl therefore belongs on a machine that stays switched on, such as a small server or an office computer that is not shut down at night.

The second way is the command line. Here another program starts the Spider without anyone clicking, for example a deployment script right after a release. A typical call looks like this:

1ScreamingFrogSEOSpiderCli.exe --crawl https://example.com
2  --headless
3  --config "technik-audit.seospiderconfig"
4  --output-folder "C:/crawls/2026-09"
5  --export-format csv
6  --export-tabs "Internal:All,Response Codes:Client Error (4xx)"
7  --save-crawl --overwrite
Copy

Read from top to bottom, the command means this:

• --crawl https://example.com names the start URL, just like the address field in the interface.
• --headless starts the Spider without a window so that it also runs on a server without a screen.
• --config "technik-audit.seospiderconfig" loads a saved configuration, for example the one from the section above. The automated run then crawls with the same settings you would use by hand.
• --output-folder "C:/crawls/2026-09" defines the folder the results are written to.
• --export-format csv selects the file format. Alternatively you can use xlsx or gsheet for Google Sheets.
• --export-tabs "Internal:All,Response Codes:Client Error (4xx)" exports only the tabs and filters you care about. Here that is all internal URLs and all 4xx errors.
• --save-crawl --overwrite additionally saves the complete crawl file and replaces the one from the previous run.

On macOS and Linux the program file has a different name, but the options stay the same.

The real benefit comes from the chain behind it. The crawl runs automatically, the CSV files land in a folder, and a small script compares them with the previous run and reports only the differences, such as new 404 pages or lost titles. How you build this part is shown in our article on Python for SEO.

Exporting data and saving crawls

All data from the upper window goes out via the export button as CSV, Excel, or Google Sheet. For large quantities, you use the Bulk Export in the main menu. In database storage mode, the spider saves continuously. Under File > Crawls, you open or duplicate previous runs. Always keep old crawls as a basis for comparison.

Screaming Frog for competitor analyses

You may also point the spider at foreign domains. This provides you with valuable insights into the content strategy of the competition. Crawl a competitor's website and sort by the Inlinks column in the Internal tab. Pages with a noticeably high number of internal links are the hub pages they invest in. With Bulk Export > Links > All Anchor Text, you get all internal anchor texts. The frequency distribution shows the keywords the structure is designed for.

Three decisions arise from the result. First, you see topics that the competitor occupies and you do not cover at all. Second, you recognize by the directory depth how they cut categories. Third, you compare their internal link density with yours. Translate this into your own structure instead of copying it blindly. Keep the crawl speed low and respect the robots.txt.

Limits of the SEO Spider

No tool covers everything. You should know these points before you buy a license.

• Desktop program. The crawl only runs when your computer is running. You only get permanent monitoring via your own machine plus scheduling.
• No off-page data. The spider does not provide backlinks, domain authority, and visibility indices.
• No real mobile data. You can simulate mobile browsers, but you do not measure real end devices.
• Prior knowledge required. The spider shows data, it does not evaluate. What counts is clarified in our article on technical SEO.
• Hardware dependency. For very large websites, the RAM decides whether the crawl finishes. Cloud crawlers do not have this problem.
• License per user. Working together on the same crawl is impractical because the database is local.

Alternatives and when they fit better

The choice is decided by the operating model. It is about desktop versus cloud and single audit versus permanent monitoring. For one-off audits with full control, the SEO Spider remains our first choice. For ongoing monitoring, we resort to a cloud solution because nobody wants to leave a computer running for it.

Conclusion

The SEO Spider shows most accurately what crawlers find on a website. Its strength lies in its configurability and speed with large URL volumes. Its weakness is the operating model as a desktop program. Those who know the costs, hardware, and limits get significantly more out of it with scheduling and the command line than with simple click crawls.