Vector Embeddings: SEO Use Cases for Vectorizing the Web with Screaming Frog

Search engine optimization (SEO) is constantly evolving, and new technologies keep emerging that change the playing field. One of these innovations is vector embeddings – a method that makes semantic relationships between pieces of content visible in a completely new way.

You may have heard that Google has been using such technologies for years, while many of us still rely on traditional, lexical models. But that is changing right now: with tools like Screaming Frog, you can now use the power of vector embeddings for your own SEO strategies!

In this blog post, we show you how the technology works, why it offers so much potential, and how you can use it to expand your SEO insights.

What are vector embeddings?

Have you ever wondered how search engines like Google manage to understand not just the words in a search query, but also their meaning and context? The answer is (BERT) vector embeddings.

Imagine every word, every sentence, or even an entire document being represented as a point in a multidimensional space. The position of this point is not random – it reflects the semantic meaning and the relationships to other words. Words that are similar or used in the same context sit closer together.

Compared to traditional methods like TF-IDF, which are mainly based on word frequency, modern vector embeddings go much deeper. They are "denser", meaning they can also capture meaning, context, and even ambiguity (polysemy). A word like "bank" can be interpreted as a financial institution or a place to sit, depending on the context.

This capability has fundamentally changed the way we understand semantic search. Google has been using vectors for years to model queries, entities, sentences, authors, or even entire websites, using its own model called BERT. Thanks to LLM providers like OpenAI, however, anyone can now generate vector embeddings.

What is the best way to explain this with an example?

Suppose you have a multidimensional space in which you place the words "dog", "cat", and "car". The position of the words depends on their meaning and their relationships to each other. For example, "dog" and "cat" sit close together because they are both animals and share similar properties. The word "car", on the other hand, is further away because, unlike the animals, it is an object and thus has a different context. Each of these words is represented by a vector. Now imagine that an entire URL consists of several groups of vectors, and these vectors represent the content of a page.

Why do vector embeddings matter for SEO?

With the increasing integration of machine learning into search engine ranking systems, vector embeddings are becoming more and more important. They help capture the meaning and relevance of content not only through keyword matches, but also on a deeper semantic level. For SEOs, this means that a deeper understanding of the semantic relationships between pieces of content is required to succeed in the search results.

Screaming Frog and vector embeddings

With a custom JavaScript function, you can generate vector embeddings directly in Screaming Frog – and that opens up entirely new possibilities for your SEO strategy.

Screaming Frog gives you the freedom to run your own JavaScript functions and even retrieve third-party data via API calls. This means you can use the feature "(ChatGPT) Extract embeddings from page content", for example, to make semantic relationships within your content visible.

To get started, you just need to set up Screaming Frog accordingly: this includes entering your OpenAI API key and making a few adjustments to the crawl settings. Don't worry, it's easier than it sounds – and you'll quickly see the benefits!

If you prefer to try other embedding models, that's no problem either. Screaming Frog supports alternative options such as Google's Vertex AI or open-source tools like Meta's Llama. This gives you the flexibility to use exactly the technology that suits you best.

How to use vector embeddings with Screaming Frog

1. Data extraction: Crawl your website with Screaming Frog and export the required data, such as page titles, meta descriptions, and main content.
2. Create vector embeddings: Use a pre-trained language model to generate vector embeddings for your content. This can be done with programming languages like Python and libraries like TensorFlow or PyTorch.
3. Analysis and visualization: Use data visualization tools to map the relationships between pieces of content and identify patterns.
4. Derive actions: Based on the analysis, you can make optimizations, e.g. set internal links, create new content, or rework existing content.

How do you create vector embeddings?

With vector embeddings, you can identify internal linking opportunities and significantly improve your SEO efforts. This data-driven approach not only helps you uncover hidden linking opportunities, but also increases the relevance of the links – both for search engines and for users.

Here is a step-by-step guide that makes the process easier, even if you have no prior experience with Python or vector embeddings:

Step 1: Get an OpenAI API key

First, create a new secret API key on the OpenAI website and copy it. You will need this key later in Screaming Frog.

Step 2: Set up Screaming Frog

• Open Screaming Frog and go to Configuration > Custom > Custom JavaScript.

• Click Add from library and select (ChatGPT) Extract embeddings from page content.

• Insert your OpenAI API key into the code.

• Test the setup with any URL of your website to make sure everything works.

Then adjust the Screaming Frog settings:

• Enable JavaScript rendering: Go to Configuration > Crawl Config > Spider > Rendering > JavaScript and enable JavaScript rendering.
• Adjust crawl settings: Under Configuration > Crawl Config > Crawl you can specify that only textual content, and anything else you specifically need for your use case, is crawled.
• Configure data extraction: Go to Configuration > Crawl Config > Spider > Extraction and select only the data you need.

Then start the crawl. After about a minute, you will see numbers next to the URLs in the Custom JavaScript tab.

What you can do with the extracted data

Once you have extracted the vectors with Screaming Frog, there are many ways to process them further and gain valuable insights.

The extracted vectors can be processed for various SEO analyses:

• Keyword mapping: Vector similarity searches allow keywords to be assigned to the most relevant pages in order to optimize keyword targeting and internal linking.
• Keyword relevance: The cosine similarity between keywords and URLs helps evaluate the relevance of content for specific search terms.
• Internal linking & redirects: Semantic relationships between pages enable targeted internal linking and the optimal mapping of redirects during website migrations.
• Link building: Comparing vector embeddings to evaluate potential backlink sources in order to identify relevant and valuable links.
• Content clustering: Thematic grouping of content to optimize structure and identify outliers, e.g. with BERTopic.
• Similarity and diversity measurement: Analyzing the differences and similarities between data points for various use cases.
• Anomaly detection: Spotting unusual or deviating elements within the data.

For the calculations, the vectors from Screaming Frog, which are stored as comma-separated strings, must be converted into numerical values. Python with NumPy can be used for this. For vector search, tools such as SCaNN (Google), FAISS (Facebook), or Annoy (Spotify) are suitable. Alternatively, the data can be used in BigQuery for vector searches. Vectorizing keyword lists with CSV files is also possible for targeted analyses.

By using vector embeddings, you can not only better understand the semantic meaning of your content, but also develop targeted, data-driven SEO strategies. Integrating this technology into tools like Screaming Frog opens up new possibilities for optimizing your website and discovering SEO opportunities. By applying these techniques, you can not only improve your SEO results, but also develop a deeper, long-term strategy for the discoverability and relevance of your website. So it is well worth using vector embeddings to boost your website's competitive performance.