{"id":17504,"date":"2024-05-08T17:41:21","date_gmt":"2024-05-08T21:41:21","guid":{"rendered":"https:\/\/ipullrank.com\/?p=17504"},"modified":"2025-07-31T15:51:35","modified_gmt":"2025-07-31T19:51:35","slug":"vector-embeddings-is-all-you-need","status":"publish","type":"post","link":"https:\/\/ipullrank.com\/vector-embeddings-is-all-you-need","title":{"rendered":"Vector Embeddings is All You Need: SEO Use Cases for Vectorizing the Web with Screaming Frog"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Since learning of the importance and magnitude of vector embeddings, I have been proposing that a link index should vectorize the web and make those representations of pages available to SEOs. Fundamentally, with the further integration of machine learning in Google\u2019s ranking systems, vector embeddings are even more important to what we do than an understanding of the link graph. Currently, only search engines and large language modelers have this data at scale. I believe every SEO tool should be providing this data about pages and keywords.<\/p>

Google has been leveraging vector embeddings to understand the semantics of the web since the introduction of Word2Vec<\/span><\/a> in 2013 (here\u2019s a simpler explanation<\/span><\/a>) through the update we knew as Hummingbird, but the SEO software industry has continued to operate predominantly on the lexical model of natural language understanding. Lesser known tools like InLinks, WordLift, MarketMuse, MarketBrew and the various keyword clustering tools have all done things with this technology, but the popular SEO tools have not surfaced many semantic features.<\/p>

To some degree this showcases the gap between modern information retrieval and search engine optimization as industries. The common (simplified) understanding of Google in our space is that their systems simply crawl pages, break the content into its lexical components, counts the presence, prominence, and distribution of words, reviews linking relationships, ranks pages by expanding queries and breaking them into n-grams. Once it retrieves posting lists based on the n-grams, it intersects the results, scores that intersected list, sorts by the score and presents the rankings. Then it reinforces what ranks based on user signals. Structurally, the model architecture looks like this:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

That model is not wrong per se, because Google still does these things. It\u2019s just not at all indicative of the state of the art with which Google Search operates because it does so much more.<\/p>

Distinguished Google Researcher, Marc Najork, in his Generative Information Retrieval presentation<\/span><\/a> (where the above image comes from) discussed how the state of the art has evolved to fusion-based approaches that are a hybrid between lexical and semantic models. Citing his Google Research team\u2019s own 2020 paper Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approach<\/span><\/a> where they showcase open source libraries to implement and examine the viability of the method. They combine BM25<\/span><\/a> (the lexical retrieval model SEO effectively still operates on) with Google\u2019s SCaNN<\/span><\/a> package for vector search with BERT<\/span><\/a> for dense vector embeddings and combine and re-rank the results with a method called RM3.\u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"the\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

One of the key differences in the models is the advent of nearest neighbor searching with dense vectors. This is why rankings often no longer behave the way we anticipate. Semantic matching, and Google\u2019s specific improvements on it \u201cneural matching,\u201d<\/span><\/a> is often a \u201cfuzzier\u201d understanding of relevance when you\u2019re used to seeing the explicit presence of words in page titles, h1 tags, and distributed across body text and more links with targeted anchor text.\u00a0<\/p>

In fact, our mental model of how Google works is quite out of date. Based on what Najork presents in his deck, it looks a lot more like this:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"architecture\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

As natural language processing technology has yielded denser embeddings (as compared to the sparse embeddings featured in approaches like TF-IDF), Google has improved its ability to capture and associate information on a passage, page, site, and author level. Google moved on a long time ago, but with the rapid advancements in vector embeddings we can catch up.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

What are Vector Embeddings?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

As I explained in my \u201cRelevance is not Qualitative Measure for Search Engines\u201d<\/span><\/a> piece, the vector space model is what powers the understanding of relevance between queries and documents. Vector embeddings are used to represent the query and the documents in that model.<\/p>

I explained this in that post, but in the spirit of saving you a click and improving the relevance for this post, vector embeddings are a powerful technique in natural language processing (NLP) that represent words, phrases, or documents by plotting them as coordinates in multi-dimensional space. These vectors capture the semantic meaning and relationships between words, allowing machines to understand the nuances of language.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"embedding\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

While TF-IDF and its variants yielded simplified word vectors that indicated the presence of words based on a given page’s vocabulary, the history of modern vector embeddings goes back a little over a decade ago. Seminal works like the aforementioned word2vec were introduced in 2013 and capabilities have rapidly improved since the advent of Google\u2019s Transformer<\/span><\/a>. These models learned word relationships by analyzing large text corpora, positioning similar words close together in the vector space. Transformer improved upon a concept called Attention<\/span><\/a> wherein the language model developed the ability to also understand context and polysemy. So when I have the sentence \u201cShe bats her eyelashes flirtatiously at her date across the table\u201d and the sentence \u201cAt dusk, bats emerged from the cave, flitting about in search of insects,\u201d modern language models can now understand that the second usage of the word \u201cbats\u201d means the noun representing the animal while the first usage of the word \u201cbats\u201d is the verb representing the physical action.<\/p>

Dense vector embeddings revolutionized search by improving upon semantic search (or vector search), which goes beyond keyword matching to understand the intent behind a query and the meaning of the documents being considered. Search engines can now identify synonyms and related concepts, leading to more relevant and accurate results.\u00a0 When we say we\u2019ve moved form keywords to concepts this is what we\u2019re talking about. Information Retrieval is no longer solely reliant on the presence of specific words, a concept<\/i> can be represented and measured.<\/p>

Further still, vectors allow Google to effectively model representations of queries<\/span><\/a>, entities<\/span><\/a>, individual sentences, authors<\/span><\/a>, websites<\/span><\/a>, and use those representations to fulfill the ideas behind E-E-A-T<\/a>.<\/p>

State-of-the-art vector embeddings are trained on massive datasets and incorporate contextual information to capture complex relationships between words. This ongoing development continues to improve the accuracy and efficiency of search algorithms.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

Enter Screaming Frog SEO Spider\u2019s Custom JavaScript<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Screaming Frog SEO Spider<\/a><\/span> has been nudging the SEO space forward for over a decade. The team has continued to innovate in ways that the industry needs, and the SaaS tools are too slow to do. Being in the cloud has its advantages for scalability and speed, but there is nothing any of those tools can do that SFSS can’t, and plenty that it can do that those other tools won’t. When the SF team launches cutting edge features other tools should consider altering their roadmap.<\/p>

Version 20 is no different. What’s great for my cause is that it seems they are in agreement that someone should help us vectorize the web.<\/p>

Their new Custom JavaScript functionality allows users to run bespoke JS functions on pages as the spider crawls them. You can now do customized analysis and make calls to third party sources to enhance your crawl data.\u00a0<\/p>

While a lot of SEOs are going to use this upgrade to turn SFSS into Scrapebox, one of the operations that comes with the tool is code to generate vector embeddings from OpenAI as you crawl. This is specifically what is going to help you make the upgrade from lexical analysis to semantic.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t

JavaScript Functions in Screaming Frog<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

I was going to explain the different functions for accessing things in SFSS, but instead I made you a custom GPT called Kermit<\/span><\/a> with the documentation<\/span><\/a> to help you write your code. Thank me later.<\/i><\/p>

The main functional capabilities to know are that you can:<\/p>