Since learning of the importance and magnitude of vector embeddings, I have been proposing that a link index should vectorize the web and make those representations of pages available to SEOs. Fundamentally, with the further integration of machine learning in Google’s ranking systems, vector embeddings are even more important to what we do than an understanding of the link graph. Currently, only search engines and large language modelers have this data at scale. I believe every SEO tool should be providing this data about pages and keywords.

Google has been leveraging vector embeddings to understand the semantics of the web since the introduction of Word2Vec in 2013 (here’s a simpler explanation) through the update we knew as Hummingbird, but the SEO software industry has continued to operate predominantly on the lexical model of natural language understanding. Lesser known tools like InLinks, WordLift, MarketMuse, MarketBrew and the various keyword clustering tools have all done things with this technology, but the popular SEO tools have not surfaced many semantic features.

To some degree this showcases the gap between modern information retrieval and search engine optimization as industries. The common (simplified) understanding of Google in our space is that their systems simply crawl pages, break the content into its lexical components, counts the presence, prominence, and distribution of words, reviews linking relationships, ranks pages by expanding queries and breaking them into n-grams. Once it retrieves posting lists based on the n-grams, it intersects the results, scores that intersected list, sorts by the score and presents the rankings. Then it reinforces what ranks based on user signals. Structurally, the model architecture looks like this:

That model is not wrong per se, because Google still does these things. It’s just not at all indicative of the state of the art with which Google Search operates because it does so much more.

Distinguished Google Researcher, Marc Najork, in his Generative Information Retrieval presentation (where the above image comes from) discussed how the state of the art has evolved to fusion-based approaches that are a hybrid between lexical and semantic models. Citing his Google Research team’s own 2020 paper Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approach where they showcase open source libraries to implement and examine the viability of the method. They combine BM25 (the lexical retrieval model SEO effectively still operates on) with Google’s SCaNN package for vector search with BERT for dense vector embeddings and combine and re-rank the results with a method called RM3. 

One of the key differences in the models is the advent of nearest neighbor searching with dense vectors. This is why rankings often no longer behave the way we anticipate. Semantic matching, and Google’s specific improvements on it “neural matching,” is often a “fuzzier” understanding of relevance when you’re used to seeing the explicit presence of words in page titles, h1 tags, and distributed across body text and more links with targeted anchor text. 

In fact, our mental model of how Google works is quite out of date. Based on what Najork presents in his deck, it looks a lot more like this:

As natural language processing technology has yielded denser embeddings (as compared to the sparse embeddings featured in approaches like TF-IDF), Google has improved its ability to capture and associate information on a passage, page, site, and author level. Google moved on a long time ago, but with the rapid advancements in vector embeddings we can catch up.

What are Vector Embeddings?

As I explained in my “Relevance is not Qualitative Measure for Search Engines” piece, the vector space model is what powers the understanding of relevance between queries and documents. Vector embeddings are used to represent the query and the documents in that model.

I explained this in that post, but in the spirit of saving you a click and improving the relevance for this post, vector embeddings are a powerful technique in natural language processing (NLP) that represent words, phrases, or documents by plotting them as coordinates in multi-dimensional space. These vectors capture the semantic meaning and relationships between words, allowing machines to understand the nuances of language.

While TF-IDF and its variants yielded simplified word vectors that indicated the presence of words based on a given page’s vocabulary, the history of modern vector embeddings goes back a little over a decade ago. Seminal works like the aforementioned word2vec were introduced in 2013 and capabilities have rapidly improved since the advent of Google’s Transformer. These models learned word relationships by analyzing large text corpora, positioning similar words close together in the vector space. Transformer improved upon a concept called Attention wherein the language model developed the ability to also understand context and polysemy. So when I have the sentence “She bats her eyelashes flirtatiously at her date across the table” and the sentence “At dusk, bats emerged from the cave, flitting about in search of insects,” modern language models can now understand that the second usage of the word “bats” means the noun representing the animal while the first usage of the word “bats” is the verb representing the physical action.

Dense vector embeddings revolutionized search by improving upon semantic search (or vector search), which goes beyond keyword matching to understand the intent behind a query and the meaning of the documents being considered. Search engines can now identify synonyms and related concepts, leading to more relevant and accurate results.  When we say we’ve moved form keywords to concepts this is what we’re talking about. Information Retrieval is no longer solely reliant on the presence of specific words, a concept can be represented and measured.

Further still, vectors allow Google to effectively model representations of queries, entities, individual sentences, authors, websites, and use those representations to fulfill the ideas behind E-E-A-T.

State-of-the-art vector embeddings are trained on massive datasets and incorporate contextual information to capture complex relationships between words. This ongoing development continues to improve the accuracy and efficiency of search algorithms.

Enter Screaming Frog SEO Spider’s Custom JavaScript

Screaming Frog SEO Spider has been nudging the SEO space forward for over a decade. The team has continued to innovate in ways that the industry needs, and the SaaS tools are too slow to do. Being in the cloud has its advantages for scalability and speed, but there is nothing any of those tools can do that SFSS can’t, and plenty that it can do that those other tools won’t. When the SF team launches cutting edge features other tools should consider altering their roadmap.

Version 20 is no different. What’s great for my cause is that it seems they are in agreement that someone should help us vectorize the web.

Their new Custom JavaScript functionality allows users to run bespoke JS functions on pages as the spider crawls them. You can now do customized analysis and make calls to third party sources to enhance your crawl data. 

While a lot of SEOs are going to use this upgrade to turn SFSS into Scrapebox, one of the operations that comes with the tool is code to generate vector embeddings from OpenAI as you crawl. This is specifically what is going to help you make the upgrade from lexical analysis to semantic.

JavaScript Functions in Screaming Frog

I was going to explain the different functions for accessing things in SFSS, but instead I made you a custom GPT called Kermit with the documentation to help you write your code. Thank me later.

The main functional capabilities to know are that you can:

• run actions on the page 
• Run extractions from the page
• save files based on either 
• load external scripts
• run multiple operations
• perform operations from the Chrome Utilities API

Unless you’re a JavaScript wizard, I recommend leaning heavily on the custom GPT to get yourself started. I also recommend contributing what you build to this public repository of SFSS custom JS scripts.

How to Vectorize a Site with Screaming Frog SEO Spider

If you’ve used custom extractions in SFSS before, the new Custom JS extraction functionality is an expansion of that. You can define what you want to execute at runtime and store it in unique columns. To get you started, the SF team has prepared a series of templates, including one for capturing embeddings from OpenAI as you crawl. Here’s how you do it:

1. In the Crawl Config select Custom > Custom JavaScript

This brings you to a dialog box where you can add and test your custom JS code. Click Add from Library to get the party started.

2. Select the ChatGPT extract embeddings from page content

In the system tab you’ll find a series of custom extractions and actions. There are things that can be pulled without external functionality as well as things that can use ChatGPT’s API or a local LLM via ollama. For our purposes, you’ll want the “(ChatGPT) Extract embeddings from page content” function.

3. Enter your OpenAI API key.

Once you’ve selected the extraction, you’ll need to configure the code by adding your OpenAI API key. You can test it on the right by adding a URL and clicking test. What you’ll get back is a series of decimal numbers. These are your embeddings.

4. Configure your crawl as you normally would and make sure to enable JavaScript rendering and that crawling external links is enabled. Then let it run as normal.

When you get your data back your Custom JavaScript tab will look like this:

By default, the embeddings will only be computed on pages that are of text/html type, but embeddings can be multimodal, so if you wanted to compute them on images you could. For that you’d have to adjust the Content Types that the JS fires on and pass the images as bytes.

Do I Have to Use OpenAI?

No, you don’t. In fact, according to the HuggingFace Massive Text Embedding Benchmark (MTEB) Leaderboard, they are not considered state of the art at this point. Google’s text-embedding-preview-0409 embeddings model is smaller with lower dimensionality and outperforms OpenAI’s text-embedding-3-large embeddings model in all tasks.

So, if we want closer parity to what Google is using in production, then we’d want to use the embeddings models in their Vertex AI offering.  You should know that Google’s embeddings are a fraction of a cent more expensive than OpenAI’s. OpenAI’s text-embedding-3-small is $0.00002 / 1K tokens while Google’s text-embedding-preview-0409 is $0.000025 / 1K tokens. If you did 100 million tokens (or the equivalent of two thousand novels not written by Stephen King), you’re spending $2 with OpenAI and $2.50 with Google. However, if you do batch requests to Google, the pricing is exactly the same. Although, I wonder if this will change based on announcements at the upcoming Google I/O conference.

Accounting for Token Limits

The length of your content is also a factor since Google only accepts 3,071 input tokens, whereas OpenAI accepts 8,191. If your content is too long, you’ll get an error message that looks like this:

 

Error: {
  "error": {
    "message": "This model's maximum context length is 8192 tokens, however you requested 11738 tokens (11738 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.",
    "type": "invalid_request_error",
    "param": null,
    "code": null
  }
}

In these cases you’d have to chunk the content and manage the embeddings into a single set for our use cases. It’s common practice to average the embeddings from the chunks into a single set of embeddings as follows:

 

const OPENAI_API_KEY = 'your_api_key_here';
const userContent = document.body.innerText;

function chatGptRequest() {
    if (new TextEncoder().encode(userContent).length > 8191) { // Checking byte length approximation for tokens
        // Function to break the string into chunks
        function chunkString(str, size) {
            const numChunks = Math.ceil(str.length / size);
            const chunks = new Array(numChunks);

            for (let i = 0, o = 0; i < numChunks; ++i, o += size) {
                chunks[i] = str.substring(o, o + size);
            }
            return chunks;
        }

        // Divide content into manageable chunks
        const chunks = chunkString(userContent, 8191);

        // Function to request batch embeddings for all chunks
        function chatGptBatchRequest(chunks) {
            return fetch('https://api.openai.com/v1/embeddings', {
                method: 'POST',
                headers: {
                    'Authorization': `Bearer ${OPENAI_API_KEY}`,
                    "Content-Type": "application/json",
                },
                body: JSON.stringify({
                    model: "text-embedding-3-small",
                    input: chunks,
                    encoding_format: "float",
                })
            })
            .then(response => {
                if (!response.ok) {
                    return response.text().then(text => { throw new Error(text); });
                }
                return response.json();
            })
            .then(data => {
                if (data.data.length > 0) {
                    const numEmbeddings = data.data.length;
                    const embeddingLength = data.data[0].embedding.length;
                    const sumEmbedding = new Array(embeddingLength).fill(0);

                    data.data.forEach(embed => {
                        embed.embedding.forEach((value, index) => {
                            sumEmbedding[index] += value;
                        });
                    });

                    const averageEmbedding = sumEmbedding.map(sum => sum / numEmbeddings);
                    return averageEmbedding.toString();
                } else {
                    throw new Error("No embeddings returned from the API.");
                }
            });
        }

        // Make a single batch request with all chunks and process the average
        return chatGptBatchRequest(chunks);
    } else {
        // Process single embedding request if content is within the token limit
        return fetch('https://api.openai.com/v1/embeddings', {
            method: 'POST',
            headers: {
                'Authorization': `Bearer ${OPENAI_API_KEY}`,
                "Content-Type": "application/json",
            },
            body: JSON.stringify({
                model: "text-embedding-3-small",
                input: userContent,
                encoding_format: "float",
            })
        })
        .then(response => {
            if (!response.ok) {
                 return response.text().then(text => {throw new Error(text)});
            }
            return response.json();
        })
        .then(data => {
            console.log(data.data[0].embedding);
            return data.data[0].embedding.toString();
        });
    }
}

// Execute request and handle results
return chatGptRequest()
    .then(embeddings => seoSpider.data(embeddings))
    .catch(error => seoSpider.error(error));

Using this code you won’t have to worry about the input length error. However, the 8.1k input tokens should work for most pages.

Using Google’s Embeddings

Google’s REST APIs require OAuth (the annoying pop up window for authentication), so it’s not as simple as just making an HTTP request to an endpoint with an API key like with OpenAI. Since SFSS does not support OAuth for Custom JS (nor does it need to), you’d have to stand up some middleware between it and the Vertex AI API. What I do is setup a local server with an API that makes the API request to VertexAI

Here’s the code to do so using Flask:

import logging
import sys
import os
from flask import Flask, request, jsonify
from google.auth import load_credentials_from_file
import tiktoken 
import numpy as np

from google.cloud import aiplatform
from google.oauth2 import service_account
from typing import List, Optional
from vertexai.language_models import TextEmbeddingInput, TextEmbeddingModel
import vertexai.preview

app = Flask(__name__)


os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = ‘[insert path to service account file here]’

def authenticate():
    """Load credentials from the environment variable."""
    credentials, project = load_credentials_from_file(os.environ["GOOGLE_APPLICATION_CREDENTIALS"])
    return credentials

credentials = authenticate()

def token_count(string: str, encoding_name: str) -> int:
    encoding = tiktoken.get_encoding(encoding_name)
    num_tokens = len(encoding.encode(string))
    return num_tokens

def split_text(text: str, max_tokens: int = 3000) -> List[str]:
    words = text.split()
    chunks = []
    current_chunk = []
    for word in words:
        if len(current_chunk) + 1 > max_tokens:
            chunks.append(' '.join(current_chunk))
            current_chunk = []
        current_chunk.append(word)
    if current_chunk:
        chunks.append(' '.join(current_chunk))
    return chunks

def embed_text(text: str, task: str = "RETRIEVAL_DOCUMENT",
               model_name: str = "text-embedding-preview-0409", dimensionality: Optional[int] = 256) -> List[float]:
    model = TextEmbeddingModel.from_pretrained(model_name)
    text_chunks = split_text(text)
    inputs = [TextEmbeddingInput(chunk, task) for chunk in text_chunks]
    kwargs = dict(output_dimensionality=dimensionality) if dimensionality else {}
    chunk_embeddings = [model.get_embeddings([input], **kwargs) for input in inputs]
    embeddings = [embedding.values for sublist in chunk_embeddings for embedding in sublist]
    average_embedding = np.mean(embeddings, axis=0)
    return average_embedding.tolist()

@app.route('/embed', methods=['POST'])
def handle_embed():
    data = request.json
    if not data or 'text' not in data or 'task' not in data:
        return jsonify({"error": "Request must contain 'text' and 'task' fields"}), 400
    text = data['text']
    task = data['task']
    try:
        embedding = embed_text(text, task)
        return jsonify({"embedding": embedding})
    except Exception as e:
        return jsonify({"error": str(e)}), 500

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0', port=5000)

To get this working, you’ll need to install the dependencies, enable Vertex AI, and get a service key. Here is the pip install one liner for the dependencies:

pip install flask numpy google-cloud-aiplatform google-auth tiktoken vertexai

Once you have the server running, you can setup a custom JS extraction to pull the data as follows:

const userContent = document.body.innerText;

function vertextAiRequest() {
    return fetch('http://127.0.0.1:5000/embed', {
        method: 'POST',
        headers: {
            "Content-Type": "application/json",
        },
        body: JSON.stringify({
            task: "RETRIEVAL_DOCUMENT",
            text: `${userContent}`
            })
    })
    .then(response => {
        if (!response.ok) {
             return response.text().then(text => {throw new Error(text)});
        }
        return response.json();
    })
    .then(data => {
        return data.embedding.toString();
    });
}

return vertextAiRequest()
    .then(embeddings => seoSpider.data(embeddings))
    .catch(error => seoSpider.error(error));

Here’s what the Vertex AI embeddings output will look like:

What About Open Source options?

This is all quite inexpensive, but you could also set up an embedding server locally via ollama and generate your embeddings for free using one of the open source pretrained models. For example, if you wanted to use the highly-rated SFR-embedding-mistral embeddings model in the same way, follow these steps:

1. Download, install, and start ollama 
2. Confirm that it’s running by going to http://localhost:11434
3. At the command line download and run the model with this command: ollama run avr/sfr-embedding-mistral. To verify things are working properly you can run a query in Postman or with cURL.

4. Once you’ve confirmed it’s running you can use this code as a custom extraction for generating the embeddings.

Here’s the code:

const userContent = document.body.innerText;

function getEmbeddings(userContent) {
    const apiUrl = 'http://localhost:11434/api/embeddings';

    const postData = {
        "model": "avr/sfr-embedding-mistral",
        "prompt": userContent,
    };

    const fetchOptions = {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify(postData)
    };

    return fetch(apiUrl, fetchOptions)
        .then(response => {
            if (!response.ok) {
                return response.text().then(text => {throw new Error(text)});
            }
            return response.json();
        })
        .then(data => {
            return data.embedding;
        });
}

return getEmbeddings(userContent)
  .then(embeddings => seoSpider.data(embeddings))
  .catch(error => seoSpider.error(error));
  

Depending on the specs of your machine, ollama may slow down your crawl too much to use it for generating embeddings. It may timeout before the data is returned. Make sure to test it on a few URLs before you let your crawl go.

Preparing Your Embeddings for Analysis

Before we get into use cases, you need to know how to prepare the data for analysis. Embeddings are stored in SFSS as comma-separated strings, but embeddings are numerical and need to be converted back into floats to be used for analysis. I prefer to export the XLSX file rather than a CSV because too many tools have done me dirty when it comes to formatting. I don’t want any potential formatting issues to damage my hard-won data. However, my testing has shown that CSVs can work just fine here too.

Nevertheless, the conversion is simple with numpy. Here’s a function to make it happen after you load your file into a dataframe:

def convert_strings_to_float(df, col, new_col_name):
  df = df[df[col].isna() == False]
  df[new_col_name] = df[col].str.split(',')
  df[new_float_col] = df[new_float_col].apply(lambda x: np.float64(x))
  df['EmbeddingLength'] = df[new_float_col].apply(lambda x: x.size)

  return df

Now you have your embeddings in a dataframe and ready to use for analysis.

Indexing Your Vector Embeddings

The first thing we want to do is build an index of the vectors so we can search them for various use cases. For vector searching, we’ll use Google’s SCaNN package. Here’s the code:

def scann_search(dataset:np.ndarray, queries: np.ndarray, n_neighbors = 10, distance_measure = "dot_product", num_leaves = 2000, num_leaves_to_search = 100):
  normalized_dataset = dataset / np.linalg.norm(dataset, axis=1)[:, np.newaxis]

  searcher = scann.scann_ops_pybind.builder(normalized_dataset, n_neighbors, distance_measure).tree(
      num_leaves=num_leaves, num_leaves_to_search=num_leaves_to_search, training_sample_size=250000).score_ah(
      2, anisotropic_quantization_threshold=0.2).reorder(100).build()

  return searcher

def convert_scann_arrays_to_urls(arrays: np.array, df: pd.DataFrame,column):
    results = []
    for arr in arrays:
      results.append(df.iloc[arr.flatten()][column].tolist())
    return results

siteDf = siteDf[siteDf['openAiEmbeddings'].isna() == False]
siteDf['openAiEmbeddingsAsFloats'] = siteDf['openAiEmbeddings'].str.split(',')
siteDf['openAiEmbeddingsAsFloats'] = siteDf['openAiEmbeddingsAsFloats'].apply(lambda x: np.float64(x))
siteDf['EmbeddingLength'] = siteDf['openAiEmbeddingsAsFloats'].apply(lambda x: x.size)

if siteDf['EmbeddingLength'].unique().size == 1:
  d = siteDf['EmbeddingLength'].unique() #Number of dimensions for each value
else:
  print('Dimensionality reduction required to make all arrays the same size.')

dataset = np.vstack(siteDf['openAiEmbeddingsAsFloats'].values)
queries = dataset

siteSearcher = scann_search(dataset, queries)
siteSearcher.serialize(index_directory+'/site_scann_index')

I’m using SCaNN, but you could use another package like Facebook’s FAISS or Spotify’s Annoy.

Note: If you don’t want to do this with Python, you could also push the data to BigQuery and use its engine for vector searches.

Vectorizing your Keyword List

In the vector space model, vectors for queries are compared to vectors for documents to determine what are the most relevant documents for a user’s search. So, for much of your comparative analysis, you will want to vectorize your list of keywords to compare against with nearest neighbor searches and other operations. You can use similar code on a CSV of keywords with their landing pages. We’ll want to maintain the landing pages so we can compare against the pages that are considered the most relevant.

Here is the approach to doing it with OpenAI using an export of keyword data from Semrush:

# Function to get embeddings and flatten them for SCANN
def get_openai_embeddings(keyword):
    response = openai.embeddings.create(
        input=keyword,
        model="text-embedding-3-small"  # Make sure to use the same embeddings as Screaming Frog
    )
    # Extract and flatten the embedding
    embedding_vector = response.data[0].embedding
    return np.array(embedding_vector).flatten()

semrushFile = 'ipullrank.com-organic.Positions-us-20220415-2024-05-05T16_03_31Z.csv'
keywordDf = read_file(semrushFile, 'CSV')
display(keywordDf)

# Loop through the DataFrame and get embeddings for each keyword
embeddings = []
for keyword in keywordDf['Keyword']:
    embeddings.append(get_openai_embeddings(keyword))

keywordDf['embeddings'] = embeddings

# Create a temporary DataFrame for Excel output with embeddings converted to strings
tempDf = keywordDf.copy()
tempDf['embeddings'] = tempDf['embeddings'].apply(lambda x: str(x))
tempDf.to_excel('semrush-embeddings.xlsx', index=False) # Save with embeddings as strings

# Display the updated DataFrame
print(keywordDf.head())

If you’re using Google’s embeddings they make a specific distinction between document and query embeddings. So, the code we used earlier will require the “RETRIEVAL_QUERY” task type to be specified. The only change that we make is calling the embed_text() function with the task variable set to RETRIEVAL_QUERY.

Here’s the adjustment to the code to make that happen:

# Loop through the DataFrame and get embeddings for each keyword
embeddings = []
for keyword in keywordDf['Keyword']:
    embeddings.append(embed_text(keyword,”RETRIEVAL_QUERY”)

Now let’s create a SCaNN index of the keyword list:

embeddings_matrix = np.vstack(keywordDf['embeddings'])

keywordSearcher = scann.scann_ops_pybind.builder(embeddings_matrix, 10, "dot_product").tree(
    num_leaves=200, num_leaves_to_search=100, training_sample_size=250000
).score_ah(2, anisotropic_quantization_threshold=0.2).reorder(100).build()

In case you’re wondering, you should not compare embeddings from different sources because they are not the same length nor are they composed by the same language model. Be consistent with the embeddings model that you use across your analysis.

SEO Use Cases for Vectorized Crawls

Ok, now we can unlock some new capabilities that can enhance the level of analysis we can do. Typically, machine learning engineers use embeddings to do a variety of things, including:

• Clustering –  Clustering is the process of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar to each other than to those in other groups.
• Classification – Classification involves assigning categories to objects based on input features, using trained models to predict the category for new, unseen data. I’m not going to cover classification today because that is worth its own post and I want to collect more spam and helpful content data and show you how to build an embeddings based classifier.
• Recommendations – Recommendation systems suggest relevant items to users based on their preferences and past behavior.
• Similarity and Diversity Measurement – This involves assessing how similar or different objects are from each other, often used in systems that need to understand variations or patterns among data points.
• Anomaly Detection – Anomaly detection identifies rare items, events, or observations which raise suspicions by differing significantly from the majority of the data.
• Information Retrieval – Information retrieval is the process of obtaining relevant information from a collection of resources that satisfies the information need from within large datasets.
• Machine Translation – Machine translation automatically translates text from one language to another, using complex models to understand and convert languages.
• Text Generation – Text generation is the process of automatically producing text, often mimicking human-like writing, using various algorithms and statistical techniques.

This new SF feature unlocks your ability to apply these techniques to drive deep insights for SEO.

Keyword Mapping

The keyword-to-keyword and keyword-to-page relationships are the most important aspects we can directly impact as content creators and SEOs. Optimizing pages to improve their keyword targeting through copy adjustments and linking strategies is best reinforced through a determination of what page owns what keywords. In some cases, due to the array of ranking factors, you’ll find that what ranks for the keyword is not the best page on your site. To remedy that at scale, you can loop through your keyword vector embeddings and perform nearest neighbor searches on your document SCaNN index. Wherever the highest ranking URL does not match the current landing page, that’s a linking opportunity for optimization.

To do that we perform the search, add the URL to the dataframe and an indication of whether it’s a match. Very quickly we have an understanding of how where we need to improve our keyword targeting.

queries = np.vstack(keywordDf['embeddings'].values) #Stacking all individual embeddings vertically into matrix

kwSearcher = scann_search(dataset, queries) # dataset is the same as before
nearest_neighbors = kwSearcher.search_batched(queries, final_num_neighbors=1)
matched_urls = convert_scann_arrays_to_urls(nearest_neighbors, siteDf, 'Address')

keywordDf['BestMatchURL'] = convert_scann_arrays_to_urls(neighbors, siteDf, 'Address')
keywordDf['BestMatchURL'] = keywordDf['BestMatchURL'].apply(lambda x: x[:1][0])

display(keywordDf)

When we run this, the data tells me that our guide to enterprise SEO is considered more relevant for enterprise SEO queries than our enterprise SEO landing page. Granted, our enterprise SEO page performs better, but looks like we need to optimize our content a bit better if we want the enterprise SEO landing page to rank better.

Keyword Relevance Calculations

The measure of relevance is the function of distance between embeddings. That is calculated in several ways: euclidean distance, dot product, and my preference, cosine similarity. I prefer it because of its simplicity and the ease of converting it into a score between 0 and 100. With the keyword and URL embeddings we can compare the mapped keyword to the URL to determine how relevant it is. You could also crawl competitor pages with SFSS and do the same. The comparison is simple. Find the embeddings for the URL and the keyword in their respective dataframes and perform cosine similarity.

# Function to normalize embeddings
def normalize_embeddings(embeddings):
    norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
    return (embeddings / norms).tolist()  # Normalize and convert to list

# Normalize the embeddings and convert them to lists for DataFrame storage
keywordDf['NormalizedEmbeddings'] = normalize_embeddings(np.vstack(keywordDf['embeddings'].values))
siteDf['NormalizedEmbeddings'] = normalize_embeddings(np.vstack(siteDf['OpenAI Embeddings 1ConvertedFloats'].values))

# Function to calculate cosine similarity
def cosine_similarity(embedding1, embedding2):
    return np.dot(embedding1, embedding2)

# Initialize a list to store cosine similarity results
cosine_similarities = []
relevance_values = []

# Loop through each keyword to calculate cosine similarity with its corresponding URL in siteDf
for index, row in keywordDf.iterrows():
    keyword_url = row['URL']
    keyword_embedding = row['NormalizedEmbeddings']  # This is now a list

    # Find the corresponding URL in siteDf
    if keyword_url in siteDf['Address'].values:
        # Get the embedding for the matching URL, which is also stored as a list
        url_embedding = siteDf.loc[siteDf['Address'] == keyword_url, 'NormalizedEmbeddings'].iloc[0]
        # Convert list to numpy array for calculation
        similarity = cosine_similarity(np.array(keyword_embedding), np.array(url_embedding))
        relevance = similarity * 100
    else:
        similarity = None  # Set similarity to None if no matching URL is found
        relevance = None

    cosine_similarities.append(similarity)
    relevance_values.append(relevance)

# Store the cosine similarities in the keywordDf
keywordDf['CosineSimilarity'] = cosine_similarities
keywordDf['Relevance'] = relevance_values


# Display or use the updated DataFrame
print(keywordDf[['Keyword', 'URL', 'CosineSimilarity','Relevance']])

keywordDf.to_excel('keyword-relevance.xlsx')

This is what our tool Orbitwise does. 

What you’re seeing here is an indication of low middling relevance for these keywords versus these landing pages.

Internal Linking and Redirect Mapping

Link relevance is about parity and the higher the relationship between the source and target URLs, the more valuable the link.

When Overstock was migrating to bedbathandbeyond.com, I talked about how they could map the redirects as scale using nearest neighbor searches. This same concept can be applied to identifying where to build internal links.

As we have mapped our keywords to landing pages, we now have an understanding of the best pages to own which keywords. Such analysis is especially useful when dealing with millions of pages versus millions of keywords. Assuming we want to build 10 links from different pages across the site, we can determine internal link sources for a given page by using keyword searches on the document index or we can do it by doing document searches on the document index.

The code is the same as what we did for keyword mapping, we just want more results. Let’s search for 10 neighbors this time.

# Search siteDf for keywords, return 10 neighbors per keyword

queries = np.vstack(keywordDf['embeddings'].values) #Stacking all individual embeddings vertically into matrix

kwSearcher = scann_search(dataset, queries) # dataset is the same as before
neighbors, distances = siteSearcher.search_batched(queries, leaves_to_search = 150)

nearest_neighbors = kwSearcher.search_batched(queries, final_num_neighbors=5)

matched_urls = convert_scann_arrays_to_urls(nearest_neighbors, siteDf, 'Address')

keywordDf['InternalLinkSuggestions'] = convert_scann_arrays_to_urls(neighbors, siteDf, 'Address')
keywordDf['InternalLinkSuggestions'] = keywordDf['InternalLinkSuggestions'].apply(lambda x: x[1:])
display(keywordDf)

# Create a temporary DataFrame for Excel output with embeddings converted to strings
tempDf = keywordDf.copy()
tempDf['embeddings'] = tempDf['embeddings'].apply(lambda x: str(x))
tempDf.to_excel('keyword-internal-link-mapping.xlsx', index=False) # Save with embeddings as strings

For the document version, we select the document embedding and use it to perform the search on the document index.

queries = dataset

siteSearcher = scann_search(dataset, queries)

neighbors, distances = siteSearcher.search_batched(queries, leaves_to_search = 150)

nearest_neighbors = siteSearcher.search_batched(queries, final_num_neighbors=10)

matched_urls = convert_scann_arrays_to_urls(nearest_neighbors, siteDf, 'Address')

siteDf['PageToPageLinkMapping'] = convert_scann_arrays_to_urls(neighbors, siteDf, 'Address')
siteDf['PageToPageLinkMapping'] = siteDf['PageToPageLinkMapping'].apply(lambda x: x[2:])
display(siteDf)
siteDf.to_excel('page-to-page-link-mapping.xlsx')

Here are the results for our Enterprise SEO page based on the page to page calculations.

['https://ipullrank.com/resources/guides-ebooks/modern-enterprise-seo-guide/chapter-1', 'https://ipullrank.com/', 'https://ipullrank.com/seo-for-the-procurement-professional', 'https://ipullrank.com/services/technical-seo', 'https://ipullrank.com/services', 'https://ipullrank.com/author/andrew-mcdermott/page/3', 'https://ipullrank.com/resources/guides-ebooks/modern-enterprise-seo-guide', 'https://ipullrank.com/11-common-enterprise-seo-problems-and-solutions']

For redirect mapping, you’d crawl the old site and the new site to generate embeddings for both. Then search the target site’s index with embeddings to the target site with top k set to 1. Using this data, you can determine the redirect relationships and limit what Google might perceive as soft 404s.

migratingSiteDf = read_file('migrating-site.xlsx', 'Excel')
migratingSiteDf = migratingSiteDf[migratingSiteDf['OpenAIEmbeddings'].isna() == False]
migratingSiteDf = migratingSiteDf[~migratingSiteDf['OpenAIEmbeddings'].str.contains('error')]
migratingSiteDf['OpenAIEmbeddingsFloats'] = migratingSiteDf['OpenAIEmbeddings'].str.split(',')

migratingSiteDf['OpenAIEmbeddingsFloats'] = migratingSiteDf['OpenAIEmbeddingsFloats'].apply(lambda x: np.array(x, dtype = float))

queries = np.vstack(migratingSiteDf['OpenAIEmbeddingsFloats'].values) #Stacking all individual embeddings vertically into matrix

migrationSearcher = scann_search(dataset, queries) # dataset is the same as before
neighbors, distances = migrationSearcher.search_batched(queries, leaves_to_search = 150)

nearest_neighbors = migrationSearcher.search_batched(queries, final_num_neighbors=1)

migratingSiteDf['MigrationTargetSuggestions'] = convert_scann_arrays_to_urls(neighbors, siteDf, 'Address')
migratingSiteDf['MigrationTargetSuggestions'] = migratingSiteDf['MigrationTargetSuggestions'].apply(lambda x: x[:1][0])

display(migratingSiteDf)
migratingSiteDf.to_excel('migration-recommendations.xlsx')

Link Building Target Identification

This is another story for another day, but I do not believe the volume approach for link building works anymore. On the back of the advancements in natural language processing, Google is better at understanding and modeling relevance parity between the source and target of links. Links built from sources that are completely irrelevant to the subject matter are invalidated in modern PageRank calculations. My hypothesis is that this is an aspect of how SpamBrain works.

To that end, we can vectorize a list of pages we are considering for link building and compare them against the target page using cosine similarity to determine how relevant the page source page of the link is.

For this process, we’d:

1. Identify a series of link targets using a tool like Ahrefs, Semrush, Pitchbox, or Respona
2. Crawl those pages with Screaming Frog to collect their embeddings. 
3. Compare them against the embeddings for your site to get the cosine similarity.

That yields a table that looks like the one below. When I sort ascending, that lets me know all the URLs that are not good fits for me to get links from. When we look at the scores, if they are not a 0.6 or higher, they are not relevant enough to build links from.

Here’s the code to make it happen:

import pandas as pd
import numpy as np
from scipy.spatial.distance import cdist

# Load the DataFrame
linkProspectsDf = pd.read_excel('link-prospects.xlsx')
linkProspectsDf = linkProspectsDf[linkProspectsDf['OpenAI Embeddings Long Inputs 1'].notna()]
linkProspectsDf = linkProspectsDf[~linkProspectsDf['OpenAI Embeddings Long Inputs 1'].str.contains('error|TypeError', regex=True)]

# Convert the string of numbers into a list of floats
def convert_embeddings(embedding_str):
    try:
        # Split the string into a list of strings, then convert each to float
        return np.array([float(num) for num in embedding_str.split(',')])
    except ValueError:
        # Return None or np.nan in case of conversion failure, which should be handled or filtered later
        return np.nan

linkProspectsDf['OpenAIEmbeddingsFloats'] = linkProspectsDf['OpenAI Embeddings Long Inputs 1'].apply(convert_embeddings)

# Remove rows where embeddings conversion failed (if any)
linkProspectsDf.dropna(subset=['OpenAIEmbeddingsFloats'], inplace=True)

# Normalize the embeddings
linkProspectsDf['normalized_embeddings'] = linkProspectsDf['OpenAIEmbeddingsFloats'].apply(lambda x: x / np.linalg.norm(x))
siteDf['normalized_embeddings'] = siteDf[new_float_col].apply(lambda x: x / np.linalg.norm(x))

# Specific URL to search for
specific_url = 'https://ipullrank.com/enterprise-seo'  # Change this to your specific URL

# Retrieve the normalized embedding for the specific URL
specific_embedding = siteDf[siteDf['Address'] == specific_url]['normalized_embeddings'].values[0]

# Prepare the embeddings array from the second dataframe
embeddings2 = np.stack(linkProspectsDf['normalized_embeddings'].values)

# Calculate cosine similarity
cosine_similarity_scores = 1 - cdist([specific_embedding], embeddings2, 'cosine')[0]

# Create a dataframe to store the results
results = pd.DataFrame({
    'Search Address': specific_url,
    'Target Address': linkProspectsDf['Address'],
    'Cosine Similarity Score': cosine_similarity_scores
})

# Optionally, sort the results by scores
results = results.sort_values(by='Cosine Similarity Score', ascending=False)
results.to_excel('link-prospect-relevance.xlsx')
display(results)

Clustering Content

Clustering the content can help us form topical clusters and also identify anomalies where the content is not relevant to any other content on the site. As sites grow larger, it becomes more difficult to manage how often the same topics are covered. Using BERTopic with our embeddings we can build and visualize a topical map of our content. When thinking about how you might want to do some content pruning to further reinforce your clusters this is a great approach.

When we run clustering on the embeddings using BERTopic it automatically puts the content into meaningful groups. BERTopic integrates with ChatGPT which allows you to generate human-readable names of the topic that was modeled. This is a vast improvement over other topical modeling approaches that use keywords from the content as representations rather than user-friendly labels. 

Once we run our clustering we can visualize them a few different ways. First as a clustered scatter plot:

We can also quickly look at the distribution of topics in a bar chart.

And, we can cluster the topics hierarchically. 

You can also see how topics are related and not related to each other: 

To do this we’ll also need the content itself so we can extract features from it to name the clusters. Capturing the content via SFSS is trivial. The code for the custom function is a one liner:

return seoSpider.data(document.body.innerText);

If you did not capture the content before, that’s fine we can just merge the two dataframes as you see in the code below:

def cluster_and_visualize_content(df, embeddings_col):
    print("Starting the topic modeling process for keywords...\n")

    # Prepare data
    df['Page Content'] = df['Page Content'].astype(str)
    keywords = df['Page Content'].tolist()
    embeddings = np.vstack(df[embeddings_col].tolist())  # Ensure embeddings are properly shaped
    embeddings = normalize(embeddings)  # Normalize embeddings for cosine similarity

    prompt = """
      I have topic that is described by the following keywords: [KEYWORDS]
      I am attempting to categorize this topic as part of 2-4 word taxonomy label that encapsulates all the keywords.
      Based on the above information, can you give a short taxonomy label of the topic? Just return the taxonomy label itself.
      """
    client = openai.OpenAI(api_key=openai.api_key)
    representation_model = OpenAI(client, model="gpt-3.5-turbo", prompt=prompt,chat=True)
    # Initialize BERTopic
    topic_model = BERTopic(representation_model=representation_model,calculate_probabilities=True)

    # Fit BERTopic
    topics, probabilities = topic_model.fit_transform(keywords, embeddings)
    df['topic'] = topics  # Adding topic numbers to the DataFrame

    # Visualize the topics with t-SNE
    print("Reducing dimensions for visualization...")
    tsne = TSNE(n_components=2, random_state=42, metric='euclidean')
    reduced_embeddings = tsne.fit_transform(embeddings)

    plt.figure(figsize=(10, 8))
    plt.scatter(reduced_embeddings[:, 0], reduced_embeddings[:, 1], c=topics, cmap='viridis', s=50, alpha=0.6)
    plt.colorbar()
    plt.title('Content Topics Visualization with t-SNE')
    plt.xlabel('t-SNE Feature 1')
    plt.ylabel('t-SNE Feature 2')
    plt.show()

    # Probability distribution visualization
    min_probability = 0.01
    if any(probabilities[0] > min_probability):
        print("Visualizing topic probabilities...")
        fig = topic_model.visualize_distribution(probabilities[0], min_probability=min_probability)
        fig.show()
    else:
        print("No topic probabilities above the threshold to visualize.")

    # Intertopic distance map
    print("Visualizing intertopic distance map...")
    fig = topic_model.visualize_topics()
    fig.show()

    # Hierarchical clustering
    print("Visualizing hierarchical clustering...")
    fig = topic_model.visualize_hierarchy()
    fig.show()

    # Extract and name topics
    df['topic_name'] = df['topic'].apply(lambda x: topic_model.get_topic(x)[0][0] if topic_model.get_topic(x) else 'No dominant topic')

    # Display DataFrame with topic names
    display(df)

    # Export the DataFrame with topic labels
    df.to_excel('content-clusters-bertopic.xlsx', index=False)

pageContentDf = read_file('ipr-content.xlsx', 'Excel')

contentEmbeddingsDf = siteDf.merge(pageContentDf, on='Address', how='inner')
#print(contentEmbeddingsDf)

cluster_and_visualize_content(contentEmbeddingsDf, 'OpenAI Embeddings 1ConvertedFloats')

The Value of a Vector Index of the Web

The democratization of the link graph gave us a series of measures that allowed us to understand the value of websites in the way that Google attributes authority. Granted, those metrics are only approximations of what Google may use, but they have driven the SEO space for nearly two decades.

And, that was enough prior to Google Search’s transition to becoming a heavily machine learning-driven environment. In a hybrid fusion environment, the link graph matters less because Google is taking signals derived from vector embeddings and using them to inform ranking. 

Dare I say, the link graph and link indices are less valuable than they were in the past. Whereas all of the above could be native functionality for link indices that make them more valuable for doing SEO moving forward.

Until someone gives us such an index, Screaming Frog has armed us with what we need to catch up to Google.

So, What Are Your Use Cases?

The shortcomings of SEO software has yielded a strong community of Python SEOs. People have been leveraging state of the art technologies to cover the chasm between what SEO software can do and what Google does do.

So, I’m curious, what are your use cases for vector embeddings? How do you anticipate that Screaming Frog’s new feature will help you do your job even better? In the meantime, you can play with all the code I shared in this Colab and contribute your own custom JavaScript snippets at this GitHub. I’ll be back soon with some classification use cases.

Let me know if there’s anything you want me to cook up for you.

Next Steps

Here are three ways iPullRank can help you combine SEO and content to earn visibility for your business and drive revenue:

1. Schedule a 30-Minute Strategy Session: Share your biggest SEO and content challenges so we can put together a custom discovery deck after looking through your digital presence. No one-size-fits-all solutions, only tailored advice to grow your business. Schedule your consultation session now.
2. Get Our Newsletter: AI is reshaping search. The Rank Report gives you signal through the noise, so your brand doesn’t just keep up, it leads. Subscribe to the Rank Report.
3. Enhance Your Content’s Relevance with Relevance Doctor: Not sure if your content is mathematically relevant? Use Relevance Doctor to test and improve your content’s relevancy, ensuring it ranks for your targeted keywords. Test your content relevance today.

Want more? Visit our blog for access to past webinars, exclusive guides, and insightful blogs crafted by our team of experts. 

The post Vector Embeddings is All You Need: SEO Use Cases for Vectorizing the Web with Screaming Frog appeared first on iPullRank.

Get iPullRank’s AI in Content and SEO Free Guide – DOWNLOAD

WEBINAR REPLAY: AI Content for SEO – WATCH

One of the biggest predictions in the marketing world for 2023 was an increase in the use of artificial intelligence in content creation. Between Microsoft’s recent $10 billion interest in OpenAI and how a new AI tool seems to launch each week, this prediction seems likely to come true. Even Gartner offered its perspective on the changing landscape of content creation. 

Resistance to new technology never seems to pan out for the naysayers. Take the printing press. When mass production started, the few that owned printing presses protested, fearing it would dilute the quality of written work available. While the new wealth of printing presses definitely led to some bad books, it also made it much quicker to share information than ever before. 

The same could be said about the internet. When it was introduced to the public, many dismissed it as a fad. 27 years after hitting the mainstream, the Internet permeates our life so much that SEO is crucial to a brand’s visibility and success.

While AI tools offer a lot of value to content production, it’s not the end-all-be-all solution. There is currently no resource to produce AI content that’s optimized, engaging, and voice-accurate. Even with these new AI advancements, good content still requires human creators. 

Before you decide your position on AI, let’s look at the business use cases, the benefits, limitations, and what you can possibly expect from future iterations.

What is Generative AI?

As the name suggests, generative AI is an automated platform that generates new content. The new explosion of ever-evolving AI platforms creates new content by scraping existing, relevant pages available on the web, then compiling that information into a post based on prompts you provide. 

These prompts are phrases and queries you submit to an AI platform. Much like instructions given to a human writer, they give the platform a set of guidelines. It uses those guidelines to create the content you want. 

Generative AI can help create many types of content, including:

• Copy briefs
• Meta descriptions 
• Title generation
• Header tags 
• Custom video and image 
• Full copy creation

With the right prompts and phrasing, AI can write nearly anything. We’ve used it to create everything from copy briefs and generated posts to food recipes.

The Benefits of Generative AI

For organizations hoping to reduce the cost of content production, generative AI may be the missing piece. By combining prompt analysis with open access to the Internet, these generative models can learn and write much faster than humans can. Using an AI resource can make a number of content production processes much more efficient:

• Research
• Outlining
• Drafting
• Revisions (to a limited extent)

Instead of spending hours writing a first draft, you can prompt the AI to create one in a matter of minutes. This can be a game-changer for organizations with short production deadlines. 

Using generative AI for the grunt work of production can ease the burden on your content team. This can free up your marketers and creators to focus on higher-level things like:

• Tying content to relevant topic clusters.
• Finding opportunities for information gain.
• Expanding your internal linking strategy.
• Aligning existing content with your new content plan.
• Mapping out your content calendar to align with a marketing campaign.
• Improving your content quality to beat the competition.

While generative AI can’t take the place of human writers, it can seriously reduce their workload. When used intelligently, it can help your organization produce more and better-coordinated content at a lower price.

Can Generative AI Create Original Content?

Generative AI can’t create original content – at least, not yet. While it can mimic how a human approaches content creation, resulting in well-written paragraphs and realistic images, the outcome is anything but original. 

As we mentioned, resources like Open AI create content by scraping the internet for information relevant to the prompt. They then stitch that information together in a way that makes sense. That means anything they write is made entirely of statements recycled from existing content. 

An AI content platform is also limited by the number of sources it has access to, as well as the amount of data it is able to access. Currently, ChatGPT-3 only has information up to 2021, so using it to create content related to events later than that will result in additional manual research or trying other prompts for more information.

 

 

Even with the entire internet at its disposal, generative AI can’t solve all your content issues. Though many in the content marketing and SEO community fear AI will replace them, content creation is still a human-driven process.

Read More: AI Content is not the SEO threat they want you to think it is 

The Limitations of Generative AI

AI writing is still in its early stages. Even with all the resources available, there are some things it just can’t handle yet:

• Voice and tone
• On-page optimization
• Internal linking

In addition, there are some other limitations to keep in mind when using generative AI:

• It doesn’t know which sources are reliable, so it’s important to double-check all information for accuracy.
• The content it produces may be thin or overly broad, so be prepared to do some additional research or prompting.
• It can’t produce original content, so you’ll need to check all text thoroughly for plagiarism and adjust language as needed.
• It can’t reliably source or produce images relevant to the content. If you’re using images, be ready to find appropriate ones yourself.
• It struggles to produce good subjective content. Opinion pieces it generates are often shallow or generic.

If your organization is considering AI for content generation, there are a few things to keep in mind. 

Writing Effective AI Prompts

Everyone using generative AI needs to understand how to properly use the prompts, because prompt quality has a huge effect on the resulting piece. A bad prompt may result in inaccurate or out-of-context content. 

Avoiding Plagiarism and AI Detection

Along with the emergence of AI like ChatGPT came a rise in AI detection and advanced plagiarism checkers. Because AI scrapes existing web pages to create new content, AI-generated pieces almost always have several instances of plagiarism. It’s important to adjust them thoroughly so they’re truly new and unique.

In addition, Google has been vocal about penalizing AI-generated content on SERPs, and it claims to have the resources to do so. That means that in order to rank, it’s important that your piece doesn’t look like it was created by AI.

As said before when asked about AI, content created primarily for search engine rankings, however it is done, is against our guidance. If content is helpful & created for people first, that’s not an issue.https://t.co/3rs3Yrrrk1https://t.co/TlFEbdXGAphttps://t.co/Yl9XWr5CAN pic.twitter.com/gFTE2C2wq1

— Google SearchLiaison (@searchliaison) January 12, 2023

 

Voice, Tone, and Content Quality

Generative AI isn’t great at meeting a particular voice and tone. It does its best, but it’s still just a bot that is trying to compile all of the relevant information you request. This means someone will have to proofread and edit the piece. 

As a result, AI-created content often requires more editing than a human-written piece. The editorial team will need to update to match the intended voice and tone. They’ll also need to make sure it’s engaging, flows logically, and is actually about the topic you meant to discuss.

 

On-Page SEO

AI can’t currently optimize a piece of content for search engines. It can assist in optimization efforts like meta information development and keyword research, but it has very little ability to create content that follows best practices for on-page SEO.

How to Use Generative AI for Content Development & More

There are other ways to use generative AI beyond content development. While many tools create full content pieces, other resources like Frase and Notion use AI for different purposes. These resources offer insights on how to improve existing content by suggesting related URLs, keywords, and topics to include. 

Some generative AI tools even extend beyond the written word. A variety of platforms specialize in other types of content creation like image generation or even game design. With effective prompts and the right AI tools, organizations can improve efficiency in a wide variety of ways. 

AI Tools: Work Smarter, Not Harder

Ignoring the potential of AI to simplify processes like writing, research, brief development, or even image design is the antithesis of “work smarter, not harder.” By integrating artificial intelligence tools into the workflow, writers can spend less time and energy on the heavy lifting and focus on the overall quality of the piece of content. 

At this point, AI is still in the early stages of development. Human experts can work with AI to produce high-quality content that resonates with the target.

With the limitations that various AI platforms have, what’s the best way to integrate it into a team’s workflow? What situations are perfect for generative AI content, and where does it really struggle? We’ll continue to explore the opportunities AI presents and how to get the most out of each platform throughout the year. 

Watch the Webinar Replay

The post The Role of Generative AI in Content Production appeared first on iPullRank.

The search engine has developed a wide range of new opportunities for e-commerce brands to attract customers, provide critical shopping information, and visually represent products across an enhanced search results page.

E-commerce experienced exponential growth during the pandemic. In the following chart, you can see how the lockdowns during the pandemic have affected global e-commerce sales, giving them a boost. In addition, all e-commerce sectors are forecast to see steady and continuous revenue growth through 2024.

GRAPH: Global eCommerce Revenue Forecast 8% Growth 2020-2025

When physical stores began to close for the health and safety of the public, online stores became essential to the business health of many commercial brands. Whether they were launching an e-commerce website or improving the performance of existing stores, smart businesses knew e-commerce was a lucrative financial investment.

In the world of e-commerce, Google has lost significant market share to Amazon. The e-commerce search engine has increasingly become the default option for customers researching and purchasing products, despite criticisms of Amazon favoring its own products and bullying retailers.

This has led Google to make a series of major e-commerce changes.

“We’re advancing our plans to make it free for merchants to sell on Google. Beginning next week, search results on the Google Shopping tab will consist primarily of free listings, helping merchants better connect with consumers, regardless of whether they advertise on Google”.  Bill Ready – President, Commerce

This new approach reduces the exclusivity of Google’s pay-to-play model. While selling products on Google used to be the preserve of advertisers, now small and medium-sized businesses can also sell products on Google organically. But how can you get free listings on Google for your products? In this article, I’ll tell you the key to success for your e-commerce SEO and how you can use it to get better search visibility across all of Google’s products and ultimately, increase your sales.

How To Get Free Listings On Google For Your Products 

While it used to be the investment in advertising that made the difference, now with the new free listings, it’s the data that determines whether or not your products show up on Google. 

Google has expanded the eligibility of websites that use structured data for both commercial listings and product snippets in Google search results. 

Whereas before you could share information about the product group you wanted to advertise on Google, now with free listings it is essential to provide the search engine with all the information it needs to know what you are selling and help users to conclude the purchase. This helps Google present your products more accurately in organic search. 

Using structured data is the key to getting your products to show up in Google’s free listings. This is because it provides the search engine with all the information it needs to understand your products and display them to users. 

Therefore, it is important to have an accurate and complete data set. This is the only way to get free listings on Google, attract more visitors to your e-commerce and make more sales.

What Is A Product Knowledge Graph And Why It Is Important For Your E-Commerce SEO

The Product Knowledge Graph (PKG) is an e-commerce-specific form of knowledge graph designed to improve product discoverability and end-user experience by enriching a brand’s content with data. 

Visual example of the Product Knowledge Graph

It is used to link products and all the information about them, creating the dynamic architecture behind your website that helps the search engine understand and index the content and show it to those who are interested and searching for that particular product. It also allows you to link editorial content to product content, helping consumers in their customer journey and bridging the gap between them and the brand. 

As we know, the customer journey is not linear (we have already talked about the messy middle several times) and there are numerous steps that the user goes through before completing a purchase. The PKG allows you to drive the user on their journey to find the product they are looking for and find it in your e-commerce store. 

Let us take an example. Silvia is looking for a new pair of sunglasses to wear while hiking. Silvia’s customer journey starts with a question. She will search for information, ask questions, and read different content to find out which product best fits her needs. 

In an e-commerce context, it’s all about connecting your site’s editorial content with purely transactional information such as prices, availability, types, product numbers, payment methods, and delivery information. This is what the PKG does for your data and your customers. 

It allows you to create a data set that is not only available to the customer (Silvia in our example) but can also be used by various types of automated applications that Silvia (and other potential and future customers) use to navigate your content (search engines, chatbots, personal assistants, etc.).

The Power Of Structured Data For E-Commerce 

As you have seen above, the power of structured data is immense, and a Product Knowledge Graph is critical to making your products visible in Google free listings and improving SEO for your e-commerce. 

Specifically, there are 3 benefits that you can achieve as you can see below. 

Better visibility for your products

The Product Knowledge Graph makes your content visible because it provides machine-readable and highly interconnected descriptions. Just like Google’s Knowledge Graph does.

However, the Product Knowledge Graph fulfills a special need: discoverability. It is a specialized graph that provides answers that can ultimately be read by robots. The Product Knowledge Graph focuses on discoverability and enables improved e-commerce marketing by providing the data structure for targeted, richly linked, product-enriched content.

Creating immersive content for your audience  

To increase audience engagement, it is important to create a dynamic and carefully linked environment. Product Knowledge Graph connects editorial content with product categories so that the user can easily navigate through the editorial content and, when ready, proceed to purchase the product they have chosen by instantly seeing prices, rates, availability, types, delivery information, etc. 

Enabling interoperability of your data set

With a Product Knowledge Graph, data can be easily shared across different devices, platforms, and software. You can provide APIs to other parties, such as search engines and retailers, and run business-critical “hubs” that collect data and make it available to users, such as Merchant Center, Google Shopping, voice assistants, etc.

How You Can Build A Product Knowledge Graph For Your Online Shop 

In this scenario, optimizing an e-commerce company’s products by curating and enriching its data means involving both the marketing and development teams in the workflow. This relationship, as we know, is not always easy.

With WordLift, it is possible to make this process smoother and more accessible to all, without the need for developers. Indeed, you can add structured data about the products in your e-commerce and build your Product Knowledge Graph, which allows you to list your products on Google Shopping for free. 

To build your Product Knowledge Graph, you can start with the data you have about the products in your e-commerce. You can use your product information management system, where you store all the product attributes, or you can use your merchant feed, which allows you to view your products on different interfaces. Otherwise, you can choose to crawl your website to extract additional attributes that will help you build the Knowledge Graph.

Here are three important challenges:

• Data scarcity. Identify product attributes and focus on those that are important to customers in their search. Product identifiers, color, material, product type, brand, etc 
• Data completion. Ensure data (structured and unstructured) is complete and consistent. Adding multiple formats for each product image is a good example. Another example is checking if the free description in the table contains missing information.
• Data quality. Check the reliability of the data. In SEO, it is important to ensure that data presented as structured data matches data submitted via XML feeds (e.g. Google Merchant Feed). It also helps to prevent product descriptions from conflicting with structured data.

The result of WordLift is the curation and linking of data. In other words, the available information about your products is not only enriched with structured data to make it readable by Google and search engines but your data is also linked to other data sources such as Wikidata and DBPedia. 

Entity-based SEO gives context to your website. That’s why it’s important to work with entities, because they help connect the world’s information to get relevant results when searching.

The solution developed by WordLift works with proprietary CMS as well as other e-commerce platforms such as Shopify and Magento. It all starts with the data sent to WordLift, which is implemented and enriched to build the Product Knowledge Graph.

Recently, WordLift has developed the Product Knowledge Graph Builder. If you have an e-commerce feed and a merchant feed, you can use this feature to import the data by enriching it with structured data and creating your Product Knowledge Graph in a few simple steps.

Data Harmonization And Data Reconciliation

Data synchronization and reconciliation are critical to your e-commerce SEO.  

The first ensures that the information comes from the Merchant Feed and matches the information in the Merchant Feed. This helps you avoid confusing Google and not being found for one or more specific search queries.  

The second is making sure there are no duplicates that could cause Google to refuse to show products in the free listings. A lack can be confusing and cause your products to not rank well on Google for one or more specific search queries. This is made possible by a unique product identifier (UPI) assigned to each individual product in your e-commerce.

AI-Generated Content For E-Commerce 

As Google challenges Amazon to become the leader in e-commerce online search, we expect to see more features to improve the user experience, more opportunities for visibility, and easier access leading to more competition in the search results.

What other ways do we envision eCommerce brands scaling their visibility in search?

AI-generated content has become a popular trend in tech and it’s the perfect tool to leverage in e-commerce content and SEO. In fact, our founder Andrea Volpini wrote extensively on the power of AI to automatically create content for product descriptions and FAQs. Meanwhile, iPullRank’s founder Mike King addresses why AI content is not yet a threat to SEO.

Be careful, though, because creating content with AI does not mean you have to forgo human input. In fact, this is still crucial to ensure the validity of the action and its consistency with the brand and its characteristics. 

In WordLift, an advanced validation workflow has been developed that allows editorial teams to scale SEO, saving time and energy within optimal results. 

Similarly, data and AI can be used to generate FAQs and e-commerce 404 smart pages that provide users with an increasingly relevant experience that facilitates the buying process.

Most e-commerce brands are not leveraging structured data to improve the visibility of their products and even fewer are using a Product Knowledge Graph to develop topical authority in their space. 

Take this competitive advantage and become the most likely product to appear in your buyer’s search results.

Want help creating a Product Knowledge Graph and a complete content journey for your e-commerce brand? Contact iPullRank for a consultation and WordLift for technical implementation.

The post E-Commerce SEO: How To Increase the Visibility of Your Products By Building a Product Knowledge Graph appeared first on iPullRank.

At that scale, Googlebot doesn’t always crawl every page, let alone index them.

• Broken links.
• Slow response times.
• Low crawl frequency.

With such quantity, you need a way to analyze website performance and catch technical issues before your organic search traffic takes a hit.

Log file analysis is a not-so-secret and critical, though sometimes underutilized, element of an SEO strategy. With the growing complexity of the web and increased competition online, it is an even more important tactic than ever to gain data-backed insights into how your website is performing.

So what is log file analysis and why does it matter for SEO? 

Your web servers record and measure the crawl behavior of search engines. These logs can reveal issues and opportunities that impact the way that Google sees your site. In other words, monitoring and analyzing log files lets you see your website how Google sees it.

SEOs analyze those logs to determine how efficiently Google interacts with your website. Proper log analysis can highlight and inform optimizations and fixes to better guide search engines to your most important and strategic pages. As a result, you maximize traffic, revenue, and the overall impact of organic search.

In the words of Dana Tan, Director of SEO at Under Armour:

“Getting server logs takes the conjecture out of SEO and it’s 100% scientific. It’s data. You can’t argue with cold, hard data. And that really makes SEO a science instead of guesswork.” 

Where are server log files stored and how can you access them?

In many cases, just getting access to the server logs themselves is the hardest part. Access to your server logs depends on your server and hosting setup – and you most likely will need to engage with the DevOps or Engineering team to get the appropriate permissions.  In most scenarios, traffic is delivered through a Content Delivery Network (CDN) such as Akamai, Cloudflare, or Fastly, and can be delivered to an AWS S3 bucket or database. 

The anatomy of a clean log line 

Before delving into your CDN, let’s take a look at a clean log line for a crawl from Googlebot:

Although log data is usually simple in format, it’s important to realize the common hurdles that arise when attempting to retrieve and convert the data into “readable” metrics. Too often, SEOs end up with a lot of manual work, unstable files, missed data, or inaccurate analyses.  

• Log files for a large website can quickly add up to gigabytes of data, becoming quite cumbersome. This data is often too large for desktop analysis tools, so you will need to have the files compressed and then decompressed when retrieving the data.
• Some log files may require multiple downloads in a single day.
• Log files need to be parsed to extract the specific data that is useful for SEO, and then transcribed into SEO terms such as crawl volume, visit volume, unique URLs, and more. 

Instead of trying to manage log files yourself or manually parse through millions of cells worth of data, you can have your log files automatically fed into a usable interface made specifically for SEOs. In our case, it’s into the Botify LogAnalyzer within the Botify Analytics suite. 

Botify can ingest a subset of the SEO-relevant data from the web server log files based on direct delivery from the CDNs, parse that data, and do so programmatically and on a recurring basis. 

By handling log files in this manner, you are also ensuring the highest level of security. 

When ingesting the logs, Botify uses the IP address from your log files to validate the user agent to avoid inflating data with spoofing, while leveraging the referrer data to provide visit insights. 

Botify never processes personal data. If a personal IP address is not removed by the client and surfaces in the server log files, Botify permanently removes all of this information before processing any log files. Botify is GDPR and SOC2 compliant, providing the highest standard of data privacy and security for brands. 

Reading a log file analysis report: what could you uncover?

Quite a lot, actually! Log file analysis allows you to:

• See if Google is spending time on unimportant pages, like duplicate content or orphan pages.

• Evaluate crawls by segments and analyze search engine behavior on specific sections of your website.
• Understand the distribution of bad status codes and server errors such as 3XX, 4XX, or 5XXs  that may impact Google’s ability to properly and efficiently crawl your website and discover content.
• Compare crawl volume to unique URLs crawled and learn of any new URLs that weren’t in a previous crawl.
• Know how frequently Googlebot is crawling your website, helping you chart the relationship between crawl frequency, crawl budget, and traffic fluctuations and determine if the frequency is aligned with your publishing schedule. 

Below are some valuable visuals from Botify’s LogAnalyzer. Within Botify’s tool, specifically, you have the flexibility of viewing data such as crawl volume or unique number of URLs for the entire website – or based on specific site segments, customizable within the platform. 

For example, if you’re managing an ecommerce website that sells sportswear, apparel, athletic gear, and accessories, you can isolate portions of the website for analysis, whether by category, i.e. sneakers, by content type, i.e. tennis, or by gender, i.e. women. Alternatively, if managing a news site, you may choose to segment the website based on content type, i.e. articles, or sections i.e. international news.  This not only allows for easier sharing of data across teams, but also helps to pinpoint where issues may be on the website. This becomes incredibly useful when there are specific events such as a product launch or major news event.

In fact, one of Botify’s publisher customers experienced a short-term boost in traffic followed by a steep decline in traffic. A recent news event was responsible for the boost, and when they isolated their pages by content, they were able to see that their evergreen content was suffering. The only way to identify the problem was by splitting both types of content through segmentation.

It’s also important to monitor which pages are being crawled frequently in order to gauge if search engine bots are coming back to your website often enough and finding new content when they do return. If you notice that they are coming back to pages that remain largely unchanged, this can be a waste of crawl budget that would be better used on pages with fresh content.

Monitoring 304s and static resources

Within LogAnalyzer, you can see all of your pages that return a 304 status code. This status code signals to a web crawler that it has not been modified and should be cached, helping lower the crawl frequency to these pages and freeing up crawl budget to be allocated to other pages. Reviewing 304s is also a useful tactic for site audits to determine if your server is telling search engines that pages that have, in fact, been updated, have not been. 

Similarly, you can monitor the frequency at which search engine bots are crawling static resources such as JavaScript files, fonts, CSS files, and more – but also the number of resources they are spending to render this content. This not only affects crawl budget, but also render budget.

Render budget looks at how many pages a search engine bot is able to fully execute and the resources it takes to do so as compared to a simple HTML page. If 40+% of your crawls are going towards these types of resources, you may potentially have a render budget issue, which would impact how much content Google will crawl and ultimately render in a given timeframe.

Log file analysis reports can uncover a wealth of information about the discoverability and performance of your website and should help guide your SEO priorities. And, with so many people changing various portions of the website, it’s an excellent diagnostic tool to pinpoint anomalies and mitigate disasters – one that should be utilized regularly. 

But as you know, SEO professionals are often asked to do a lot with a little. Working with a trusted agency partner like iPullRank, who has the technical SEO acumen to utilize a sophisticated platform like Botify, the experience to run the breadth and depth of analyses necessary to keep up, and the expertise to help turn analysis into action, maximizes the value of your overall investment in organic search. 

How to diagnose which pages are not being crawled by Google

Google might not be crawling your entire site. You need to discover which pages aren’t being crawled and why.

Every time Googlebot crawls your site, your team may have deployed something that prevents the spider from picking up on your content. You need recurring monitoring in place to alert you of those page drops every time they happen. 

With Botify’s SiteCrawler, you can overlay log and crawl data and simplify the challenge of finding those gaps.

One of the most popular charts within the Botify platform, within the Search Engine section of SiteCrawler, is a simple Venn diagram comparing what was crawled by Botify only, crawled by both Botify and Google, and crawled by Google only. This gives an easy visual of what your website crawl coverage is, with the ability to drill down further and compare characteristics across URLs or sets of URLs, across crawls. 

Within Site Crawler, you can also find additional insights surrounding:

Inlinks, redirects, and sitemap maintenance

Problem: If you have inlinks to 4XX or 3XX pages, redirects to 4XX pages, or 3XX or 4XX in sitemaps, you can be wasting crawl budget, as well as creating a poor user experience for both search engines and users. 

• Inlinks to 4XX or 3XX or Redirects to 4XX:  Sending search engines to pages that don’t exist, or a page where they are then redirected once again, wastes crawl budget and disrupts the experience of finding relevant content on your site.
  • Solution: Within the Inlinks subsection in SiteCrawler, set a filter to view your URLs by status code, quickly understand the volume of URLs impacted, drill into them specifically, and work with your team to remove or replace.

• 3XX or 4XX in sitemaps: sitemaps should only contain clean URLs, i.e. those with a 200 status code. If search engines see URLs with 3XX or 4XX, they will start to lose trust in your sitemaps and no longer reference it as a signal for indexing.
  • Solution: Within SiteCrawler, visit the Sitemaps section and filter your URLs based on status code (or by showing all but 2XX status codes). You can see the breakdown based on your website’s page type, category, country, domain, and more, and see a list of each URL and the specific status code. From there, you can prioritize removing them, and for 3XX, replacing them with the destination URLs.

Non-self-referencing canonicals

Problem: URLs may get linked with parameters and UTM tags leading Google to pick up that URL as the canonical version and multiple versions of the same page getting indexed separately. So a self-referencing canonical lets you specify which URL you want to have recognized as the canonical URL.

Solution: Checking canonicals one by one is nearly impossible. Botify checks for common canonicalization mistakes and has reports that give you visibility into whether or not your canonical tags are following best practices. The platform shows you everything from content similarity between a page and its canonical to pages that canonical to a URL with errors.

Depth of content 

Problem: If strategic pages are buried too deep within a website, search engines may not discover them, in turn not index them and eliminate the chance of traffic from organic search. Search engines may also not return and try to find them often enough, impacting crawl frequency. 

Solution: Look at how many pages search engines are finding in a given crawl, and how many have a page depth of three or more. Typically, if a page needs more than three clicks to be reached, it will perform poorly, and excessive page depth can be a threat to your SEO. 

In order to mitigate page depth issues, consider your pagination, limit navigation filters, and move or remove tracking parameters.

URL response time

Problem: If the response time of your URLs is too slow, typically over two seconds, there’s the possibility that Googlebot will not crawl them. If they’re not crawled, they won’t be indexed, eliminating their ability to drive traffic. A page that loads in 500ms or less is considered extremely fast in the eyes of Google.

Solution: Use Botify’s SiteCrawler to find your URLs that are indexable, but are loading too slowly and compare it to what is actually being crawled, indexed, and rendered fully, as elements on the page might be the cause of the slow response time. 

The ability to integrate crawl and log data in a single platform for easy comparison is also a key distinction between Botify and some of the other log file tools. Botify unifies this crawl and log data with keyword data and website analytics for the most comprehensive analysis of your website. Logflare or Screaming Frog’s Log Analyser, for example, are more manual, less automated, and not efficiently integrated with other data sets, posing risks for inaccuracy and missed opportunities. Furthermore, using machine-based storage instead of cloud-based storage makes it more difficult to share data and views with key stakeholders. 

Finding the right partners

With lean teams, the constantly changing digital and SEO landscape, and heightened consumer expectations, brands need to minimize the busy work and develop an organic search strategy that combines the best of technology and expertise.  iPullRank up levels the strategy of many enterprise brands utilizing Botify’s technology – and can do the same for you. 

Learn more about Botify’s LogAnalyzer and for a complete log file audit, contact iPullRank. 

The post Log File Analysis: A Must-Have for SEO Pros appeared first on iPullRank.

In this article, I will explain what the desktop page experience update means for you, and how you can help your site to rank better with the new features.

What is Page Experience?

Page experience refers to the signals that evaluate a user’s experience with a web page. It goes beyond the value of the content on the page and includes core web vitals that measure loading rate, visual stability, as well as mobile-friendliness. Google’s target is to improve the experiences of users who search for information on the platform.

Some years back, hackers began using spam techniques to manipulate the way pages ranked on Google. As a result, monetized sites appeared more relevant to search queries than the real content. At the time, the experiences of users were not considered.

However, when users gave negative feedback to the rise in irrelevant content, Google re-strategized. This gave birth to HTTPS (security), mobile-friendliness (device context), and AMP (accelerated mobile pages – for speed) as ranking signals. The page experience signal is simply icing on the cake to ensure user satisfaction.

Why Does the Device Matter for Page Experience?

Previously, Google indexed only the desktop version of web page content. Since more people now search for information using their mobile devices, Googlebot indexes and ranks content differently for both desktop and mobile devices.

In fact, search engines, especially Google, now display more search results for mobile-friendly pages than desktop. When you search for information with your smartphone, you are more likely to want local information since you are on the move (naturally that’s changed with the pandemic). But with a desktop, the results will include more text, images, and videos. Hence, the ranking for both devices will differ.

Additionally, people want to see quick results, so responsive websites ( a dynamic version of the site that adapts to the device depending on the viewport size) are more likely to rank well. If a website is not responsive, it might be more visible in desktop search results. This implies that if someone searches for the same keyword using a mobile device, your website may not even show up. Hence, the mobile-friendliness of a site matters.

Sometimes, however, a user’s experience on a website may be influenced by the speed of their internet connectivity and the type of device in use. As a result, different users will use different devices for personalization.

In 2019, Google rolled out its mobile-first indexing, which indexes and ranks mobile version content first. Eventually, Google plans to move to a mobile-only index, though that’s been delayed to some undefined time in the future. However, the page experience desktop update emphasizes providing the same user experience on both desktop and mobile.

How Do You Check Your Page Experience for Desktop?

Google Search Console

To check your page experience for desktop, do the following:

• Sign in to Google Search Console on your computer.
• You will find the “Experience” section on the left side of the screen after the “Index” section. Below it, you will find a menu containing Page experience, Core web vitals, and Mobile Usability as shown below.

• Click on “core web vitals” to see a desktop report of how your site is faring.
• Next, click on “page experience”. This will display the percentage of good URLs on your site and the total impressions on the good URLs. Depending on your SEO strategies and best practices, your ratings could either be good or bad. 
• Google Search Console also displays a graph of your URLs’ performances over a period. You can click on any portion of the graph to see the performance for a particular day or week. You can also evaluate the performance of individual URLs from the search console.

• Below the chart, you will see the desktop page experience signals that Google uses to rank sites: core web vitals and HTTPS. The ratings can either be good or bad too. 

The report does not include mobile usability since desktop sites do not need it to achieve a good score for desktop page experience.

Lighthouse

Lighthouse is an open-source tool that you can use to improve your web pages. 

It audits individual URLs on a website for five parameters: 

• Performance
• Progressive web applications
• Best practices
• Accessibility
• SEO

From the audit report, you can see how your web pages are faring on web applications, desktops, and mobile devices.

Lighthouse runs in Chrome DevTools. You can also install it as a chrome extension or use the Node CLI. And it is very easy to use. 

Simply input the URL of one of your web pages on the tool, then allow it to run an intensive audit. It will display a report of the parameters that are doing well and those that need improvement. The tool is generous enough to explain why those parameters are not too good and what you can do to fix them.

Let’s see how to audit a web page using Chrome DevTools

• Install Google Chrome on your computer.
• In the search bar, enter a link to the page you want to audit and open it.
• Right-click, then select inspect to open Chrome DevTools.
• A page will open on the right side of the window. Click on the right-shift operator (>>) on the header of that page, then go down and select Lighthouse.
• Under “categories”, ensure that the five parameters for the audit are checked. For device, select “desktop”.

• Click on “Generate report”. It will take between 30 seconds to 1 minute to see the audit report.

If you installed some extensions on your Google Chrome browser, run the audit in an incognito window. Chrome extensions may reduce the load performance of the page.

Below is an example of Lighthouse audit reports for The New York Times vs Washington Post.

The performance score gives you an idea of how each page is faring based on the desktop core web vitals. If you scroll down, you will see opportunities and diagnostics, which are suggestions that can help the page to load faster. They may not directly influence the performance score though.

It is important to note that performance scores can fluctuate due to certain factors. These factors include the type of desktop used during the audit, antivirus software, browser extensions, and ads on the site.

After diagnostics, you will find a section called “passed audits”. The items that are checked in green are the applicable audits while those that are unchecked do not apply to the web page. 

From the photo shared above, the performance scores of The New York Times and Washington Post are below par. A good score in Lighthouse is between 50 and 89 while an excellent score is between 90 and 100.

Furthermore, The New York Times has an excellent accessibility score compared to Washington Post. Accessibility measures whether people with impairments can navigate the site or not.

In this case, The New York Times has settings that allow site visitors to increase the font size. On November 8th, 2017, the site started including closed captioning (CC) in videos featured on the site. In 2019, it enabled transcripts for both videos and audio.

Washington Post, on the other hand, has accessibility features such as closed captioning, alt text for images, color contrast, and keyboard navigation.

Best Practice score checks whether the web page utilized methods that prove trustworthiness and safety. SEO (search engine optimization) refers to the page optimization that enables it to rank high in search results. You can see that The New York Times scored 100 in SEO. Overall, it is safe to say that The New York Times is more user-friendly than Washington Post.

Here’s another example of two popular e-commerce sites: Amazon vs Etsy

Amazon has a performance score of 88, which is good, while Etsy’s score is excellent at 92. This implies that Etsy is more user-friendly compared to Amazon; online shoppers can easily find the products they want on the site and purchase them.

When it comes to accessibility, Amazon shines brighter than Etsy. This implies that more users can access the site’s content. When a site is accessible, there is a huge potential for visitors to convert to paying customers.

In the SEO segment, Etsy does better than Amazon. On a general note, both sites are not doing poorly. If the site managers adjust a few items, following the suggestions on Lighthouse, there will be significant improvements. 

In addition to using the developer tools in the Chrome Browser, you can use another free Google tool: Page Speed Insights.  This tool uses the Lighthouse framework but lives independently of the Chrome developer tools. Using Page Speed Insights, you can more efficiently toggle between Desktop and Mobile versions of your site:

How Does Page Experience Impact SEO?

The anticipated desktop page experience update does not rule out the need to create quality content. It is a call for marketing teams to strategically ensure that the desktop versions of their sites are more user-friendly. So, if your site is user-friendly but does not offer quality content, it will rank low. Thankfully, you can always reach out for a content strategy consultation to help with that.

In a podcast, Google’s John Mueller, Martin Splitt, and Gary Illyes discussed something about a tie-breaker with respect to page speed, a part of the core web vitals. 

Does this mean we should be bothered about the page experience update, especially if we stick to the new rules but a web page loads slowly due to factors beyond our control? 

Does it really matter since our SEO tools can make recommendations for improved site visibility in search results pages?

This is a confusing situation because when Google started using HTTPS as a ranking factor, there was a tie-breaker too. Google claimed to prioritize HTTPS sites over non-HTTPS sites if there was a tie. 

Well, the right thing to do is increase your knowledge base of core web vitals since they affect the way visitors use your site. The metrics go beyond the traditional optimization strategies, utilizing complex SEO tips and tricks. The reward for doing this is that your site will earn a good spot in search results because Google promotes content that a lot of users find valuable. 

Remember that page experience is not just about the content but also how users interact with the entire website. That is why it is important for CMOs to focus on content, page experience, and SEO if they really wish to increase conversion at the end of the day.

Other Considerations for Page Experience Beyond SEO

Take off your SEO hat for just a minute and you will realize that your marketing strategy requires a rewarding page experience. At the base of this experience is a web page that loads faster than the speed of light.

Imagine that you visit a website with the excitement of seeing what they’ve got only for you to meet a site that loads endlessly. Even if the excitement sustains you for 10 seconds, probably because you got a hint of what the website is offering, you’d be turned off and could leave eventually. Yeah, a slow web page is a huge turn-off.

According to Cloudflare, the first five seconds of loading a page impacts the conversion rates. Anything beyond that will reduce your conversion rate by 4.42 percent on average.

Some factors responsible for slow loading are:

• The response time of the server
• Redirecting visitors to a different page
• Embedding heavy videos and large-sized images
• Using too many JavaScript, CSS, and HTML lines of code than necessary

The truth is that people are willing to forgo the fancy themes, images, and videos just to access a fast-loading web page. You stand a better chance of converting visitors to customers if they can access your website within the first 2 seconds.

Thankfully, there are several steps you can take to increase how fast your web page loads.

How to increase the speed of your web page

• Choose a hosting platform that is built for speed. If you are just starting, you may be tempted to go for shared hosting because it is cheap. However, bear in mind that the activities of other server tenants will impact your loading time.
• Boost your web page speed by leveraging a content delivery network.
• Use an image optimization plugin to compress large image files. Also, compress your HTML and CSS files as they usually contain duplicate lines of codes and white space.
• Host videos on platforms like YouTube, then embed the links on your site.
• Reduce the number of links that redirect people to another page. Some backlinks are not necessary; get rid of them.
• Enable page cache to reduce the TTFB (time-to-first-byte). What this does is present the static features of your site immediately after a user tries to access it. Afterward, it can gradually load the rest of the site’s features like images, fonts, videos, and so on. You can also enable caching for the browser to prevent stylesheet, HTML, and JS from loading all over again when someone visits the site.

Now that you are aware of Google’s desktop page experience update, it is time for CMOs, SEO managers, content developers, and all concerned to go back to the drawing table. 

Delivering great content is amazing, but how well can users access them? Is your site secure and trustworthy? Does your page load within 0 to 2 seconds? These and other metrics in the core web vitals will help you to align your website with Google’s new ranking factor.

Are you ready to crush your marketing goals and get the conversion rates your business deserves? Download the Ultimate Personas Guide eBook to help you understand and relate with your target audience at a deeper level. 

The post Page Experience for Desktop: Your Preparation Guide appeared first on iPullRank.