Mike King: Please welcome James Cadwallader. 

James Cadwallader: Hi. Nice to meet you. Hey, Lily. How you doing? I’m James, co founder, CEO here at Profound. We’re based in Union Square, so we’re just around the corner. Yeah, definitely got some impostor syndrome, me here talking to so many smart people about the future of an industry that I’d argue you all know much better than I do. We’re a tech company. We launched back in August of last year. And I think, I’ll be really quick. I think this is one of the last or if not the last one of the day. Yeah. So, I’ll probably be able to whiz through this in like ten minutes. And I think it’s going to be pretty top level. I think there’s lots of really fantastic granular data there that you can chew on. But yeah, we’re building ProFound. We’re a tech company at our core. So, we’re a team that’s ex Uber, ex AMD, ex Microsoft, ex Datadog. And, we are approaching this like a tech play. Our view is that answer engines represent one of the biggest platform shifts in the history of the internet. And, I think all of this, I would be shocked if it’s not just the same stuff you’ve been hearing because it’s obvious that this stuff is so monumental. But, maybe there’ll be a bit of framing here that, I don’t know, helps solidify things a bit or it’ll be – there’ll be some interesting take outs at the end. Now, I’ve got you all very excited, obviously, with my British self deprecation.

So, today I want to talk about what we see coming which is the agentic internet. So, the way that we’re thinking about this is to an extent, and this is a framer, this is high level and we’ll kind of narrow in a little bit more. But the traditional search was a two sided marketplace. You had the consumer would go to that list of blue links, and they would discover what they want, they would interpret the information, and they would consume the information that they found contained there within. But, we’re now at this inflection point where humans no longer need to visit websites. And that’s kind of the crux of the whole thing. Right? Instead of browsing an index of blue links, we send an agent on our behalf. And more than just browsing that content, the agent will actually interpret and prepare the content and then feed it up to the end user on a silver platter and say, hey, here’s the answer. And sure, you know, there’s a PDF thing. That’s meant to be the user chewing. The agent will tell the user where the content came from. It will provide sources and citations. But, frankly, who cares where that content came from if it tastes good. I think our point is I’m not clicking those sources and citations. I don’t think people will long term. That’s just vibe based. Yeah. 

Search was this two sided marketplace where consumers visited brands directly. Answer engines steal the end user from the brand. And, I think that’s the kind of that’s the big point I’m trying to lead with here. And eventually, our take is that we will enter a zero click future and the consumer will only interact with the answer engine and it’s so hard to stay up to speed because obviously since I sent Mike my slides, there’s now been the ChatGPT integration with Shopify and that’s becoming clearer and clearer as we go. Every week, this becomes clearer, this thesis. And, agents will be the only visitors of websites or the primary visitors of websites is what we’re saying. And, I think Karpathy, who’s one of the godfathers of OpenAI and from Tesla, he’s got this fantastic line which you probably all saw doing the rounds on Twitter or LinkedIn where he says 99.9% of attention is about to be LLM attention. And for what it’s worth, I think we’re using LLM attention liberally here. We say answer engines instead of LLMs because I think when you’re talking about just the pre-trained LLM, I think that’s very different to the actual end product of ChatGPT, which now has system prompts, memory, personalization, a ton of retrieval. Through Profound, our platform, we’re seeing like sometimes 70, 80, 90% of commercial search will return via what is being referred to as grounding or what we would say as web retrieval.

So, we now think – this just framing, this concept of agent experience. We’re all very familiar with this concept of UX. You know, the overall experience of a person using and enjoying a website. This is a discipline in its own matter. But what if people aren’t using websites anymore? Operator from OpenAI, which is what this screen showed you. I should have put a label on that. But that’s operator from OpenAI will browse on our behalf. And I think, you know, I’ve got friends who work at the big models and the stuff that those guys are cooking behind the scenes that’s just around the corner is where our heads are at right now. Operator, computer use from Anthropic, this is here, it’s happening, period. So, agent experience is the intentional design of a digital environment to ensure that your content, your brand, your products are legible and actionable and they can be interfaced with via an agent. Again, I’m using agent maybe a little bit liberally here. That could just be a web, that could be a retrieval agent for intensive purposes, ChatGPT user agent, as we’re all familiar with. But if SEO, to sort of go there, this is it. This is the two slides articulate what I’m saying. If SEO is the art of ensuring that your stool was well placed in this marketplace for the end user, ax is just straight up understanding that you have an entirely customer. You have one VIP customer now, which is an agent. Agents are going to become first class citizens of the internet. Marketers, every brand on the planet, is going to need to learn how to understand they’re going to have to connect with consumers via agents. Again, it’s all very wide aperture. AX experts, maybe that’s not the term that sticks, but this new discipline is the idea of rethinking brand, product and content for the new end user. I think, Mike, that’s some of the stuff that we’ve spoken about. It’s like rethinking this discipline. Shopping is gonna be a huge push for us at Profound. We think shopping is inevitable. Agents won’t just provide answers to the end user. They will autonomously decide where best to shop. And then, they’ll transact on the user’s behalf as well.

I mean, Sam Altman, this is March. We’re talking. This is weeks old, this statement. If anyone wants it, it’s the most recent interview with Sam Altman on the Strathecary newsletter. He has some really interesting takes on this stuff and I’m pulling this from there. He talks about, hey, we’re not so sure about ads. What if we could do a 2% affiliate fee instead? What if we could just avoid this grubby world of advertising and instead, we’ll just send agents to transact and take a little 2% affiliate fee?  It’s interesting. And OpenAI, I mean, if you want a hot take while I’m standing here, this is unrehearsed, I personally think OpenAI are going to create a monopoly here. I think that’s super possible. I think memory will make that product the stickiest product in the history of software. It’s already sticky. I would never shift to Gemini or Grok right now because ChatGPT has all of my memory installed in there. 

So, this is the Adobe case study that everyone in the room, I’m sure, has seen. But, you know, 87% said they’re more likely to use AI for long or complex purchases. The x y axis that we’re always referring to is the do AI answers matter to my brand? On the x axis, you have high consideration.  Are you a high consideration purchase? Yes or no? Bam. Okay. High consideration, high up on the x axis. On the y axis, you have That’s one of those nightmare moments where you forget what’s on the y axis. Josh? Oh, the y axis, yeah, I guess is just cat. Yeah, yeah. Okay. So, that was all off the top of the dome. Bear with me. I think, yeah, this moment of memory, I think is This is kind of the this is the damn busting moment is memory. For anyone that’s not using ChatGPT with memory right now, highly advise you go and try O3 in particular with memory turned on. It’s it’s this is it’s groundbreaking. It’s like it’s a whole new product. And, yeah. OpenAI have introduced this new sophisticated memory. The way that we believe this is working is this concept of a dream state which runs over historical conversation logs. So, you have your previous conversation logs, all of the conversations you’ve had with ChatGPT and then you have this dynamic LLM call that’s being run over and over and over again and creating, think of it like a dynamic system prompt that gets pushed into the system prompt to create your answer. 

It’s not just a rag call is my point. This is becoming a very sophisticated system. So, and I think this quote from ChatGPT underlines it perfectly. The more you use ChatGPT, the more useful it becomes. New conversations build upon what it already knows about you to make smoother, more tailored interactions over time. And yeah, I think ChatGPT will use these tools on the end user’s behalf. Proactive conversation. I think this will be the next big breakthrough of like push pull conversations, which is – this is absolutely inevitable. What does this mean for everyone in the room? It’s a push pull conversation. It’s that true walking Phoenix, her type moment. And yeah, this is another quote from April. So this is weeks old. The upload happens bit by bit. Soon, ChatGPT will proactively push things to you. That’s from Sam Altman himself. So you can imagine this happening. At the moment, ChatGPT just responds. What happens when it starts giving you suggestions that are based on context, location, and that personalization of memory? 

Advertising. I think it’s uncertain. I think there’s a hot take here that ChatGPT won’t integrate advertising. I think I stand by that. I think they have a corporate structure that makes it really finicky for them to introduce advertising. I think they have a very research heavy background. I think that would be very, very interesting for everyone in this room if billions of eyeballs have all migrated into one place, but you can’t access them through paid ads. Then all of a sudden, everyone in this room becomes pretty interesting. But, if they do do it, I think, don’t just think of it as advertising. I think it’s this combination of generative advertising plus the agentic internet, everything we just spoke through, plus memory creates this mind meltingly different environment where what happens when the agent comes back with a response, but then suggests something that’s perfectly tailored to that exact moment where you really wanted to maybe consider an alternative and gives it to you in that moment there.

Or one step further, B2A. If you have B2B, B2C advertising, what happens when you can advertise two agents as an interesting side point operator from OpenAI actually already considers advertising that it sees, is what we’re seeing, which is very interesting. So, AX has arrived. The first role of AX is to pay attention to how agents are interacting with your brand across all channels. So, I guess this is where my company steps in, Profound. Using AI, so using Gen AI, we study the patterns and behaviors across data sets and then use these insights to inform and create new content with principles. So, these are just some of the things that we’ve been learning so far. Mike’s prompt was like come with some actionable things. Think, yeah, I’ll whiz through these very quickly. 

We’ve, so at Profound, our platform, our website is tryprofound[.]com. We look at all of the, so we essentially help brands understand and control how they show up in answer engines. So, we’re looking at hundreds of millions of data points. Every week we send inordinate amounts of prompts and understand how these answer engines work. We see that agents don’t care about traffic when they’re going to visit sites. They don’t care if the authority of a site doesn’t matter. They also don’t seem to care about backlinks, which is pretty interesting. We see this super frequently. They don’t care if you’ve loads of backlinks. All they care about is the information on the page. Agents mostly don’t interact with JavaScript, bar operator maybe. Also don’t, remember they don’t interact with images on the page. So, that’s pretty interesting. I’m talking about retrieval crawlers here. I’m not rendering JavaScript predominantly. They love structured data. So, of course, serve it up to them with LLMs.txt. Someone’s probably done a session on LLMs.txt today, I’d imagine. 

We’re seeing the semantic URLs. Everyone will probably have a little laugh to themselves here because I’m 20 years too late to the game here. But semantic URLs do seem to really matter to agentic crawling in how you get pulled into answer engines. And, they pay a ton of attention to metadata. So, we’re seeing the agents effectively will go and do the pre-reading in the metadata to decide how they want to answer that question. They’re kind of lazy, as you’d expect, to save compute. So, when I sent Mike these slides, this was true. It’s actually maybe since the introduction of o3 become less true, which is interesting. What we’ve been seeing is, I’ll go back and forth, but we see that comparative listicles. This is across 177 million web sources that are being spat back by answer engines. We see comparative listicles being pulled through super frequently. But, you know, and the reason when you think about it is logical. Why would this agent, if you ask it best running sneaker, it doesn’t want to go to Adidas and Nike and go and visit those sites and run the inference. It will just instead go to a comparison site and cheat. Lazy agents. But, what we’re seeing now is o 3 is doing the hard work. o 3 is doing so much web retrieval. It’s the future. It’s mind blowing to us what’s happening there.

So, This is one of our customers. We’ve seen them. They increase their citation share and then their visibility goes up. Boom, boom, boom, boom. Citation goes up, visibility goes up. Agents like shiny new things. This is my slides getting scuffed up here. Okay, what this does say is newness is very important. So, ensuring the content is very new seems to be preferred. This also says agents like social media. This is what that slide was meant to say. So, yeah. Yeah, we see Reddit of course getting pulled in a bunch, but there’s a bunch. We see YouTube getting pulled in a bunch. We work with a major consumer electronics company. You definitely all know this brand. And even with brand direct queries in ChatGPT, we’re seeing it run at YouTube and go to the brand’s owned YouTube channels, which is pretty cool. 

But yeah, this is just the beginning. I think the bad news is that SEO will likely become an antiquated function just as an output of search becoming an antiquated function over time. But I think everyone in this room, I really really mean this, it’s so obvious to me and I think Mike, this is some of the stuff that we’ve spoken about. Everybody in this room has this kind of like once in a century opportunity to own what might well be one of the most important functions in the future of marketing. So, I think that’s pretty rad. So yeah, at Profound, we believe that AX will become the most important function of marketing and we’re building technology. We’re not an agency. We’re here to build a platform for everyone in this room to use to help you and your teams and your clients and brands navigate and win this new agentic internet. So, track your AI visibility, find out what matters most, build your strategies as you have done, understand the details. Yeah. And thank you very much for having me. Appreciate it.

Mike King: Frontier SEO Research and Data Science, please welcome Manick.

Manick Bhan: SEO Week. How we doing? What’s up? Mike, thank you for having me. It’s great to see you, brother. And I’m really excited to share with you guys some research today. But first, I wanna do a quick crowd poll. Where are my white hat SEOs at? Up here in the front? Actually, white hat? Come on. Keeping it squeaky clean? What about my black hat SEOs? Where’s our Snake? There he is. Awesome. Never underestimate the power of the dark side. And for those of you that said nothing, I guess you guys aren’t hat people, and that’s fine. But you’re one of these, I guarantee it.

We kind of think of of SEOs, as Mike alluded to, there is no one thing that is SEO. We all sort of look at it a little bit differently, and we have different approaches. We we have SEO sheep, we have SEO parrots that just repeat what they what they see on Search Engine Journal, or anything from Danny Sullivan. We’ve also got other people on here. We’ve SEO babies that are still, you know, reading the elementary Moz blog out there, and then we’ve got clown hat SEOs that are still pounding their sites with GSA backlinks, like it’s going out of style. But what I think we need more of today than anything else, all jokes aside, is we need more scientists and researchers like JR, like Jori, like the people here that are sharing their knowledge with the rest of us so we can all advance together. And I hope that we all feel inspired by what we’re learning and that as a result of some of the research that’s being shared, that we collaborate even more fully to learn more about SEO. And for those of you that don’t wanna do it, there’s always easy mode, which is I’ll just wait for Mike King to tweet and do whatever he says and still get good results.

The thing that’s exciting today is that if you widen your aperture, search is more dynamic and interesting and more of an opportunity than it ever has been before. And this really gets me excited because search is not just about Google anymore, there’s all these other places that we can focus on, that we can create visibility. These are the platforms where the game is gonna be played. Search is based on information retrieval science and mathematics. And the algorithms behind it, I would say they’re not pure, they’re subjective. Putting Reddit into the search results was not objectively the right decision. There’s someone that’s deciding what they think is right, what creates the best search experience. And obviously, we get user signals to validate or contrast that, but there is no objective, correct algorithm like the Pythagorean theorem for search. It’s being discovered, it’s being written, it’s being understood live in real time. And I think all of us here believe that we know important parts of how this algorithm works.

The thing though is that the search volatility is at all time highs. We have never seen volatility higher than what we’ve seen in the last one year. Beginning with November, the scale of that volatility was unprecedented. And, what is becoming very clear to me and to I think many other people in the room is that Google is playing their own game. It’s not the SEOs in this room against other SEOs in this room. I bet you, if we surveyed the room, there’d be very few keywords that we’re competing with each other over. It’s us against them. And so, in order for us to triumph and win, we have to collaborate, we have to work together, we have to share our research and knowledge to advance the industry forward because I believe, as an industry, SEO is at least a decade behind the science that the search engine is doing. We’re seeing patents or books in information retrieval systems published in 2009 that we’re only now starting to talk about as an industry. And so my mission as the CTO of Search Atlas is to help decode the search engine and to share what we’ve learned with the community and to adopt an SEO approach that stems from holistic SEO. It’s not just content or links or technicals, it’s all these things, including click signals, and to bring the industry forward. And so, today, I hope to take you guys through my research and I hope that each of us, everyone in the room will learn something from this that will help you in your practice, whether you’re an agency or you’re a brand, that we can advance all of our SEO together through this research. 

So today, let’s step into the SEO lab. I’m gonna take you through the research that we’ve been doing to decode Google for the last six months and I’m gonna give you a lot of free stuff that we built. It took us a long time and I’m excited to share this stuff with you guys. So buckle up, buttercup. It’s gonna be a wild ride. Let’s do it. So first, I’ve been doing SEO research for the last seven years. I believe that decoding this algorithm is worth over a $100 billion. Right? It’s highly, highly valuable. And so, to decode that algorithm is gonna be an exercise not in futility, but in great triumph if we can figure it out. And so, we’ve been doing single variant tests and sharing our research. We spent over $30,000 a month on the team that’s doing this work. And I spend most of my time engaging with them and pushing that research forward. Today, I’m gonna give you guys some of the work we’ve done for free. Okay? 

My GitHub has a variety of things on it that may be of value to you. One of the things that I’m really excited to share is Patent Brain, which is a GPT with 9,500 Google patents, as well as some other stuff on here. So, if you are interested in that, check out my GitHub, and I’ll take you guys through what we did. So, 9,500 patents. And, you know, I myself have probably read about 200 patents. I would have read more, except my wife is like, please put the patents down. We want to see you. We want to hang out with you. So, thankfully, we can train the GPT with all that data, and we can learn. We can survey that information. We can get way more insight, way faster. If you guys wanna try it out, you can go to Patent Brain on Search Atlas where we’re hosting it and ask it questions like these questions. These are some of my favorite questions to ask that I found to be the most revealing, the most insightful, where we can actually get from the patent literature algorithms, the exact algorithms that are being used mathematically. We can ask it questions about which ranking factors they’re using. And this is all theoretical initially, but that theoretical moves forward into science. 

Let’s start with semantic distance. Much has been said of topicality here, rightfully so. It’s an important aspect of the algorithm that we’ve seen. We’ve seen it empirically through case studies that people have done like Core I, amazing case studies of topical authority, but we’ve never quantitated it. And so, to begin, we wanted to understand if we could plot and visualize the topicality of a website much in the same way that Google is doing it, putting all that content into vector space and the nodes that are close together are thematically related. And so, can we actually do this? And we know that this is important because we can see it in the leak. Thanks to Irfan and Mike and Rand sharing that leak with all of us, we’ve learned that there’s three variables that are interesting that should be studied. The site embedding, which is that vector space Who is it that presented that earlier today? Jeff. Jeff, where are you? Jeff presented this fascinating work on visualizing that vector model. We also see a variable called site focus score and site radius. But our industry, we’ve never even tried to quantify these things yet. 

And so, the first thing we wanted to do was try to quantify that. So, this is an analysis using HDB scan of the page embeddings of my website. You guys can take my code and plug in any site you want. You can plug in your own sites and you can analyze them from a topicality perspective and look at how your sites are organized. The first analysis we did was based on heading vectors. So, is actually how we calculated it. And then, you guys can actually get the code where you can look at it line by line and see how we did it. But we used this library that I can’t kind of pronounce the name of. It’s like Traffalatura. Nice. Anyone know is that Italian? And that allowed us to remove all the crap from the from the documents and then extract their their real meaning and then run them through sentence transformers to then calculate the site radius, which is the distances on average of all the nodes from the site centroid, which is really interesting. Then, as we did more research, and you guys can put this into patent brain, ask it how Google actually looks at documents and you’ll see something called truncated term vectors, which was an interesting concept for me. We looked we looked it up and we actually tried to understand mathematically how are they deriving that, and then we replotted it based on those TFIDF terms, the top twenty. You can change that to top ten if you want, just change one number inside the code and you’ll see a slightly different visualization. So, is really interesting.

There are two applications of this. One is reducing your site radius by pruning. This is an approach that we know works. We have a case study where we took a very large online casino, gambling site, and we we noticed that what happened to these guys is they polluted the site with a bunch of irrelevant content. Right? They polluted its topicality. Hey, Lily. Hey. They polluted it with with a bunch of irrelevant content. And then what we did is we actually used this approach to remove that irrelevant content and we saw rankings go up because the site radius went down, the site focus score went up. And you’ll see another term on here called DCO, which I’ll explain later in my talk. The second application of this is to benchmark sites and to analyze their topicality and to see visually if we can actually understand how do these sites compare, who has more depth of topicality in a certain space. So, what we did is we – this is an example of this concept where we took two brands, Mudwater and Everyday Dose, and here you can see Everyday Dose actually has more content, more documents in that cluster than Mudwater does, which is one of the reasons that it outranks Mudwater. So, that was cool. Let’s go deeper. So, the next step was, can we actually calculate topical relevance? Right? That’s actually what Google calls it. They don’t call it topical authority. There’s no reference to that. It’s called topical relevance of websites. And the topics themselves are called topic identifiers. And the Twiddlers use those topic identifiers and the site score to re rank the information retrieval scores of the documents. So, we learned about how they’re looking at this from this important patent by, I think it’s Numbneet Panda, about user context based search engine. And so, essentially, they take the Internet and they organize that information into knowledge domains, and they build terms lists for each knowledge domain. So, that was interesting. Then we went deeper and we found other patents through Patent Brain that were related to that.

And so, these are the four that I think are the most important. So, if you want to go deeper, these are the patents to focus on. From these patents, we were able to get the algorithm that we believe that they’re using for topical relevance. Remember, as an industry, we know that it is important to quantify this, but we’ve never done it before. We don’t even know what our topical relevance scores are versus our competitors. So, this was something we felt was important to do research into and try to quantify. So, what we did is we discovered the sort of two approaches that Google talk about. One is an engram analysis of the terms on the website itself and its content. The other approach we see them reference is doing this on the keyword data. In the patents, they’re not always very clear which approach. They they kind of postulate about different approaches. And so, you have to kind of trace each approach and then run a correlation study, I guess, to figure out which one’s right. So that’s exactly what we did. We actually wrote the algorithm, we scored all these sites on the x axis. These are all SEO sites.

And, one of the things that wasn’t clear to us is how granular are Google’s topic identifiers. It’s not like they’re just going to hand it to us on a silver platter and show us what their topic IDs are. I wish. So, we had to try to derive them. And so, we actually did multiple levels, and I’ll show you that in a second. But what we found, you’ll notice there’s a site here, I don’t think I have a laser pointer, but on the far left and the bottom is LinkBuilder. Which was acquired by my friend Mark at the Hoff. And that site is very interesting because they outranked every other site for link building related keywords, even though they had almost no backlinks. And we’re wondering, how are they doing it? What type of dark magic are they doing to outrank all these other sites, including us? And then, when we did the topicality analysis, the story became clear that this is something that Google is using. So then, here’s another way to look at it. If you’re more of a bubble person versus a table person, but the story is the same, we can actually build our topical authority in these granular clusters. But is this BS? Like, is that mathematics just nonsense and bunk? Like, you shouldn’t trust my scores just because I created this thing and Mike has me on stage. You’ve to ask the question.

First is, how strongly does that topical relevance correlate with rankings? That’s an important question. Then the second thing was we wanted to know how does the topical relevance score compete relative to domain strength scores. Right? Which one is more effective or important? So, did the correlation study. 57,000 SERPs for every SERP keyword URL position, and we have three levels of topical scoring. We have the granular topic, then we have the hypernym and the hypernym above that, which is the knowledge domain. As I said, we didn’t know how granular Google is actually taking topicality. So, for example, is SEO as an industry, is it just SEO? Or is there SEO services and SEO tools and link building and all these other more granular contexts? So, we let the correlation be our guide, And this is what we found out. We found out that the topical score, the most specific topical score that we have, was the most predictive of rankings. And the crazy thing is the p value was .001, which means this is not chance. This is predictive, highly predictive. So that was absolutely astonishing. Like, it was so cool to see that. You rarely see that type of statistical significance when you’re doing this type of analysis. And the other thing that was really cool is that we saw that it’s actually more correlated with rankings. The topical signal is more correlated than our authority measurements. We looked at domain rating, which is from Ahrefs, and then domain power, which is something that my team at Search Atlas created. And it turned out that domain power significantly outperforms domain rating when it comes to predicting rankings. Domain power is a metric based on traffic. We believe that Google is no longer using PageRank in the classical way that our SEO vendor tools are spoon feeding it to us. As Mike said, the toolkit needs to evolve, and yet, it has not. We’re still living in the stone age. But thanks to this sort of research, we can actually open our eyes and learn some new insights.

So, as I mentioned, the statistical significance was very high, which is really exciting. And what we learned also is that don’t look at topical relevance as a generalized thing. It’s very granular. It can be very granular and you can win. You don’t have to dominate SEO as a whole, you have to dominate your slice of it. And SEO could be skiing, could be snowboarding, could be insurance, could be anything. You can be granular. And we also wanted to validate that we can actually use this knowledge to translate into ranking results because that’s the other important thing. It’s not just knowledge, you’ve got to take it and apply it and see if we can actually get something practical from it. And so, it worked very well for us. Have many case studies, but this is another one that showed that this actually really does work. 

Okay. That was fun. How many of you guys learned something? All right. Good. Let’s shift gears. Who are my local SEOs in the house? All right. Mac’s up in the front. Right here. Cool. Cool. All right. So, I’ve all I’ve for a long time believed that the local Like Google Maps, the ranking equation’s very simple. Way simpler than search. So, we’re like, let’s let’s break it. Let’s figure it out. Can we break it? So, those were our key questions. We wanted to understand, can we actually build the ranking equation? And then, the second thing is we wanted to know, is it the same ranking equation for every industry? Is it one or do they have sort of different weightings of the factors for each industry? So, we had about 10,000 GMBs, 20,000 keywords. We segmented those keywords by intent type. We removed all the branded keywords from the dataset because we knew that those weren’t going to give us a clean signal. They’d throw the numbers off. And you can see on the right the variables that we were measuring. We had to create these variables. We had to create these relevance scores. And so we did vectors based mathematics to understand relevance for the keyword versus a bunch of different components of the GMB. What do you guys think is the number one most correlated variable with rankings? Shout it out. Uh-huh. Interesting. Yeah. See? We don’t all agree. Whoever Who said proximity, distance? Nice, Lily. Sweet. What’s your name? Matt. Matt got it right, guys. Matt right over here on the left. Good job, Matt. Here’s what we actually found. We found that distance is the most correlated across all industries and sectors. But what was second most correlated, it wasn’t just reviews, it was the keyword relevance of the review. It’s not just how many reviews do we have, it’s does the review specifically bring about relevance to the keyword we want to rank for, which makes so much sense. Right? I’m a vegan, so when I go to dinner, I’m looking for restaurants that have good vegan food. If the reviews talk about vegan food, obviously, it should influence rankings. So, that was interesting to see this. Business sector is very important as well. Who said business sector back there? Was that you, Garrett? I was in category. Category? Same thing. Same thing, basically. Yeah. Well done. 

We also saw some other interesting findings. From Patent Brain, we asked the patents, what are the correlation factors? And one of the factors that we got was this thing called web score, is what Google calls it for for GMB ranking purposes. And it essentially is the relevance of the website for the keywords the GMB is ranking for. That was really interesting. So, we see the content of the site is actually important for rankings. It’s not just the GMB, it’s also the site it’s linked to. We’ve known that. I’ve believed that for a long time, but I’ve never seen data to prove it. And this was really cool, that now we can finally tell our for the local SEOs in the house, tell the clients, don’t just focus on the GMB, you’ve got to also do the site. Right? Now, can show them data that says that it’s important, which is cool. And the other one that I thought was really interesting on here is domain rating is at the absolute bottom. It’s not about backlink authority for GMBs. Okay? It’s not. Google has enough authority measurement signals on the GMB itself, they don’t really care that much about the strength of the backlink. 

There’s one additional piece of research that we didn’t get a chance to do, which I’ll mention in a second, which is the anchor text, really, of the backlinks. We know that they’re probably important from what we’ve learned from Patent Brain, but we didn’t get a chance to include them in the study. So, there’s more work to be done. We also then looked industry by industry to see, is it the same generalized ranking factors for all industries? Turns out that there are variations. And for some industries, like car repair, it’s actually less about the distance. And there, it matters more, the reviews, which is interesting. So, it doesn’t seem like it’s all one unified equation. They’re weighted differently depending on the business, which is interesting. If you guys are wondering what’s what’s the, let’s say, the weighting model for the industry you guys care about, all of this data is in a Shiny app. So, you guys can play with it, you can set your industry, you can set which SERP you want to keep in the analysis and run it for yourself. And I’ll give it to you. Just It’ll be on my GitHub. So, I hope that helps.

So, plenty of interesting insights. And next directions. One of the things we also want to test is the importance of Q & A and posts, GPT posts, because we know that they’re related. We also want to expand the data panel. All right. So, can we actually take this knowledge and improve our GPT rankings? Hell, yeah, we can. So, what we did is we we actually took this knowledge and we tried to implement it in an automated programmatic fashion across those GBPs on behalf of the businesses. The results were amazing. This is all automated SEO, essentially. And I’ll show you guys how we automated it at the very end. But plenty of of success. 

Okay. Next up, crawl behavior and GSE data correlations. So, Jory showed something really cool where she showed us all how we can By the way, Jory, where are you? Are you here? Jori? Yeah. Phenomenal work that she did with that GTM JavaScript code using GA 4 basically as the database for who’s crawling you. I’d never seen anyone present that before. We actually wanted to understand more about Google crawl behavior and how whether it’s predictive of rankings or how Google is viewing a website. So we took our JavaScript pixel and we started downloading all the data from the sites and we built a massive database of crawl hits across a population of 25,000 sites with over 200 million crawl hits in the database. Because all of our data is centralized, we can now run correlation studies with this data. So, this was super cool. This is the relationship between daily crawl frequency, Okay, and the amount of impressions at the page level. Can you believe that Google would recrawl a page ten thousand times in a day? Crazy. I couldn’t believe that. And you can see the strength of this relationship is so strong. It’s a very strong, like, linear signal here that we can see that the crawl frequency is very much determined by the search demand of that document. The more eyeballs that are searching it, the more they need to recrawl it to make sure that the document still exists, that the data is accurate. So, very predictive. This is including even the outliers in the data set. And you can see, even at the tail ends, for pages that are crawled 80,000 times a day, the relationship still holds. Looks to me like there’s a linear equation inside their database based on the search impressions, how many times they need to recrawl the page. Seems like it’s pretty clear. It’s pretty cool. 

We then looked at it with traffic, very similar relationship because obviously traffic impressions are related. We looked at it by average position, no relationship. So it has nothing to do with the average position. We also looked at the response time versus traffic and rankings. And as you can see, pages that are fast, way more likely to have traffic. Pages that are slow, way less likely to have meaningful traffic. So, we do see a relationship between response time of the page, and we’re able to get that, by the way, inside the JavaScript. Right? So, one thing that you guys can do if you do use Jori’s code, try to get the page load time inside that JavaScript and send that also into GA4 because that’s another data signal that we can use. You guys get it? Cool. We also looked at the page size. Like, how does page size influence rankings? And what we see is that once pages get beyond a certain size, they’re not likely to rank on page one of Google. We’ve always sort of known this, but it’s cool to see this in the data. 

All right. This part six is about content semantics and correlations. This is I’m really excited about this because much has been said about semantic SEO and content. I personally think we’re at an age where there’s a lack of respect for the written word. When LLMs can gibberish split out content left and right, a lot of people in SEO don’t really look at the word the same way that we used to. We just treat it as like this byproduct that’s just something that ChatGPT is spitting out. But can we go back to first principles and understand, even though this is subjective and content quality is subjective, how did Google make it objective? How did they do that? And so, to help us understand that, we essentially try to derive all of the different types of signals that they might be using. We know that they’re using a neural network that they’ve trained on content quality. And so, can we identify features of that neural network that can help us improve our ability to execute SEO campaigns? 

So, we had a 52,000 keyword data set, about a million URLs are in that data set, and we extracted semantic triples, factuality. We looked at the micro semantics of the content, we looked at the content vectors, We looked at the entity scores, the heading vectors and query relevance. And what we see is that the score, the higher the Scholar score, the better the rankings. Right? So, if you have a kind of a compilation of all these factors together, you’re more likely to rank well. So, that’s really interesting. It means that there is predictive power of that model. And some of the components were very predictive, like query relevance scoring. So, query relevance is essentially a metric we made up that is taking the keyword, making it into a vector, taking the headings of the page, making that into a vector and looking at the dot product, the similarity between the two. And we see a very strong relationship here between query relevance and rankings. That’s encouraging. We also see user alignment being very important, user intent alignment. So, the type of content that we’re seeing dominate in the SERPs. If you align with that, you rank better, which is interesting. I was surprised because one of the things we know about Google is they like a diversity of perspective. They don’t want to just have one type of page in the SERP. But, if you know what is the dominant type of page in the SERP that’s ranking the best, it’s better to align with that. And also, to create a page for each of the other intent types because you can rank really any of them in that SERP. We also saw domain power. I’m sorry. The desktop interactive millisecond time be somewhat predictive as well of rankings. We got that from PSI. So, we ran all those pages through PSI. And we also ran them through domain power. We see, actually, site traffic does correlate with site with URLs that are able to rank more highly on Google, which is great to see that. And when we put all of it together, we see that there are several components that are predictive, but there’s also many things we tested that are no longer predictive that we can see. Word count, not predictive at all. Actually, triples, like the number of triples in the content actually negatively correlated with ranking well. So it’s not just about filling your content with factuality. If you do that, it actually seems to negatively impact your rankings. So it’s very interesting.

What about off page? Well, we compared all the different off page metrics that we could. We looked at the domain rating from Ahrefs, we looked at trust flow, citation flow from Majestic, we looked at just OT and domain power and domain power one, hands down. I think this is leading us to understand that it’s not a page rank driven backlink graph anymore, So, it’s time to move on beyond those page rank concepts. The other thing that was interesting is we cross sectioned this by search intent type, and we found that the correlations were stable on all the search intents except for navigational, where things work differently. That kind of makes sense. Right? Like, when you’re navigating to a site, we’re not expecting to see the same type of signals inside the search results. 

Looks like I’m running out of time.  All right. LLM visibility and interrogation. So, how do we get our content into LLMs? And how does ChatGPT determine information consensus? Oh, Okay. So, this is essentially how ChatGPT brings its data in. And one of the things that we did is we said, can we actually interrogate it? Can we ask it 135 million questions and get the answers from it? And then can we also ask it, well, ChatGPT, where did you get that data from? Can we actually get the URL, the source document that they got that information from? And so we were able to do this and pull the source documents from ChatGPT. And we learned a little bit about where they’re getting their data from. The thing I think that’s most interesting is that it’s Common Crawl. Common Crawl is very important. I’ve got about 700 sites, publications that I bought over the last five years. And a lot of the sites have no OT, but they are in Common Crawl. And the LLMs have that content, even though those sites are totally bunk, the LLMs are not doing a very good job of filtering the content quality that’s coming in. So, you can get yourself into Common Crawl, you can get yourself into the LLM. If you wanna know if your site is in Common Crawl, you can use this free tool on their site to plug in your domain and you’ll see if you’re in there.

Krishna talked about IndexNow. I think it’s everyone in here should be using it. We were able to get 160% increase in Bing traffic. Some other ways of creating consensus, think it’s really interesting, is actually guest posting and getting on sites that are in Common Crawl. You can also use press releases, you can use Reddit, you can use your branded assets. You can also use Bing results to get into the 40% of searches that are enabled on on search GPT. You can also do petri-dish SEO where if you Google, Snackachusetts – how many of you guys have been to Snackachusetts? It’s this 52nd state right outside of Massachusetts that I made and turns out that if you create the content on Google, you can then ask Gemini and Gemini is none the wiser. It believes that this is something real. So, we can actually start to play with this and see how long until Snackachusetts shows up in ChatGPT. I’m waiting. Not yet. We’re going wait until they update their content.

And I want to show you guys one last thing, micro agents. So, micro agents are essentially prompts that run and refactor your content. And so, we created these 10 micro agents that we know are related to reducing the cost of retrieval scores for search engines. So, if you can run content through a battery of these micro agents, you can objectively enhance it according to that PQ scoring that we know they’re doing with the neural network. And so, case study, DCO was dynamic content optimization, where we were pushing the content through the micro agents and republishing it on the site without really doing anything by hand, ranking improvement. Because the content, we reduced the fluff, we increased the decisiveness, the specificity, the granularity and significantly improved the results there. And then, sorry, I guess I had a 10th thing about SEO automation. We built a bridge to bring AI models to the site and attempted to automate now about 70% of holistic SEO. And we implemented this on 152 websites. Every single site got a pop. So, we’re seeing that SEO techniques, they can be automated and they will be automated. We’re starting to step into that brave new world. It’s a little bit uncomfortable, but actually Max was sharing with me a case study of his own where he was using our technology and and got incredible results for a dentist, I believe. For a dentist? Dentist. And I saw it on SEMRush right after I used Auto and Search Atlas. Awesome. So, it’s not just reported as improving in our tool, it’s also on other third party tools. And more case studies. Okay, guys. Thank you. I’m a little bit over. This is super fun.

Mike King: A gentleman by the name of JR Oakes. Yes. Is like when I see him post stuff, I get jealous. I’m like why didn’t we come up with that? And JR, actually worked with him and a gentleman named Patrick Stocks, another gentleman by the name of Matthew Kay. And we throw another conference called Tech SEO connect in the Raleigh, North Carolina, area. So if you want more of this, come to that in December. But yeah, back to JR Oakes. JR is the VP of Strategy at Locomotive, leveraging 10 years of SEO expertise to scale business success. He’s a former architectural glass artist with notable installations for SAS, US Army and Coach K. JR now uses creativity to fuel everybody okay? Strategic growth. He’s a tech enthusiast and coder, always exploring new dark beers, jamming on base, and enjoying life with his family. Presenting Benchmarking Brand Discoverability, LLMs Versus Traditional Search, please welcome JR Oakes.

JR Oakes: Thank you. I’m gonna need that again. How are y’all? I thought Mike said that the the production value is gonna be high here. I don’t know. Amazing. My name is JR Oakes. I am the VP of Strategy at Locomotive. This is this image of me is a little old here. Mike, thank you so much for inviting and for everybody behind us. I know it’s a lot of work and it’s amazing. At Locomotive, we do a lot of open source. Here’s three projects we’ve worked on right here. Taxonomy, the latest one is called Crawling Chat. It’s cool because it’s a complete rag system in a box. You can stick in your XML site map and it will push out API for GPTs or MCP’s. We really like pushing out concepts as code. So when I started thinking of something for SEO Week, I really wanted to kinda do something different. And I was seeing a lot of people thinking about like rank tracking and language models in the same way and I don’t think they are. I I think of the concept of like a language model saying, domain one or brand one is really terrible, but brand two and three are really great. Right? So if you’re number one, you’re not in too good of a space there. So I’m not sure that they exactly fit.

So I started thinking about the idea of just like discoverability. Of if I have a brand and I’m thinking about like where people are gonna find me and get me into the set of brands that they’re gonna be researching for for procurement or whatever. Then I wanna think about like how I’m gonna be found. So here’s the key to this. We’ll start first with G2. I’m sure everybody knows what G2, it’s a software SaaS review site. G2 was really interesting for me because it’s kind of a homogenous set of the same category of intents. Right? There’s a bunch of different product categories or products on there, but the the intent clusters kind of around all of them are very similar. It’s obviously had some up and downs, but in Ahrefs, there’s a ton of category data. There’s a ton of search data. And it was the most comprehensive kind of grouping a consistent data across a bunch of different companies and brands. So what I did was this is a little hack, but if you wanna find hidden site maps, you can go to Wayback Machine and just search for XML files. And so I found the XML file of the categories for g two and then threw those into a trefs. And then we found the best base term for every category. Right? So if it was CRM tools, the best base term might be best or like CRM software. Right? And then we also looked through that and we wanted to find all of the modifiers. And so we found a ton of modifiers in the data set and I’m assuming everybody knows what a modifier is here. It’s kind of like Nike shoes and free Nike shoes, where free is the modifier.

So from that, we produced a ton of different search terms and then built out a rig to essentially hit most major models from the last three years across Anthropic, Gemini, OpenAI, as well as Google search and AI Overviews. So we essentially hit every one of the models with the search terms. We pulled back the content, extracted brands out of the content, and then ran it through a little magic tool to get the domain for each of the brands. And then once we had the domain, we matched it up with a trust data and then another dataset called the company’s API. The final dataset was 1.9 thousand industries, 63 search modifiers, 1.9 million brand results, and 57,000 companies. So not a huge data set, but I think a significant one. So looking at the data, one thing I think is important to call out here is that when you go to ChatGPT, there’s a lot of stuff in between you and the actual base model. That can be the GPT has memory of prior things you’ve put in. It could be tools that it’s calling, analysis tools, or it can be like search retrieval. Right? So none of that is in here. What we’re doing here is looking at the base model. And one other note here is that for Google, we’re kind of treating Google like a model. Right? You put a phrase into into ChatGPT and it does a wall of text. We’re kind of saying the same thing. You put a phrase into Google and you get a bunch of text back. So we’re looking at the text and we’re looking at brand mentions within that text. Make sense? Yeah. Okay. 

Alright. So here’s the first view. This is the top 30 domains mentioned in lM models and organic models by search volume. The things that are really interesting here for me are how much more SEO-y the organic results are with aggregated results like Gartner, G2, Captera, etc. And then obviously, how dominant Microsoft is in the B2B SaaS space. I had like no idea. Looking at the business type distribution of this data, obviously that’s a there’s a long long tail of B2B SaaS companies and a lot are probably held. Predominantly United States headquartered companies. And this struck me as odd that there are actually B2B companies that don’t have LinkedIn. Like what the hell are they doing? Anyway and then we can look at the distribution by revenue, which most of the results of companies were in that the smaller second smaller bucket. 

Okay. So now we wanted to look at some comparisons since we had data for all the LLMs. This is probably one of my favorite charts here. This takes the top industries by search volume. And then we look at average brand mentions across Google AI Overviews as well as all of the individual models. I think this is cool. Like just to see we should all be thinking thinking Google. Like how impressive is that? And especially how impressive is it going from GPT 2, GPT 3.5 and the number of brands that it does per industry category to now. And just how impressive it is, how much visibility brands get out of GPT-I mean out of Google compared to the other models. This is another one that I think is really interesting because I think it really goes to show like of the smaller companies in this dataset, how critical search visibility in Google is to their even being found by users looking for their products and services. And then we can kinda drill into that bucket a little bit and see all the different companies in there, which I know a few of these. But obviously Google and LLM know a lot more. One thing that I think was kind of shocking for me was just how closely the results tracked for LLMs and traditional search. I expected a lot more variation in this based on company size or company, I I would say, brand equity. And they’re very very similar. We’ll see other mentions of that. Like this one. I I think this is was pretty shocking for me, but like if you’re using we obviously had a Ahrefs data. So if we’re using Ahrefs keyword count as kind of a proxy for content production and success. Right? Then I think there are a lot of parallels between Google results and language models. So I think maybe that gives us a little bit of solace that the things that we’re doing from an SEO perspective and writing good content and growing content and growing authority reflects well for Google results, but also reflects well for language model results. I also think that there’s a lot of opportunity. This essentially just puts the models against each other from a brand overlap perspective. Right? And we can see how similar, obviously, Google AI Overviews and Google are. But looking at all the different models, even between models in the same by the same provider, there’s a lot of opportunity to look at each of these models where they’re being used and to at least measure. If you can’t measure it, you can’t manage it. At least start measuring like where you’re being found, where you’re being talked about, what the different models are saying.

I threw this one in there. I just thought it was interesting. It’s just if you have like low traffic B2B sites, you have to pay for your visibility. And as you kind of get bigger, you don’t as much. So kind of a throwaway here, but interesting. This is the one that everybody at my work was like fawning over because look how how aligned domain rating from Ahrefs is from visibility and language models. Like you expect this from a AI overview and Google search search percentage, but almost every model follows the same curve there. And like I actually asked them to do it in a different way and they did. And it’s the same curve. Right? So the more authority you have in search engines, the more presence you have online, the more authority and brand authority you have overall, we see representations of that from a a language model perspective. Next is geography and time. So we had location data on all the companies and then we also know when the models were produced. So that led to some interesting outcomes like this one. Obviously, you can see I wish I had a pointer. On the 5 to 9, you see the disparity between traditional search and language models. That’s obviously the reason why we have RAG today is to fill in the knowledge gaps and the recency gaps with language models. That’s definitely there. We even see that to a little bit, the 0 to 4. But maybe more interestingly, how quickly the models are catching up and and covering that gap with just more frequent model releases and model updates. We definitely see that. So they’re definitely getting better at new information.

I’m good now. Where’d it go? Yeah. Also, I’d love to talk to somebody about this here. Obviously, most of our Google search results were taken from the US, but we weren’t like sticking an implicit or explicit US into our model results. But most of the results were from the US. So it would be interesting to see if there’s query modification or something like that. Even though we were hitting the API, they the models are heavily balanced towards US results. But it does seem like at some level they’re getting more worldly overall. This one is kinda hard to explain, so I’ll try. We took the average share of voice by industry across all states where there were companies in that industry in more than one state. So multiple states represented and which companies, which state which states had companies that had the strongest share of voice in that. And I think, obviously, this is B2B SaaS, so California is gonna be really heavily influenced in that. But maybe the and really interesting thing here is just like Washington, Texas, and New York. Not kind of felt like they made some ground in language models. Maybe not completely as black and white as organic is here. And then I had to throw in a social media slide, so that’s what this is. I don’t really feel there’s any impact with social media on on either of these. But I do think that companies that do marketing and are more found online tend to do social media, so that’s that.

The next section is we’ll go into a little bit of the modifiers. So this is what the modifiers look like. We segmented them and I I forgot to mention this, but once we share this, you can get the data and see all the modifiers and see all the search terms. And that’s linked from one of the earlier slides. But we have all the modifiers here, 63, and they’re all in nice little neat categories here. So we were able to go in and do some comparisons across different modifiers. So what did this modifier do? So looking at the overlap between some of the modifiers for LLMs and traditional search, we see again, here is an opportunity for us to be looking at LLMs and seeing how those results are different than search. We’re seeing a lot of like very high quality, highly qualified leads come in who have done a lot of research and procurement research in language models. And so making sure that your brand is even like in the discussion there for some of these qualifications, I think is very important. This is interesting because this is looking at the base search terms. So Nike shoes, what show what what brands are found there versus red Nike shoes. Right? Versus the modifiers. And we can see that affordable, free, some of these are very very close between language models and organic results. But a lot of them have some pretty large disparities between the two. So again, looking to see where those disparities are in the models. And like all it takes is going to the actual models and searching. I’m not sure at a level that you need to be spending $10,000 a month searching 40,000 search terms to figure this out. But the models are pretty easy and accessible for $20 a month to be able to figure some of this out. 

This one is here because I really wanted the violin plot. Does everybody like violin plots? Okay. And then because I asked for a violin plot, they gave me two. So I think at least I think this shows an example that language models are at least getting more deliberate about producing brands and highlighting brands and talking about brands. So hopefully they get linked more. Here’s I think this reinforces the point here about other AI being different from Google. We can see how similar AI overviews are to Google results. And then just how dissimilar each of the other models are to Google results. Which again is opportunity to optimize those channels, which I think the two speakers after me are gonna go probably into a little more detail. I think this one is my favorite finding that I’m probably gonna make into a tool. The thing I like about this is kind of treating modifiers as fingerprints. Right? So if you think about it, we have like 63 modifiers. And for Google here, we can see that, you know, models think of their affordability is they’re really good for affordability. Right? We can see that models think they’re really good for for being free. Right? And essentially, how your brand is represented and peaks and valleys in the modifiers that are being used. I think that would be really cool to track over time to see how kinda your brand narrative and how it’s being reflected in language models and search results over time.

And then one other highlighting, just the amazingness of Google above all over other models in talking about brands and returning brands. And just their overall knowledge of brands, I think eclipses the other models. So some takeaways, and I think I’m running a little short. I would say just the diversity. I think that’s one interesting thing is that if you the same search term is going to return wildly different results in every model you give it. So if you’re doing any kind of rank tracking, it really matters what model you’re using and where you’re which model your prospective customers are using in terms of how you even get found. Right? So I think that’s super interesting. I think the other piece for me is the transferability. Like, just how much of the metrics of SEO that matter for us also matter from a language model perspective. And that if you’re doing the right things from an SEO perspective, we expect, at least from this data, to see that reflection in language models. Maybe you don’t have to scale out 10,000 pieces of content on other sites to try to influence models. You can just be really intentional and effective on your own messaging and branding and content. 

I think trying to understand, especially for companies and B2B SaaS companies that maybe wanna get into the US market or our other markets, understanding that there are some biases in the language models with which types of companies they preference. And that’s, I think, at a national level as well as maybe even regional level and how those are different from search and and AI. And then I would say the other piece is just LLMs lag, query deserves freshness freshness and the immediacy of of of information retrieval results. And this is something that I think you need to be aware of, encounter as more and more people are doing a lot of their heavy research and language models. You may not even be there if you’re an early stage company, so you need to be found in other channels.

Last one is, you know, if we go away from ten blue links and it it really is just a heavily heavy presence of of language models. I don’t know if there’s a place for aggregator sites at that point because the language model is the aggregator at that point. Right? So that’s it. We put together a little interactive thing in Streamlit that you can go here and check out. And it has all of the categories, industries, and you can kinda pick your industry and see the data associated with it. And then also wanted to thank Patrick Stocks and Carrie Sugarman for going through and sanity checking my dataset and data. And I really appreciate it, and that’s all I got.

Annie Cushing: Thank you. Yeah. This is my first conference since, like, 2018. I feel like I’ve been in witness protection. So thanks for having me. Okay. So I am going to talk about some of the hype, the hubris. I promise you it’s not – I don’t subscribe to the hysteria of, you know, hide your family, hide your kids, hide your wives, you know, AI is killing everybody out there, you know. But I wanna start with before I get into some of the issues, I wanna start with I mean, it is like what a time to be alive. There is so much good that AI is accomplishing in the world.

Harvard Medical School developed an AI model that they dubbed Chief that reportedly achieved 94% accuracy in identifying cancer. And unlike other AI models in the cancer space that usually identify a handful of cancer types, this was successfully tested on 19 different cancer types. As a cancer survivor, that is really exciting. In fact, I was treated right here at Memorial Sloan Kettering. One of the scariest things my oncologist said to me was, Akisha, you’re really healthy and and that’s good, but we have no data to guide your treatment. I was like, you have no idea how much that that rocks my world. So I’m actually part of a 10-year study so that they will have data. Harvard Medical School developed an AI oops. Sorry. It didn’t – researchers at Rutgers University developed AI tools to help prevent collisions with whales in the North Atlantic. So that’s been a huge issue, with the critically endangered North Atlantic whale. There are only 70 reproducing whales left. That’s how critically endangered it is. So with that preservation in mind, they combined two databases and now they are using AI to guide ships so that they can avoid the whales. So there was a children’s hospital in Texas that donated in an AI well, it’s AI software for a hospital in Malawi, Africa which really struggled with stillborns and an infant mortality, specifically neonate neonatal deaths. In the three years since they donated this software, there has been an 82% drop in stillborns and neonatal deaths. This one is especially cool. Of the 4,000 indigenous languages worldwide, one dies every two weeks. So there’s an AI engineer, he he has the coolest name, Michael Running Wolf, and he founded Indigenous in AI, and his team has been building speech recognition models to preserve more than 200 endangered indigenous languages in North America.

You can track good news like this with my it’s totally free. It’s an AI timeline. Specifically, it’s an AI for good tag. You can also use it to stay abreast of AI news and developments. You can filter by company. So if you wanna track DeepSeek, not saying why you would necessarily wanna track that, but you can filter by company. I have just under 100 tags that you can filter by or you can query it. With all of the positives, there are challenges associated with AI that require vigilance. I’ll make a case for two in particular that are specific to AI app development. So I work as an AI strategist. I’ve been out of the SEO space for quite a while now. But the two biggest issues I see are one, lack of strategy, that’s on the development side. And then lack of transparency, that’s on the model provider side. And you’ll see in this presentation, those issues can literally cost companies millions of dollars.

The first issue I’m going to touch on is those building AI apps, many times they just don’t know how to pick the best model for a specific task. So a model that might work swimmingly for one task might completely bite the dirt for another task and they can be hamstrung with issues like hallucinations, lag time, unnecessary expense. We will see very specific examples of unnecessary expense. One term I’m going to be throwing around quite a bit is leaderboard. So let’s establish what a leaderboard is. A leaderboard is just a dashboard that compares the performance of models against a set of evaluation tasks. So basically, what these leaderboard providers, these benchmark providers are doing is they’re taking a bunch of tasks and they’re stress testing these models. And and then they assign them scores, you know, for these tasks and then they rank them with other models. These leaderboards tend to be pretty low fidelity. They generally kind of focus around a table and a few of them have charts but most of them are a table and they vary wildly from one leader board to another. So I spent the entire month of December just digging into these all these different leader boards to learn how to use them to create an app that will help reduce your learning curve. It’s also a free app. I’ll show you that. This is a pretty typical leaderboard. This is the Math Eval leaderboard. As the name suggests, this is very helpful if you’re doing any kind of app that includes math, which I’ve worked on. Math is very still very, very challenging for AI models. And this is this is pretty common. It’s not even the the worst example, but they have these filters at the top and benchmarks ability and grade, those are all pretty straightforward. But then you get to shots and there’s nothing that guides the user like, okay, what would be the benefit to me changing from the default overall highest to zero shot or few shot? And so, it’s like, you know, we’re going through these. I was like, ugh, where are the info icons? You know, like, why do I I kept having to go back and forth between, you know, Claude saying, what what does this mean? What does this do? You know, it’s like, how how would I use this? And I just built this massive JSON object and that’s what I used to build the app, which you’ll see in a minute. Here’s another example. This is the V Bench leaderboard. This is a leaderboard for video generation models. It’s a decent leaderboard. Again, another table. But there’s some ambiguity in the filters. So, if you’re looking at this, it’s not apparently clear. Like, what’s the difference between this select quality dimension, select semantic dimensions versus evaluation dimension? Things like that. I’m like, oh, if it just had like an info icon with some tool tips, like that would be really helpful because then I would know, okay, from a business perspective, what would be the benefit to switching up this filter? This is very common. This is also from the bench. Oops. Sorry. Not to pick on the bench, but you will typically see in these leaderboards that it has all of these benchmarks and again, no info icons, no tool tips. You you look at them and you’re like, and some of them are pretty esoteric like object class, multiple objects, human action. I mean, are some kind of evaluation task that it’s measuring the models on, but you’re like, what does that actually mean? This particular leaderboard, it actually has a really good filter evaluated by V Bench team. Again, I think this would be really impactful evaluated to have an info icon to explain what this means. What it’s saying is if you go with the default, it filters out self reporting models. And that is one of the biggest issues that’s causing transparency concerns with the model providers because a lot of these leaderboards, they allow models to self report.

So there’s no transparency. There’s no accountability. These model providers, they’re making outrageous claims. And you as you as you’ll see later in the presentation, like, you go back to these leaderboards and you’re like, well, that’s weird. Why don’t I see this score that you proclaim that all of these publishers just pair it and, you know, and and there’s just no there there’s no verification. So I personally wish that leaderboards would not allow any models to self report. I just think they are putting way too much trust in humanity. But at minimum, if they do allow self reporting, they should absolutely disclose that in my opinion. But when it comes to the info icons, you know, now I’ve built a handful of apps, I on the side of including info icons and tool tips for everything. I mean, in my opinion, if you’re going to on one side or the other, err on the side of being obnoxiously helpful. Don’t make people leave your tool and try to figure out what exactly it means because these model decisions have significant implications. They can they they impact the lag time of the apps. They impact the cost of the apps. It I mean, it it impacts so many things downstream.

Live Code Bench, okay, they are brutal. So those red highlights, those are not my highlights. What they used to do, they no longer do it anymore, which makes me so sad, they used to flag models that they suspected of cheating. And in the AI world, we don’t actually call it cheating, we call it model contamination because we are princesses. But it is cheating. So what they what they decided is that if a model ran its evaluation shortly after the evaluation or the the validation dataset was published and it performed unusually well or its answers were too close to the answers that they published, they’re like, I think they cheated. And you can see in the the ranks, they don’t even assign a rank. Like, if they thought you cheated, you didn’t you didn’t get a rank. So it it was awesome, but they they stopped doing that.

Okay. So I built this app. It’s totally free. I built three apps to help people understand, like, how do you pick an AI model and then you’ll see one, how do you how do you pick a machine learning model and then the AI timeline is like I said, it’s completely free. The way it works is you choose the task that or tasks that your AI app needs to perform, and then you select what benchmarks you want to compare. Now the tasks are listed alphabetically, but the benchmarks are actually listed in frequency because all of these benchmarks have all all of these leaderboards, excuse me, have quality benchmarks. Very few of them include context window. And when you do that, it will generate this network graph. I love network graphs for datasets where you have, like, a variable number of data points. So you can see over within the bottom right hand corner, the Helmlight leaderboard, that has a lot of benchmarks around it. The gray nodes, which are just, you know, it’s just geek speak for dots. The gray dots are the benchmarks and the green nodes are the leaderboards. And but, like, if you look up at the cost and and speed, you don’t have as many benchmarks around those. When you generate this, depending on how many nodes you have on the screen, they can get a little tangled up. So you just you but it’s totally interactive. So you can just click and drag them until you detangle them. And because I am a hardcore perfectionist, I didn’t like how the chart then kinda canted to the left or the right, so I added an orientation slider which you can minimize afterwards. That’s more important on mobile, so I optimized all of my apps for mobile and wanted to optimize the the space. Now, if you select the green nodes, it will trigger this leaderboard card. And this is a pretty typical leaderboard card. You have a brief summary of what that leaderboard does, then you have links to the leaderboard as well as the leaderboard’s methodology. Most times that’s going to be an archive paper. Pretty dreadful reads, but, you know, it’s there for the taking. Then the middle column, those are the tips that I have compiled from just going through the leaderboard, finding the gotchas like, oh, wow. Okay. I wasn’t expecting this. I’ll add that. You know, any any tips like a lot of times there are filters that are hidden under a like more options link and I’m like, ah, these filters are amazing. These should be, you know, surfaced to the top. And then any like practical applications. So I’m, you know, working with these AI apps quite a bit. And so I’ll just share some of my experience. Like, hey, you know what? This is really worth the money on this one. You may wanna really focus on this or, you know, if you filter this chart this way, this could help you, you know, identify hallucinations, things like that. And then in the right column is just an alphabetized list of the benchmarks specific to that particular task. So if you’re looking at quality for agentic models, they’re gonna be specific to agentic leaderboards and quality. 

Okay. Next issue. Organizations spend way too much money on foundation models. This is a huge, huge issue. We’ll look at a dramatic example of how easy that is to do in a bit, but this is still somewhat dramatic here. So this screenshot is from one of my favorite leaderboards. It’s called the artificial analysis leaderboard. And they actually have many different leaderboards. So you go to artificial analysis, and then, like, this specific chart came from text to image leaderboard. So these are all text to image models. And one of my favorite charts in that leaderboard is this quality versus price chart. I go right for that chart, whatever I’m doing. I love it because it puts price along the x axis and then whatever the performance measurement is on the right axis. I explain what all of these what what all these like performance measurements actually mean, how they measure them, stuff like that in in the app. But one thing I love about this is that it provides that like a these visual cues of like a wall of fame and a wall of shame, like you can’t really see the the gray yeah. You I can see it more there. So the green quadrant shows you, okay, these are the models that are the top performers and the gray, not so much. So one thing I did, and now this has been consistent ever since I started building the app, was December. You will frequently see along this line at about ten fifty, multiple models that are performing on a very similar quality level. Talk with my hands too much. But Ideogram, Midjourney, Dolly, they’re coming in at about $80. Now, this particular leaderboard, they measure it by 1,000 images generated. By sharp contrast, Flux 1, which is an open model, is coming in just under $8. Same quality, 10% of the price.

Now, one one thing you need to be aware of when you’re dealing with these cost metrics, they do a little bit of mathematical witchcraft to to pull this in because you typically have input tokens, output tokens, and different pricing models for them. And so all of the leaderboards I’ve included in my in my app, they all operate under the assumption that the output tokens are three times the cost of the input tokens. They’re actually closer to four times and and north of that, but it’s still better than having to have two different charts where you have input costs and output costs versus performance. So I’ll allow it. Also, keep in mind that model pricing can vary pretty wildly from one API provider to another. So what I did in this chart is I filtered for the LAMA 3.3 70 billion parameter model that’s Meta’s open model. And each one of those data points is a separate API provider. And you can see the price varies from twenty cents per million tokens to ninety four cents per million tokens. But this is a good example to to highlight because in a case like this, there may be other factors that make that ninety four cents worth it. So what I did in the same tool, I looked at the output speed. And the output speed for Cerebras, which was the one that came in, the API provider that came in at ninety four cents, was more than ten times faster than the next closest model. So if speed is particularly important for your model, the juice might be worth the squeeze. 

Okay. Now this is where the inmates take over the asylum. This is an agent leader board called Galileo. It’s a really good leader board. They combine date four different datasets. So they have quite a wide range of tasks. AgenTik models are models that perform some kind of task. Like maybe it schedules something on your calendar, books a flight, it runs a web search, like that. Excuse me. And here I drew the line at 90. So basically, all of these models up north of 90, they’re like the geeks in your school who ruined the curve for everyone else. And you can see a number of models line up along that and and or higher. What is crazy is you might wanna notice that x axis, that’s a log scale. You have a log scale, specifically a log ten scale, and you have that when there’s a really big differential in cost. You don’t want to be the reason some analyst needs to throw the x axis on a log scale. Y axis, that’s a different that’s a different game. But if you look up here, Gemini 2.0 flashlight, that that model is coming in at thirteen cents per million tokens by sharp contrast. At an even lower performance level, GPT 4.5 preview is coming in at just under $94 per million tokens. So, by way of summary, Gemini 2.0 flashlight achieved a higher performance at 0.1% of the cost of GPT 4.5. Now, if you wanted a model that really knocks it out of the park, like Claude 3.7 SONNET right now, that is the model to beat. I think it came in it’s coming in at about a 97. But it’s only $6 per million tokens, which is still just 6% of GPT 4.5. So if you look over on that right side, it’s it’s OpenAI. OpenAI is the reason that some analysts had to put this scale on a on a log scale. Otherwise, it just would have gone on five ever. Okay. So if you wanna check out the Galileo dashboard, you can just fire up a filter by chain agents. That’s the task. And you can see with this particular dashboard, there are a lot of tips. It’s a thick and chewy cookie. 

Okay. Next up, overuse of AI when alternative options are actually better. And by better, I mean cheaper, more reliable, less less prone to hallucination, etc., etc. So I’m gonna share a few examples working having worked on on some apps.  The first one is it comes from the education space. I’ve actually worked on four different projects in the education space because right now the education space is absolutely desperate for AI models. But we were working on this one app and it was really important that the app maintained compliance with US standards. So we were in an on on-site meeting with a client and someone on the client side said, how are you going to ensure that the app maintains compliance? And the lead data scientists on the project said, we’re gonna use GPT 4. And how it was like, no.  I mean, it hadn’t been updated at that point, I think in about nine months. It didn’t even perform web searches. But even if it did and I mean no disrespect to the data science, it say data scientists, I mean, we’re all crawling along this learning curve. But in this particular case, this was a this was a pretty egregious misstep because for something that is updating all the time like US state standards, you need an API. And there were two different APIs we had to to choose from. So we had to course correct on on that one. Another education app, this was a student-facing app and it was a homework app and history is so boring, you know, for students. So we were trying we were banding about ideas for how to make history more interesting. Someone threw out the idea, what if we used AI to generate political cartoons? I was like, or, you know, I just did a quick search while we were in the meeting because I was looking to see, is there an API that will actually return political cartoons, authentic political cartoons where the people don’t have six fingers, you know. And so, I I looked I I just did a quick search. The Library of Congress has a database of more than nine thousand political cartoons from the late seventeen hundreds to current and it’s free. So I was like, alternatively, how about we use this API and pull in actual political cartoons from that day? So if you have a student who’s learning about the Bay of Pigs disaster, they could just, you know, whimsically say, okay, I wanna see a political cartoon from that time. 

Another example, I was part of an email chain where, like, when prospective clients would would come through, we would just talk about like different ideas of how to approach a project. And there was a client, they weren’t looking for AI. They just said, we have a customer churn issue. And in this chat, especially like upper management, they were like, what? We’ll sell them on AI. And I’m like, well, unless they wanna talk about their churn issue, I don’t think that this is an AI problem. This is this is garden variety logistic regression. You can ask ChatGPT what that is. But, like, in this particular case, like, you could use a machine learning model. You don’t need to use AI. First of all, AI would be outrageously expensive, you know, to throw all of this data, but it’s just an it’s it’s unnecessary. Now, if you wanted to add a chat component to it, you could do that at a fraction of the cost. And in that case, you like in the app, you would choose chat and look at the benchmarks specific to chat. One really good one is this it’s a benchmark called IfEval. And it’s offered by Hugging Face. It it’s all in in the app. But it just measures how well a model performs on a specific request and includes the formatting of the response. 

Okay. Last example. We had client come through. They were building they wanted to build an AI app for dermatologists that would help them diagnose like rarer skin conditions. And they had some labeled images. They were pretty nasty of like different, you know, skin diseases. But and and again, you know, no disrespect to my teammates, but they were like, that’s all on AI. I was like, alternatively, we could use a convolutional neural network. So I knew in being on these projects, I knew how challenging it was to just like always be contrarian and say, well, yeah, because I’m really excited about AI, but if you can do something better for cheaper, then you should do it. You you should do that. So I built this was the last of my trifecta apps, also totally free. It’s a machine learning model picker. And the way that it works, I’ll just walk through like an example of if you were to try to figure out what machine learning model would be best for labeled images. So you just pick your category and and then when you choose a category, the subcategories update their specific to that category. If you don’t know what something is, you could just hover over it and you get a tool tip and explain it like I’m five tool tip. And then when you click the or select the generate flowchart button, it opens up this flowchart that answer asks you a series of yes/no questions. And depending on what you answer, it ultimately leads you to a model. So in this particular case, are you working with structured data? Well, structured data is like CSVs and stuff. So, no, we’re working with images. So we go down here. Are you analyzing text data? No. We’re analyzing images. But if something isn’t clear, you just click on it and a tool tip. And then ultimately, it’s like, oh, okay. You want a CNN. Now if you select that node, you open up a model card that has the name, it has a description, then I also threw in, like, example use cases, even alternative models that you could use, as well as the difficulty level. So the difficulty level, I have a methodology page in the footer for all of my apps. And so you can see how I how I determine difficulty level. And then a link to a Python and an R tutorial.

Okay. Last up. Accept model creator performance claims as facts. Again, this is a really is a really big issue. Okay. So February 2nd of this year, OpenAI announced its deep research agent. We’ve heard a little bit about deep research already. They claimed it achieved a score of 26.6 in the humanities last exam benchmark. It’s the most difficult benchmark for models. It’s supposed to measure how close we get to AGI, which is artificial general intelligence. Everyone’s saying, oh, they you know, we’re so close to AGI. Anyone who has ever used AI to code knows we are nowhere near AGI. Okay? So I hate to rain on that parade, but we are nowhere near AGI. But they also claimed a 72.57 on the Gaia benchmark. So I didn’t find it until February 6th, but I was like, we will see. So I looked up humanity’s last exam, their leader board, didn’t see it. So then I looked up the Gaia leader board and I’m like, what the hell? Didn’t see it. So one one caveat is humanity’s last exam, their leaderboard, they’ve already said they are not going to update it regularly, but Gaia is updated very regularly. So the next day, I emailed a member of the Gaia team. And this person said that OpenAI never submitted the its deep research model to be tested and went so far as to question the veracity of their claims. I extracted one sentence from that email. Their results have little legitimacy as one, they did not submit officially, so we have no guarantee that they used the correct scripts. Two, the validation set is public, so can be cheated on often accidentally. They’re being very gracious, very generous. Whereas, the test set has never been revealed. And that is the de facto standard. So what these these benchmark providers do, they publish their validation dataset. And then the model creators, they just, like, pinky promise, like, we definitely didn’t train our model on your validation dataset.

Okay. So there’s one leaderboard. It’s the MMLU leaderboard that it actually discloses the data source, and it discloses if the if the model was self reported or submitted their model for  independent testing. And I just highlighted the ones that actually submitted their model for independent testing. You can see they are in the majority. Now imagine if we allowed students to just, like, self report their SAT results to universities. They’d be like, oh, dope. Okay. Yeah. Come on in. I mean, it’s so crazy, but no people just don’t question it. So this is live footage of me. Every time a model announces, you know, some grandiose achievement, I’m like, alright, fire up this leaderboard. Not there. Fire up this leaderboard. And I used to be able to look at that one, the live code bench leaderboard to see if they’ve cheated in other respects and kinda use that as as, you know, a test of how honest they are. Now I now I can’t do that. But this is another example of typical shenanigans. XAI, this was also February, they announced that they tested Grok 3 on the 2025 American Invitational Mathematics exam abbreviated as AIM, seven days after the questions and answers were published. So this is a really big test and all of the math models, they like hawk over this test and the day after the test, they publish the questions and the answers. Again, we are operating under the assumption that these models aren’t then saying, okay. Thank you. And like hungry little hippos, you know, training their models on the test because they have the answers. But so this was seven days after they were published. We are supposed to believe that in those seven days that Grok did not touch that test. And they claimed that Grok 3, their think model achieved an astounding 93.3% on what is arguably the most difficult math test out there. Maybe. Probably not.

Okay. So remind you of anything. I mean, we’ve basically come full circle. Like, I used to be in SEO in the early days where, you know, people were putting like white text on a white background to, you know, like boost their rankings. That’s where we are, but for AI. So be vigilant. Thank you.

Mike King: With 15+ years of experience growing startups in global enterprises. She is a program leader at Northwestern University, mentoring future marketing leaders. And she’s presenting hybrid engine optimization, a crawler driven approach to maximizing search and AI visibility. Please welcome Jori Ford.

Jori Ford: Mostly because people are gonna wonder, why did I call it hybrid engine optimization? And I’m gonna be honest with all of you about why I did that instead of calling it GEO. I wanted all of you to be able to go home and say, went to the most epic conference ever, NYC, SEO, where I learned about SEO, AIO, LLMO, GEO, and then HEO, plus a little bit of EEAT sprinkled on top. That’s why I did it. But that’s not really why I did it. Jeff kind of set me up for success here because I think that we’ve looked at how we frame things as SEO and search and AI. And I don’t think that’s like one or the other kind of situation here. I think that we really have to look at it in the landscape of how does it impact our business. And from a landscape of how it impacts our business, many of us are still working in search. It’s not going away right now. So you guys remember when it was just SEO and it felt easy? Before like AI came in and started being disruptive to everything that we were doing? Our minds were blown. We’re like AI. And I’m like, don’t be afraid. It’s going to be okay. We’ve got new prospects on the horizon. And ultimately, hybrid just means it doesn’t have to be that hard. Like, we can figure out how to create balance. Because guess what? SEOs have been doing this for a very long time. I think Mike talked about the evolution of time. But I want you guys to think about this in your jobs and your careers. Right? There’s been a disruption across search every, what, three to five years. Have you lost your jobs? No. We’ve risen to the challenge and we’ve changed. We’ve adapted, we’ve grown, so we will continue to evolve. We test the crap out of everything and we evolve with it. So, I don’t know if you know who this guy is, but he’s been telling us that the whole time. Right? We’re going to change and evolve. It’s all going to move, and relevance is our new realm. I don’t know why, but he keeps saying that. So, we have to reframe our thinking on what works. I had that slide before he went through everything he did, you guys. I was like, oh, no. He’s going to tell.

So, things I want to go about today. I want to talk about the new landscape as it is because we’re going to talk a lot about the future later, and we’re going to talk about the things we can do. But, let’s talk about, like, right now where we’re at. We’ll talk about crawl logs because I think of myself as like the crawler queen. Every single time I talk, people are like, Oh, Jori, not the crawl logs again. But, I actually think they’re one of the most vital tools we have, so that’s what I’m going to talk about. Something that I hear a lot is, but what if you can’t get crawl logs? I’m going to tell you how to get around all the developers telling you no. And we’ll talk about building insights and dashboards that help evolve what we’re doing so that people understand how you’re changing and challenging the landscape and how you are adapting to AI right now in your roles. And then, we’ll talk about strategy. How do we implement these things? So, it’s a brave new world. The landscape is changing. AI is taking I’m just playing. So, really, here are the things that have changed. Right? We think of the journey as very linear or at least we did. I think a lot of us as marketers have been like, hey, you’d start here and it’s part of the funnel. And search has been part of that funnel, so we’ve been part of the solution for every problem across every step. But it’s changing, right?

We all know the journey is never ending. It’s cyclical. Right? Things keep evolving. People keep needing to learn. So we have to keep being involved. And now, we shove some AI in there. And so what does that do for us? Really, it just changes queries into questions. We go from queries in search to questions in search and agents because there’s more information for people to take in and distill. It shouldn’t be that bad. Right? Again, reframing how we think. So we know that search and discovery is going to change. Organic search is not just about position, it’s not about rankings anymore, it’s about presence. So, we have to adapt to our marketers within because for some reason, I don’t know why, but people don’t think SEOs are marketers. And I’m talking in the technical track and I still know I’m in marketing. So, it’s about presence in the AI-generated summary. And this is the piece I want you guys to take away. Right? If you’re not part of the journey, if you’re not part of the conversation, you’re not there. They exploded keyword rankings. So we run around and we’re like, what are we going to do? That’s what we that was our bread and butter. Right? Every tool that’s here right now has had to adapt to that. We can’t do keyword rankings. That’s okay. The surface changed a lot. That’s okay. We don’t have all of our eggs in one basket anymore, and that’s okay.

I’m not going to go too much into this, but when you think about how do you know you’re part of the journey, there’s lots of tools. People are going to talk about them all day, so I’m not going go into it. But, like, a little bit, we could just go into our analytics. We can do that. That’s easy. Look for referral traffic. We can go to Also Ask, dump some queries in, get a bunch of stuff out, and then if you want, you guys can steal my little AI mention tracker because I’m cheap, so I don’t pay for all the really fancy tools. No, I’m lying. I paid a little bit here because I use the GPT plugin and everything like that. But when you dump it in, what you’ll get is a quick search of whether or not you’re visible. That’s one of the questions you’re going to get asked as SEOs. Am I visible in the AI agents? And it doesn’t actually matter if it’s not good for your business, but if you want to answer them, use my tool. If you don’t want to use my tool and you want to pay a little money, not a lot of money, use something like Use Haul. If you want to be really sophisticated, there’s several tools. DemandSphere is one of them. 

But let’s get into the crawl logs because that’s that’s my bread and butter. Right? So, many of you have asked for crawl logs. What do you do if you get them? That’s like the weird part. Right? Because when do we ever get them? But, if we got them, well, you might have to parse through like 226 bots because in March, Cloudflare did this really awesome thing where they analyzed all the data that we give them and filter through them, and they identified 226 bots and crawlers. That’s a lot of work for you guys to have to filter through if you get into the crawl logs. But here’s the funny part. We shouldn’t care about that. 226 bots does not matter to you guys or your businesses. What might matter is the ones that are actually visiting your site. And the thing is is Cloudflare is not going to actually give you the user agents without you paying them. I don’t know if you guys heard them a little bit cheap, but the guys over at Momentic, Tyler, if you’re here, I know. I can’t look dead at you and say it. But amazingly good list because all the agents are listed here. So if you want to figure out who you should be following or what’s actually going on, go over to this resource and it distills it down into 24 and they actually keep it up to date. So you don’t have to do a lot of the busy work because the agents is what matters, not just how many bots are out there. 

So, why do I care so much about crawl logs? Why is it the only thing I constantly talk about? It’s because we, as SEOs, have overindulged in the use of tools that are hypothetical sometimes. Right? Keyword rankings are hypothetical. Can you validate all of them? No. Search is personalized. So you can look at your tool and you can say, I think this is what’s happening, but you don’t really know. Logs tell you the truth about everything that is happening on your website. That’s why I say that it’s your real qual equity because when you look at log files, it’s actually what the bots cared about. They’re only crawling what they care about and you can now see exactly what that is. So, if you happen to get really nice developers who are willing to assist you in this, we make the mistake of over asking without actually giving direction. I say this because I used to be an engineer and people used to ask me for things and I’m like, you don’t even know what you’re asking for. Don’t do that. So, if you want access to the logs, you can make it simple. Ask them for access logs. That’s very specific. And then, you’re going to get a developer that says, what about security? What about these things? Hold your horses, just ask for timestamp, the user agent, the URL path. Let them understand that it’s not secure information we’re worried about because we only want the get methods. We only want the things that the crawlers want. We don’t care about the rest. 

So how do you distill this down into information? What are the pieces that you actually need? So the IP address, that one’s an actual GPT bot. The reason I say that if you get crawl logs, you want the IP addresses, those 226 crawlers that Cloudflare found out, they found them out because they did IP mining and reverse lookups. Because here’s the beautiful new world of AI that you’re not going to be used to. AI lies. AI doesn’t have any standards. They just want to get what they need. So guess what? You have 226 unknown bots traversing your planet and you don’t know who they are, unless you have their IP address. Guess where you can get that? Your log files. They have to give it to you. Timestamping your request lets you know some frequency. HTTP method, you guys can read the slide, I don’t have to read it to you. You’re all smart. But what can you get with that? Every marketer asks these questions. These are normal day to day questions. But did you know that the log files can answer most of these day to days in a way that you don’t expect? Who? Which user agent? Which bots are visiting you? Because if your business tells you you need to prioritize bots, I don’t always say that you should listen to them. But if you decide to do that, find out if they’re actually visiting. Did you impact their visits? What pages were important? Are you optimizing the right things so that they can get there? Can you link from the pages that they actually are accessing? When? How frequently are you getting crawled? If you’re not getting crawled that much, you’re probably not that important to the agent. So, you’ve got to either figure out how to be more important or explain that you’re not that important, and that’s not a channel that you need to worry about. Where is my audience looking for me? I think that’s huge. Right? That’s part of the relevance, it’s part of the change because we don’t give these tidbits where we can walk into a room and say, I’m in SEO and I know where your brand actually is. I know what’s happening. And then, how am I doing and why am I falling behind my competition? If you know that your competition shows up in ChatGPT and you know that ChatGPT does not visit you frequently or index any of your pages, you know how your competition is beating you. They’re using ChatGPT. But what if they’re not? Then why are you worried about it? So that’s how you would look at these IP logs if you’re looking at raw log files.

But I’m not going to do that to you guys. I love you. So what I’m going to teach you is how to get around the developers. You’re not going to necessarily get the IP addresses, but from a bot perspective, you can track them on your own. You can find out if they’re important before you ask the developer for access or before you give the push back that you might need not to need to give. So I’m going to show you how to set up Google Tag Manager, connect it to your GA4, and merge in some of your Search Console stuff. So these are things we’re all familiar with. Right? But sometimes it can get a little inundating. But it’s really not that hard. Three simple steps. That’s how easy it is for you guys to all track search engine bots and AI bots. That’s what makes it hybrid. Right? You get to see it all in the same place. You set up a variable, you set up a trigger, you set up a tag. I think that sounds simple. So here’s what you really need to do. Now, again, cheating, lazy, and Ilias gave you guys toolkits. I’ll give you tools too. That’s why we’re part of the technical track. This script, it’s a resource that I’ll give to you. Just take it. It’s already in a Google collab. But you’re going to set up your variable and set up this custom script. And that’s going to let people like GA know which bot’s coming. Then you’re going to set up a trigger. So you don’t just want to know which bots are coming, you want to know when they come and on what pages. You set up a trigger, and now you’re isolating around that. That regular expression seems really complicated. It’s not. It’s just separating pipes on the same names that are in your JavaScript file. Then, we’re going to set up the GA4 event tag because without the tag, none of this fires and you don’t collect anything. Once you set this up to match the parameters, that’s going to get passed right through to your GA4. Guess what now you have? 

Alright. Yeah. Yeah. That’s it. You guys got it. Got it. Okay. So you have two ways to track, you have two ways to understand the agents, and now you have these really, really pretty reports. You get to see the bot name. You understand the events being handled. You can now customize this for your own view. You know how often and frequently they’re visiting you. And if you want to know how I got this into a GA account, I actually built a Google Collab script that runs this for demos. The reason I do that is because you don’t want to work on some of this stuff in a production environment because there are other tags set up. So I will give you this as well so you can set up your own demo accounts. You can now parse and sort it out by page frequency. Oh my goodness. Look at all of that. The not set’s my fault because I didn’t set it on the day that I ran it. Just ignore it, Okay? But in this hypothetical situation, you can see which pages are the most important to this crawler. So now you’ve got log files and you’ve got reports. But don’t you want to take that from there to the pretty stuff that I made? Okay. Well, it’s not that hard to do this. We all have access to Looker. Looker is free. Right? So in our free Looker account, we can set up these bot distribution reports. Also, I’ll give you this free template, free of charge. It’s because I’m nice. Don’t worry. But easy things that you can see really quickly. Overlapping pages where you see high impression and high event counts from the bot. That’s your existing opportunity to evolve. Right? So stuff you already own that you could just do things to. Go optimize it. In this opportunity, high bots, low quicks, missed opportunities. It’s rocket science I’m creating up here. Alright? So, you go to that landing page, now you know what else you can optimize. 

You have a plan. So, what do you do with the plan, Jori? Nobody’s going to ask that? Fine. Okay. So, what you do is you go get shit done. And this is when you get hybrid about it. Because if you know that your business needs to prioritize it, or you are getting traffic from AI agents, you are getting traffic from search, and you need to evolve, you need to expand your traffic set, get more conversions, do whatever it is you need to do. One, search is still important. I’m not going to say that it’s not. The reason search is still important right now, not in the future because we should prepare for it, but right now, most of your jobs hinder on search. Results in search, results in AIO because that’s what your teams know. And until you educate them out of that mind thinking, you still have to do your jobs. So today, Google still needs to be a priority. Why? Because Google actually still has Gemini. It that’s an AI too. So if you want to show up in their AI agent, you should probably work on, you know, what they use in their index for their rag model. They’re grounded in it, so that’s just me. Log and tag all the important bots. I say important because not every one of them is important to your business. 

This is where we become more strategic. Only track what you need to track. Other things are obsolete and noise. That’s one of the problems we have. We are analysts at heart. Right? You want to get lost in the noise, lost in the sauce. And if you do that, you kind of lose your audience, you lose the business. Train the LLMs on you. People wonder how that happens. You can actually talk to the LLMs about you until they just show up. And somebody actually called me out on this. They’re like, are you sure that that can happen? Yes, I’m sure. I’ve literally done it. I typed in Aspen Dental over and over and over because my buddy told me that I couldn’t do it, and Aspen Dental showed up in the LLMs where it was not. And I just kept asking it questions about if I needed dentures, should I get Invisalign, where could I get those things, and it happened. Create answer assets for retrieval on your pages. What does that kind of look like? Well, clearly isolate your brand on the pages, create product definitions. These are things we have access to. It’s just content. It’s not big wow’s. FAQs, we did that for a long time. You guys remember the ugly FAQs at the very top and bullets and people are like, these SEOs are screwing up our content. Why are they doing that? Now, can bring it back and go, guys, I know you hated that idea I had, but if we want to show up in AI, you have to let me do that really crappy thing again. I’ll show you a prettier way to do it though. Named entity references. Your brand matters. We’ve all talked about how we need to shift the dynamic towards brand. Reference your brand, reference your features, reference your categories. And guess what they’ll realize? SEOs care about branding. SEOs care about content and how the business operates and we get to be part of the business discussions now. Submit to Perplexity Pro, it’ll just put you in the knowledge base. It’s pretty easy. And then, if you’re not getting crawled, make sure you do the basics. XML sitemap and your robot text files, get them crawled. 

Create answer assets. Remember I said I’ll show you a better way to not do the ugly FAQs? It’s up there, over there on the right. I can’t look that far. I’m going to look down here. But once you do that, you can also add high quality explainer blocks. Does this page look ugly to you? Does it look like it’s not useful? It doesn’t to me. It looks broken up. It looks like something I can synthesize in chunks because just like people have to analyze, scan and absorb information because we do this every day. Right? The Internet is like this, constant information overload. But when you get to a page where things are simple, broken up, chunked out, I don’t know, I’ve heard a lot of people talking about chunks of information, elements that get indexed, something about AI and how they relate to information, set it up the same way and the LLMs love you. Use bulleted summaries. It’s pretty easy. Step by step guides, glossaries, linkable anchors. 

All of these things are things we have access to that make it really simple. Because I think we overcomplicate things sometimes and we don’t need to. Open graph and meta descriptions, they’re getting extracted, leverage them. And you want to know why you should do this and listen to what I said? Don’t listen to me because I’m cool. Aleyda said so too. So listen, if Aleyda says it, I’m right then. Just remember that. Okay? Thank you guys for your time.