iPullRank

Great thread. I'm one of those people that was poking around and trying to figure this out a few years back. Obviously, there's a lot I know you can't say, but I'd love to know your general thoughts on how far off we were: http://ipullrank.com/googlebot-is-chrome https://moz.com/blog/just-how-smart-are-search-robots

My personal opinion is that open sourcing googlebot[0] and tensorflow, as well as a the numerous other tools like elastic search and frameworks like hadoop make it pretty easy to make narrow search tools. So someone like me, a pretty shitty programmer, can use an language built on top of googlebot, like nodejs, and just index the stuff I care about and instead of that information being returned to google, I can just send it back to bots viewport directly and not even see the webpage, just the info if I want. The problem is discovery (what google solved in 1999), no one knew where to find websites, like not even info, just what sites even were on the web. This is a huge pain point again because while there are a lot of great websites there are a ton of shitty ones. This is pretty obvious if you look at sites like reddit or hacker news. They are developed around the idea of aggregating good resources because there are so many it is overwhelming. So, I think, that of the 50% of the people that are already running googlebot, some subset will start probably using it to get info they care about. Eventually, someone will write code you can use to make this easier for normal people, and then google whill either sell the silos of info to them instead of advertising, or use it in it's new meta structure to solve big problems. [0]http://ipullrank.com/googlebot-is-chrome/ [1]http://www.sitepoint.com/browser-trends-august-2015-chrome-exceeds-50/

>We are expecting to get the same chance of ranking as normal pages. Then you should probably adjust this expectation. You say in your article: >While having this sort of HTML fallback was technically possible, it added a lot of extra work to public-facing single page apps, to the point where many developers dropped the idea... A JS-driven site with an HTML fallback is a normal page. Then you don't need any tricks or force Google to run your application and hopefully make pages out of them. Start with the fall-back and enhance. This is a serious mistake with consequences. Tor bundle and Firefox shipped with JavaScript support, because disabling JS broke too much of the current web. It causes accessibility issues (remember when Twitter changed to hash-bang URL's?), if not for Googlebot, then for regular users (From the Webmaster Guidelines): >Following these guidelines will help Google find, index, and rank your site. >Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site. >Make pages primarily for users, not for search engines. I am still going on the assumption that you created a one-page application without a time-consuming fallback, and you rely on Google to make rankable pages from them. Then…

If you read the original publication on this[1], they mention there’s a stray commit publishing the internal variant of the SDK intended for the actual Google warehouse database. So the code bases probably live close enough together for someone to accidentally pass the wrong folder name or something. This has been fixed, but the commit and all it’s changes are out there—and tragically, published alongside a copy of the Apache 2.0 license (intended for the document warehouse API SDK), which officially sanctioned freely copying and using the code. So there is really nothing Google can do about it. [1] https://ipullrank.com/google-algo-leak

The "lies" there are such bullshit. They basically consist of "Google claimed they didn't do X" now it looks like they probably do X and therefore google lied. They are somewhat circumspect about the fact that each example is actually, 8+ years ago Google made a claim about what they did at the time and now there's evidence that they are currently doing doesn't match that claim. That's not a lie. For example, they link to this tweet from 2016 as such a claim: https://ipullrank.com/wp-content/uploads/2024/05/image21.png Or a later example where they link to this 2016 talk: https://www.youtube.com/watch?v=iJPu4vHETXw Later on, they use claims from 2012: https://www.seroundtable.com/google-chrome-search-usage-15618.html

There is something truly strange about the idea than people "trust" a website operator and can rely on it to provide them with useful information when that same operator is well-known to be secretive, deceptive and dishonest in order to protect its own interests. It's like imagining that a fact witness who tells the truth on some occasions and lies on others is credible. https://ipullrank.com/google-algo-leak

> Though the documents aren’t exactly a smoking gun Smoking gun of what? The article never seems to actually say. The only accusation I see is that SEO grifters were lied to by Google Search docs, but you look at the article like[1] it's over things like "Google always said Domain Authority isn't a thing, but we found a variable site_authority. We don't know what it's for or what it does, but clearly we were right". Which, even if true, isn't a thing I care about at all... [1] https://ipullrank.com/google-algo-leak

Everything Google has said that the bot does, plus a knowledge of coding. They aren't using a LYNX style browser to render javascript and determine the position of ads on pages. They have to be using a headless type of actual browser for that - and it's a safe bet they aren't using firefox or IE or safari. They aren't telling us how Googlebot is coded, but it's pretty easy to deduce. http://ipullrank.com/googlebot-is-chrome/ for more details

Reminds me of "Googlebot Is Chrome": http://ipullrank.com/googlebot-is-chrome/ Obviously the claim probably isn't literally true, but it they could certainly share a JS engine (V8) at the very least, and the idea that the motivation behind developing that JS engine may have been spidering JS-dependent pages doesn't seem too far-fetched.

A while back a blog post popped up here arguing that Chrome is a repackaging of a new Google bot: that Chrome was developed first as a crawler, then later repurposed as a desktop browser. http://ipullrank.com/googlebot-is-chrome/ There's no real proof for this of course, but it makes this change to the Google bot's behaviour make sense and explains Google's massive investment of programmer effort into Chrome and everything surrounding it (e.g. WebKit/Chromium, V8, Chrome's update mechanism).

There was an article a while back about how Googlebot is likely a version of Chrome that fully renders a page to process it. [1] You're right, without a full browser engine, you could pull a whole host of tricks to make the page appear clean but then put in ads later. Since Googlebot is probably a full featured browser, it becomes a lot harder to trick. [1] http://ipullrank.com/googlebot-is-chrome/

It wouldn't have SEO implications because using javascript to load content isn't considered cloaking. (Cloaking is where you present something different to googlebot vs. a live user). It's well known that googlebot does run javascript now: https://twitter.com/#!/mattcutts/status/131425949597179904 There's also some speculation that googlebot actually runs Chrome and can render anything that Chrome does: http://ipullrank.com/googlebot-is-chrome/

Last month, there was a significant buzz among those involved with Google Search (which includes almost the entire modern internet and all its developers). "Erfan Azimi," the owner of an SEO firm, suddenly began sharing leaked documents revealing how Google's ranking system works. It's more complicated than just the search itself; it involves various APIs around it. Nonetheless, these APIs reveal a lot. The leak happened when a Google developer wrote a program to convert API calls into his preferred programming language but accidentally published everything (if you're interested, I've included a link to the commit with all these descriptions [1]). Multiple confirmations from reputable sources, including former and current Googlers, have verified the authenticity of this leak. It's not a hoax or a joke but a genuine breach of information that has piqued the interest of all SEO researchers. Here's a reliable summary of the findings: – Google has allowlists of manually optimized sites, at least for certain topics, such as the 2020 elections or COVID-19. – Domain names and subdomains are significant factors (despite Google's previous claims). – There's a sandbox for new sites, which Google has always denied. – Google directly uses data from EWOK (a system where paid users rate the quality of search results). – User behavior on sites is actively used for ranking. – Click data is collected not only from Google Analytics but also directly from the Chrome browser. – Sites…

Long time dev/designer and part-time SEO person here. The major implications are that a lot of what Google was telling SEO practitioners were a lie or cleverly disguised red herrings wrapped up in semantics. The article that was posted earlier today went through a lot of the stuff they were lying about with examples of Matt Cutts and Gary Ilyes statements that are now directly contradicted with the recent release of this code. Here is the article: https://ipullrank.com/google-algo-leak The premise: What I’ll do here is contextualize some of the most interesting ranking systems and features (at least, those I was able to find in the first few hours of reviewing this massive leak) based on my extensive research and things that Google has told/lied to us about over the years. “Lied” is harsh, but it’s the only accurate word to use here. While I don’t necessarily fault Google’s public representatives for protecting their proprietary information, I do take issue with their efforts to actively discredit people in the marketing, tech, and journalism worlds who have presented reproducible discoveries.