Google vs. Bing: Correlation Analysis of Ranking Elements

Earlier this year, Danny Sullivan of Third Door Media asked me if SEOmoz could put together some data comparing ranking elements of Google against those of Bing to help illustrate the potential biases SEOs might face when optimizing for the two engines. Today at SMX Advanced in Seattle, I presented the following data, compiled by our own Ben Hendrickson with help from the entire SEOmoz engineering team (particularly Phil & Chas on the Linkscape side). The results I’m sharing match those in the presentation, with a bit more detail added in for those interested.

Rather than include the entire slide deck, I’ve taken the charts, graphs and data directly from the presentation so those of you seeking to convince clients or motivate internal teams can use them in your own presentations. But, before we begin with the data, I’d like to share a few critical notes about this research that shouldn’t be ignored.

Goals of the Correlation Data Research

With this research, we hope to accomplish three big things:

1. Add a new source of data to SEOs’ understanding of how Google & Bing rank web pages
2. Bring more science to SEO through a repeatable, peer-reviewed dataset
3. Provide recommendations based on our own interpretations AND open the data for interpretation by others as well

Further research, including causation analysis through more sophisticated ranking models and possibly more correlation analysis on other factors are certainly part of our goals as well.

Methodology

* We collected 11,351 search results from both Google & Bing via Google AdWords suggest data for the various categories (you can see these keywords yourself via Google’s AdWords tool)
* We looked only at the first page of results (which typically included 10 results, but sometimes contained a higher or lower number). We ignored all no-standard results (meaning universal or vertical results such as video, images, local or “instant answers”)
* The correlations relate to higher/lower positional ranking on page 1 of the search results
* We controlled for search results where all (or none) of the results matched the metric. Thus, for example, if we were looking for correlation with .gov domains and no results in the set included a .gov domain, we didn’t use that SERP for that dataset.
* We’ve used Spearman’s correlation coefficient, as it is the standard (and in our opinion, best choice) for ranked datasets. You can read more about this selection via Ben’s comments here and here.

This is a very similar methodology we used for our recent information on Google PageRank correlation.

Understanding Correlation Significance

The correlation numbers we show range between -0.2 and 0.35, where a perfect correlation would be 1.0 and no correlation would be 0.0.

The standard error for each result set is also included, but tends to be so low in most cases that displaying it on the bar graph would make it nearly invisible. This is thanks to the large number of results collected – we’ve got very high confidence in the statistical significance of these.

Correlation ? Causation

It’s long been held in statistical analysis that even very high correlations do not necessarily mean one data set is the cause of the other. People holding umbrellas don’t cause rain. Ice cream sales don’t cause hot weather.

It’s critical to know that the data below, like data from other types of SEO tests, requires careful consideration and analysis. Parsing a bigger correlation as a direct sign that one should do X or Y more would be a fallacy.

Understanding Negative Correlation

In the research below, you’ll see a few data points where the correlation is actually negative, meaning that when we saw the element, it tended to predict lower placement in the results, rather than higher. For example:

The data for URL length shows that longer URLs are negatively correlated with ranking well. This isn’t particularly shocking, and it probably iswise to limit the length of our URLs if we want to perform well in the engines. However, the second data point on .com TLD extensions shouldn’t necessarily suggest that using .com as your top-level domain extension will actually negatively affect your rankings, but merely that all other things being equal, .com domains didn’t perform as well in the dataset we observed as other domain extensions.

As we go through each set below, we’ll try to explain our thinking, but certainly invite you to draw your own conclusions from the data.

As we’ve seen in the past, when more sophisticated ranking models are introduced, using machine learning against the search results, we often find that previously negative correlations turn out to be positive (or neutral) ranking factors.

That’s it! Let’s dive into the data.