How Google Gets It Wrong

I have an avid interest in search engines and search algorithms. If you've been following my blog for any time at all, you know this.

It's commonly known that the Big Three search engines (Google, Bing and Yahoo) base their ranking of results primarily on one thing: links. Yes, the documents need to be relevant to the search query, but after meeting that criteria it's the quantity and quality of links to a document that lands it in the top spot for any given set of keywords.

While links are a great way to provide a baseline of trust or authority, links are not the only way a document should be judged.

Here's an example of how using links as the primary ranking factor fails. Go to Google (or Bing) and search for:

old time radio

You'll see the #1 result is RadioLovers.com. It's a very popular site because it offers a lot of downloads of old time radio shows in MP3 format.

That's great if that's what you were after, but what if you were actually wanting to read about Old Time Radio? In that case RadioLovers.com is a terrible #1 result — it's mostly just a list of links.

The #2 result in Google isn't any better (Otr.net) — it's all links.

In fact, if you go through each of the top 10 results in Google for "old time radio" you'll see that, with the exception of the Wikipedia entry that comes in at #9, there's very little information about Old Time Radio in any of the results!

Thus the failure of using links as the primary method for ranking web sites becomes clearer.

In some cases using links is perfect, such as when looking for the official web site of a particular company. Search for "Microsoft" at Google and you'll get Microsoft.com — even though there's very little information on the home page. But if you searched for "Microsoft" in the hope of getting some information about the company, that doesn't start until Google's fourth result (the Wikipedia entry for Microsoft).

That got me thinking: while links are a great way to establish a measure of trust or authority, wouldn't it be better to have an alternative ranking method that puts sites that offer the most information first?

What I'd like to do is:

1. Pull the top 30 results from Google (or Bing or Yahoo).

2. Gather up all of the information in all of those top 30 documents.

3. Score each document to see how much of the overall body of knowledge was covered in each document.

So, for instance, if a particular web page contained 80% of the information covered by the top 30 results, that's the web page I'd want at #1 because it covers most of the information found in all 30 documents. Number two would be the next best coverage, and so on.

That would mean that I'd have to visit a lot fewer pages to get a great overall understanding of a topic.

This method would also be great for news items, since it would list the news article with the best overall coverage of a topic first — saving a lot of time reading multiple articles to try and get a balanced or complete picture.

Wouldn't that be nice?

Well, stop day-dreaming, because it's a reality! I've incorporated the methodology listed above into my search engine, Shablast:

Click here to go to Shablast.com

Now, when you enter a query into Shablast and select either "Standard" or "News" from the drop-down list next to the keyword field, Shablast will score the top 30 Bing results using the methodology listed above and sort the results to show the web pages/news articles with the best coverage first.

Search Shablast for "old time radio" in Standard mode and RadioLovers.com comes in at #13 because it only contains 19% of the overall body of knowledge contained in the top 30 results. The #1 result is MysteryShows.com, which contains 64%, followed by the Wikipedia article at #2 with 56%.

Search Shablast for "Microsoft" and the #1 result is the CNet.com news page dedicated to Microsoft because it contains a whopping 84% of all the information contained in the top 30 results for "Microsoft."

The top 30 results are pulled from Bing using their awesome API. The advantage of using the Bing API is that links still play a part. It takes a lot of links to get into the top 30 results for major keywords, which ensures a certain level of trust or authority. But then Shablast examines those top 30 documents and resorts them to show the documents with the best coverage first.

Why not give it a try and let me know what you think, either in a comment on this post or by discussing Shablast at the Shablast discussion forum?

Like what you see? Then subscribe to Marketing Insiders and reap big benefits!

By subscribing to my free Marketing Insiders email list, you will regularly receive special member-only insider information, discounts and freebies. You will also be notified when new articles are posted here at the blog.

It's absolutely free to subscribe, and you can leave the list at any time.

For subscribing today, I will give you a valuable free gift as well!

First Name:
Email:

 

85 Comments

My Site Is Gone!

Yesterday your site was in the top 5 results in Google for your keywords. Today your site is nowhere to be found. You spent the last week building new links to your site. Did you overdo it? Did you "break the rules"? What happened!?!

First, take a deep breath and relax.

Good.

Now, let's talk about why this happens, why you usually shouldn't worry about it, and when you should.

Why Your Site Disappears

Especially when your site is new, it is not at all unusual for it to be ranking very well one day, and then disappear from those rankings the next day. This is especially true after you do some link building to your site.

Here's a scenario:

1. You build a brand new site (or go back to work on a site you haven't done anything with for a while).
2. You start building links to that site.
3. After a few weeks your site jumps into the rankings for the keywords you're working on. Yes!
4. Another week or two later it disappears from those rankings.

It's at #4 that most people freak out, but you really shouldn't. That's because of #5:

5. A few days later your site pops back into the search results, often in a better position than it was before.

If you're not doing anything spammy or evil, #5 is very likely to follow #4. Let's talk about why.

Why Does Google Do This?

It might seem like Google's goal is to give you a heart attack, or at least put the fear in you, but it's really not (though those two things might be side effects of what they are doing).

What's happening is what's commonly referred to as the "Google Dance." You see, Google has a massive index, which it updates on a daily basis as it discovers new links and makes adjustments to old links, removes dead pages, etc. But because Google's index is so huge, it takes a while to recalculate all of the ranking information for your site in among all of the others.

I'm not sure why Google doesn't keep a static version of the search results up while making these adjustments, but they don't. Perhaps it would require too much overhead, so it's just easier for them to do real-time updates like this. Whatever the case, these updates mean that your site will jump around in the result list, often disappearing altogether for a couple of days, until Google finishes its little "dance", at which point your newly acquired links will often make your site appear even higher up in the results than it was before.

When Your Site Doesn't Come Back

If your site does not return to the position it was before (or at least on the same page) within a few days (at most 3 in my experience), the next step to take is to make sure that your site is still indexed by Google. To do that, go to Google and do a site search on your domain name, like this:

site:jonathanleger.com

Replacing, of course, jonathanleger.com with your own domain name. If Google returns a list of your site's pages, then you're probably fine. If it doesn't, that means Google has decided that something you did was bad and has removed your site from their index altogether.

Coming back from being deindexed is tricky, and is beyond the scope of this blog post. More often than not your site will still be indexed even if it doesn't come back into the rankings.

Why Didn't My Site Come Back?

The primary reason this happens is that you got too many links too fast and then stopped getting links altogether (or slowed down a lot).

You see, if you get a huge number of links to your site all at once and then stop growing your links, Google will rank your site well for your keywords pretty fast for a while. But once it sees that your links have stopped growing, it assumes your site's popularity was short lived and drops it back out of the rankings again.

That makes logical sense. If a site is suddenly popular because of some big event that happened which caused it to get a lot of publicity, it makes sense to keep that site high in the rankings for the duration of that notoriety. But once the jets cool off, it doesn't make sense to keep the site ranking so well.

That's why it's important to grow your links slowly over time. If you tapper off your link growth (or just get a small number of links over a long period of time), then Google doesn't usually drop your site out of the rankings like that.

So instead of getting 1,000 links in one week, getting 10 a day for 100 days will keep your rankings stronger for a lot longer! If you've chosen your keywords carefully, your site is likely to hold its position until a competitor comes along and gets more links than you — in which case it's time for you to go back to work and go through this experience all over again.

Summary

Google will often drop a site from the rankings if you get too many links too fast and then stop growing your links altogether, so be sure to grow your links slowly over time if at all possible. But even if you do everything right, Google will "dance" for a while after you grow your links, bouncing your site around in the search results for a few days before it settles back down.

One more point: once your site has been established for a while (usually six months or so of steady link building), the dancing tends to stop happening so much (if at all). At that point your site can usually sustain short bursts of big link additions without Google going crazy on you. In the beginning though, you just have to expect this kind of bouncing around.

Please post your thoughts and questions in a comment below.

Like what you see? Then subscribe to Marketing Insiders and reap big benefits!

By subscribing to my free Marketing Insiders email list, you will regularly receive special member-only insider information, discounts and freebies. You will also be notified when new articles are posted here at the blog.

It's absolutely free to subscribe, and you can leave the list at any time.

For subscribing today, I will give you a valuable free gift as well!

First Name:
Email:

 

127 Comments

Free, Easy Backlinks Using Google Alerts

Webmasters are constantly in search of new and creative ways to get backlinks to their web sites in order to improve their rankings in the major search engines.

One free, easy way to get backlinks is to monitor forums where people are discussing your areas of expertise. This is really easy to do with Google Alerts.

Simply create alerts for each of the major keywords in your industry, and set the alerts to monitor "discussions" only. Click here for a snapshot of some alerts I've setup to do this.

I personally recommend you use an RSS Reader to keep track of these alerts, which is a lot easier than keeping track of email notifications. Google even has a handy built-in RSS reader that you can use if you don't already have one (thus the "view in Google reader" link in the image above).

How this works is simple: each time Google finds a new discussion that is related to the keywords you specify, it makes a note of that in the RSS feed. You can then scan the feed for threads where you can jump in and respond. Most forums let you have a keyword-rich link to your web site in your signature, and so you get free links by joining in these discussions. Those links help your ranking in the search engines over time.

Doing this also establishes you as an expert in your field at these forums, which at a busy forum can be just as valuable as a #1 ranking for your keywords. Not to mention the direct traffic you can generate from your signature links.

How many threads will Google find that you can respond to?

Well, as an example, the "Discussions" alert for "internet marketing" over the last ten days has generated 350 question threads on 140 unique domains. By "question thread" I mean a thread whose title is a question (e.g. "Where do I find a list of blogs to submit comments?" — that's a real thread). The question threads are often the best to respond to, since they establish your expertise, but you can, of course, respond to any thread that you feel you can contribute something to.

I wrote a script to extract only the questions from the RSS feed, since I find those to be more valuable. Getting free links on 140 domains in 10 days isn't a bad deal at all! And that's no doubt going to grow over time.

So if you're needing some free links, head over to Google Alerts and get started following discussions related to your keywords!

Please post your thoughts and questions in a comment below.

Like what you see? Then subscribe to Marketing Insiders and reap big benefits!

By subscribing to my free Marketing Insiders email list, you will regularly receive special member-only insider information, discounts and freebies. You will also be notified when new articles are posted here at the blog.

It's absolutely free to subscribe, and you can leave the list at any time.

For subscribing today, I will give you a valuable free gift as well!

First Name:
Email:

 

138 Comments

Rodney's 404 Handler Plugin plugged in.