A perfect search engine would be able to read and comprehend all of the text on every site, making judgment calls about what content is actually better in terms of it's information value and quality.
That search engine doesn't exist, and won't for the forseeable future. Machines can't yet understand the content. They can only break it down into words and phrases to determine what keywords the content is relevant for. There are a few things they can use, like grammar scoring, to try and discern the technical quality of the writing, but they have near zero ability to judge informational quality.
In an attempt to circumvent this problem, search engines like Google use external factors to help gauge quality and relevance. These factors are called "off-page" factors in the world of search engine optimization.
By far the most important off page factor is the links coming into your site. Generally speaking, the more links you have coming into a page, the better it's chances of ranking in Google for the keywords the content is related to.
This means that you could have written the world's greatest article on, for instance, how to train your dog, but if you never get any links to that article it will not show up in the search results for "dog training." in fact, with no links to it at all, Google may ignore the page completely, not even adding it to it's index of pages.
While this may be frustrating, and is clearly a limitation of the current generation of search engines, it's also a natural reflection of the world we live in. A business that fails to advertise it's wares may produce the greatest product in its market and still go bankrupt for lack of customers. A musical genious that rivals Beethoven but never performs before an audience or records his work will go unknown and unappreciated.
So it shouldn't surprise you that the sites with the most money and resources tend to outrank everybody else -- even if their quality isn't the best.
The problem is that once a site reaches a certain "critical mass" of popularity, it has very little need of search engine optimization as a rule. This is because the nature of it's popularity gives Google what Google wants.
For instance, a big site like Amazon can't sneeze without ten million reporters rushing to write an article explaining the sneeze, what kind of tissue Amazon chose to use, the physicians opinion on the far-reaching impact of the sneeze, and so on. That results in millions of additional links pouring into Amazon's website, further reinforcing it's domination in the search results.
On the other hand, when you, the lone webmaster, sneeze, you're doing good if anybody bothers to glance in your direction for more than a fleeting moment. Nobody offers a tissue, nobody cares what the implications are for your business. No links get generated, and your rankings remain unchanged.
Does that mean it's a hopeless, unwinnable battle? Should you give up now and decide that you are doomed to a life of financial slavery to some evil boss?
Um, no. It jus means that you have to be smart about the way you work. It means you have to leverage what you have and make it work harder with less effort from you.
For instance, the number of writers a big web property can afford to pay full time means they can have fresh, quality content pouring into their site every day. You're only one person, and if you can't afford to pay writers it means you need to make the content you can create stretch further. Many marketers use a content spinner or private label rights content (or both) for this purpose. Others put their own original content on their "money" sites and use those other tools for content they submit elsewhere.
As to gaining links to your site, the big web properties get those easily, too, as I mentioned earlier. But the lone webmaster can fight back here as well, taking advantage of link networks and blog networks to grow the links to your site without having to manually submit content to thousands of sites.
What you do with the profit you make once you do start to see some success is also important. Rather than getting all excited at that first great affiliate commission check and buying yourself a new big screen television, you need to reinvest as much of that as possible back into your business. Use your early successes to finance the tools and services you need to reach the next level of success. Of course, it's important to reward yourself to keep your motivation strong, jut don't go crazy.
Little by little your successes will get larger and larger if you stick to the plan. There will be failures and setbacks, of course, but you will succeed if you stay the course.
In time, your business will also reach the point of "critical mass", where you have to struggle much less for each subsequent success. In fact, if you stay focused and disciplined, you may even get to the point where it all comes rather easily. And let me tell you, that's a beautiful place to be.
Please leave your thoughts in a comment below.