Step 11 - Rank Checking

Introduction

One of the rewards of a well optimized website is the increase in search engine ranking. Search engine rankings can fluctuate a great deal almost on a daily basis, and keeping an accurate record of rankings over time is very important. Webmasters want to ensure they get an accurate picture of their own website's position in order to react to any changes in search engine algorithms or competing website strengths.

Websites that rely on visitor traffic from the search engines for leads, sales, and any other types of search engine marketing want to know how well their target keywords are performing. Whether you are in the middle of a campaign targeting new keyword phrases for a well established website, or your site has just passed the first month of being online, you want to get an accurate report of your rankings.

In addition to checking on your own rankings, you may also want to check on your competition from time to time to see if they are gaining or loosing positions compared to your own rankings.

The manual and automatic rank checking options

We don't think we have to state the obvious, manually checking your search engine rankings is time consuming and tedious, not to mention the lack of accuracy and the record keeping aspect. The search engines can make searching for a specific site a little easier by giving users the ability to retrieve a larger search result per page, usually up to one hundred per page. Even with this advanced feature, the ranking check can take a long time to complete if you have hundreds of keywords in your list and a handful of websites. So, if you have a site that you are serious about, you can't pass up the ease and convenience of automated ranking reporting tools.

The search engines' view of automated rank checking tools

There is one main drawback to checking your search engine rankings with automated tools. The search engines don't like automated tools downloading their search results pages without an actual human visitor seeing them. Why, you may wonder? The answer lay in the search engine's revenue model. Search engine profits are driven by the number of "eyeballs" seeing not only the natural search results, but the dozens of advertisers' ads who are bidding on the same keywords the user typed in. It's no wonder, the search engines have taken steps to limit the automated robots querying their results.

One of the methods used by the search engines to detect automated "robots" is the frequency of the query requests within a certain period of time. If the number of requests exceed a specific threshold per minute or second, the search engines block the users access temporarily based on their PCs IP address. The ban normally stays in effect until the user types in a security keyword in response to an image displayed or a specified amount of time passes without any further suspicious activity.

Please note your own website can never be penalized for running automated queries on the engines. It's impossible for the search engines to find out which website URL you are checking. The only information the search engines can derive from automated queries is the IP address of the PC running the queries and the keywords being used.

How does SEO Studio address safe rank checking?

In order to avoid getting your PC's IP address banned, the SEO Studio Ranking tool offers a number of safe methods of automated rank checking.
 

• The SEO Studio Ranking tool has the ability to use the legal, publicly available application interface search methods. APIs, short for "application programming interfaces," give developers a "back door" access to their search results without "scrapping" the result pages. Not only these APIs provide safe and legal access for the developers to search results, but the developers can create online and off-line tools using these API features and offer them to the public.
  The SEO Studio Ranking checker can use your existing  Google SOAP API key or you can sign up for a Yahoo! Developer API key.

• Using proxy servers to avoid detection of automated queries. Public or private proxy servers hosted on external sites give you the ability to query the search engines without revealing your own IP address. If a ban occurs, the proxy servers' IP address used to query the search engines will be banned and not your own IP address. You can find public proxy servers at Public Proxy Servers or you can sign up for a paid proxy server service at Find Not. You can also install a desktop based IP address concealing solution. One of our favorites is Ghostsurf, a fairly inexpensive package that does a great job at hiding your PC's identity. With all of the solutions we have listed, you need to give SEO Studio the IP address of the proxy server using the Public Proxy server dialog found in the  Tools>Options>Connection Settings menu path.

• Increasing the time delay between search queries. The search engines can easily detect the time passed between search queries, and an unnaturally high number of queries executed in a short amount of time suggests there is an automated robot behind the search queries. This is why you should increase the search query time delay to about 1,000 milliseconds as a starting point using the Connection setting options.

Running a ranking check using SEO Studio

Once again let's illustrate the SEO Studio's Ranking tool's features with an example. Prior to opening the Ranking tool, we have already input our own Yahoo! and Google API keys in the Profile Settings screen. We have also increased the query delay between each search query to 2,000 milliseconds to avoid our PC's getting banned by the engines.

1. Open the Ranking Tool

First we'll name our report for easy retrieval later. Our ranking report name will be called "Sample Ranking Report.” See the "Report Description" text box below.

We have selected the "Exact URL Reporting" option as well. This will enable us to view the full URL found in response to our search. For example, using the keyword "discount import car parts,” the engines may return a URL that is not the home page, but a sub page on a specific domain. We'll want to know which page is ranked for that specific keyword phrase, and that is why we have checkmarked the "Exact URL Reporting" option.

We have also chosen to search up to 50 positions for our keyword phrases using the Google and Yahoo! APIs. This means SEO Studio will retrieve the first 50 search results for each keyword phrase and will check if the thepartsbin.com site and one of its competitor's site carpartsimports.com is found in the returned results.

Now, we are ready to hit the Start button to begin our search.

Conclusion

Checking your website's ranking becomes necessary once your site achieves a minimum level of popularity and traffic. You need to get a detailed picture of your site's ranking by keyword and search engines so you can better target your optimization of specific sub pages of your website or increase inbound links to pages that needs some more "link juice" to get higher ranking. We recommend using the Yahoo! API to safely check your ranking without the risk of getting penalized.

Google doesn't currently have an API interface for rank checking. Since last year Google switched from the SOAP based API to an AJAX based API and the new AJAX search API is not suitable for rank checking. The reason the Google search AJAX API is not suitable is because it only retrieves the first 9 results. When clicking on the "Next" link to retrieve additional results pages, the API switches to the normal results pages making it impossible to retrieve ranking results. Only existing owners of SOAP API keys can use Google for rank checking since Google refuses to give out any new SOAP API keys.

The fact remains, you need to run ranking reports at least once a month with or without APIs, and there are some recommended solutions you can implement to avoid getting your IP address banned. Some of the alternative Internet connection methods include web or PC based proxy servers which can hide your PC's IP address. You should also increase the time delay between search queries to make your ranking checks search engine friendly in using the Tools>Connections menu.