WordPress SEO for Real Estate Websites-Robots.txt

icon_bigIt seems almost eery that within the week I scheduled to write a post on WordPress SEO about the Robots.txt file,  I get two…not one, but TWO calls from Realtors who just can not figure out why their sites are not ranking as they should. After careful observation, BOTH of them turn out to be robots command issues… Now granted one was an on page robots (no index) issue, but the other was a robots.txt file issue…  Each carry the same message…  HEY GOOGLE, DONT INDEX THIS SITE (PAGE) K?

Probably not the message YOU want to send to Google, now is it…

So today, we will speak about the robots.txt file…  in a week or so, (probably not next week, as two robots posts in two weeks would even bury my eyelids…) we will talk about robots commands, another sure fire way to increase slightly, or DECREASE GREATLY your SEO.

So, What Is Robots.txt?

Robots.txt is a text (not html) file you add to you root directory that tells the  Search Engine Spiders which of your pages NOT to visit. Search Engines do not guarantee they will abide by this information, but typically they do.  Most spiders will not go where they are told not to go.  A robots.txt is a strong suggestion to a search engine not to index your page.  However, it is not a rule…  It is a bit like asking someone not to go in a room while you are not home…  the room is unlocked, but you trust they will not.

Just as you would not leave VALUABLES in that room, do not PROTECT sensitive information by adding it to your robots.txt file. There are way better ways to secure pages.

The location of robots.txt is very important. Search engines will not search your site for it. They simply scan the main directory and if they don’t find it, they assume that your site does not have a robots.txt file and they proceed to index everything they find.

So HOW do you write a Robots,txt file.

Well the simplest way, is to open NOTEPAD and create the file there.  robots.txt is exactly what it says, it is a text file named robots not an HTML file.  So, notepad is the logical choice.

The structure of a robots.txt is pretty simple, but frankly NOT very flexible– it is an endless list of user agents and disallowed files and directories. Basically, the syntax is as follows:



User-agent” are search engines’ crawlers such as Google Bot or Yahoo and disallow: is a lists of the pages or directories tyou wish NOT to be indexed. You may also see lines of text proceeded by #.  These are simple comments and intended for human eyes as opposed to the spiders.

For example, if I did not want the spiders to index my wp-admin page (something I do on all my sites to keep from leaking link-juice to meaningless pages) I would write it this way…

# All user agents are disallowed to see the /wp-admin page

User-agent: *

Disallow: /wp-admin/

The first line is for me, the second states “OK, this is for all spiders and bots” and the third line, “Pretty please, don’t index my logon page.”

The concept and structure of robots.txt is a decade old, and there are plenty of resources, such as http://www.robotstxt.org/ or you can go straight to the Standard for Robot Exclusion

Robots.txt files can get lengthy and the exact syntax, punctuation and SPELLING (most of the errors I see are mis-spelling of user agents, directories or missing colons, slashes, etc.) is of great importance. So be careful…

Here is an example of Virtual Results robots.txt file.

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /comments
Disallow: */trackback
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads

Dont be totally discouraged if you dont get all of this, as you will have another chance.  In coming weeks, look for my WordPress SEO for Real Estate Websites post on wordpress DUPLICATION,  a pretty large SEO issue with WordPress out of the box.  WIthin this post, we will give tips and tools to use wordpress plugins to create robots.txt to reduce content duplication…

Next time?  Creating Great WordPress titles for SEO

WordPress SEO for Real Estate Websites-Permalinks

icon_bigIn WordPress, what some call a URL, we call Permalinks…

Permalinks are literally permanent links to your blog posts, yeah, the ones that live in your address bar because they contain the address to your blog post (or page).

WordPress sets them by default to look something like this…


Now, having important keywords in your URL (permalinks) is pretty darn important to the major search engines, (arguably, one of the top 5 most important things in the Google algorithm) So, unless your post HAPPENS to be about the number 123, we should really change this.. In addition, it gives your readers a better idea what your post is about…

Change to what? and how? Well here again, there are two schools of thought…
1) www.yourdomain.com/the-title-of-your-post/
2) www.yourdomain.com/category/the-title-of-your-post/

Both make a ton of sense in the proper situation… I tend to favor number two when I can build keyword rich, important categories and write proper descriptions for those categories…

How to change to “pretty Permalinks”
In WP 2.7, simply locate the link that says “permalinks” near the bottom of your left hand tool bar, under “settings”…

next choose the radio button next to “custom settings” and finally enter the permalink structure youhave decided on…

For 1) www.yourdomain.com/the-title-of-your-post enter /%postname%/
For 2) www.yourdomain.com/category/the-title-of-your-post enter /%category%/%postname%/.

Thats it your done?

*Note that if your .htaccess file is not set to be writeable, WordPress will produce a set of codes which you’ll need to copy and paste to your .htaccess file!

Now, I know this is really simple stuff for a lot of you, but please hang with me and SUBCRIBE, because it is going to get a lot more interesting….

Next Post? When is the last time you invested some time in your robots.txt file?