Friday, 4 March 2011

Admin Notes: Setting up robots.txt on your Domino server

So, you're running a Domino server, and one of your co-workers finds the Domino help databases on your server indexed through Google.

What do you do?  

The development team wants them left there so that they can be used, but they don't want to have to login to use them.

How do you get them unlisted from Google's index?

Easy, using a robots.txt file.   But you say, how do I create it, what database do I put it in?   And how do I get Google to remove the listing of my databases?

Create a Robots.txt
I like this site for creating my robots.txt files.   Simple, easy to use.   It handles 99% of my needs.

Where to put Robots.txt on a Lotus Domino server
The Robots.txt file goes in the 'domino/html/' sub-directory of the Lotus Domino data directory.  Not in a specific database.   It will get ignored if it is not in the root of your site.  (http://server.com/robots.txt)

Validating the Robots.txt file
There are lots of tools that will validate your robots.txt file.  I like this one.

Removing sites from Google
With Google, you do have the option of blocking your server with a Robots.txt and then waiting for the results to drop off the radar.

If you need it faster, you need to use the Google Site Removal tool.   It's located here: https://www.google.com/webmasters/tools/removals