So, you're running a Domino server, and one of your co-workers finds the Domino help databases on your server indexed through Google.
What do you do?
The development team wants them left there so that they can be used, but they don't want to have to login to use them.
How do you get them unlisted from Google's index?
Easy, using a robots.txt file. But you say, how do I create it, what database do I put it in? And how do I get Google to remove the listing of my databases?
Create a Robots.txt
I like this site for creating my robots.txt files. Simple, easy to use. It handles 99% of my needs.
Where to put Robots.txt on a Lotus Domino server
The Robots.txt file goes in the 'domino/html/' sub-directory of the Lotus Domino data directory. Not in a specific database. It will get ignored if it is not in the root of your site. (http://server.com/robots.txt)
Validating the Robots.txt file
There are lots of tools that will validate your robots.txt file. I like this one.
Removing sites from Google
With Google, you do have the option of blocking your server with a Robots.txt and then waiting for the results to drop off the radar.
If you need it faster, you need to use the Google Site Removal tool. It's located here: https://www.google.com/webmasters/tools/removals
What do you do?
The development team wants them left there so that they can be used, but they don't want to have to login to use them.
How do you get them unlisted from Google's index?
Easy, using a robots.txt file. But you say, how do I create it, what database do I put it in? And how do I get Google to remove the listing of my databases?
Create a Robots.txt
I like this site for creating my robots.txt files. Simple, easy to use. It handles 99% of my needs.
Where to put Robots.txt on a Lotus Domino server
The Robots.txt file goes in the 'domino/html/' sub-directory of the Lotus Domino data directory. Not in a specific database. It will get ignored if it is not in the root of your site. (http://server.com/robots.txt)
Validating the Robots.txt file
There are lots of tools that will validate your robots.txt file. I like this one.
Removing sites from Google
With Google, you do have the option of blocking your server with a Robots.txt and then waiting for the results to drop off the radar.
If you need it faster, you need to use the Google Site Removal tool. It's located here: https://www.google.com/webmasters/tools/removals
You can put the robots.txt in a database. The only thing you should do is Redirection document on your server, with the name robots.txt and redirect this to the url of the robots.txt in a database.
ReplyDeleteWe use this method for some years, and Google is not complaining
Any tips on how to restrict access across mulitple web domains all hosted on the one server, using robots.txt. We have all sites under /data/web/sitedomain1/site1.nsf, /data/web/sitedomain3/site3.nsf, etc. Can each /sitedomain#/ foldder or site#.nsf have a robots.txt file?
ReplyDeleteCheers
Ian
You can put the robots.txt in a database. The only thing you should do is Redirection document on your server, with the name robots.txt and redirect this to the url of the robots.txt in a database.
ReplyDeleteWe use this method for some years, and Google is not complaining