When Google started to indicate documents like PDF, RTF or Word-files and made them visible at google.com, many webmasters had to think about the fact that they probably did not configure there servers the right way. But now there are some hints that, at least in some cases, this was maybe wrong.
The company Klaus Schallhorn Omnisearch proved that Google indicates so-called secure servers (ssl), too. Although they say that they do not. Meanwhile there are more then 3.8 million secure sites indicated by Google.
Even worse: Google announced that they are fully compliant to the robots.txt-standard but they definitly do not when indicating secure servers. The Google-spiders have already executed over 161,000 cgi-programs one or more times.
If you check the search results of corresponding requests you can find servers that forbid via the robots.txt access to the cgi-bin directory even under the top 10 results.
Webmaster have to take new measures to protect their sites from being indicated by spiders.
Further information is available at the title link (German).
(ps) (Translation: sk)
[News message: 04. Mar. 2002, 12:29] [Comments: 0]
[Send via e-mail] [Print version] [ASCII version]
