User-Agents

Note: See the Server log file section for a complete list of environmental variables and more about your server log files.

User-Agents are one of many environmental variables that the server gets from the visitor. ( HTTP_USER_AGENT ). For example, when a visitor visits this site, with Windows 98, running Microsoft Internet Explorer 5.5, the HTTP_USER_AGENT would look something like this:

Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)

Every visitor that visits your site leaves user-agent info that is able to be logged, however there are utilities that will allow you to return a custom user-agent to the server.

There are certain user-agents that you DO NOT want to see in your log files however. Some can waste your bandwidth, causing your site to slow down, others can harvest your e-mail address, some of them will copy your website, others are just plain rude.

Here is a brief list of search engine spider user-agents..these are a good thing to see in general.

• Googlebot/2.1 - Google.com
• Scooter-3.0.FS - Altavista.com
• FAST-WebCrawler/2.2.5 - Lycos/Alltheweb/Fast

Here is a brief list of User-agents you do NOT want to see.

• WebZIP - Copies websites to hard drives
• EmailSiphon - E-mail harvester
• Wget - Very malicious web crawler

Check out our log file section for more on environmental variables and how to analyze your log files. Click here for a few scripts that will help you block these user-agents.

If you have any questions, feel free to visit our forums.

Table of Contents