All about Junk E-mail ( SPAM )

I've been doing some research on a hot topic that's been around since the early days of the web, It's called mail harvesting, and it's the number 1 way an e-mail address gets collected and used for unsolicited e-mail. This will help you all in probably answering a common question, "Why do I get so much spam?"

The answer all comes down to mail harvesting programs. Now I could rant on for hours about this, but I'm going to try to sum it all up as much as possible.

Today on the Internet lies many marketing companies that use mailing lists that send out letters to addresses. Some companies use opt-in lists ( meaning you have to add your address to the list, you SUBSCRIBE ) but many more use generated lists, this is where the unsolicited part comes in. There are literally hundreds of 'mail harvesting' programs out there that can do 1 or more of the following things:

A. Follow links for keyword searches through search engines to sites, harvesting e-mail addresses from pages. For example, if a marketing company wanted to build an e-mail database of web designers, all they would have to do is enter "web design services" for example, into the harvesting program, and it will follow the returned results from engines to the specific sites and harvest the addresses from there.

B. Visit a domain all in itself, pulling e-mail addresses from pages.

This includes USENET groups, discussion forums ( which are commonly hit ), and ISP's.

There are several ways you can protect your e-mail address from harvesting programs....

1. Don't post your e-mail address in any USENET groups. As far as message forums go, Don't put your e-mail address in your message body. If the forum has a profile for its users, you shouldn't enter your e-mail address in there either as it is still subject to harvesting. The most advanced harvesters are looking for preprogrammed patterns in the most popular message boards such as the UBB.

2. A good Harvester will only follow CGI links if they find a certain pattern. Most amateur spammers still use programs that specify how man directories deep they go into a site. From my results using http://domain.com/a/b/c/d/e/f/email.html ( for example ) will cut down on a lot of spam. If you have a cgi-bin use an e-mail form which stores the e-mail in the cgi code ( or PHP ). Also just to be safe change the name of the cgi mail to something different...other then using the word 'mail'.

3. If you are entering your e-mail address on a web page, use this sample javascript to help stop harvesting programs:

Substitute the data for variable "firsthalf" with your username and the data for variable "secondhalf" with your actual domain for the e-mail address. Most web browsers are JavaScript enabled so that shouldn't be much of a problem and basically safe to use as a simple solution. However one of the problems with this is that harvesting programs will attempt to put together any combination of phrases that even look like an e-mail address, but this would help.

Blocking user agents from domains with several well-known harvesting programs remains to be an option, but as I stated before there are far to many to block them all. Many other methods also embed tons of fake e-mail addresses into their web pages, in hopes the harvester will pick them up, resulting in a lot of bounced mail.

The pro spammers are now using reverse DNS lookup when compiling and sorting their lists. In the past its easy to generate false domains and e-mail addresses for spammers
to suck up. Now its easy for them to sort the lists if the domain isn't a current one (i.e. false) its eliminated, therefore most of the anti-spam programs are ineffective. I've heard there is a way around this but its still being tested out.

I would recommend to those with web space to put up
some sort of script on your index page that will allow you to track the user-agent of visitors. Here is a brief list of the user-agents of more popular harvesting programs:

EmailSiphon
EmailWolf
ExtractorPro
Mozilla.*NEWT
Crescent
CherryPicker
WebBandit
NICErsPRO
EmailCollector

Just because a user-agent with a harvester program visited your page, doesn't mean your address was successfully harvested

Table of Contents