FAQ FAQ | Forum Rules and Policies Forum Policies
 
Memberlist Member List | Calender Community

Navigation

Home
Tutorials

Sell Your Website
References
Web design tips
Dating
Graphic Design tips
Search Engine Promotion
User-Agent database
About the Internet
Web Technology
Web Screenshot Archive
Our Newsletter
Search

Free Stuff
Free graphics
Free desktop wallpaper
Free HTML templates
Free scripts

Free web-based tools
Graphic Design Contest
Affordable Web Hosting
Website Business Tips
Diaper Cakes


Services
Website Business Ideas
Services summary
Web design services
Web design packages
Logo design services
Graphic design services
Pricing
Portfolio
Custom Website Design
Unique Baby Gifts

Inside i.D.
Our partners
Advertise
Site Map
FAQ
Contact
About

Spider List - Frequent Web Crawlers and Spiders

Here is a list of some of the spiders that have visited Icehouse Designs. With any luck they will aid you in your search engine spider identification. This list was last updated on April 2nd, 2002. If you have access logs ( stats ) for your site, you should see similar visitors below. As you probably know by now, they aren't human visitors, but rather little programs designed mostly to index data for search engines.

Keep in mind that just because a spider visits your site, it doesn't mean that your site was added at that time, or will ever get added for that matter. Some sites may take several days to several weeks to index after a particular crawl. Some, even take months.

 

IP / Hostname
User-Agent
Owner / Purpose
63.236.92.151
teomaagent [crawler-admin@teoma.com]
Teoma / Indexing / http://www.teoma.com
66.35.208.60
Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)
Wisenut / Indexing / http://www.wisenut.com
k1.fujitsu.co.jp
"InfoNaviRobot(F107)"
www.infoweb.ne.jp
natasha.waltham.northernlight.com Gulliver/1.3

Norternlight / Indexing/ http://www.northernlight.com

france.excite.com ArchitextSpider Excite, Indexing
marvin.northernlight.com Gulliver/1.3
NorthernLight / Indexing/ http://www.northernlight.com
godzilla.bos.lycos.com
Lycos / Indexing / http://www.lycos.com
scooter4.sv.av.com
Scooter-3.0.FS
Altavista / Indexing / http://www.altavista.com
scooter5.sv.av.com
Scooter-3.0.FS
Altavista / Indexing / http://www.altavista.com
cr011r01-test.sac2.fastsearch.net FAST-WebCrawler/3.2 test Fast crawler / Lycos
crawler21.bos2.fast-search.net
crawler@fast.no; http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html Fast / AlltheWeb / Indexing/ http://www.alltheweb.com
rob029.goo.ne.jp
"moget/2.0 (moget@goo.ne.jp)"
Goo
wg1.en.webtop.com
"Mozilla/4.0 (compatible; MuscatFerret/2.0; http://www.webtop.com/)"
WebTop / Indexing / http://www.webtop.com
*.inktomi.com Mozilla/3.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Multiple sites use Inktomi data, http://www.inktomi.com
*.inktomi.com Slurp Multiple sites use inktomi data, http://www.inktomi.com
crawl1.googlebot.com
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Google / Indexing / http://www.google.com
crawl2.googlebot.com
Googlebot/2.1 (+http://www.googlebot.com/bot.html) Google / Indexing / http://www.google.com
crawl3.googlebot.com
Googlebot/2.1 (+http://www.googlebot.com/bot.html) Google / Indexing / http://www.google.com
crawl4.googlebot.com
Googlebot/2.1 (+http://www.googlebot.com/bot.html) Google / Indexing / http://www.google.com
crawl5.googlebot.com
Googlebot/2.1 (+http://www.googlebot.com/bot.html) Google / Indexing / http://www.google.com
crawl6.googlebot.com
Googlebot/2.1 (+http://www.googlebot.com/bot.html) Google / Indexing / http://www.google.com
crawl7.googlebot.com
Googlebot/2.1 (+http://www.googlebot.com/bot.html) Google / Indexing / http://www.google.com
crawl8.googlebot.com
Googlebot/2.1 (+http://www.googlebot.com/bot.html) Google / Indexing / http://www.google.com
crawl9.googlebot.com Googlebot/2.1 (+http://www.googlebot.com/bot.html) Google / Indexing / http://www.google.com
http://directory.mozilla.org Robozilla/1.0 DMOZ - Open Directory / link checker
ezspider307.directhit.com
Mozilla/2.0 (compatible; Ask Jeeves) Ask Jeeves / Direct Hit

 

Menu
   

 

 

 

 

 

 


Cool Sites

Website Templates
Free Hit Counter
$2.95/month Web Hosting
Reseller Hosting

 



Logo Design | Photoshop Tutorials | Dreamweaver Tutorials | Non GMO Canola
© 2000-2004 IceHouse Designs, Inc. View Privacy Statement.

Valid HTML 4.01!

Valid CSS!