1 |
AdsBot [Google] |
AdsBot-Google |
|
Yes |
2 |
Alexa [Bot] |
ia_archiver |
|
Yes |
3 |
Alta Vista [Bot] |
Scooter/ |
|
Yes |
4 |
Ask Jeeves [Bot] |
Ask Jeeves |
|
Yes |
5 |
Baidu [Spider] |
Baiduspider+( |
|
Yes |
6 |
Bing [Bot] |
bingbot/ |
|
Yes |
7 |
Exabot [Bot] |
Exabot/ |
|
Yes |
8 |
FAST Enterprise [Crawler] |
FAST Enterprise Crawler |
|
Yes |
9 |
FAST WebCrawler [Crawler] |
FAST-WebCrawler/ |
|
Yes |
10 |
Francis [Bot] |
http://www.neomo.de/ |
|
Yes |
11 |
Gigabot [Bot] |
Gigabot/ |
|
Yes |
12 |
Google Adsense [Bot] |
Mediapartners-Google |
|
Yes |
13 |
Google Desktop |
Google Desktop |
|
Yes |
14 |
Google Feedfetcher |
Feedfetcher-Google |
|
Yes |
15 |
Google [Bot] |
Googlebot |
|
Yes |
16 |
Heise IT-Markt [Crawler] |
heise-IT-Markt-Crawler |
|
Yes |
17 |
Heritrix [Crawler] |
heritrix/1. |
|
Yes |
18 |
IBM Research [Bot] |
ibm.com/cs/crawler |
|
Yes |
19 |
ICCrawler - ICjobs |
ICCrawler - ICjobs |
|
Yes |
20 |
ichiro [Crawler] |
ichiro/ |
|
Yes |
21 |
Majestic-12 [Bot] |
MJ12bot/ |
|
Yes |
22 |
Metager [Bot] |
MetagerBot/ |
|
Yes |
23 |
MSN NewsBlogs |
msnbot-NewsBlogs/ |
|
Yes |
24 |
MSN [Bot] |
msnbot/ |
|
Yes |
25 |
MSNbot Media |
msnbot-media/ |
|
Yes |
26 |
NG-Search [Bot] |
NG-Search/ |
|
Yes |
27 |
Nutch [Bot] |
http://lucene.apache.org/nutch/ |
|
Yes |
28 |
Nutch/CVS [Bot] |
NutchCVS/ |
|
Yes |
29 |
OmniExplorer [Bot] |
OmniExplorer_Bot/ |
|
Yes |
30 |
Online link [Validator] |
online link validator |
|
Yes |