爬虫大全\网络蜘蛛 源代码
const king_robots=":robot/1.0@Neofonie.de|0.1_hseo(at)cs.rutgers.edu@Rutgers.edu|AbachoBOT (Mozilla compatible)@Crawler.de|AbachoBOT@Crawler.de|Abbr@http://www.suoxie.com|abcdatos_botlink@Abcdatos.com|abot/0.1 (abot; http://www.abot.com; abot@abot.com)@Abot.com|abot/0.1@Abot.com|About/0.1libwww-perl/5.47@About.com|accoona@Accoona.com|AcoiRobot@Acoi Picture Robot|Acoon Robot v1.50.001@Acoon.de|Acoon Robot v1.52@Acoon.de|Acoon Robot@Accon.de|ActiveWorlds/3.xx@ActiveWorlds.com|admin@crawler.de@Crawler.de|admin@maxbot.com@maxbot.com|ADSARobot@ADSARobot|AESOP_com_SpiderMan@Aesop.com|agadine/1.x.x@Agada.de|Agent-admin/@Daum.net|ah-ha.com crawler (crawler@ah-ha.com)@Ah-ha.com|ah-ha.com crawler@Ah-ha.com|AIBOT/2.1@21Seek.com|aipbot/1.0 (aipbot; http://www.aipbot.com; aipbot@aipbot.com)@NameProtect.com|aipbot/1.0@NameProtect.com|aipbot@NameProtect.com|Aladin/3.324@Aladin.de|Aleksika Spider/1.0@Aleksika.de|AlkalineBOT/1.3@Vestris.com|AlkalineBOT/1.4@Vestris.com|AlkalineBOT@AlkalineBOT|Allesklar/0.1 libwww-perl/5.46@Allesklar.de|AltaVista-Intranet@Altavista.co.uk|AmfibiBOT@Amfibi.com|Amiga-AWeb/3.4.167SE@AWeb.com|amzn_assoc@Amazon.com|AnswerChase PROve x.0@AnswerChase.com|AnswerChase x.0@AnwserChase.com|AnswerChase@AnswerChase.com|Antibot@AntiSeach.net|antibot-V1.1.5/i586-linux-2.2@AntiSeach.net|AnzwersCrawl/2.0@Anzwers|A-Online Search@Jet2Web.com|Aport@Aport.ru|appie 1.1@Walhello.com|appie@Walhello.com|arachnoidea@euroseek.net@Euroseek.net|Arachnoidea@Euroseek.com|Aranha@Girafa.com|ArchitectSpider@Excite.com|archive.org_bot@Archive.org|archive_org@Archive.org|ask jeeves@Ask Jeeves|ask.24x.info@Ask.24x.info|ASPSeek/1.2.5@ASPSeek.org|ASPseek/1.2.9d@ASPSeek.com|ASPSeek/1.2.x@ASPSeek.org|ASPSeek/1.2.xa@ASPSeek.org|ASPseek/1.2.xx@ASPseek.org|ASPSeek/1.2.xxpre@ASPSeek.org|ASPSeek@ASPSeek.org|asterias/2.0@SingingFish.com|Asterias@SingingFish.com|AtlocalBot/1.1 +(http://www.atlocal.com/local-web-site-owner.html)@Atlocal.com|AtlocalBot@Atlocal.com|Atomz/1.0@Atomz.com|Atomz@Atomz.com|Augurfind@Augurnet.ch|augurnfind V-1.x@Augurnet.ch|autohttp@Elsop.com|AV Fetch 1.0@Altavista.com|AVSearch@Altavista.com|AVSearch-3.0@Altavista.com|AxmoRobot@Axmo.com|Baiduspider+@Baidu.com|BaiDuSpider@Baidu.com|BanBots/1.2@Banbots.com|BarraHomeCrawler@BarraHome.com|bdcindexer_2.6.2@Business.com|BDFetch@Brandimensions.com|BDNcentral Crawler v2.3@Bdncentral.com|beautybot/1.0@Cosmoty.com|BigCliqueBOT/1.03-dev@BigClique.com|Bilbo/2.3b-UNIX@Biblo Bot|BlackWidow@BlackWidow Bot|Blaiz-Bee/1.0 (+http://www.blaiz.net)@Blaiz.net|Blaiz-Bee/1.0@Blaiz.net|BlitzBOT@tricus.net (Mozilla compatible)@Blitzsuche.rp-online.de|BlitzBOT@tricus.net@Blitzsuche.rp-online.de|Blog@http://www.5iya.com/blog|BlogBot/1.x@Blogdex.net|BlogBot@Blogdex.net|Bloglines Title Fetch/1.0 (http://www.bloglines.com)@Bloglines.com|blogWatcher_Spider/0.1@Blogwatcher.pi.titech.ac.jp|BlogzIce/1.0 (+http://icerocket.com; rhodes@icerocket.com)@IceRocket.com|BlogzIce/1.0 +http://www.icerocket.com/@IceRocket.com|BlogzIce/1.0@IceRocket.com|boitho.com-dc/0.xx@Boitho.com|boitho.com-robot/1.x@Boitho.com|BSDSeek/1.0@Inktomi.com|BullsEye@Intelliseek.com|bumblebee@relevare.com@Relevare.com|Buscaplus Robi/1.0@Buscaplus.com|Buscaplus Robi@Buscaplus.com|CanSeek/@Cansee.ca|ChristCRAWLER 2.0@Christcrawler.com|ChristCRAWLER@Christcrawler.com|CipinetBot@Cipinet.com|citenikbot/@Cenik.co.uk|ClariaBot/1.0@Searchscout.com|Claymont.com@Claymont.com|CLIPS-index@Cips-index.imag.fr|Clushbot/3.3-BinaryFury (+http://www.clush.com/bot.html)@Clush.com|Clushbot/3.3-BinaryFury@Clush.com|Combine/x.0@Combine Bot|Computer_and_Automation_Research_Institute_Crawler@Research Bot|contact/jylee@kies.co.kr@Kies.co.kr|Convera Internet Spider V6.x@Convera.com|cosmos/0.8_(robot@xyleme.com)@Xyleme.com|cosmos/0.9_(robot@xyleme.com)@Xyleme.com|cosmos@Xyleme.com|C-PBWF-ip3000.com-crawler@IP3000.com|CrawlConvera0.1@Converna.net|Crawler (cometsearch@cometsystems.com)@Findwhat.com|Crawler admin@crawler.de@Crawler.de|Crawler V 0.2.x admin@crawler.de@Crawler.de|crawler@alexa.com@Alexa.com|crawler@brainbot.com@Brainbot.com|crawler@fast.no@Alltheweb.com|Crawler@Crawler.de|CrawlerBoy Pinpoint.com@Pinpoint.com|CrawlerBoy@Pinpoint.com|CrocCrawler@CrocCrawler.com|Custom Spider www.bisnisseek.com /1.0@Bisnisseek.com|DaAdLe.com ROBOT/@Daadle.com|DataparkSearch/4.xx@DataparkSearch.com|DaviesBot/1.7@Wholeweb.net|DaviesBot@Wholeweb.net|dbDig@Prairielandconsulting.com|DBrowse 1.4b@DBrowse Bot|DBrowse 1.4d@DBrowse Bot|dCSbot/1.1@Openmarket.com|DeepIndex@DeepIndex.com|DeepIndexer.ca@DeepIndex.com|deepweb@Deepweb.com|Demo Bot DOT 16b@ Btopenworld.com|DiaGem/@Skyrocket.gr.jp|DiaGem/1.1 (http://www.skyrocket.gr.jp/diagem.html)@Skyrocket.gr.jp|DiaGem/1.1@Skyrocket.gr.jp|Diamond/1.0@Searchscout.com|Digger/1.0 JDK/1.3.0rc3@Diggit.com|Digimarc WebReader@Digimarc.com|DigOut4U@Arisem.com|DIIbot/1.2@Digital-Integrity.com|DIIbot@Digital-Integrity.com|DittoSpyder@Ditto.com|dloader(NaverRobot)/@Naver.com|dtSearchSpider@Dtsearch.com|Dumbot(version 0.1 beta - dumbfind.com)@Dumbfind.com|Dumbot(version 0.1 beta)@Dumbfind.com|dumrobo(NaverRobot)/@Naver.com|EARTHCOM.info/1.4beta@Earthcom.info|EasyDL/3.02@Keywen.com|EasyDL/3.03@Keywen.com|EasyDL/3.04@Keywen.com|EasyDL/3.xx http://keywen.com/Encyclopedia/Bot@Keywen.com|EasyDL/3.xx@Keywen.com|EchO!/2.0@Voila.com|egothor/3.0a (+http://www.xdefine.org/robot.html)@Xdefine.com|egothor/3.0a@Xdefine.com|EgotoBot/4.8@Egoto.com|Enterprise_Search/1.0.xxx@Innerprise.net|Enterprise_Search/1.0@Innerprise.net|erik@malfunction.org@Malfunction.org|EroCrawler@Erocrawler.com|ES.NET_Crawler/2.0@Innerprise.net|eseek-larbin_2.6.2@Exactseek.com|ESISmartSpider@Travel-Finder.com|ExactSeek Crawler/0.1@Exactseek.com|exactseek-crawler-2.63@Exactseek.com|Exalead NG/MimeLive Client@Exabot.com|Excalibur Internet Spider V6.5.4@Excalib.com|ExperimentalHenrytheMiragoRobot@Mirago.co.uk|EyeCatcher (Download-tipp.de)/1.0@Download-tipp.de|EyeCatcher@Download-tipp.de|EZResult@Ezresults.com|Fast Crawler Gold Edition@Alltheweb.com|FAST Data Search Crawler@Alltheweb.com|FAST Enterprise Crawler 6 (Experimental)@Alltheweb.com|FAST Enterprise Crawler 6 used by FAST (FAST)@Alltheweb.com|FAST Enterprise Crawler 6 used by FAST@Alltheweb.com|FAST Enterprise Crawler 6@Alltheweb.com|FAST Enterprise Crawler/6 (www.fastsearch.com)@Alltheweb.com|FAST Enterprise Crawler/6@Alltheweb.com|FAST FirstPage retriever (compatible; MSIE 5.5; Mozilla/4.0)@Alltheweb.com|FAST FirstPage retriever@Alltheweb.com|Fast PartnerSite Crawler@Alltheweb.com|FastCrawler 3.0.1 (crawler@1klik.dk)@Alltheweb.com|FastCrawler 3.0.1@Alltheweb.com|FAST-WebCrawler/2.2.10@Altheweb.com|FAST-WebCrawler/2.2.6@Altheweb.com|FAST-WebCrawler/2.2.7@Altheweb.com|FAST-WebCrawler/2.2.8@Altheweb.com|FAST-WebCrawler/3.2 test@Altheweb.com|FAST-WebCrawler/3.2@Altheweb.com|FAST-WebCrawler/3.3@Altheweb.com|FAST-WebCrawler/3.4/Nirvana@Altheweb.com|FAST-WebCrawler/3.4/PartnerSite@Altheweb.com|FAST-WebCrawler/3.5@Altheweb.com|FAST-WebCrawler/3.6/FirstPage@Altheweb.com|FAST-WebCrawler/3.6@Altheweb.com|FAST-WebCrawler/3.7/FirstPage@Altheweb.com|FAST-WebCrawler/3.7@Altheweb.com|FAST-WebCrawler/3.8/Fresh@Altheweb.com|FAST-WebCrawler/3.8@Altheweb.com|FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no)@Altheweb.com|FAST-WebCrawler/3.x Multimedia@Altheweb.com|FAST-WebCrawler@Alltheweb.com|Faxobot/1.0@FaXo.com|Felix - Mixcat Crawler@Mixcat.com|Felix@Mixcat.com|FFC Trap Door Spider@Frequentfinders.com|fido/1.0 Harvest/1.4.pl2@Planet Search|FileExtension@http://www.kuozhanming.com|FinaleRobot@expressus.com|FindAnISP.com_ISP_Finder_v99a@FindAnISP.com|FineBot@Finesearch.com|Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5)@Fireball.de|Firefly/1.0@Fireball.de|FlickBot 2.0 RPT-HTTPClient/0.3-3@DivX.com|Fluffy the spider@SearchHippo.com|Flunky@Metacarta.com|Francis/1.0 (francis@neomo.de http://www.neomo.de/)@Neomo.de|Francis/1.0@Neomo.de|FreeFind.com-SiteSearchEngine/1.0@Freefind.com|FyberSearch@Fybersearch.com|FyberSpider@Fybersearch.com|Gagglebot@Gagglebot|GAIS Robot/1.0B2@Seed.net.tw|Gaisbot/3.0+@Seed.net.tw|Gaisbot/3.0@Seed.net.tw|GalaxyBot/1.0 (http://www.galaxy.com/galaxybot.html)@Galaxy.com|GalaxyBot/1.0@Galaxy.com|GalaxyBot@Galaxy.com|GammaSpider/1.0@Gammasite.com|gazz/1.0@Infobee.ne.jp|gazz@nttrd.com@Infobee.ne.jp|Geckobot@Geckobot.com|geckobot@Geckobot.com|GenCrawler@Gendoor.com|generic_crawler/01.0217/@Cmu.edu|GentleSpider@Research.att.com|GeonaBot@Geona.com|GigaBaz@Brainbot.com|GigaBazVStheWeb@Brainbot.com|GigaBlast@Gigablast.com|Gigabot/1.0@Gigablast.com|Gigabot@Gigablast.com|GigabotSiteSearch/2.0 (sitesearch.gigablast.com)@Gigablast.com|GigabotSiteSearch/2.0@Gigablast.com|Giskard@oralco.com|GNODSPIDER@Gnod.net|Goblin/0.9 (http://www.goguides.org/)@Goguides.org|Goblin/0.9.x (http://www.goguides.org/goblin-info.html)@Goguides.org|Goblin/0.9.x@Goguides.org|Goblin/0.9@Goguides.org|GoForIt.com@Goforit.com|Goofer/0.2@Wanadoo.fr|Google@Google.com|Googlebot/1.0@Google.com|Googlebot/2.1@Google.com|Googlebot/Test@Google.com|googlebot@googlebot.com@Google.com|Googlebot@Google.com|Googlebot-Image/1.0@Google.com Image Bot|GrigorBot 0.8@Grigor.biz|grub crawler(http://www.grub.org)@Grub.com|grub-client@Grub.org|gsa-crawler (Enterprise; GID-01422; jplastiras@google.com)@Google.com|gsa-crawler (Enterprise; GID-01742;gsatesting@rediffmail.com)@Google.com|gsa-crawler@Google.com|Gulliver/1.2@Northernlight.com|Gulliver/1.3@Northernlight.com|Gulliver@Northernlight.com|Gulper Web Bot 0.2.4@Gulper Bot|Gulper Web Bot@Gulper Bot|Hatena Antenna/0.4 (http://a.hatena.ne.jp/help#robot)@a.hatena.ne.jp|Hatena Antenna/0.4@a.hatena.ne.jp|HeinrichderMiragoRobot@Mirago.de|HenrytheMiragoRobot@Mirago.co.uk|HenryTheMiragoRobot@Mirago.co.uk|Hippias/0.9 Beta@ Hippias Bot|holmes/x.x@Morfeo.centrum.cz|HomePageSearch(hpsearch.uni-trier.de)@Hpsearch.uni-trier.de|Homerbot: www.homerweb.com@Homerweb.com|Homerbot@Homerweb.com|htdig/3.1.6 (http://computerorgs.com)@Computerorgs.com|htdig/3.1.x (root@localhost)@Htdig.org|htdig/3.1.x@Htdig.org|http://www.abcdatos.com/botlink/@Abcdatos.com|Hubater@Hubat.com|ia_archiver/1.6@Archive.org|ia_archiver@Archive.org|IAArchiver-1.0@Archive.org|icsbot-0.1@Icseoul.org|ideare - SignSite/1.x@Ideare.com|Ideare - SignSite@Ideare.com|IlTrovatore-Setaccio (+http://www.iltrovatore.it)@Iltrovatore.it|Iltrovatore-Setaccio/0.3-dev@Iltrovatore.it|IlTrovatore-Setaccio/1.2@Iltrovatore.it|iltrovatore-setaccio/1.2-dev@Iltrovatore.it|IlTrovatore-Setaccio@Iltrovatore.it|IncyWincy@Incywincy.com|IndexTheWeb.com Crawler7@IndexTheWeb.com|Inet library@Inetlibrary.com|info@searchhippo.com@Searchhippo.com|InfoFly/1.0 (http://www.versions-project.org/)@Versions-project.org|InfoFly/1.0@ Versions-project.org|InfoSeek Sidewinder/0.9@Infoseek.com|InfoSeek Sidewinder/1.0A@ Infoseek.com|InfoSeek Sidewinder/1.1A@ Infoseek.com|Infoseek SideWinder/1.45 (Compatible; MSIE 10.0; UNIX)@ Infoseek.com|Infoseek SideWinder/1.45@ Infoseek.com|Infoseek SideWinder/2.0B (Linux 2.4 i686)@ Infoseek.com|Infoseek SideWinder/2.0B@ Infoseek.com|InfoSeek Sidewinder@infoseek.com|Inktomi Search@Yahoo.com|Inktomi@Yahoo.com|InnerpriseBot/1.0 (http://www.innerprise.com/)@Innerprise.net|InnerpriseBot/1.0@Innerprise.net|InsumaScout@InsumaScout|Internet Ninja 6.0@Dti.ne.jp|InternetSeer.com@ InternetSeer.com|InternetSeer@InternetSeer|InternetSeer@InternetSeer.com|ip3000.com@ip3000.com|ip3000.com-crawler@ip3000.com|IPiumBot laurion(dot)com@Laurion.com|ipiumBot@Laurion.com|Iron33@Verno.ueda.info.waseda.ac.jp|Jabot/6.x (http://odin.ingrid.org/)@Ingrid.org|Jabot/6.x@Ingrid.org|Jabot/7.x.x (http://odin.ingrid.org/)@Ingrid.org|Jabot/7.x.x@Ingrid.org|Jack@Domanova.co.uk|jan.gelin@av.com@Altavista.co.uk|jeeves@Ask Jeeves|Jetbot/1.0@Jeteye.com|JoocerBot@Joocer.com|Jyxobot/x@Jyxo.cz|Katriona@KatrionaKE_1.0/2.0 libwww/5.2.8@Voila.fr|Kenjin Spider@Kenjin.ne.jp|KIT_Fireball/2.0@Fireball.de|KIT-Fireball/2.0 (compatible; Mozilla 4.0; MSIE 5.5)@Fireball.de|KIT-Fireball/2.0@Fireball.de|KIT-Fireball@Fireball.de|Knowledge.com/0.x@Knowledge.com|Kototoi/0.1@s.u-tokyo.ac.jp|kuloko-bot/0.2@Kuloko.com|kuloko-bot/0.x@Kuloko.com|Kulokobot@Kuloko.com|kulturarw3/0.1@National Library of Sweden|LEIA/3.01pr (LEIAcrawler; [SNIP])@Gseek.com|LEIA@Gseek.com|LexiBot@Lexibot.com|LNSpiderguy@Lexis-nexis.com|Look.com@Look.com|lwp-trivial/1.27@Search4free.com|lwp-trivial/1.34@Search4free.com|Lycos_Spider_(T-Rex)/3.0@Lycos.com|Lycos_Spider_(T-Rex)@Lycos.com|lycos@Lycos.com|Mackster( http://www.ukwizz.com )@Ukwizz.com|Mackster@Ukwizz.com|Mag-Net@Wind.it|mailto:webcraft@bea.com@Beasys.com|mammoth/1.0 (+http://www.sli-systems.com/)@Sli-Systems.com|mammoth/1.0@Sli-Systems.com|MantraAgent@Looksmart.com|mapper@teradex.com@Mapper.teradex.com|Mariner/5.1b [de] (Win95; I ;Kolibri gncwebbot)@Kolibri.de|Mariner/5.1b@ Kolibri.de|Martini@Looksmart.com|Marvin v0.3@Marvin Bot|marvin/infoseek@Webseek.de|marvin-team@webseek.de@Webseek.de|Mata Hari/2.00@Lexibot.com|MediaCrawler-1.0 (Experimental)@Mediacrawler.de|MediaCrawler-1.0@Mediacrawler.de|Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)@Google.com Adsense|Mediapartners-Google/2.1@Google.com Adsense|MediaSearch/0.1@Www.fi|MegaSheep v1.0 (www.searchuk.com internet sheep)@SearchUK.com|MegaSheep v1.0@SearchUK.com|Mercator@Altavista.com|Mercator@Altavista.com|Mercator-1.x@Altavista.com|Mercator-2.0@Altavista.com|Mercator-Scrub-1.1@Altavista.com|metabot@Metabot|Metaspinner/0.01@Meta-Spinner.de|MFC_Tear_Sample@Microsoft.com|MicrosoftPrototypeCrawler@Microsoft.com|Minibot@Minibot|MiracleAlphaTest@Nec.co.jp|MnogoSearch/3.2.xx@mNoGoSearch.org|moget/2.0@Goo.ne.jp|moget/2.1 (moget@goo.ne.jp)@Goo.ne.jp|moget/2.1@Goo.ne.jp|moget@goo.ne.jp@Goo.ne.jp|Mole2/1.0@Intags.de|Morris - Mixcat Crawler (+http://mixcat.com)@Mixcat.com|Morris - Mixcat Crawler@Mixcat.com|Morris@Mixcat.com|Mouse-House/7.4@Spidermonkey.ca|mozDex/@Mozdex.com|mozDex/0.04-dev@Mozdex.com|Mozilla (Mozilla@somewhere.com)@Somewhere.com|Mozilla/2.0 (compatible; Ask Jeeves)@Ask Jeeves|Mozilla/2.0 (compatible; EZResult -- Internet Search Engine)@DirectHit.com|Mozilla/3.0 (compatible; MuscatFerret/1.5.4; claude@euroferret.com)@Euroferret.com|Mozilla/3.0 (compatible; MuscatFerret/1.5; olly@muscat.co.uk)@Euroferret.com|Mozilla/3.0 (compatible; MuscatFerret/1.6.x; claude@euroferret.com)@Euroferret.com|Mozilla/3.01 (Compatible; Links2Go Similarity Engine)@Links2go.com|Mozilla/4.0 (compatible; FastCrawler3, support-fastcrawler3@fast.no)@AllTheWeb.com|Mozilla/4.0 (compatible; grub-client-2.x)@Grub.org|Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) TrueRobot; 1.5@Echo.com|Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot)@MSN.com|Mozilla/4.0 (compatible; SpeedySpider; www.entireweb.com)@EntireWeb.com|Mozilla/4.0 (compatible; www.galaxy.com)@Galaxy.com|Mozilla/4.0 [en] (Ask Jeeves Corporate Spider)@Ask Jeeves|Mozilla/4.0(compatible; Zealbot 1.0)@LookSmart.com|mozilla/5.0 (compatible; genevabot +http://www.healthdash.com)@HealthDash.com|Mozilla/5.0 (wgao@genieknows.com)@GenieKnows.com|Mozilla/5.0 wgao@genieknows.com@GenieKnows.com|Mozilla/5.0 (compatible; Yahoo! Slurp;)@Yahoo.com|MP3Bot@mp3bot.de|MSN@MSN.com|MSNBOT/0.1@MSN.com|msnbot@MSN.com|MultiText/0.1@MultiText|MultiText@MultiText|MuscatFerret@Webtop.com|NationalDirectoryAddURL/1.0@NationalDirectory.com|NationalDirectory-SuperSpider@Nationaldirectory.com|NationalDirectory-WebSpider/1.3@NationalDirectory.com|Nazilla@Websmostlinked.com|NEC Research Agent@NEC Research Agent|NetCruiser@NetCruiser|NetResearchServer(http://www.look.com)@Look.com|NetResearchServer@Look.com|NetZippy@Netzippy.com|NextopiaBOT@Nextopia.com|NG/1.0@Exabot.com|Noago Spider@Noago.com|NP/0.1 (NP; http://www.nameprotect.com; npbot@nameprotect.com)@NameProtect.com|NP/0.1@NameProtect.com|NPBot@NameProtect.com|NPBot-1/2.0@NameProtect.com|nttdirectory_robot@Navi.ocn.ne.jp|Nutch@Nutch.org|NutchCVS/0.0x-dev@Nutch.org|NutchOrg/0.0x-dev@Nutch.org|NutchOrg@Nutch.org|NZBot@NavigationZone.com|ObjectsSearch/0.01@ObjectsSearch.com|ObjectsSearch/0.0x@ObjectsSearch.com|oBot ((compatible;Win32))@Cobion.com|oBot@Cobion.com|OliverPerry@Claymont.com|OntoSpider/1.0 libwww-perl/5.65@Ontospider.i-n.info|OntoSpider/1.0@Ontospider.i-n.info|OntoSpider@Ontospider.i-n.info|Openbot/@Openfind.com.tw|Openbot/3.0+@Openfind.com.tw|Openbot/3.0@Openfind.com.tw|Opencola@Opencola.com|Openfind piranha@Openfind.com|Openfind Robot/1.1A2@Openfind.com.tw|http://www.5iya.com|OpenTextSiteCrawler/2.9.2@Opentext.net|Oracle Ultra Search@Oracle.com|Oxxbot1@Oxxfordinfo.com|Pansophica@Pansophica|ParaSite/1.0b (http://www.ianett.com/parasite/)@Ianett.com|ParaSite/1.0b@Ianett.com|pd02_1.0.0 pd02_1.0.0@dzimi@post.sk@Post.sk|peerbot@Peerbot.com|Phoaks@Phoaks|phortse@hanmail.net@Daum.net|PICgrabber@PICgrabber|PicoSearch/@PicoSearch.com|PicoSearch/1.0@PicoSearch.com|PictureOfInternet@PictureOfInternet|PintaSpider@PintaSpider|pipeLiner/0.10@Pipeline-Search.com|pipeLiner@Pipeline-Search.com|Pita@Pita Bot|Pizilla++ ver 2.45@He.net|PJspider/3.0 (pjspider@portaljuice.com; http://www.portaljuice.com)@Portaljuice.com|PJspider/3.0@Portaljuice.com|PJspider@Portaljuice.com|polybot 1.0 (http://cis.poly.edu/polybot/)@ PolyBot|polybot 1.0@PolyBot|PolyBot@PolyBot|Pompos/1.x http://dir.com/pompos.html@Dir.com|Pompos/1.x pompos@iliad.fr@Iliad.fr|Pompos/1.x@Free.fr|Popdexter/1.0@Popdex.com|PortalBSpider/2.0 (spider@portalb.com)@Portalb.com|PortalBSpider/2.0@Portalb.com|Project XP5 [2.03.07-111203]@ XP5 Bot|Project XP5@XP5 Bot|PROve AnswerBot 4.0@Answerchase.com|psbot/0.1 (+http://www.picsearch.com/bot.html)@Picsearch.com|psbot/0.1@Picsearch.com|psbot@Picsearch.com|Qango.com Web Directory (http://www.qango.com/)@Qango.com|Qango.com Web Directory@ Qango.com|QPCreep Test Rig ( We are not indexing, just testing ) @Quepasa.com|QPCreep Test Rig@Quepasa.com|QuepasaCreep ( crawler@quepasacorp.com )@Quepasa.com|QuepasaCreep v0.9.1x@Quepasa.com|QuepasaCreep@Quepasa.com|QueryN Metasearch@Queryn.com|rabaz (rabaz at gigabaz dot com)@Brainbot.com|Rabaz@Brainbot.com|RaBot@Daum.net|ramBot xtreme x.x@Intersearch.de|Reaper [2.03.10-031204] (http://www.sitesearch.ca/reaper/)@Sitesearch.ca|Reaper [2.03.10-031204]@Sitesearch.ca|Reaper/2.0x (+http://www.sitesearch.ca/reaper)@Sitesearch.ca|Reaper/2.0x@Sitesearch.ca|Reaper@Sitesearch.ca|roach.smo.av.com-1.0@Altavista.com|roach.smo.av.com-1.0@Altavista.com|RoboCrawl (www.canadiancontent.net)@CanadianContent.net|RoboCrawl@CanadianContent.net|Robot/www.pj-search.com@PJ-Search.com|Robot@SuperSnooper.Com@SuperSnooper.com|robot@xyleme.com@Xyleme.com|robot-master@expressus.com@Expressus.com|robot-response@openfind.com.tw@Openfind.com.tw|Robozilla/1.0@Dmoz.org|Rotondo/3.1 libwww/5.3.1@Qualigo.de|ru-robot@Rutgers.edu|Scooter/1.0 scooter@pa.dec.com@Altavista.com|Scooter/1.0@Altavista.com|Scooter/1.1 (custom)@Altavista.com|Scooter/2.0 G.R.A.B. V1.1.0 @Altavista.com|Scooter/2.0 G.R.A.B. X2.0@Altavista.com|Scooter/3.3.QA.pczukor@Altavista.com|Scooter/3.3.vscooter@Altavista.com|Scooter/3.3_SF@Altavista.com|Scooter/3.3@Altavista.com|Scooter_bh0-3.0.3@Altavista.com|Scooter_trk3-3.0.3@Altavista.com|Scooter@Altavista.com|Scooter2_Mercator_3-1.0@Altavista.com|Scooter2_Mercator_x-x.0@Altavista.com|Scooter-3.0.EU@Altavista.com|Scooter-3.0.FS@Altavista.com|Scooter-3.0.HD@Altavista.com|Scooter-3.0.VNS@Altavista.com|Scooter-3.0QI@Altavista.com|Scooter-3.2.BT@Altavista.com|Scooter-3.2.DIL@Altavista.com|Scooter-3.2.EX@Altavista.com|Scooter-3.2.JT@Altavista.com|Scooter-3.2.NIV@Altavista.com|Scooter-3.2.SF0@Altavista.com|Scooter-3.2.snippet@Altavista.com|Scooter-3.2@Altavista.com|Scooter-3.3dev@Altavista.com|Scooter-ARS-1.1@Altavista.com|Scooter-ARS-1.1-ih@Altavista.com|scooter-venus-3.0.vns@Altavista.com|Scooter-W3.1.2@Altavista.com|Scooter-W3-1.0@Altavista.com|Scrubby/@Scrubtheweb.com|search.at V1.2@Search.at|search.ch V1.4.2@Search.ch|search.ch V1.4@Search.ch|SearchExpress Spider0.99@SearchExpress.com|Searchit-Now Robot/2.2 (+http://www.searchit-now.co.uk)@Searchit-now.co.uk|Searchit-Now Robot/2.2@Searchit-now.co.uk|SearchSpider.com/1.1@Searchspider.com|Searchspider/@Searchspider.com|Searchspider/1.2@Searchspider.com|sebastien.ailleret@inria.fr@Inria.fr|Seeker.lookseek.com@Lookseek.com|SeznamBot/1.0 (+http://fulltext.seznam.cz/)@Seznam.cz|SeznamBot/1.0@Seznam.cz|sohu-search@SOHU|sohu-search@Sohu|Shark@Openfind.com|sherlock_spider@Sherlock.com.cn|SightQuestBot/@Sightquest.com|SiteSpider +(http://www.SiteSpider.com/)@Sitespider.com|SiteSpider@Sitespider.com|SiteXpert@Xtreeme.com|Skampy/0.9.x@ Skaffe.com|Skimpy/0.x@Skaffe.com|Slarp/0.1@Only.com|Slider_Search_v1-de@Slider.com|Slurp.so/1.0@ Yahoo.com|Slurp/2.0j@Yahoo.com|Slurp/2.0-KiteHourly@Yahoo.com|Slurp/2.0-OwlWeekly@Yahoo.com|Slurp/3.0-AU@Yahoo.com|slurp@inktomi.com@Yahoo.com|slurp@inktomi@ Yahoo.com|Slurp@Yahoo.com|SlySearch/1.0@Turnitin.com|SlySearch/1.x@Turnitin.com|SlySearch@Turnitin.com|speedfind ramBot xtreme 8.1@Speedfind.de|speedfind ramBot xtreme@Speedfind.de|Speedy Spider (Beta/x.x; speedy@entireweb.com)@EntireWeb.com|Speedy Spider@EntireWeb.com|Speedy_Spider (http://www.entireweb.com)@EntireWeb.com|Speedy_Spider@EntireWeb.com|Spida/0.1@Only.com|Spider TraficDublu@Traficdublu.ro|Spider/maxbot.com admin@maxbot.com@Maxbot.com|Spider/maxbot.com@Maxbot.com|spider@aeneid.com@Yahoo.com|Spider_Monkey/@Spidermonkey.ca|spider_monkey@Spidermonkey.ca|SpiderMonkey/7.x@Spidermonkey.ca|Sqworm/2.9.81-BETA (beta_release; 20011102-760; i686-pc-linux-gnu)@Aol.com|Sqworm@Aol.com|Steeler/1.x (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/)@Steeler Bot|Steeler/1.x@Steeler Bot|Steeler@Steeler Bot|Suchbot@Suchbot.de|Suchknecht.at-Robot@Suchknecht.at|super-robot@super.navi.ocn.ne.jp@Navi.ocn.ne.jp|support@canseek.ca@Canseek.ca|Surfnomore Spider v1.1@Surfnomore.com|SwissSearch V1.2@SwissSearch|SynoBot@Synomia.fr|TaWWWantula@TaWWWantula|teoma_admin@hawkholdings.com@Teoma.com|teoma_agent1@Teoma.com|TeraCrawl@TeraCrawl|Teradex_Mapper@Mapper.teradex.com|The Informant@The Informant|TheSuBot/0.1 (www.thesubot.de)@TheSuBot.de|TheSuBot/0.1@TheSuBot.de|thumbshots-de-Bot (Version: 1.02, powered by www.thumbshots.de)@ThumbShots.de|thumbshots-de-Bot@ThumbShots.de|Toutatis 2.5-2@Hoppa.com|Toutatis 3.3 (hoppa.com)@Hoppa.com|Toutatis 4-19.6 (hoppa.com)@Hoppa.com|TurnitinBot/x.x (http://www.turnitin.com/robot/crawlerinfo.html)@Turnitin.com|TurnitinBot/x.x@Turnitin.com|TurnitinBot@Turnitin.com|Tutorial Crawler@Tutorgig.com|Tv<nn>_Merc_resh_26_1_D-1.0@Altavista.com|Tv<nn>_Merc_resh_26_1_D-1.0@Altavista.com|TygoBot@Tygo.com|UCmore@Ucmore.com|UdmSearch@Mnogosearch.org|UK Searcher Spider@Uksearcher.co.uk|Ultraseek@Infoseek.com|unlostBot@Unlost.com|updated/0.1beta@Updated.com|URLBlaze@Urlblaze.net|UrlScope@UrlScope|Vagabondo/1.x MT (webagent@wise-guys.nl)@Wise-guys.nl|Vagabondo/2.0 MT (webagent at wise-guys dot nl)@Wise-guys.nl|Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl)@Wise-guys.nl|Vagabondo/2.0 MT@Wise-guys.nl|Vagabondo@Wise-guys.nl|vspider@Verity.com|webbandit/4.xx.0@WebBandit|Webbandit@WebBandit|Webclipping.com@WebClipping.com|webcollage/1.xx@WebCollage.com|webcollage@WebCollage.com|WebCompass@WebCompass|WebCrawler/3.0 Robot libwww/5.0a@WebCrawler|WebCrawler-AddURL/2.0@WebCrawler|WebFindBot(http://www.web-find.com)@Web-find.com|WebFindBot@Web-find.com|WebGather@WebGather|WebGenie@WebGenie|webmaster@intags.de@Intags.de|whatUseek_winona/3.0@Whatuseek.com|whatuseek@Whatuseek.com|WhizBang! Lab@Whizbanglabs.com|WIRE WebRefiner@Wire.co.uk|wisenutbot@Looksmart.com|WSCbot@WorldSearchCenter.com|www.inktomisearch.com@Yahoo.com|www.WebWombat.com.au@Webwombat.com.au|YahooSeeker/CafeKelsa@Yahoo.com|Yandex@Yandex.com|Yellopet-Spider@YellowPet.com|YottaShopping_Bot/4.12@YottaShopping.com|Zao/0.1 (http://www.kototoi.org/zao/)@Zao Bot|Zao/0.1@Zao Bot|Zao/0.2@Zao Bot|Zao-Crawler@Zao Bot|zerxbot/Version 0.6 libwww-perl/5.79@Zerx.com|Zippy v2.0 - Zippyfinder.com@Zippyfinder.com|ZyBorg@Wisenut.com"