User-agent: .info Disallow: / User-agent: 192.comAgent Disallow: / User-agent: 2Bone_LinkChecker/1.0 libwww-perl/5.64 Disallow: / User-agent: 4anything.com LinkChecker v2.0 Disallow: / User-agent: 82-2-3011-1954 / nhnbot@naver.com Disallow: / User-agent: 8484 Boston Project v 1.0 Disallow: / User-agent: ABCdatos BotLink/5.xx.xxx#BBL Disallow: / User-agent: AESOP_com_SpiderMan Disallow: / User-agent: ASPSeek/1.2.5 Disallow: / User-agent: ASPSeek/1.2.x Disallow: / User-agent: ASPSeek/1.2.xa Disallow: / User-agent: ASPSeek/1.2.xxpre Disallow: / User-agent: ASPseek/1.2.9d Disallow: / User-agent: ASPseek/1.2.xx Disallow: / User-agent: ASSORT/0.10 Disallow: / User-agent: AV Fetch 1.0 Disallow: / User-agent: AVSearch-1.0 Disallow: / User-agent: AVSearch-2.0-fusionIdx-14-CompetitorWebSites Disallow: / User-agent: AVSearch-3.0(AltaVista/AVC) Disallow: / User-agent: Aaron@bol.com.br Disallow: / User-agent: AbachoBOT Disallow: / User-agent: AbachoBOT (Mozilla compatible) Disallow: / User-agent: Aberja Checkomat Disallow: / User-agent: About/0.1libwww-perl/5.47 Disallow: / User-agent: Accelatech RSSCrawler/0.4 Disallow: / User-agent: Accoona-AI-Agent/1.1.1 Disallow: / User-agent: Accoona-AI-Agent/1.1.2 Disallow: / User-agent: AcoiRobot Disallow: / User-agent: Acoon Robot v1.50.001 Disallow: / User-agent: Ad Muncher v4.xx.x Disallow: / User-agent: Ad Muncher v4x Build xxxxx Disallow: / User-agent: AgentName/0.1 libwww-perl/5.48 Disallow: / User-agent: Aladin/3.324 Disallow: / User-agent: AlkalineBOT/1.3 Disallow: / User-agent: AlkalineBOT/1.4 Disallow: / User-agent: Allesklar/0.1 libwww-perl/5.46 Disallow: / User-agent: AltaVista Intranet V2.0 AVS EVAL search@freeit.com Disallow: / User-agent: AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.net Disallow: / User-agent: AltaVista Intranet V2.0 evreka.com crawler@evreka.com Disallow: / User-agent: AltaVista V2.0B crawler@evreka.com Disallow: / User-agent: AmfibiBOT Disallow: / User-agent: Amfibibot/0.06 Disallow: / User-agent: Amfibibot/0.07 Disallow: / User-agent: AnnoMille spider 0.1 alpha - http://www.annomille.it Disallow: / User-agent: Anonymizer/1.1 Disallow: / User-agent: AnswerChase PROve x.0 Disallow: / User-agent: AnswerChase x.0 Disallow: / User-agent: Apexoo Spider 1.x Disallow: / User-agent: Aplix HTTP/1.0.1 Disallow: / User-agent: Aplix_SANYO_browser/1.x (Japanese) Disallow: / User-agent: Aplix_SEGASATURN_browser/1.x (Japanese) Disallow: / User-agent: Aport Disallow: / User-agent: ArchitextSpider Disallow: / User-agent: Arikus_Spider Disallow: / User-agent: Asahina-Antenna/1.x Disallow: / User-agent: Asahina-Antenna/1.x (libhina.pl/x.x ; libtime.pl/x.x) Disallow: / User-agent: Atomz/1.0 Disallow: / User-agent: Azureus 2.x.x.x Disallow: / User-agent: BDFetch Disallow: / User-agent: BIGLOTRON Disallow: / User-agent: BMCLIENT Disallow: / User-agent: BMLAUNCHER Disallow: / User-agent: BSDSeek/1.0 Disallow: / User-agent: BStop.BravoBrian.it Agent Detector Disallow: / User-agent: BaiDuSpider Disallow: / User-agent: Barca/2.0.xxxx Disallow: / User-agent: BarcaPro/1.4.xxxx Disallow: / User-agent: BeebwareDirectory/v0.01 Disallow: / User-agent: Big Fish v1.0 Disallow: / User-agent: BigBrother/1.6e Disallow: / User-agent: Bilbo/2.3b-UNIX Disallow: / User-agent: Bitacle bot/1.1 Disallow: / User-agent: BlackWidow Disallow: / User-agent: Blaiz-Bee/2.00.xxxx Disallow: / User-agent: BlitzBOT@tricus.net Disallow: / User-agent: BlogBot/1.x Disallow: / User-agent: Bobby/4.0.x RPT-HTTPClient/0.3-3E Disallow: / User-agent: Bot mailto:craftbot@yahoo.com Disallow: / User-agent: BravoBrian SpiderEngine MarcoPolo Disallow: / User-agent: BravoBrian bstop.bravobrian.it Disallow: / User-agent: BullsEye Disallow: / User-agent: BunnySlippers Disallow: / User-agent: CFNetwork/x.x Disallow: / User-agent: CJ Spider/ Disallow: / User-agent: CJ Spider/ Disallow: / User-agent: CJB.NET Proxy Disallow: / User-agent: Catall Spider Disallow: / User-agent: Catherine@kcmsd.net Disallow: / User-agent: China Local Browse 2.6 Disallow: / User-agent: China Local Browse 2.6 Disallow: / User-agent: ChristCRAWLER 2.0 Disallow: / User-agent: ClariaBot/1.0 Disallow: / User-agent: Claymont.com Disallow: / User-agent: Claymont.com Disallow: / User-agent: CoBITSProbe Disallow: / User-agent: CoBITSProbe Disallow: / User-agent: Combine/2.0 http://combine.it.lth.se/ Disallow: / User-agent: Combine/3 http://combine.it.lth.se/ Disallow: / User-agent: Combine/x.0 Disallow: / User-agent: ContactBot/0.2 Disallow: / User-agent: ContentSmartz Disallow: / User-agent: Convera Internet Spider V6.x Disallow: / User-agent: ConveraCrawler/0.2 Disallow: / User-agent: ConveraCrawler/0.9d Disallow: / User-agent: ConveraMultiMediaCrawler/0.1 Disallow: / User-agent: CoolBot Disallow: / User-agent: CoteoNutchCrawler/Nutch-0.9 Disallow: / User-agent: CougarSearch/0.x Disallow: / User-agent: Cowbot-0.1 Disallow: / User-agent: CrawlConvera0.1 Disallow: / User-agent: CrawlConvera@yahoo.com Disallow: / User-agent: Crawler Disallow: / User-agent: Crawler V 0.2.x admin@crawler.de Disallow: / User-agent: Crawler admin@crawler.de Disallow: / User-agent: CrawlerBoy Pinpoint.com Disallow: / User-agent: CurryGuide SiteScan 1.1 Disallow: / User-agent: DBrowse 1.4b Disallow: / User-agent: DBrowse 1.4d Disallow: / User-agent: DIIbot/1.2 Disallow: / User-agent: DISCo Pump x.x Disallow: / User-agent: DSurf15a 01 Disallow: / User-agent: DSurf15a 71 Disallow: / User-agent: DSurf15a 81 Disallow: / User-agent: DSurf15a VA Disallow: / User-agent: DTAAgent Disallow: / User-agent: DataCha0s/2.0 Disallow: / User-agent: DataFountains/DMOZ Downloader Disallow: / User-agent: DeepIndex Disallow: / User-agent: DeepIndexer.ca Disallow: / User-agent: Demo Bot DOT 16b Disallow: / User-agent: Diamond/x.0 Disallow: / User-agent: DiamondBot Disallow: / User-agent: DigOut4U Disallow: / User-agent: Digger/1.0 JDK/1.3.0rc3 Disallow: / User-agent: DittoSpyder Disallow: / User-agent: DoCoMo/1.0/Nxxxi/c10 Disallow: / User-agent: DoCoMo/1.0/Nxxxi/c10/TB Disallow: / User-agent: DoCoMo/1.0/P502i/c10 Disallow: / User-agent: DoCoMo/2.0 P900iV(c100;TB;W24H11) Disallow: / User-agent: Dumbot Disallow: / User-agent: EBrowse 1.4b Disallow: / User-agent: EMPAS_ROBOT Disallow: / User-agent: ESISmartSpider Disallow: / User-agent: ESurf15a 15 Disallow: / User-agent: EasyDL/3.xx Disallow: / User-agent: EchO!/2.0 Disallow: / User-agent: Educate Search VxB Disallow: / User-agent: EgotoBot/4.8 Disallow: / User-agent: EmailSiphon Disallow: / User-agent: EmailWolf 1.00 Disallow: / User-agent: EmeraldShield.com WebBot Disallow: / User-agent: Enfish Tracker Disallow: / User-agent: Enterprise_Search/1.0 Disallow: / User-agent: Enterprise_Search/1.0.xxx Disallow: / User-agent: EroCrawler Disallow: / User-agent: Everest-Vulcan Inc./0.1 Disallow: / User-agent: Exabot-Images/1.0 Disallow: / User-agent: Exabot-Test/1.0 Disallow: / User-agent: Exabot/2.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: Exalead NG/MimeLive Client Disallow: / User-agent: Excalibur Internet Spider V6.5.4 Disallow: / User-agent: ExperimentalHenrytheMiragoRobot Disallow: / User-agent: Expired Domain Sleuth Disallow: / User-agent: ExtractorPro Disallow: / User-agent: EyeCatcher (Download-tipp.de)/1.0 Disallow: / User-agent: FAST Enterprise Crawler 6 Disallow: / User-agent: FAST Enterprise Crawler 6 used by FAST Disallow: / User-agent: FAST FirstPage retriever Disallow: / User-agent: FAST MetaWeb Crawler Disallow: / User-agent: FAST-WebCrawler/2.2.6 Disallow: / User-agent: FAST-WebCrawler/2.2.7 Disallow: / User-agent: FAST-WebCrawler/2.2.8 Disallow: / User-agent: FAST-WebCrawler/3.2 test Disallow: / User-agent: FAST-WebCrawler/3.3 Disallow: / User-agent: FAST-WebCrawler/3.4/Nirvana Disallow: / User-agent: FAST-WebCrawler/3.4/PartnerSite Disallow: / User-agent: FAST-WebCrawler/3.5 Disallow: / User-agent: FAST-WebCrawler/3.6 Disallow: / User-agent: FAST-WebCrawler/3.6/FirstPage Disallow: / User-agent: FAST-WebCrawler/3.7 Disallow: / User-agent: FAST-WebCrawler/3.8 Disallow: / User-agent: FAST-WebCrawler/3.8/Fresh Disallow: / User-agent: FAST-WebCrawler/3.x Multimedia Disallow: / User-agent: FFC Trap Door Spider Disallow: / User-agent: FSurf15a 01 Disallow: / User-agent: Factbot 1.09 Disallow: / User-agent: Fast Crawler Gold Edition Disallow: / User-agent: Fast PartnerSite Crawler Disallow: / User-agent: FastBug http://www.ay-up.com Disallow: / User-agent: FastCrawler 3.0.1 Disallow: / User-agent: FastSearch Web Crawler Disallow: / User-agent: Faxobot/1.0 Disallow: / User-agent: Feed Seeker Bot Disallow: / User-agent: Feed24.com Disallow: / User-agent: FeedChecker/0.01 Disallow: / User-agent: FeedForAll rss2html.php v2 Disallow: / User-agent: Feedfetcher-Google Disallow: / User-agent: Feedster Crawler/3.0; Feedster, Inc. Disallow: / User-agent: Felix - Mixcat Crawler Disallow: / User-agent: Filangy/0.01-beta Disallow: / User-agent: Filangy/1.0x Disallow: / User-agent: FindAnISP.com_ISP_Finder_v99a Disallow: / User-agent: Findexa Crawler Disallow: / User-agent: FineBot Disallow: / User-agent: Firefly/1.0 Disallow: / User-agent: Firefox Disallow: / User-agent: Firefox_1.0.6 Disallow: / User-agent: FirstGov.gov Search - POC Disallow: / User-agent: Flapbot/0.7.2 Disallow: / User-agent: FlickBot 2.0 RPT-HTTPClient/0.3-3 Disallow: / User-agent: FocusedSampler/1.0 Disallow: / User-agent: Francis/1.0 Disallow: / User-agent: Franklin Locator 1.8 Disallow: / User-agent: FreeFind.com-SiteSearchEngine/1.0 Disallow: / User-agent: FreshNotes crawler Disallow: / User-agent: Full Web Bot 0416B Disallow: / User-agent: Full Web Bot 0516B Disallow: / User-agent: Full Web Bot 2816B Disallow: / User-agent: FuseBulb.Com Disallow: / User-agent: FyberSpider Disallow: / User-agent: GAIS Robot/1.0B2 Disallow: / User-agent: Gagglebot Disallow: / User-agent: Googlebot Disallow: / User-agent: Mediapartners-Google Disallow: / User-agent: Adsbot-Google Disallow: / User-agent: Janice@vr-web.de Disallow: / User-agent: Jeeves Disallow: / User-agent: Jet2Web Search Disallow: / User-agent: Margarito@jrhl.com Disallow: / User-agent: Merrill@ymcanyc.org Disallow: / User-agent: Randolph@dtiglobal.com Disallow: / User-agent: SitiDi.net/SitiDiBot/1.0 Disallow: / User-agent: Slurp Disallow: / User-agent: Yahoo-MMCrawler Disallow: / User-agent: VIAGRA ® Official Site Disallow: / User-agent: abot@abot.com Disallow: / User-agent: accoona Disallow: / User-agent: admin@fastsearch.com Disallow: / User-agent: admin@mspennyworth.com Disallow: / User-agent: agent@amfibi.com Disallow: / User-agent: aiderss.com Disallow: / User-agent: aipbot/1.0 Disallow: / User-agent: aipbot/2-beta Disallow: / User-agent: aipbot@aipbot.com Disallow: / User-agent: albertof@barrahome.org Disallow: / User-agent: amibot Disallow: / User-agent: annotate_google; http://ponderer.org/download/annotate_google.user.js Disallow: / User-agent: antibot-V1.1.5/i586-linux-2.2 Disallow: / User-agent: anzwerscrawl@anzwers.com.au Disallow: / User-agent: arachnoidea@euroseek.com Disallow: / User-agent: aranhabot Disallow: / User-agent: archive.org_bot Disallow: / User-agent: ask.24x.info Disallow: / User-agent: asterias/2.0 Disallow: / User-agent: atSpider/1.0 Disallow: / User-agent: atw-crawler@fast.no Disallow: / User-agent: augurfind Disallow: / User-agent: augurnfind V-1.x Disallow: / User-agent: autoemailspider Disallow: / User-agent: autohttp Disallow: / User-agent: bCentral Billing Post-Process Disallow: / User-agent: bellsouth.net Disallow: / User-agent: bilgi@bilgi.com Disallow: / User-agent: billbot wjj@cs.cmu.edu Disallow: / User-agent: blogsearchbot-pumpkin-3 Disallow: / User-agent: boitho.com-robot/1.x Disallow: / User-agent: bot@bigclique.com Disallow: / User-agent: bot@douban.com Disallow: / User-agent: bot@evaal.com Disallow: / User-agent: bot@execrawl.com Disallow: / User-agent: btopenworld.com Disallow: / User-agent: bumblebee@relevare.com Disallow: / User-agent: carleson/1.0 Disallow: / User-agent: cazoodlebot@cazoodle.com Disallow: / User-agent: ccubee/x.x Disallow: / User-agent: cfetch/1.0 Disallow: / User-agent: cg-eye interactive Disallow: / User-agent: collage.cgi/1.82 Disallow: / User-agent: combine/0.0 Disallow: / User-agent: cometsearch@cometsystems.com Disallow: / User-agent: crawler@1klik.dk Disallow: / User-agent: crawler@alexa.com Disallow: / User-agent: crawler@burstfind.com Disallow: / User-agent: crawler@crawlly.com Disallow: / User-agent: crawler@exactseek.com Disallow: / User-agent: crawler@fast.no Disallow: / User-agent: crawler@flaptor_period.com Disallow: / User-agent: crawler@freshnotes.com Disallow: / User-agent: crawler@ilab.sztaki.hu Disallow: / User-agent: crawler@sphsearch.sg Disallow: / User-agent: csci_b659/0.13 Disallow: / User-agent: dCSbot/1.1 Disallow: / User-agent: daypopbot/0.x Disallow: / User-agent: dds explorer v1.0 beta Disallow: / User-agent: deepak-USC/ISI Disallow: / User-agent: dev-spider2.searchpsider.com/1.3b Disallow: / User-agent: disco-crawl@discoveryengine.com Disallow: / User-agent: disco/Nutch-0.9 Disallow: / User-agent: disco/Nutch-1.0-dev Disallow: / User-agent: dlman Disallow: / User-agent: dloader(NaverRobot)/1.0 Disallow: / User-agent: dodgebot/experimental Disallow: / User-agent: dssb@dataspear.com Disallow: / User-agent: dtSearchSpider Disallow: / User-agent: eStyleSearch 4 Disallow: / User-agent: ejupiter.com Disallow: / User-agent: epicurus@gmail.com Disallow: / User-agent: exactseek-crawler-2.63 Disallow: / User-agent: exactseek-pagereaper-2.63 Disallow: / User-agent: exactseek.com Disallow: / User-agent: fastbot crawler beta 2.0 Disallow: / User-agent: favo.eu crawler/0.6 Disallow: / User-agent: filangy-agent@filangy.com Disallow: / User-agent: fileboost.net/1.0 Disallow: / User-agent: findlinks/x.xxx Disallow: / User-agent: firstgov.webmasters@gsa.gov Disallow: / User-agent: firstsbot Disallow: / User-agent: flunky Disallow: / User-agent: fly/6.01 libwww/4.0D Disallow: / User-agent: flyindex.net 1.0 Disallow: / User-agent: francis@neomo.de Disallow: / User-agent: helpdesk@fast.no Disallow: / User-agent: helpdesk@fastsearch.com Disallow: / User-agent: http://asked.jp Disallow: / User-agent: http://balihoo.com/index.aspx Disallow: / User-agent: http://bulkfeeds.net/ Disallow: / User-agent: http://cantor.ee.ucla.edu/~networks/crawl Disallow: / User-agent: http://career.drecom.jp/bot.html Disallow: / User-agent: http://crawler.burstfind.com Disallow: / User-agent: http://dataparksearch.org/bot Disallow: / User-agent: http://de.searchengine.com/spider Disallow: / User-agent: http://download-tipp.de/ Disallow: / User-agent: http://dssb.dataspear.com/bot.html Disallow: / User-agent: http://everest.vulcan.com/crawlerhelp Disallow: / User-agent: http://fast.no/support.php?c=faqs/crawler Disallow: / User-agent: http://fast.no/support/crawler.asp Disallow: / User-agent: http://freefind.com Disallow: / User-agent: http://fulltext.seznam.cz/ Disallow: / User-agent: http://help.yahoo.co.jp/help/jp/search/indexing/indexing-27.html Disallow: / User-agent: http://icerocket.com Disallow: / User-agent: http://ivia.ucr.edu/useragents.shtml Disallow: / User-agent: http://keywen.com/Encyclopedia/Bot Disallow: / User-agent: http://lucene.apache.org/nutch/bot.html Disallow: / User-agent: http://mixcat.com Disallow: / User-agent: http://pauillac.inria.fr/~fpottier/ Disallow: / User-agent: http://search.denmex.com Disallow: / User-agent: http://search.innerprise.net/ Disallow: / User-agent: http://webarchive.cs.ucla.edu/bruinbot.html Disallow: / User-agent: http://wkd.iis.sinica.edu.tw/LiveTrans/ Disallow: / User-agent: http://wortschatz.uni-leipzig.de/findlinks/ Disallow: / User-agent: http://ws.daum.net/aboutkr.html Disallow: / User-agent: http://www.123spider.de Disallow: / User-agent: http://www.21seek.com Disallow: / User-agent: http://www.MyNewFavoriteThing.com/fsb.php Disallow: / User-agent: http://www.SharewarePlaza.com Disallow: / User-agent: http://www.abot.com Disallow: / User-agent: http://www.ackerm.com/ Disallow: / User-agent: http://www.acoon.de Disallow: / User-agent: http://www.acorn.isara.org Disallow: / User-agent: http://www.adaxas.net/ Disallow: / User-agent: http://www.agada.de Disallow: / User-agent: http://www.aipbot.com Disallow: / User-agent: http://www.aleksika.com/ Disallow: / User-agent: http://www.amfibi.com Disallow: / User-agent: http://www.andrew.cmu.edu/~brgordon/webbot/index.html Disallow: / User-agent: http://www.answerbus.com/ Disallow: / User-agent: http://www.apassion4jazz.net/bebopbot.html Disallow: / User-agent: http://www.araby.com Disallow: / User-agent: http://www.asaha.com/ Disallow: / User-agent: http://www.atlocal.com/local-web-site-owner.html Disallow: / User-agent: http://www.attentio.com Disallow: / User-agent: http://www.authoritativeweb.com/crawl Disallow: / User-agent: http://www.axada.de/ Disallow: / User-agent: http://www.axmo.com Disallow: / User-agent: http://www.baboom.us Disallow: / User-agent: http://www.baidu.com/search/spider.htm Disallow: / User-agent: http://www.baidu.com/search/spider_jp.html Disallow: / User-agent: http://www.bdncentral.com/robot.html Disallow: / User-agent: http://www.beammachine.net Disallow: / User-agent: http://www.bigclique.com Disallow: / User-agent: http://www.bigsearch.ca Disallow: / User-agent: http://www.bilgi.com Disallow: / User-agent: http://www.bisnisseek.com /1.0 Disallow: / User-agent: http://www.bitacle.com Disallow: / User-agent: http://www.blaiz.net Disallow: / User-agent: http://www.blogbridge.com/ Disallow: / User-agent: http://www.bloglines.com Disallow: / User-agent: http://www.blogsnow.com/ Disallow: / User-agent: http://www.boitho.com/bot.html Disallow: / User-agent: http://www.boitho.com/dcbot.html Disallow: / User-agent: http://www.btbot.com/btbot.html Disallow: / User-agent: http://www.buildcms.com/crawler Disallow: / User-agent: http://www.buscaplus.com/robi/ Disallow: / User-agent: http://www.cazoodle.com/cazoodlebot Disallow: / User-agent: http://www.chilkatsoft.com/ChilkatHttpUA.asp Disallow: / User-agent: http://www.cipinet.com/bot.html Disallow: / User-agent: http://www.clush.com/bot.html Disallow: / User-agent: http://www.cometrics.de Disallow: / User-agent: http://www.cougarsearch.com/faq.shtml Disallow: / User-agent: http://www.crawlly.com Disallow: / User-agent: http://www.croccrawler.com Disallow: / User-agent: http://www.cuasar.com/spider_beta/ Disallow: / User-agent: http://www.cydral.com Disallow: / User-agent: http://www.dataparksearch.org/ Disallow: / User-agent: http://www.dataspear.com/spider.html Disallow: / User-agent: http://www.die-kraehe.de Disallow: / User-agent: http://www.discoveryengine.com Disallow: / User-agent: http://www.douban.com Disallow: / User-agent: http://www.downloadaccelerator.com Disallow: / User-agent: http://www.dumbfind.com/dumbot.html Disallow: / User-agent: http://www.earthcom.info Disallow: / User-agent: http://www.egoto.com/about.htm Disallow: / User-agent: http://www.emeraldshield.com/webbot.aspx Disallow: / User-agent: http://www.en.deepindex.com Disallow: / User-agent: http://www.envolk.com/envolkspider.html Disallow: / User-agent: http://www.envolk.com/envolkspiderinfo.php Disallow: / User-agent: http://www.eurip.com Disallow: / User-agent: http://www.eventax.de/ Disallow: / User-agent: http://www.execrawl.com/ Disallow: / User-agent: http://www.factbites.com/robots Disallow: / User-agent: http://www.factbites.com/webmasters.php Disallow: / User-agent: http://www.fast.no/faq/faqfastwebsearch/faqfastwebcrawler.html Disallow: / User-agent: http://www.fastbot.de Disallow: / User-agent: http://www.fastsearch.com Disallow: / User-agent: http://www.favo.eu Disallow: / User-agent: http://www.feedmap.net Disallow: / User-agent: http://www.filangy.com/filangyinfo.jsp?inc=robots.jsp Disallow: / User-agent: http://www.fileboost.net Disallow: / User-agent: http://www.findexa.no/gulesider/article26548.ece Disallow: / User-agent: http://www.flaptor.com Disallow: / User-agent: http://www.floorstransformed.com Disallow: / User-agent: http://www.flyindex.net Disallow: / User-agent: http://www.fooky.com/scorpionbots Disallow: / User-agent: http://www.fybersearch.com/fyberspider.php Disallow: / User-agent: http://www.google.com/feedfetcher.html Disallow: / User-agent: http://www.htmlvalidator.com/ Disallow: / User-agent: http://www.icerocket.com Disallow: / User-agent: http://www.innerprise.net/es-spider.asp Disallow: / User-agent: http://www.lidan.com Disallow: / User-agent: http://www.loc.gov/minerva/crawl.html Disallow: / User-agent: http://www.lr.pi.titech.ac.jp/blogWatcher/ Disallow: / User-agent: http://www.megaproxy.com Disallow: / User-agent: http://www.micro-sys.dk/products/keyword-research/ Disallow: / User-agent: http://www.micro-sys.dk/products/sitemap-generator/ Disallow: / User-agent: http://www.nearsoftware.com Disallow: / User-agent: http://www.neofonie.de/loesungen/search/robot.html Disallow: / User-agent: http://www.neomo.de/ Disallow: / User-agent: http://www.netwu.com Disallow: / User-agent: http://www.nutch.org/docs/en/bot.html Disallow: / User-agent: http://www.prairielandconsulting.com Disallow: / User-agent: http://www.rawgrunt.com Disallow: / User-agent: http://www.scifihifi.com/cocoalicious Disallow: / User-agent: http://www.scirus.com/srsapp/contactus/ Disallow: / User-agent: http://www.searchengineworld.com/cgi-bin/robotcheck.cgi Disallow: / User-agent: http://www.shunix.com/robot.htm Disallow: / User-agent: http://www.sicher-durchs-netz.de/bot.html Disallow: / User-agent: http://www.simpy.com/bot.html Disallow: / User-agent: http://www.skyrocket.gr.jp/diagem.html Disallow: / User-agent: http://www.sql-und-xml.de/freeware-tools/ Disallow: / User-agent: http://www.sygol.com Disallow: / User-agent: http://www.sync2it.com/susie Disallow: / User-agent: http://www.telekom.com.my Disallow: / User-agent: http://www.tkl.iis.u-tokyo.ac.jp/~crawler/ Disallow: / User-agent: http://www.uchoose.de/crawler/beautybot/ Disallow: / User-agent: http://www.uchoose.de/crawler/elfbot/ Disallow: / User-agent: http://www.unchaos.com/ Disallow: / User-agent: http://www.vigiltech.com/esensedisclaim.html Disallow: / User-agent: http://www.walhello.com Disallow: / User-agent: http://www.wholeweb.net Disallow: / User-agent: http://www.xdefine.org/robot.html Disallow: / User-agent: http://www.yama.info.waseda.ac.jp/~yamana/es/ Disallow: / User-agent: http://www.zetbot.com Disallow: / User-agent: http://www.zipcommander.com/ Disallow: / User-agent: info@attentio.com Disallow: / User-agent: info@blogpulse.com Disallow: / User-agent: info@coteo.com Disallow: / User-agent: info@enhancededge.com Disallow: / User-agent: info@eventax.de Disallow: / User-agent: kasparek@naparek.cz Disallow: / User-agent: kastaneta03@hotmail.com Disallow: / User-agent: kevin.watters@fastsearch.com Disallow: / User-agent: mm dash crawler@fast.no Disallow: / User-agent: msnbot Disallow: / User-agent: psbot Disallow: / User-agent: nimakhaj@gmail.com Disallow: / User-agent: nutch-agent@askaboutoil.com Disallow: / User-agent: nutch-agent@lists.sourceforge.net Disallow: / User-agent: nutch-agent@lucene.apache.org Disallow: / User-agent: peter.turney@nrc.ca Disallow: / User-agent: pvincent@pagesjaunes.fr Disallow: / User-agent: research@bdc Disallow: / User-agent: rhodes@icerocket.com Disallow: / User-agent: robot@balihoo.com Disallow: / User-agent: robot@xyleme.com Disallow: / User-agent: scirus-crawler@fast.no Disallow: / User-agent: search_comments@sensis.com.au Disallow: / User-agent: spider@banbots.com Disallow: / User-agent: spider@dataspear.com Disallow: / User-agent: spiderinfo@freefind.com Disallow: / User-agent: sts@comperio.no Disallow: / User-agent: vadim_gonchar@unchaos.com Disallow: / User-agent: wardi@uni-muenster.de Disallow: / User-agent: * Disallow: /affies Disallow: /affiliates Disallow: /images Disallow: /*.exe$ Disallow: /*.gif$ Disallow: /*.Gif$ Disallow: /*.GIF$ Disallow: /*.html$ Disallow: /*.jpg$ Disallow: /*.Jpg$ Disallow: /*.JPG$ Disallow: /*.jpeg$ Disallow: /*.Jpeg$ Disallow: /*.JPEG$ Disallow: /*.pdf$ Disallow: /*.Pdf$ Disallow: /*.PDF$ Disallow: /*.php$ Disallow: /*.png$ Disallow: /*.txt$ Disallow: /*.zip$ Disallow: /*.Zif$ Disallow: /*.ZIP$