User-Agent: ArchitextSpider # excite User-Agent: Ask Jeeves # ask.com User-Agent: FAST-WebCrawler User-Agent: Freecrawl # euroseek.net User-Agent: Googlebot # google User-Agent: Mediapartners-Google* User-Agent: Gulliver # Northern Light User-Agent: ia_archiver # web.archive.org User-Agent: InfoSeek User-Agent: Lycos User-Agent: Scooter # AltaVista User-Agent: Slurp # Inktomi User-agent: mozDex # open source :D Disallow: /browse/ Disallow: /fav/ Disallow: /friends/ Disallow: /journal/ Disallow: /near/ Disallow: /polls/ Disallow: /scraps/ Disallow: /star/ Disallow: /watchedby/ Disallow: /watching/ User-Agent: * Disallow: /