# file: $Id: robots.txt,v 1.1 2006/03/20 00:00:00 tangxiwen Exp $ # www.dukuai.com 都快网 # 按照robots.txt的标准写法,规定一些不允许爬虫爬的页面或目录。 # robots.txt 的写法参照 # Format is: # User-agent: # Disallow: | User-agent: * Disallow: /bbs/admin/ Disallow: /bpage/ Disallow: /help/question/action/add Disallow: /index2.html Disallow: /index_zt.htm Disallow: /passport/login Disallow: /passport/logout Disallow: /uploadfile/ly/rain.pps Disallow: /user/information/action/edit User-agent: Microsoft URL Control - 6.00.8862 Disallow: /