# Robots.txt file from http://www.wxnyjs.net/ # All robots will spider the domain User-agent: * Disallow: /bbs/SN_Music/ Disallow: /jsunion/music/ Disallow: /jsunion/rm/ Disallow: /jsunion/download/ Disallow: /images/album/ Disallow: /photo/ Disallow: /download/ User-agent: slurp Crawl-delay: 10000 User-agent: sosospider Disallow: / User-agent: spider Disallow: /