User-agent: * Disallow: /home$ Disallow: /custom-*s/ Disallow: /fas*.cfm Disallow: /fo*.cfm Disallow: /*?*headerID=* Disallow: /*?*headerId=* Disallow: /*?*HeaderID=* Disallow: /*?*HeaderId=* Disallow: /media/* Disallow: /*?* Disallow: *?* Disallow: /day-* Disallow: /month-* Disallow: /year-* Disallow: /page-1* crawl-delay: 10 Disallow: /home/$ Disallow: /download.cfm Disallow: /download/doc # Block trendkite-akashic-crawler User-agent: trendkite-akashic-crawler Disallow: /