# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# Disallow: /

# Each intersect page links to one page per intersecting root, one root deeper,
# so the reachable URL space grows exponentially. Crawling it saturated the app.
# The pages are combinatorial views of content already indexable at /root_words,
# so nothing unique is lost by keeping crawlers out. The prefix covers the query
# string, which is where the roots are passed.
# One /words/:id page exists per row of the 77k-row words table, so each URL is
# unique and every crawl request is a cache miss that reaches the origin. The
# single dyno renders these at ~380ms and 200KB apiece, well under the rate a
# distributed scrape arrives at. The same lemma concordance is reachable from
# /root_words, which stays crawlable.
User-Agent: *
Disallow: /root_words/intersect
Disallow: /words/