Recognizing the Value of the Robots.txt File