Is it necessary to include robot.txt in a website or is it just for signalling Google to do not index or crawl the page/website? - Quora
![🍌 John 🍌 on Twitter: "We never supported crawl-delay in robots.txt -- the numbers given there tend to be unrealistic for the most part.… https://t.co/nPYTj5XNZh" 🍌 John 🍌 on Twitter: "We never supported crawl-delay in robots.txt -- the numbers given there tend to be unrealistic for the most part.… https://t.co/nPYTj5XNZh"](https://pbs.twimg.com/media/ELnIVQQXkAIBwkz.png)
🍌 John 🍌 on Twitter: "We never supported crawl-delay in robots.txt -- the numbers given there tend to be unrealistic for the most part.… https://t.co/nPYTj5XNZh"
![Google is going to stop supporting crawl-delay, nofollow, and noindex in robots.txt | by Umair Hashmi | Medium Google is going to stop supporting crawl-delay, nofollow, and noindex in robots.txt | by Umair Hashmi | Medium](https://miro.medium.com/max/2560/0*FEaPt7XtY2Q7S-_Z.png)