10 Reasons Why Your Website Went to Hell: Part Ten – Bad Robots Formatting
Believe it or not – the internet has been invaded by robots. Spiders too. Now the robots file is basically used to tell search engines what to look for on your site – it even tells the search engines when to look for it. Poorly formatted robots files – or no robots file altogether – can make your website suck even worse. If for whatever reason Google takes issue with your robots file it’s as good as not inviting Google at all. Once that happens – no one may ever see your website again.