Robots.txt file in MVC3

Recently I was asked to look at adding a robots.txt file to a client's website that we built on ASP.NET MVC3. So I did some research and found out some interesting information about the robots.txt file.

1. The file is just a suggestion and bots are not required to follow what you ask them to do via the robots.txt file.  
2. The file is really an exclusion list instead of an inclusion list.  This means you have to put the places you don't want the bots to view which could be a bad idea in that it would give the bad bots areas they should focus on hacking.

After my research we decided to not put a robots.txt file on the website initially. Soon after our deploy we noticed in the ELMAH logs that we were seeing a considerable amount of errors which contained this error message:

The controller for path '/robots.txt' was not found or does not implement IController.

So now we decided we at least needed an empty robots.txt file out there to prevent all these unnecessary errors. So I did some more research and developed a solution for MVC3:

• Basically you just add the physical robots.txt file to the website by adding it to the project at the root level. It could be empty or could contain the basic level of content required in a robots.txt.

Now that you have the physical file on the website it will ignore the ASP.NET MVC3 routing as long as you haven't changed the default setting of the RouteExistingFiles property of the RouteCollection which will ignore routing if a physical file is found that matches the URL pattern.

To ensure that the physical file will always be served up even if someone changes the RouteExistingFiles property you can add the following ignore route code to the global.asax.cs file:
routes.IgnoreRoute("{robotstxt}", new {robotstxt=@"(.*/)?robots.txt(/.*"});

Your mileage may vary with the robots.txt file and it might not be a bad idea to have a robots.txt with some exclusions if you really need to exclude some of your content from web crawlers or bots.

This particular client didn't really need one because most if not all the content of their website required that you log into their website so bots and web crawlers wouldn't get much content from crawling their entire site.

_ViewStart.cshtml Info

In ASP.NET MVC3 “master pages” are handled in the _ViewStart.cshtml file.  As the name suggests the code in this file is executed before each view is rendered (see Scott Gu’s blog post (http://weblogs.asp.net/scottgu/archive/2010/10/22/asp-net-mvc-3-layouts.aspx) for more details).

Now that you understand the basic use of _ViewStart.cshtml file lets go over the scope applied to these files.  The _ViewStart.cshtml file will affect all views in the same directory and below the location of it.  Also you can have another _ViewStart.cshtml file under a sub-folder which will be executed after the top level _ViewStart.cshtml.  Using this feature you can in effect override the top level _ViewStart.cshtml with one closer to the view.

Now when the Index.cshtml View under the Home folder is rendered, it will first execute the /Views/_ViewStart.cshtml file and then it will render the Index.cshtml View.

However, when the Index.cshtml View under the DifferentMasterPage folder is rendered, it will first execute the /Views/_ViewStart.cshtml file, then it will execute the /Views/DifferentMasterPage/_ViewStart.cshtml file, and then it will render the Index.cshtml View.