Detect request is a crawler/bot - asp.net
Sometimes you want to avoid bot from crawling the page with sensitive information like phone number and email address, so you need to have a method to detect either the request is a visitors or bot.
This code snippet shows how you can determine the request is a crawler or not.
Code Behind
/// <summary>
/// Returns whether browsing device is search engine crawler (spider, bot).
/// </summary>
public bool IsCrawler()
{
if (HttpContext.Current != null)
{
HttpRequest currentRequest = HttpContext.Current.Request;
return (currentRequest.Browser != null) && currentRequest.Browser.Crawler;
}
return false;
}
You can put this method under a static class and call it from the Global.asax file.