Q:

Test web by crawling all pages

I develop web site in ASP.NET MVC 3 and since Views are compiled after first visit it sometimes happens to me, that I release web site with some Vies with errors (like typos).
I'd like to create something like unit test which will try to visit all pages, to avoid any errors in views.
It should be quite easy to create another program which will crawl web on the localhost but I'd like to integrate it to unit test loop. Is this possible?
Should I write write simple crawler myself or should I use something existing?
Also It would be nice to get list of pages and their number of connections to see which pages are accessible by how many links. The crawler should crawl only my web, not external links. Also It can help to find pages which are not accesible from starting point (index page).
My web has like 100 pages so performance is not the problem :)
Thanks in advance!

A:

Try "Search Engine Optimization Toolkit" very good tool:
Use Site Analysis module
Search Engine Optimization Toolkit
Site Analysis Features
Fully featured crawler engine
Configurable number of concurrent requests to allow users to crawl their Web site without incurring additional processing. This can be configured from 1 to 16 concurrent requests.
Support for Robots.txt, allowing you to customize the locations where the crawler should analyze and which locations should be ignored.
Support for Sitemap files allowing you to specify additional locations to be analyzed.
Support for overriding ‘noindex’ and ‘nofollow’ metatags to allow you to analyze pages to help improve customer experience even when search engines will not process them.
Configurable limits for analysis, maximum number of URLs to download, and maximum number of kilobytes to download per URL.
Configurable options for including content from only your directories or the entire site and sub domains.
View detailed summary of Web site analysis results through a rich dashboard
Feature rich Query Builder interface that allows you to build custom reports
Quick access to common tasks
Display of detailed information for each URL
View detailed route analysis showing unique routes to better understand the way search engines reach your content
"Linkbot" is another tool you can look that can be helpful:
Linkbot is an award winning suite of Web site management utilities that helps webmasters track down and repair problems on their sites. Linkbot contains all the tools needed to maintain an error-free site in one integrated package: high performance multitasking engine for rapid site analysis, find and repair broken links, locate slow and outdated pages, find orphaned files that are no longer being used, map out the structure and organization of the site, check password protected and secure areas, comprehensive HTML reporting, automatically generate a what's new page, create gallery of all the images on your site and much more!

