Q:

Crawling limited number of pages

I'am writing a spider for a forum. I am trying to to is to crawl recursively some pages of a section. So I set the rules below:
rules = (
    Rule (SgmlLinkExtractor(allow=('&order=desc&page=\d'),restrict_xpaths=("//td[@class='navpage']/a"))
    ,callback='parse_items', follow= True),)

Is it possible to set a page limit so the spider could crawl a limited number of pages? For example to crawl from page 1 to page 8. 

A:

The SgmlLinkExtractor's allow parameter takes a regex. So you can modify your allow regex to look for any digit (\d) to look for only range [1-8] by doing:
SgmlLinkExtractor(allow=('&order=desc&page=[1-8]'),restrict_xpaths=("//td[@class='navpage']/a"))
,callback='parse_items', follow= True),

