Q:

What is the right xpath/css to use to get each paragraph text in a single string?

First I create a HtmlResponse And read it using scrapy:
from scrapy.http import HtmlResponse
from scrapy.selector import Selector

body = """
<div class="a">
  <p>
      text1<br> text2
  </p>
</div>
<div class="a">
    <p>
         text3
    </p>
</div>
 """
response = HtmlResponse(url='http://example.com/', body=body)
sel = Selector(response)

Now, I would like to extract text from this html But I get a list with 2 elements. This is what I have so far tried:
 sel.xpath('//div[@class="a"]/p/text()').extract()
 # [u'\n      text1', u' text2\n  ', u'\n         text3\n    ']

As you note I get 3 text elements for 2 paragraphs? How can I do to get only 2 text elements?
  [u'text1 text2',u'text3']

Note that I prefer not to use BeautifulSoup since performance is a requirement here.

A:

With CSS selectors (including Scrapy's ::text extension):
>>> from scrapy.http import HtmlResponse
>>> from scrapy.selector import Selector
>>> 
>>> body = """
... <div class="a">
...   <p>
...       text1<br> text2
...   </p>
... </div>
... <div class="a">
...     <p>
...          text3
...     </p>
... </div>
...  """
>>> response = HtmlResponse(url='http://example.com/', body=body)
>>> sel = Selector(response)
>>> [u''.join(paragraph.css('::text').extract()).strip() for paragraph in sel.css('div.a > p')]
[u'text1 text2', u'text3']
>>> 

