Q:

Python BeautifulSoup Returning Empty List

I'm trying to create a Python script to pull prices of Yugioh Card prices from tcgplayer.com using BeautifulSoup. When you search for a card on this website, it returns a page of search results with several prices from different sellers. My goal is to pull all of these prices. In the below example, I'm opening the search result for a card called "A" Cell Breeding Device:
import urllib2
from bs4 import BeautifulSoup
html = urllib2.open('http://shop.tcgplayer.com/productcatalog/product/show?newSearch=false&ProductType=All&IsProductNameExact=false&ProductName=%22A%22%20Cell%20Breeding%20Device')
soup = BeautifulSoup(html, 'lxml')
soup.find_all('span', {'class': 'scActualPrice largetext pricegreen'})

A few days ago, running the soup.find_all line correctly gave me the information I needed. However, running this now gives me an empty array []. I've searched pretty extensively about BeautifulSoup returning an empty array, but I'm not sure if any of them apply to me since it was working just fine a few days ago. Can someone help point me in the right direction? Thank you in advance!

A:

You should use selenium to scrap using real browser:
from selenium import webdriver

driver = webdriver.Chrome('/path/to/chromedriver')
driver.get('http://shop.tcgplayer.com/productcatalog/product/show?newSearch=false&ProductType=All&IsProductNameExact=false&ProductName=%22A%22%20Cell%20Breeding%20Device')
prices = driver.find_elements_by_css_selector('.scActualPrice')
for element in prices:
    print(element.text)
driver.quit()

