Q:

How can i scrape Aliexpress product data

I'm trying to scrape Aliexpress product data of a product page with Python and BeautifulSoup.
I run the next code making reference to an YML file (aliexpress.yml) with Aliexpress CSS selectors created with SelectorLib:
from selectorlib import Extractor
import requests
import json
import argparse

argparser = argparse.ArgumentParser()
argparser.add_argument('url', help='Amazon Product Details URL')
# Create an Extractor by reading from the YAML file
e = Extractor.from_yaml_file('./aliexpress.yml')

user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246'
headers = {'User-Agent': user_agent}
# Download the page using requests
args = argparser.parse_args()
r = requests.get(args.url, headers=headers)
# Pass the HTML of the page and create
data = e.extract(r.text)
# Print the data
print(json.dumps(data, indent=True))

I receive the next response:
{
 "name": null,
 "price": null,
 "image": null
}

I have other YML files for Amazon and other ecommerce and works properly, but with Aliexpress it returns null. Is strange because in SelectorLib i can view the preview data correctly, so the YML file is correct.
Could be Aliexpress blocking my scraping? How i can scrape Aliexpress data without null response?

A:

Page is loaded via JavaScript, So requests will not help you to render the JS. you can use selenium or whatever modules which render JS.
I noticed that your desired data is located within script tag, which you can actually loads it within JSON and then parse it.
The reason why you are able to see the elements within the inspect element of your driver because the browser is actually visualize it for you, But if you checked the page source, you will notice it's not located with it if you disabled the JS.

Here I've loaded the script within JSON, you can access it as JSON dict
import requests
import json
import re

target = ["title", "itemDetailUrl", "imagePath"]

def main(url):
    r = requests.get(url)
    match = re.search(r'data: ({.+})', r.text).group(1)
    data = json.loads(match)
    goal = [data['pageModule'][x] for x in target] + \
        [data['priceModule']['formatedActivityPrice']]
    print(goal)

main("https://es.aliexpress.com/item/32601027783.html")

Output:
['Nueva versión Xiaomi Mi cargador de coche Dual USB versión de carga rápida 37W Max con puntas de luz LED para iPhone Huawei Xiaomi para Samsung|charger dual|dual usb|mi car - AliExpress', 
'https://es.aliexpress.com/item/32601027783.html', 
'https://ae01.alicdn.com/kf/H6f0b93e98bb646d5a1183161b65707e3P/Nueva-versi-n-Xiaomi-Mi-cargador-de-coche-Dual-USB-versi-n-de-carga-r-pida.jpg', 
'€ 9,31 - 10,86']

