Q:

beautifulsoup extracts only first 10 elements

Im trying to extract information from Volkswagen page on kununu. For example "Pro" information. 
url = 'https://www.kununu.com/de/volkswagen/kommentare'
page = requests.get(url)

soup = bs(page.text, 'html.parser')
divs = soup.find_all(class_="col-xs-12 col-lg-12")

for h2 in soup.find_all('h2', class_='h3', text=['Pro']):
    print(h2.find_next_sibling('p').get_text())

But as an output I have only first 10 "Pro".
Looks like that it shows only top 10 commentaries by default, however all unvisible comments are under "col-xs-12 col-lg-12" class... Or maybe Im missing something
Can you help me extract all data, not just first 10?

A:

You can load these comments mimicking the XHR requests a browser is sending to load more comments dynamically. 
Working code (note: uses f-strings, so 3.6+; use .format() if using an earlier Python version):
from bs4 import BeautifulSoup
import requests

comments = []
with requests.Session() as session:
    session.headers = {
        'x-requested-with': 'XMLHttpRequest'
    }

    page = 1
    while True:
        print(f"Processing page {page}..")

        url = f'https://www.kununu.com/de/volkswagen/kommentare/{page}'
        response = session.get(url)

        soup = BeautifulSoup(response.text, 'html.parser')
        new_comments = [
            pro.find_next_sibling('p').get_text()
            for pro in soup.find_all('h2', text='Pro')
        ]
        if not new_comments:
            print(f"No more comments. Page: {page}")
            break

        comments += new_comments

        # just to see current progress so far
        print(comments)
        print(len(comments))

        page += 1

print(comments)

Note how we instantiated and used requests.Session() object which provides performance benefits when sending multiple requests to the same host.

