log IT

Just for reference a small example of scrapping https://www.alcopa-auction.fr/ for new cars on specific brand.

It can be improved a lot but just as a reference of the code.

from bs4 import BeautifulSoup
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
import time

# Set up a headless browser
driver = webdriver.Chrome()  


# Load the website with JavaScript
def alcopascrap(brand):

    driver.get(f"https://www.alcopa-auction.fr/ro/cautare?brand={brand}")

    # Wait for JavaScript to execute (adjust wait time if needed)
    driver.implicitly_wait(180)  


    WebDriverWait(driver, 1).until(
    EC.element_to_be_clickable((By.ID, "CybotCookiebotDialogBodyButtonAccept"))
).click()

# Get initial scroll height
    last_height = driver.execute_script("return document.body.scrollHeight")

    while True:
    # Scroll down
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    # Wait for loading
        time.sleep(2)  # Adjust wait time as needed

    # Calculate new scroll height and compare with previous
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            break
        last_height = new_height

# Get the HTML source with JavaScript modifications
    html_source = driver.page_source


# Parse the HTML with BeautifulSoup 
    soup = BeautifulSoup(html_source, 'html.parser')

#Create a list to store the data

    list_of_cars = []
# Now you can use BeautifulSoup to extract the data you need
# ...
    ratings = soup.find_all('div',{'data-brand':brand})
    for rating in ratings:
        data = {}
        data['title'] = rating.find('li',{'class':'mb-2 list-group-item border-0 p-0'}).get_text()
        data['price'] = rating['data-prices']
        data['details'] = [ element.get_text().strip() for element in rating.find_all('li',{'class':'list-group-item border-0 p-0'})]
        data['link'] = rating.find('a', {'class':'no-decoration'}).get('href')
        list_of_cars.append(data)
   
# Print the data
    for car in list_of_cars:
        print(car, end='\n')

# Close browser

    driver.quit()
    

alcopascrap('nissan')

As for the code description, here is the one from Github Copilot, you judge if it’s good or not.

This Python script uses Selenium and BeautifulSoup to scrape car data from the Alcopa Auction website. Here’s a step-by-step breakdown:

1. Import necessary libraries: BeautifulSoup for parsing HTML, Selenium for browser automation, and time for controlling the script’s timing.
2. Set up a headless browser using Selenium’s webdriver. This script uses Chrome as the browser.
3. Define a function alcopascrap(brand), which takes a car brand as an argument and scrapes data for that brand from the website.
4. The function first navigates to the Alcopa Auction website’s search page for the specified brand.
5. It waits for the page to load, then clicks the accept button on the cookie consent dialog.
6. It then enters a loop where it continually scrolls to the bottom of the page, waits for more content to load, and repeats until no more new content is loaded. This is done to ensure all the dynamic content loaded by JavaScript is captured.
7. After all content is loaded, it gets the page’s HTML source.
8. It then parses the HTML using BeautifulSoup.
9. It creates an empty list list_of_cars to store the car data.
10. It finds all div elements with a data-brand attribute matching the specified brand. For each of these elements, it extracts the car title, price, details, and link, and stores them in a dictionary. Each dictionary is then appended to list_of_cars.
11. It prints out the data for each car.
12. Finally, it closes the browser.
13. The function is then called with ‘nissan’ as the argument, so it scrapes data for Nissan cars.

This script is a good example of how to use Selenium and BeautifulSoup together to scrape data from a website that uses JavaScript to load content.