Tag Archives: Selenium

Selenium Basic – What is it?

Selenium is a powerful and versatile open-source library that provides a comprehensive set of tools for automating web browsers. While it is commonly used for web testing, it has gained immense popularity in the field of web scraping as well. Here are some reasons why Selenium is considered a great choice for web scraping:

  1. Dynamic Content Handling: Many modern websites heavily rely on dynamic content that is loaded or generated through JavaScript. Traditional scraping techniques, such as parsing HTML with libraries like BeautifulSoup, may not be able to handle such dynamic content. Selenium, on the other hand, excels at handling dynamic content. It can interact with the web page in real-time, execute JavaScript code, and retrieve the fully rendered content, allowing you to scrape data from websites that heavily rely on JavaScript.
  2. Browser Automation: Selenium allows you to automate web browsers, replicating human-like interactions with web pages. This means you can navigate through multiple pages, click buttons, fill out forms, submit data, and perform various other actions programmatically. By mimicking human interactions, Selenium enables you to access data that might be hidden behind login screens, interact with AJAX-based functionalities, or traverse paginated content. This level of automation makes Selenium a powerful tool for scraping complex websites.
  3. Cross-Browser Support: Selenium supports multiple web browsers such as Chrome, Firefox, Safari, and Internet Explorer, among others. This cross-browser support allows you to choose the browser that best suits your scraping needs or replicate the behavior of your target audience. You can write your scraping code once and easily switch between different browsers, ensuring compatibility and flexibility.
  4. Robust Element Selection: Selenium provides a wide range of methods to locate elements on a web page, including by ID, class, XPath, CSS selectors, and more. This flexibility allows you to precisely target the elements you want to scrape. Additionally, Selenium offers advanced element interaction methods, enabling you to extract text, attribute values, perform clicks, and handle various user interactions effortlessly.
  5. Ecosystem and Community Support: Selenium has a vast and active community of developers, which means you can find plenty of resources, tutorials, and discussions to help you with your web scraping projects. The extensive ecosystem around Selenium includes frameworks, wrappers, and third-party tools that provide additional features and make web scraping more efficient.

While Selenium is a powerful tool for web scraping, it’s worth noting that it may be overkill for simple scraping tasks. If the website you’re targeting doesn’t heavily rely on dynamic content or JavaScript, or if you only need to extract static HTML data, using a lightweight library like BeautifulSoup or Requests may be more appropriate. However, when faced with complex scraping scenarios or dynamic websites, Selenium’s capabilities shine, making it a top choice for many web scraping projects.

Here is an example of Selenium Basic with VBA:

Sub AutomateWebTask()
    Dim driver As New SeleniumWrapper.WebDriver
    Dim element As SeleniumWrapper.WebElement
    
    ' Open a browser and navigate to a website
    driver.Start "chrome", "https://www.example.com"
    driver.Get "/"
    
    ' Find an input field by its ID and enter text
    Set element = driver.FindElementById("inputField")
    element.SendKeys "Hello, Selenium!"
    
    ' Find a button by its XPath and click it
    Set element = driver.FindElementByXPath("//button[@id='submitButton']")
    element.Click
    
    ' Wait for the page to load
    driver.Wait 5000
    
    ' Extract the text from a specific element
    Set element = driver.FindElementByClassName("resultText")
    MsgBox "Result: " & element.Text
    
    ' Close the browser
    driver.Quit
End Sub

Please make sure you have the Selenium Basic library installed and referenced in your VBA project before running this code. You can find the Selenium Basic library and installation instructions on its official GitHub page: https://github.com/florentbr/SeleniumBasic

Selenium – How to update ChromeDriver using VBA

The macro below updates the local ChromeDriver to the most recent version using VBA:

    Sub DownloadChromeDriver()
        Dim url As String
        Dim fileName As String
        Dim req As Object
        Dim stream As Object
        
        ' Set the URL and file name for the latest ChromeDriver
        url = "https://chromedriver.storage.googleapis.com/LATEST_RELEASE"
        fileName = "chromedriver.zip"
        
        ' Make an HTTP request to get the latest ChromeDriver version number
        Set req = CreateObject("MSXML2.XMLHTTP")
        req.Open "GET", url, False
        req.Send
        
        ' Get the latest ChromeDriver version number from the response
        Dim latestVersion As String
        latestVersion = req.responseText
        
        ' Construct the URL for the ChromeDriver binary based on the version number
        url = "https://chromedriver.storage.googleapis.com/" & latestVersion & "/chromedriver_win32.zip"
        
        ' Make an HTTP request to download the ChromeDriver binary
        req.Open "GET", url, False
        req.Send
        
        ' Save the downloaded binary to disk
        Set stream = CreateObject("ADODB.Stream")
        stream.Type = 1 ' Binary
        stream.Open
        stream.Write req.responseBody
        stream.SaveToFile fileName, 2 ' Overwrite existing file
        stream.Close
        
        ' Extract the downloaded binary to a known location on your computer
        Dim zipFilePath As String
        zipFilePath = ActiveWorkbook.Path & "\" & fileName
        Dim extractPath As String
        extractPath = Environ("userprofile") & "\AppData\Local\SeleniumBasic"
        
        Set objShell = CreateObject("Shell.Application")
        Set objSource = objShell.NameSpace(zipFilePath).Items()
        Set objTarget = objShell.NameSpace(extractPath)
        
        objTarget.CopyHere objSource, 16
        
        
        MsgBox "ChromeDriver version " & latestVersion & " installed successfully."
    End Sub

The video below breaks it down:

SeleniumBasic – Automation error

The problem

Imagine the following scenario: You develop the project for your client using Selenium VBA as a library, everything works perfectly, in all browsers, drivers, etc. All tests pass wonderfully.

Then you send the file along with all installation instructions. But when the client runs your file, the following error is displayed:

The solution

Install .NET Framework 3.5 on the computer. You can download it here:

https://docs.microsoft.com/en-us/dotnet/framework/install/dotnet-35-windows-10

After it’s installed, run the application again.

The explanation

Although it runs in the context of VBA, which is a native Windows application, SeleniumBasic is developed under .NET Framework, specifically version 3.5, which makes its execution dependent on it.

It’s no use having only .NET Framework 2.0 or 4.0. They are independent and need to be installed separately.