Selenium quickstart¶

Selenium is a bunch of cool open source tools that let you automate web browsers. It's great because it works with all the major programming languages, and Python lovers find it very handy 🤓

What's cool about Selenium is that it uses something called the WebDriver protocol to control different web browsers, like Chrome, Firefox, or Safari. It doesn't matter if the browser is on your computer or some other computer across the network Selenium can handle it.

Selenium has been around for about 20 years now. It started off as a tool for testing how websites look and work in different browsers, but nowadays, people use it for all sorts of stuff. Need to take a screenshot of a webpage or automate some boring, repetitive tasks? Selenium is your go-to. It's also a big help for web scraping, which is like grabbing all the useful info from websites.

With Selenium, you can do things like:

Clicking buttons on a website
Filling in forms with data
Scrolling up and down webpages
Snapping screenshots of what’s on the screen
Running your own JavaScript code

One of the best things about Selenium is that it deals with websites just like a regular browser does. This is super useful, especially with those websites that are heavy on JavaScript and look more like an app than a webpage. If you tried to scrape data from these sites with basic tools, you'd end up with a bunch of JavaScript code and not much else. But with Selenium, you can really dig in and get the data you need.

Open a simple chrome browser¶

While Selenium supports a number of browser engines, we will use Chrome for the following example, so please make sure you have the following packages installed:

To install the Selenium package, as always, I recommend that you create a virtual environment (for example using virtualenv) and then:

pip install selenium certifi

In [62]:

Copied!





import os
#set your environment variable for SSL certificate
certi_path = "/Users/mac/.pyenv/versions/3.7.0/lib/python3.7/site-packages/certifi/cacert.pem"
os.environ['REQUESTS_CA_BUNDLE'] = certi_path
from selenium.webdriver.common.by import By
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options

#init proxy
from fp.fp import FreeProxy
import requests
from bs4 import BeautifulSoup
proxy = FreeProxy(country_id=['FR']).get()

# Initialize Chrome options
options = Options()
#options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3")
#disable the browser
#options.add_argument("--headless")
options.add_argument("--disable-extensions")
options.add_argument("--ignore-certificate-errors")
options.add_argument(f'--proxy-server={proxy}')

# Initialize Chrome WebDriver with the specified options
service = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=service, options=options)


# waits up to 10 seconds before throwing a NoSuchElementException
driver.implicitly_wait(10)
import os
#set your environment variable for SSL certificate
certi_path = "/Users/mac/.pyenv/versions/3.7.0/lib/python3.7/site-packages/certifi/cacert.pem"
os.environ['REQUESTS_CA_BUNDLE'] = certi_path
from selenium.webdriver.common.by import By
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options

#init proxy
from fp.fp import FreeProxy
import requests
from bs4 import BeautifulSoup
proxy = FreeProxy(country_id=['FR']).get()

# Initialize Chrome options
options = Options()
#options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3")
#disable the browser
#options.add_argument("--headless")
options.add_argument("--disable-extensions")
options.add_argument("--ignore-certificate-errors")
options.add_argument(f'--proxy-server={proxy}')

# Initialize Chrome WebDriver with the specified options
service = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=service, options=options)


# waits up to 10 seconds before throwing a NoSuchElementException
driver.implicitly_wait(10)

In [56]:

Copied!

proxy
#driver.get('https://google.com')
proxy
#driver.get('https://google.com')

Out[56]:

'http://50.171.32.227:80'

Search `find_element` method¶

WebDriver provides two main methods for finding elements : find_element for one single element and find_elements for a list of all found elements.

Both methods support eight different search types, indicated with the By class, more on the doc

Type	Description	DOM Sample	Example
`By.ID`	Searches for elements based on their HTML ID	`<div id="myID">`	`find_element(By.ID, "myID")`
`By.NAME`	Searches for elements based on their name attribute	`<input name="myNAME">`	`find_element(By.NAME, "myNAME")`
`By.XPATH`	Searches for elements based on an XPath expression	`<span>My <a>Link</a></span>`	`find_element(By.XPATH, "//span/a")`
`By.LINK_TEXT`	Searches for anchor elements based on a match of their text content	`<a>My Link</a>`	`find_element(By.LINK_TEXT, "My Link")`
`By.PARTIAL_LINK_TEXT`	Searches for anchor elements based on a sub-string match of their text content	`<a>My Link</a>`	`find_element(By.PARTIAL_LINK_TEXT, "Link")`
`By.TAG_NAME`	Searches for elements based on their tag name	`<h1>`	`find_element(By.TAG_NAME, "h1")`
`By.CLASS_NAME`	Searches for elements based on their HTML classes	`<div class="myCLASS">`	`find_element(By.CLASSNAME, "myCLASS")`
`By.CSS_SELECTOR`	Searches for elements based on CSS selector	`<span>My <a>Link</a></span>`	`find_element(By.CSS_SELECTOR, "span > a")`

One of the most interesting is method is locating by XPath or CSS. For modern website who change the randomly change the name of the classes and id for <div>

You can check this documentation about XPath syntax here referenced by the official documentation 😎

But let's start with a simple example here and scrap an amazon product page :

In [53]:

Copied!





from selenium.webdriver.common.by import By

driver.get("https://www.amazon.com/Dyson-V10-Allergy-Cordless-Cleaner/dp/B095LD5SWQ/")

title = driver.find_element(By.XPATH, value='//span[@id="productTitle"]')
current_price = driver.find_element(By.XPATH, value='//div[@id="corePriceDisplay_desktop_feature_div"]//span[@class="aok-offscreen"]')
image = driver.find_element(By.XPATH, value='//div[@id="imgTagWrapperId"]/img')

product_data = {
    'title': title.text,
    'current_price': current_price.get_attribute('innerHTML'),
    'image_url': image.get_attribute('src')
}

print(product_data)
driver.quit()
from selenium.webdriver.common.by import By

driver.get("https://www.amazon.com/Dyson-V10-Allergy-Cordless-Cleaner/dp/B095LD5SWQ/")

title = driver.find_element(By.XPATH, value='//span[@id="productTitle"]')
current_price = driver.find_element(By.XPATH, value='//div[@id="corePriceDisplay_desktop_feature_div"]//span[@class="aok-offscreen"]')
image = driver.find_element(By.XPATH, value='//div[@id="imgTagWrapperId"]/img')

product_data = {
    'title': title.text,
    'current_price': current_price.get_attribute('innerHTML'),
    'image_url': image.get_attribute('src')
}

print(product_data)
driver.quit()

{'title': 'Dyson Cyclone v10 Vacuum, Blue', 'current_price': '   $398.99 with 27 percent savings    ', 'image_url': 'https://m.media-amazon.com/images/W/MEDIAX_792452-T1/images/I/51UXR6Dn7cL._AC_SX679_.jpg'}

Screenshot¶

The beauty of browser approaches, like Selenium, is that we do not only get the data and the DOM tree, but that - being a browser - it also properly and fully renders the whole page. This, of course, also allows for screenshots and Selenium comes fully prepared here.

driver.save_screenshot('screenshot.png')

Playing with JavaScript¶

So, you've got Selenium and you're ready to do some cool stuff, right? Like taking screenshots, for instance. But hey, what if you want to capture something further down the page? No sweat, just a little bit of JavaScript and you're golden.

javaScript = "window.scrollBy(0, 1000);"
driver.execute_script(javaScript)

Or maybe you're feeling a bit artsy and want to put a red border around all the anchor tags. Easy peasy if you can do it with your browser inspector in js you can do it inside a selenium driver 🍰

Here is the js version if you want to try in your browser :

document.querySelectorAll('a').forEach(e => e.style.border='red 2px solid')

Then the selenium version :

javaScript = "document.querySelectorAll('a').forEach(e => e.style.border='red 2px solid')"
driver.execute_script(javaScript)

The execute_script() function in Selenium doesn't just run your code it also can also send back variables. Like grabbing the title of the page and popping it right into a variable in your script.

title = driver.execute_script('return document.title')

Blocking images and Javascript¶

While it’s awesome to have a fully-rendered browser experience at our fingertips, sometimes we don't need all the bells and whistles. Like, why load images or run JavaScript if you're not using them, right? Thankfully, Selenium and WebDriver let us tweak these settings.

Remember the Options class? It takes a preferences object where you can enable or disable features as you like. Here’s how to stop loading images and running JavaScript:

from selenium import webdriver

chrome_options = webdriver.ChromeOptions()

# Block images and JavaScript
chrome_prefs = {
    "profile.default_content_setting_values": {
        "images": 2,
        "javascript": 2,
    }
}
chrome_options.experimental_options["prefs"] = chrome_prefs

driver = webdriver.Chrome(
    executable_path="YOUR-CHROME-EXECUTABLE-PATH",
    chrome_options=chrome_options,
)
driver.get(URL)

It is not always recommended because the most part of modern website use js in order to render elements 😢

One of the most coolest feature of selenium I think is it's capacity to fill forms, let's try to code a simple test case and try to login in a dummy website like https://practicetestautomation.com/practice-test-login/ 🤖

In order to do so we have to :

Get the website
Find the username input field and enter the specified username
Find the password input field and enter the specified password
Find the form's submit button and click it
Wait a little for our new page to load well

In [ ]:

Copied!





from selenium import webdriver
from selenium.webdriver.common.by import By
import time 

# Your login credentials
password = "Password123"
username = "student"

driver.get("https://practicetestautomation.com/practice-test-login/")

#find the username input field and enter the specified username
username_input = driver.find_element(by=By.ID, value="username")
username_input.send_keys(username)

#find the password input field and enter the specified password
password_input = driver.find_element(by=By.ID, value="password")
password_input.send_keys(password)

#find the form's submit button and click it
submit_button = driver.find_element(by=By.ID, value="submit")
submit_button.click()

#wait for the new page to load
time.sleep(3)
from selenium import webdriver
from selenium.webdriver.common.by import By
import time 

# Your login credentials
password = "Password123"
username = "student"

driver.get("https://practicetestautomation.com/practice-test-login/")

#find the username input field and enter the specified username
username_input = driver.find_element(by=By.ID, value="username")
username_input.send_keys(username)

#find the password input field and enter the specified password
password_input = driver.find_element(by=By.ID, value="password")
password_input.send_keys(password)

#find the form's submit button and click it
submit_button = driver.find_element(by=By.ID, value="submit")
submit_button.click()

#wait for the new page to load
time.sleep(3)

Tests¶

Let's add a login test verification with assert according to to https://practicetestautomation.com/practice-test-login/ specifications in order to verify if we are indeed loged in here :

In [65]:

Copied!





from selenium import webdriver
from selenium.webdriver.common.by import By
import time 

# Your login credentials
password = "Password123"
username = "student"

driver.get("https://practicetestautomation.com/practice-test-login/")

#find the username input field and enter the specified username
username_input = driver.find_element(by=By.ID, value="username")
username_input.send_keys(username)

#find the password input field and enter the specified password
password_input = driver.find_element(by=By.ID, value="password")
password_input.send_keys(password)

#find the form's submit button and click it
submit_button = driver.find_element(by=By.ID, value="submit")
submit_button.click()

#wait for the new page to load
time.sleep(3)

#verify new page URL
print(driver.current_url)
assert "practicetestautomation.com/logged-in-successfully/" in driver.current_url, "URL check failed"

# Verify page contains expected text
page_text = driver.find_element(by=By.TAG_NAME, value="body").text
assert "Congratulations" in page_text or "Logged In Successfully" in page_text, "Text check failed"

# Verify Log out button is displayed
logout_button = driver.find_element(by=By.XPATH, value="//a[contains(text(), 'Log out')]")
assert logout_button.is_displayed(), "Log out button not displayed"

print("Test case passed according to https://practicetestautomation.com/practice-test-login/ specifications")
from selenium import webdriver
from selenium.webdriver.common.by import By
import time 

# Your login credentials
password = "Password123"
username = "student"

driver.get("https://practicetestautomation.com/practice-test-login/")

#find the username input field and enter the specified username
username_input = driver.find_element(by=By.ID, value="username")
username_input.send_keys(username)

#find the password input field and enter the specified password
password_input = driver.find_element(by=By.ID, value="password")
password_input.send_keys(password)

#find the form's submit button and click it
submit_button = driver.find_element(by=By.ID, value="submit")
submit_button.click()

#wait for the new page to load
time.sleep(3)

#verify new page URL
print(driver.current_url)
assert "practicetestautomation.com/logged-in-successfully/" in driver.current_url, "URL check failed"

# Verify page contains expected text
page_text = driver.find_element(by=By.TAG_NAME, value="body").text
assert "Congratulations" in page_text or "Logged In Successfully" in page_text, "Text check failed"

# Verify Log out button is displayed
logout_button = driver.find_element(by=By.XPATH, value="//a[contains(text(), 'Log out')]")
assert logout_button.is_displayed(), "Log out button not displayed"

print("Test case passed according to https://practicetestautomation.com/practice-test-login/ specifications")

https://practicetestautomation.com/logged-in-successfully/
Test case passed according to https://practicetestautomation.com/practice-test-login/ specifications