To use Selenium for web scraping, open the page in a browser, wait for the required content, extract fields from each record, and save them to CSV. This tutorial uses the JavaScript version of Quotes to Scrape to teach single-page extraction and verified pagination.
Key takeaways: Choose Selenium for browser rendering or interaction; an authorized API or HTTP client may be simpler when the response already contains the data. Verify fields on one page before adding pagination. In the run report,
last_pagemarks a normal pagination endpoint,page_limitmeans more pages remain, anderrormeans the run failed. A matching CSV count verifies the export, not complete site coverage.
On this practice page, JavaScript creates the visible quote cards, but the underlying data is also embedded in the HTML. The screenshots show a 10-record single-page export and a headless run with 100 records across 10 pages, last_page status, and exit code 0.
1. Set Up Python, Selenium, and Chrome
Use Python 3.10 or newer and an installed desktop Chrome browser. The official Selenium package currently lists Python 3.10+ support. Create a project folder, open a terminal there, and make an isolated environment.
On Windows PowerShell:
py -m venv .venv
.\.venv\Scripts\python.exe -m pip install selenium==4.51.0
On macOS or Linux:
python3 -m venv .venv
.venv/bin/python -m pip install selenium==4.51.0
These commands use the environment’s Python directly, so activation is optional. They pin Selenium 4.51.0. The macOS/Linux commands are platform equivalents; the run screenshots show Windows. If Windows does not recognize py but python --version works, use python -m venv .venv instead.
Modern Selenium includes Selenium Manager to resolve drivers when one has not been supplied. The webdriver.Chrome(options=options) call below invokes this process automatically when needed. First-time setup may require internet access. Restricted networks, unsupported environments, and a conflicting driver already on PATH can still need manual attention.
Save the code below as scrape_quotes.py in your project folder. Do not name it selenium.py, which can shadow the installed package.
2. Run Your First Selenium Scraper
Start with this single-page version. It uses Selenium for Python and the standard library; no additional parsing library is needed. The script waits for readable quote cards, extracts matching fields, and saves one record per CSV row.
import csv
from pathlib import Path
from selenium import webdriver
from selenium.common.exceptions import (
NoSuchElementException, StaleElementReferenceException, WebDriverException
)
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
def read_records(driver):
"""Wait until the currently rendered cards have their required fields."""
try:
cards = driver.find_elements(By.CSS_SELECTOR, ".quote")
if not cards:
return False
records = []
for card in cards:
quote = card.find_element(By.CSS_SELECTOR, ".text").text.strip()
author = card.find_element(By.CSS_SELECTOR, ".author").text.strip()
tags = [tag.text for tag in card.find_elements(By.CSS_SELECTOR, ".tag")]
if not quote or not author:
return False
records.append({"quote": quote, "author": author,
"tags": " | ".join(tags)})
return records
except (NoSuchElementException, StaleElementReferenceException):
return False
options = webdriver.ChromeOptions()
options.add_argument("--window-size=1280,900")
driver = webdriver.Chrome(options=options)
try:
driver.set_page_load_timeout(30)
driver.get("https://quotes.toscrape.com/js/")
records = WebDriverWait(driver, 15).until(read_records)
for record in records:
record["source_url"] = driver.current_url
output = Path("quotes.csv").resolve()
with output.open("w", newline="", encoding="utf-8-sig") as file:
writer = csv.DictWriter(
file, fieldnames=["quote", "author", "tags", "source_url"]
)
writer.writeheader()
writer.writerows(records)
print(f"Page: {driver.current_url}")
print(f"Saved {len(records)} records to {output}")
finally:
try:
driver.quit()
except WebDriverException:
pass
Before running, save the complete code as scrape_quotes.py, not scrape_quotes.py.txt, and open your terminal in that folder. Copy only the run command into the terminal. Page: and Saved are output messages, not commands to enter. Chrome closing automatically is expected; confirm that the terminal prints a saved-record count and open quotes.csv to check for data rows.
Run it on Windows:
.\.venv\Scripts\python.exe scrape_quotes.py
On macOS or Linux:
.venv/bin/python scrape_quotes.py
A visible Chrome window should open and close. The terminal should print the page URL, the number of saved records, and the full path to quotes.csv. The file is written in the current working directory; a later run replaces it, so save a copy if you need an earlier result.
This script only collects the current page. If startup or extraction fails, Python reports the exception instead of printing a successful save message. Check the terminal and the file timestamp: an older quotes.csv may remain from a previous run.
3. Understand and Modify the Scraper
Inspect a Record Before Writing a Selector
Open the Quotes to Scrape JavaScript practice page in Chrome. Right-click a quote and choose Inspect. One record sits inside .quote; its text is in .text, its author in .author, and its tags in .tag elements.
The script first finds all .quote containers with find_elements(). Inside each container, find_element() finds the corresponding text and author. This keeps related fields together. Collecting all authors and all quotes independently, then pairing by position, can silently misalign them when one field is missing.
Use CSS selectors for this example. An XPath is also valid, but copying a long absolute XPath from DevTools often ties the scraper to incidental page nesting. Choose attributes or relationships that identify the record you actually need.
Wait for the Required State, Not a Fixed Number of Seconds
driver.get() waits according to the browser’s page-load strategy; it does not guarantee that every later JavaScript update is finished. Selenium’s waiting documentation explains this distinction and warns against mixing implicit and explicit waits.
Here, WebDriverWait(driver, 15).until(read_records) repeatedly calls the extraction function. It succeeds when the page contains cards and all currently inspected cards have nonempty required fields. Missing or stale elements make the function retry until its timeout.
That condition fits this practice page’s rendering pattern. It is not a universal completeness check. A site that adds records in separate batches needs another signal, such as a known result count or a documented loading indicator. Even presence_of_all_elements_located can return once matches exist; its name does not mean that all eventual results have arrived.
Read Text and Save One Record per Row
Use .text for visible quote and author text. Use get_attribute("href") when you need a link destination, as the pagination code does. These are different values: the visible word “Next” is not the next page’s URL.
Tags are joined with | inside one CSV field. The multipage version below uses quote text and author as its deduplication key; source_url records where a retained row was first collected. On a product site, a stable product ID would usually be a better key than a title.
The standard-library CSV writer handles quoting for commas, quotation marks, and embedded newlines. newline="" follows Python's CSV guidance; utf-8-sig helps spreadsheet applications recognize Unicode. Count parsed records rather than physical text lines, because a quoted field can contain a newline.
4. Follow Pagination and Check How the Run Ended
Once the single-page CSV looks correct, save the complete version below as scrape_quotes_pages.py in the same folder. It includes extraction, pagination, deduplication, and a separate run-status report. Request up to 20 pages using the command for your operating system.
import argparse
import csv
import json
from pathlib import Path
from urllib.parse import urljoin
from selenium import webdriver
from selenium.common.exceptions import (
NoSuchElementException, StaleElementReferenceException, WebDriverException
)
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
START_URL = "https://quotes.toscrape.com/js/"
FIELDS = ["quote", "author", "tags", "source_url"]
def read_records(driver):
"""Wait until the currently rendered cards have their required fields."""
try:
cards = driver.find_elements(By.CSS_SELECTOR, ".quote")
if not cards:
return False
records = []
for card in cards:
quote = card.find_element(By.CSS_SELECTOR, ".text").text.strip()
author = card.find_element(By.CSS_SELECTOR, ".author").text.strip()
tags = [tag.text for tag in card.find_elements(By.CSS_SELECTOR, ".tag")]
if not quote or not author:
return False
records.append({"quote": quote, "author": author,
"tags": " | ".join(tags)})
return records
except (NoSuchElementException, StaleElementReferenceException):
return False
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--pages", type=int, default=1)
parser.add_argument("--headless", action="store_true")
args = parser.parse_args()
if args.pages < 1:
parser.error("--pages must be at least 1")
options = webdriver.ChromeOptions()
options.add_argument("--window-size=1280,900")
if args.headless:
options.add_argument("--headless=new")
driver = None
rows, seen, visited = [], set(), set()
raw_count = 0
status, detail = "error", "Browser did not start"
try:
driver = webdriver.Chrome(options=options)
driver.set_page_load_timeout(30)
wait = WebDriverWait(driver, 15)
driver.get(START_URL)
for page_number in range(1, args.pages + 1):
current_url = driver.current_url
if current_url in visited:
raise ValueError("Repeated page URL; pagination stopped")
batch = wait.until(read_records)
visited.add(current_url)
raw_count += len(batch)
for record in batch:
key = (record["quote"], record["author"])
if key not in seen:
seen.add(key)
rows.append({**record, "source_url": current_url})
print(f"Page {page_number}: {len(batch)} records | {current_url}")
links = driver.find_elements(By.CSS_SELECTOR, ".pager .next a")
if not links:
status, detail = "last_page", "No Next link on the processed page"
break
if page_number == args.pages:
status, detail = "page_limit", "More pages exist; requested limit reached"
break
expected_url = urljoin(current_url, links[0].get_attribute("href"))
old_card = driver.find_element(By.CSS_SELECTOR, ".quote")
links[0].click()
wait.until(EC.staleness_of(old_card))
wait.until(EC.url_to_be(expected_url))
except (WebDriverException, ValueError) as exc:
status = "error"
detail = f"{type(exc).__name__}: {str(exc).splitlines()[0]}"
if driver is not None:
try:
driver.save_screenshot("failure.png")
except WebDriverException:
pass
finally:
if driver is not None:
try:
driver.quit()
except WebDriverException:
pass
output = Path("quotes-pages.csv").resolve()
with output.open("w", newline="", encoding="utf-8-sig") as file:
writer = csv.DictWriter(file, fieldnames=FIELDS)
writer.writeheader()
writer.writerows(rows)
report = {
"status": status, "detail": detail,
"pages_processed": len(visited), "raw_records": raw_count,
"unique_records": len(rows), "duplicates_removed": raw_count - len(rows),
"csv": str(output)
}
Path("run-status.json").write_text(json.dumps(report, indent=2), encoding="utf-8")
print(json.dumps(report, indent=2))
if status == "error":
raise SystemExit(1)
if __name__ == "__main__":
main()
Windows PowerShell:
.\.venv\Scripts\python.exe scrape_quotes_pages.py --pages 20
macOS / Linux:
.venv/bin/python scrape_quotes_pages.py --pages 20
Use the command for your operating system; no environment activation is required. This version writes quotes-pages.csv and run-status.json, leaving the single-page quotes.csv intact. Re-running the multipage script replaces its own outputs. Twenty is a safety cap, not the site’s page count.
After extracting a page, the script checks for .pager .next a. When it finds a Next link, it saves the destination URL and an element from the old document. It clicks Next, waits for the old element to become stale, and checks the expected URL. Only then does the next iteration wait for new records.
Waiting for the same generic .quote selector immediately after clicking would be weaker: old-page elements could still satisfy it. This navigation check is for full-document pagination. A single-page application that updates content without replacing the document needs a different condition, such as a page indicator and changed record IDs.
How Do You Know a Selenium Scrape Finished Correctly?
Check navigation status and export validity separately. The stop reason tells you whether pagination reached its endpoint, stopped at your chosen cap, or failed. Comparing the parsed CSV count with unique_records checks the number of exported rows; inspect the fields as well. Read the report before using the CSV:
-
last_page: The processed page had no matching Next link. If the selector and page layout are correct, pagination reached its normal endpoint. This does not prove that every server-side record was available. -
page_limit: Another page existed, but the requested cap was reached. The run met its page limit; it did not reach the end of pagination. Increase the cap only if collecting more is appropriate. -
error: Startup, extraction, or navigation failed. The script exits with a nonzero status and attempts to export any previously collected records. A CSV file or matching row count does not make this a successful run.
The report also includes processed-page count, raw records, unique records, and duplicates removed. A page whose required fields never became readable is not counted as processed. Zero duplicates does not prove completeness; compare coverage with the intended task and inspect sample records.
On Windows, check the exit code immediately after the scraper finishes, then count parsed CSV records:
Write-Output "Exit code: $LASTEXITCODE"
Write-Output "CSV records: $(@(Import-Csv .\quotes-pages.csv).Count)"
Exit code 0 means no error was reported; check run-status.json as well, because page_limit also exits with 0. The CSV count should match unique_records. A matching count checks the export, not whether every field or every possible site record is correct.
5. Adapt Web Scraping with Selenium to Your Own Page
Change the URL and selectors together. A new URL with the old .quote selectors usually produces an empty result or a timeout, not an automatic scraper for the new site.
Replace these parts deliberately:
-
Starting URL: Set
START_URLto the permitted listing page. Confirm the browser reaches that page, including redirects. -
Record container: Replace
.quotewith the repeated item or table row. Compare the matches with the visible records. -
Fields and deduplication: Update extraction,
FIELDS, and the key together. Mark truly optional fields as optional rather than waiting forever for them. -
Pagination: Replace
.pager .next a, then choose a signal that establishes the next batch is ready.
For a Load More button, keep existing rows and wait for additional record IDs or another application-specific completion signal. Do not wait for the entire old document to become stale if the page never navigates.
For infinite scroll, collect the visible batch, scroll, and wait for new IDs. Set a maximum duration or iteration count. If a wait expires without an explicit end marker, record an uncertain stop rather than declaring the dataset complete. Some virtualized lists remove earlier records from the DOM, so extracting only once at the very end can lose data.
If the content is inside an iframe, switch to the relevant frame before locating it. Shadow DOM and canvas-based content need their own extraction approach; changing a timeout cannot turn them into ordinary document elements.
6. Fix Empty Results, Duplicate Pages, and Driver Errors
No records or repeated timeouts: Confirm the current URL and inspect the live DOM. Check the selector, frame context, and whether the expected content actually appeared. Increasing the timeout will not repair a selector that matches nothing.
StaleElementReferenceException: The referenced node was replaced or its document changed. Locate the element again after the update. Selenium’s common-error guide explains why stored references do not relocate themselves.
The same page appears repeatedly: Check the actual next URL or page indicator. An enabled-looking control may not have completed navigation. Keep the repeated-URL guard, and investigate rather than silently deduplicating the same page forever.
Chrome will not start: Separate driver-download problems from browser-startup problems. Review the exception, installed browser, conflicting driver, and operating-system restrictions. A browser proxy does not automatically fix Selenium Manager’s download network. Use an approved network or explicit driver setup when the environment requires it; do not disable browser security as a blanket fix.
The CSV is missing records: For the single-page version, check the terminal exception and quotes.csv timestamp. For the multipage version, read run-status.json and inspect quotes-pages.csv. A saved file can contain partial results. If writing fails with PermissionError, close that CSV in Excel or another spreadsheet application, confirm the folder is writable, and run again. A file-write failure can prevent the new status report from being created, so check timestamps rather than trusting older outputs.
When access is denied or a CAPTCHA appears, stop automatic retries and review permission, site terms, and available APIs. A blocked page is not a normal empty listing.
7. Run the Working Scraper in Headless Mode
After the multipage version works in a visible browser, run scrape_quotes_pages.py with --headless. This flag is supported by the multipage script, not the smaller single-page example:
Windows PowerShell:
.\.venv\Scripts\python.exe scrape_quotes_pages.py --pages 20 --headless
macOS / Linux:
.venv/bin/python scrape_quotes_pages.py --pages 20 --headless
The script uses the same 1280×900 window setting in both modes to reduce layout differences. Headless Chrome still runs a browser; it simply does not show a window. Chrome’s headless documentation describes the supported mode.
The multipage script attempts to save failure.png after an extraction or navigation error if a browser session exists. Startup failures may produce no image. Check its timestamp before using it: an older screenshot can remain from a previous run. Keep diagnostic screenshots private if they contain credentials or personal data.
Headless mode does not guarantee higher speed, lower cost, or successful access. Compare the records and run status with the visible version before scheduling it.
8. Choose a Proxy for Your Selenium Scraping Project
For projects such as price monitoring and regional content checks, a proxy lets the browser access pages through a selected network exit. Start with two questions: which location does the task require, and how long should the browser keep the same IP?
| Scraping requirement | What to check |
|---|---|
| Compare prices or content across locations | Available locations and the browser’s actual exit location |
| Collect independent pages in separate batches | Whether the exit should change between batches and how requests will be paced |
| Follow pagination or complete a session-dependent workflow | Whether the exit remains stable throughout the task |
| Run recurring checks through the same exit | Fixed-IP rental duration and maintenance requirements |
For projects that need flexible exit selection or separate collection batches, explore Socks5.IO’s rotating residential proxies. For recurring tasks that need the same IP over a longer period, compare static residential proxies. Check the selected product’s available locations, session duration, and supported authentication method before connecting it.
To connect your chosen proxy, follow the Selenium proxy server setup guide and use the Selenium proxy integration documentation for provider-specific settings. First verify the exit IP and location, then run the single-page scraper. Test pagination with the same session settings before increasing the page limit.
Chrome proxy authentication requires a supported setup; inserting username:password into --proxy-server is not enough.
Ready to connect your scraper? Choose the Socks5.IO product that fits your location and session needs, configure the endpoint, and validate a single-page export before scaling up. Keep collection within the target site’s permissions and request limits.
Valerie Quinn
Kevin Lin
