comparison

XML vs. HTML Web Scraping: Side-by-Side Python Examples

Compare XML and HTML web scraping with runnable Python examples, namespace handling, validation and CSV export, plus optional Socks5.IO proxy configuration.

Gideon ReedGideon Reed20 min read

Quick answer: XML and HTML web scraping share a collection step but require format-aware parsing. HTML extraction usually follows page elements and attributes with CSS selectors or XPath; XML data extraction follows a document’s element names, hierarchy and namespaces. Validate the resulting fields before exporting either format. Socks5.IO can supply a suitable network route for an authorized source, but a proxy cannot fix selectors, namespace errors or missing data.

This tutorial extracts the same two fictional products from HTML and XML, producing a common CSV schema. The complete Python program includes local samples, strict XML handling, validation and optional authenticated SOCKS5 routing. No companion code package is required. Start with local parsing, then adapt one approved source before making a network request.

What is the difference between XML and HTML?

format-contract

HTML defines web document structure and semantics for user-facing pages. XML defines a general markup syntax used by application-specific vocabularies, feeds and document formats. The practical difference is not that HTML cannot carry data: HTML text, attributes, tables and embedded data can all contain useful information.

Aspect HTML XML
Vocabulary Standard HTML elements, with defined extension mechanisms Application-defined names governed by the source vocabulary
Typical sources Product pages, directories, articles and tables Feeds, sitemaps, exports and integration documents
Parsing behavior HTML parsers can repair some malformed markup; trees vary by parser XML must be well-formed; strict parsing rejects malformed input
Matching names HTML element-name matching follows HTML rules Element names are case-sensitive; namespaces matter
Extraction CSS selectors or supported XPath over the parsed tree Namespace-aware paths, XPath or streaming events
Dynamic content Required values may appear only after JavaScript runs Fetching an XML document does not execute client JavaScript
Validation A matched element does not prove correct field meaning Well-formed XML does not prove schema or business validity

The HTML parsing standard and W3C XML specification describe different parsing requirements. Browser error recovery is not a reason to accept incorrect extraction, and XML syntax alone does not establish trustworthy prices or complete records. XHTML served and parsed as XML is also different from ordinary text/html; choose by the actual source contract, not a filename alone.

Which Python tools should you use?

Tool Useful role Boundary
Requests Retrieve approved HTTP responses through a configured client Does not execute JavaScript or parse a document
Beautiful Soup Navigate HTML and use CSS selectors Behavior depends on the explicitly selected parser
lxml Strict XML parsing, namespaces, XPath and optional HTML handling Configure security options; do not assume every document is trusted
Scrapy Scheduling, item pipelines and crawl management for larger permitted jobs Does not remove source restrictions or replace field validation

This example uses Beautiful Soup with Python’s html.parser for HTML, and a configured lxml XMLParser for XML. Beautiful Soup’s documentation explains parser choices and tree navigation. Neither selected library executes a page’s JavaScript. If a needed field exists only in a permitted rendered page or documented feed, use an appropriate separate adapter rather than assuming a CSS change will make it appear.

Difference between XML and HTML web scraping with examples

Both local examples describe SKU-DEMO-01, a fictional coffee product at USD 24.90, and SKU-DEMO-02, a fictional tea product at USD 18.50. The second product deliberately lacks stock status. The extractor preserves it as unknown and flags it instead of inventing availability.

Internal field HTML source XML source Validation
sku article.product[data-sku] c:product @sku Required and unique within this example response
name h2 a text c:name text Required nonempty text
amount .price text c:price text Explicit positive decimal, without currency symbols
currency .price @data-currency c:price @currency Three uppercase letters in this teaching contract
url h2 a @href c:path text Resolve against the known base; require HTTPS without credentials
stock .stock text c:stock text Missing or unrecognized values become unknown and enter review

Here, c is a parser-chosen prefix for urn:demo:catalog. It does not need to be the prefix written in the XML document. A default namespace still qualifies the element names; a bare product search would not match them.

source-fields

1. Install dependencies

Use Python 3.10 or later in a dedicated environment:

powershellpowershell
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install "requests[socks]>=2.32,<3" "beautifulsoup4>=4.13,<5" "lxml>=5,<7"

Every Windows command below uses the environment interpreter explicitly; activation is not required. On macOS/Linux, create and install the environment with:

bashbash
python3 -m venv .venv
.venv/bin/python -m pip install "requests[socks]>=2.32,<3" "beautifulsoup4>=4.13,<5" "lxml>=5,<7"

The version ranges are dependency boundaries, not a promise of identical behavior in every future release. Record your installed versions and rerun the examples after upgrades.

Core difference: selectors versus namespace-aware paths

This preview uses the HTML and XML fixtures defined in the complete program below; it is not a second standalone script.

pythonpython
html_tree = BeautifulSoup(HTML, "html.parser")
html_card = html_tree.select_one("article.product")
html_name = html_card.select_one("h2 a").get_text(strip=True)
html_price = html_card.select_one(".price").get_text(strip=True)
xml_parser = etree.XMLParser(resolve_entities=False, load_dtd=False,
                             no_network=True, recover=False, huge_tree=False)
xml_tree = etree.fromstring(XML, parser=xml_parser)
xml_product = xml_tree.find("c:product", NS)
xml_name = xml_product.findtext("c:name", namespaces=NS)
xml_price = xml_product.findtext("c:price", namespaces=NS)

The preview omits validation. The complete program below is organized as fixtures → extraction adapters → normalization → network checks → CLI/export. Copy that complete block, not the preview, to run the tutorial.

core-adapters

2. Copy the complete extractor

Save the complete block as markup_scraper.py. Default operation is local: --format html or --format xml selects a fictional embedded source. Network collection requires both --url and --approved.

pythonpython
import argparse
import csv
import ipaddress
import json
import os
import re
import sys
from datetime import datetime, timezone
from decimal import Decimal
from pathlib import Path
from urllib.parse import quote, urljoin, urlsplit, urlunsplit
from uuid import uuid4

import requests
from bs4 import BeautifulSoup
from lxml import etree

MAX_BYTES = 2_000_000
NS = {"c": "urn:demo:catalog"}
BASE = "https://catalog.example.invalid/"
HTML = b'''<!doctype html><html><body><main id="catalog">
<article class="product" data-sku="SKU-DEMO-01">
<h2><a href="/p/coffee">Demo Coffee</a></h2>
<span class="price" data-currency="USD">24.90</span><p class="stock">in_stock</p></article>
<article class="product" data-sku="SKU-DEMO-02">
<h2><a href="/p/tea">Demo Tea</a></h2>
<span class="price" data-currency="USD">18.50</span></article>
</main></body></html>'''
XML = b'''<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="urn:demo:catalog"><product sku="SKU-DEMO-01">
<name>Demo Coffee</name><price currency="USD">24.90</price>
<path>/p/coffee</path><stock>in_stock</stock></product>
<product sku="SKU-DEMO-02"><name>Demo Tea</name>
<price currency="USD">18.50</price><path>/p/tea</path></product></catalog>'''


def only(nodes):
    if len(nodes) != 1:
        raise ValueError("Expected exactly one required field or container")
    return nodes[0]


def parse_html(data):
    root = only(BeautifulSoup(data, "html.parser").select("main#catalog"))
    cards = root.select("article.product")
    records = []
    for card in cards:
        link = only(card.select("h2 a"))
        price = only(card.select(".price"))
        stock = card.select(".stock")
        if len(stock) > 1:
            raise ValueError("Ambiguous stock field")
        records.append({"sku": card.get("data-sku", ""),
                        "name": link.get_text(" ", strip=True),
                        "amount": price.get_text(" ", strip=True),
                        "currency": price.get("data-currency", ""),
                        "path": link.get("href", ""),
                        "stock": stock[0].get_text(" ", strip=True) if stock else "unknown"})
    return records


def parse_xml(data):
    parser = etree.XMLParser(resolve_entities=False, load_dtd=False, no_network=True,
                             recover=False, huge_tree=False)
    root = etree.fromstring(data, parser=parser)
    if root.getroottree().docinfo.doctype:
        raise ValueError("DTD-bearing XML is outside this tutorial contract")
    if root.tag != "{urn:demo:catalog}catalog":
        raise ValueError("Unexpected XML root or namespace")
    def text(node):
        return " ".join(part.strip() for part in node.itertext() if part.strip())
    records = []
    for product in root.findall("c:product", NS):
        name = only(product.findall("c:name", NS))
        price = only(product.findall("c:price", NS))
        path = only(product.findall("c:path", NS))
        stock = product.findall("c:stock", NS)
        if len(stock) > 1:
            raise ValueError("Ambiguous stock field")
        records.append({"sku": product.get("sku", ""), "name": text(name),
                        "amount": text(price), "currency": price.get("currency", ""),
                        "path": text(path), "stock": text(stock[0]) if stock else "unknown"})
    return records


def normalize(records, base_url, kind, fmt, run_id, stamp):
    if not records:
        raise ValueError("No product records; inspect the source instead of claiming sold out")
    rows, seen = [], set()
    for raw in records:
        row = {key: str(value).strip() for key, value in raw.items()}
        if not row["sku"] or not row["name"] or not row["path"] or row["sku"] in seen:
            raise ValueError("Missing required identity/text or duplicate SKU")
        seen.add(row["sku"])
        if not re.fullmatch(r"\d+(?:\.\d+)?", row["amount"]) or Decimal(row["amount"]) <= 0:
            raise ValueError("Require an explicit positive decimal amount")
        if not re.fullmatch(r"[A-Z]{3}", row["currency"]):
            raise ValueError("Require an explicit three-letter uppercase currency label")
        target = urljoin(base_url, row.pop("path"))
        parsed = urlsplit(target)
        if parsed.scheme != "https" or not parsed.hostname or parsed.username or parsed.password:
            raise ValueError("Expected HTTPS product URL without embedded credentials")
        row["url"] = target
        row["amount"] = str(Decimal(row["amount"]))
        row["review_reason"] = ""
        if row["stock"] not in {"in_stock", "out_of_stock"}:
            row["stock"], row["review_reason"] = "unknown", "stock_unknown"
        row.update(source_kind=kind, source_format=fmt, run_id=run_id, retrieved_at_utc=stamp)
        rows.append(row)
    return rows


def public_source(url):
    p = urlsplit(url)
    if p.scheme != "https" or not p.hostname or p.username or p.password:
        raise ValueError("Use an approved public HTTPS source without credentials")
    host = p.hostname.lower().rstrip(".")
    if host == "localhost" or host.endswith(".localhost"):
        raise ValueError("Use local fixtures, not network mode, for localhost")
    try:
        address = ipaddress.ip_address(host)
    except ValueError:
        address = None
    if address is not None and not address.is_global:
        raise ValueError("Private IP literals are outside network mode")
    return p


def make_session(use_proxy):
    session = requests.Session()
    session.trust_env = False
    session.headers["User-Agent"] = "AuthorizedMarkupTutorial/1.0"
    if use_proxy:
        host = os.environ["SOCKS5IO_HOST"]
        port = int(os.environ["SOCKS5IO_PORT"])
        if not re.fullmatch(r"[A-Za-z0-9.-]+", host) or not 1 <= port <= 65535:
            session.close()
            raise ValueError("Check issued proxy host and port")
        user = quote(os.environ["SOCKS5IO_USER"], safe="")
        password = quote(os.environ["SOCKS5IO_PASSWORD"], safe="")
        proxy = f"socks5h://{user}:{password}@{host}:{port}"
        session.proxies.update(http=proxy, https=proxy)
    return session


def get_body(session, url, accepted):
    with session.get(url, timeout=(5, 20), allow_redirects=False, stream=True) as response:
        if response.status_code != 200:
            raise ValueError(f"HTTP {response.status_code}: stop and review access/quota")
        mime = response.headers.get("Content-Type", "").split(";", 1)[0].strip().lower()
        if mime not in accepted:
            raise ValueError("Unexpected response type; check for a changed source or challenge")
        data = bytearray()
        for chunk in response.iter_content(65536):
            data.extend(chunk)
            if len(data) > MAX_BYTES:
                raise ValueError("Response exceeds tutorial limit")
        return bytes(data)


def check_exit(session):
    body = get_body(session, "https://api.ipify.org?format=json", {"application/json"})
    address = ipaddress.ip_address(json.loads(body)["ip"])
    if not address.is_global:
        raise ValueError("Checker returned a non-public address")
    return {"observed_ip": str(address), "checked_at_utc": datetime.now(timezone.utc).isoformat()}


def safe_diagnostic(error):
    if isinstance(error, etree.XMLSyntaxError):
        return "XML_SYNTAX: inspect XML well-formedness privately."
    if isinstance(error, requests.RequestException):
        return "NETWORK: inspect connection, proxy authentication and TLS privately."
    if isinstance(error, OSError):
        return "OUTPUT_IO: check destination and write permissions."
    if isinstance(error, KeyError):
        return "CONFIG_FIELD: check required configuration or checker fields."
    message = str(error)
    status = re.fullmatch(r"HTTP (\d{3}): stop and review access/quota", message)
    if status:
        return "HTTP_STATUS_" + status[1] + ": review authorization, quota or source status."
    codes = {
        "Unexpected response type; check for a changed source or challenge": "RESPONSE_MIME",
        "Response exceeds tutorial limit": "INPUT_SIZE",
        "Input exceeds tutorial limit": "INPUT_SIZE",
        "Unexpected XML root or namespace": "XML_NAMESPACE",
        "DTD-bearing XML is outside this tutorial contract": "XML_DTD",
        "Expected exactly one required field or container": "FIELD_CARDINALITY",
        "Ambiguous stock field": "FIELD_CARDINALITY",
        "No product records; inspect the source instead of claiming sold out": "NO_RECORDS",
        "Missing required identity/text or duplicate SKU": "FIELD_IDENTITY",
        "Require an explicit positive decimal amount": "FIELD_AMOUNT",
        "Require an explicit three-letter uppercase currency label": "FIELD_CURRENCY",
        "Expected HTTPS product URL without embedded credentials": "FIELD_URL",
    }
    return codes.get(message, "INPUT_OR_CONFIG") + ": inspect the corresponding input or configuration privately."


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--format", choices=["html", "xml"], required=True)
    parser.add_argument("--url")
    parser.add_argument("--approved", action="store_true")
    parser.add_argument("--proxy", action="store_true")
    parser.add_argument("--out", type=Path, required=True)
    args = parser.parse_args()
    log_path = args.out.with_suffix(".run.json")
    if args.out.exists() or log_path.exists():
        parser.error("Use new output filenames")
    run_id = uuid4().hex
    log = {"run_id": run_id, "format": args.format, "proxy_requested": args.proxy}
    if args.url:
        p = public_source(args.url)
        if not args.approved:
            parser.error("Review source permission and request budget before --approved")
        accepted = {"text/html"} if args.format == "html" else {"application/xml", "text/xml"}
        with make_session(args.proxy) as session:
            log["before"] = check_exit(session)
            data = get_body(session, args.url, accepted)
            stamp = datetime.now(timezone.utc).isoformat()
            log["after"] = check_exit(session)
        log["same_observed_ip"] = log["before"]["observed_ip"] == log["after"]["observed_ip"]
        base, kind = args.url, "user_approved_source"
        log["source"] = urlunsplit((p.scheme, p.netloc, p.path, "", ""))
    else:
        if args.proxy:
            parser.error("A local fixture is not a proxy test")
        data = HTML if args.format == "html" else XML
        base, kind = BASE, "synthetic_fixture"
        stamp = datetime.now(timezone.utc).isoformat()
        log["source"] = "local:embedded-" + args.format
    if len(data) > MAX_BYTES:
        raise ValueError("Input exceeds tutorial limit")
    records = parse_html(data) if args.format == "html" else parse_xml(data)
    rows = normalize(records, base, kind, args.format, run_id, stamp)
    log.update(source_kind=kind, records_exported=len(rows), retrieved_at_utc=stamp)
    with log_path.open("x", encoding="utf-8") as handle:
        json.dump(log, handle, indent=2)
    with args.out.open("x", encoding="utf-8", newline="") as handle:
        writer = csv.DictWriter(handle, fieldnames=list(rows[0]))
        writer.writeheader()
        writer.writerows(rows)
    print(f"Saved {len(rows)} {args.format.upper()} records; inspect review_reason.")


if __name__ == "__main__":
    try:
        main()
    except (ValueError, etree.XMLSyntaxError, KeyError, OSError, requests.RequestException) as error:
        print("Stopped:", safe_diagnostic(error), file=sys.stderr)
        sys.exit(1)

3. Run HTML scraping and XML data extraction

powershellpowershell
.\.venv\Scripts\python.exe markup_scraper.py --format html --out products-html.csv
.\.venv\Scripts\python.exe markup_scraper.py --format xml --out products-xml.csv

macOS/Linux:

bashbash
.venv/bin/python markup_scraper.py --format html --out products-html.csv
.venv/bin/python markup_scraper.py --format xml --out products-xml.csv

Both local runs export the same business fields:

SKU
Name
Amount
Currency
Stock
Review reason
SKU-DEMO-01
Demo Coffee
24.90
USD
in_stock
Empty
SKU-DEMO-02
Demo Tea
18.50
USD
unknown
stock_unknown

Product links resolve to the fictional catalog’s HTTPS paths. source_format differs between the two files, and each run has its own run_id and retrieval time. Within a single response, all rows share one timestamp. An empty review_reason means only these teaching checks passed; it is not proof of current inventory, tax completeness or cross-source product equivalence.

The manifest is written beside each CSV as .run.json. Existing filenames are not overwritten. A write failure can leave one output without the other; this example does not implement an atomic two-file transaction. If importing untrusted text into a spreadsheet, use appropriate text-import controls rather than allowing cells to execute as formulas.

local-csv-results
local-manifest

4. Exercise the differences before adding network access

  • HTML selector failure: change the fixture’s class=“product” to class=“offer” without changing the parser. Extraction stops rather than declaring that all products are unavailable.
  • XML namespace failure: change urn:demo:catalog to another namespace. The XML parser rejects the unexpected source vocabulary.
  • XML syntax failure: remove a required closing tag. Strict parsing stops; it does not silently repair the XML.
  • Business-field failure: change a price to 0 or remove the currency attribute. Normalization rejects it before either output is created.
  • Missing optional value: the second product’s stock remains unknown in both formats and stays flagged.

Use new output filenames and restore the fixtures before continuing. Duplicate SKUs are rejected because this example requires one record per SKU per response; a source with variants, locations or multiple offers needs a different documented identity contract.

How do XML namespaces, XPath and parser safety change the workflow?

For web scraping XML, namespace qualification is a common reason an apparently valid path returns no records. The expression root.findall("c:product", NS) expands the parser’s c prefix to the namespace URI. XML using another prefix for the same URI still represents the same qualified name. Renaming the URI changes the vocabulary, not merely the spelling of a prefix.

ElementTree-style findall supports a limited path language; lxml also offers a fuller XPath interface when needed. Attributes such as sku or currency in these samples are unqualified, while element names use the default namespace. Do not apply namespace assumptions from one document to all RSS, Atom or sitemap formats. Each needs its actual root, hierarchy and field mapping.

Untrusted XML needs security-aware handling. Python’s XML security guidance describes resource-exhaustion and related risks, while lxml’s parser documentation documents its configuration options. This collector disables entity resolution, external DTD loading, network access and recovery; it rejects DTD-bearing documents and leaves huge_tree disabled. It does not invoke XInclude or XSLT.

These controls and the small response cap are a conservative tutorial baseline, not a complete security sandbox. Keep parser libraries patched and enforce process-level resource/time limits for production inputs. Well-formedness is also not XML Schema validation: the code checks an expected root and required fields, not a supplied XSD or DTD.

For very large approved XML inputs, consider iterparse or another streaming design with explicit element clearing and bounded output. The example intentionally builds a small in-memory tree; it does not demonstrate constant-memory processing or benchmark one parser as universally faster.

How do you adapt an approved live source?

Inspect one authorized response and record its format, encoding, root/container, record identity, required fields and missing-value policy. Replace the relevant parser against that source, then test saved responses before changing the URL. The fixture’s article.product selector and urn:demo:catalog namespace are not current selectors for a commercial website or a generic sitemap parser.

Use a complete, deliberately non-working template only after that adaptation:

powershellpowershell
.\.venv\Scripts\python.exe markup_scraper.py --format html --url "https://catalog.example.invalid/approved-products" --approved --out approved-html.csv
.\.venv\Scripts\python.exe markup_scraper.py --format xml --url "https://catalog.example.invalid/approved-export.xml" --approved --out approved-xml.csv

macOS/Linux uses the same flags with the explicit interpreter:

bashbash
.venv/bin/python markup_scraper.py --format html --url "https://catalog.example.invalid/approved-products" --approved --out approved-html.csv
.venv/bin/python markup_scraper.py --format xml --url "https://catalog.example.invalid/approved-export.xml" --approved --out approved-xml.csv

The program accepts only its declared HTML or XML MIME types in network mode and does not automatically follow redirects. application/xhtml+xml, RSS-specific MIME types or another vocabulary require an explicit adapter decision, not a silent content-type fallback. An XML error document can be well-formed, so the expected root and fields remain necessary checks.

Network mode is for public approved HTTPS sources. It excludes localhost/private IP literals but is not a general SSRF or DNS-rebinding defense. It requests one source document; it does not crawl the product links, execute JavaScript, implement pagination or discover alternate private APIs. Relative product paths use that source URL as their base; HTML base elements are not implemented and need source-specific handling.

Where does Socks5.IO fit in XML and HTML web scraping?

The network request and document parser are separate components. A proxy changes the route through which bytes arrive; once those bytes are available, HTML selectors and XML namespaces behave the same way. For a small authorized job without a routing requirement, direct access may be sufficient.

When your job requires supported residential routing or a selected exit market, evaluate Socks5.IO rotating residential proxies. Confirm the usable locations, authentication, session behavior, traffic expiry and current purchase conditions. An advertised pool size does not establish source access or availability in one city, and a proxy does not increase a source’s permitted request quota.

Configure the issued endpoint without changing the parser

Follow the Socks5.IO account/password setup instructions: open Rotating Residential IP → Proxy Setup, choose account authentication and the proxy user, confirm location/session requirements, select the supported protocol, then generate the endpoint. Copy its host, port and proxy credentials. Create a proxy user through Sub Account Management if needed; the generated authentication username is not your dashboard login. Use the actual current interface and selected product’s supported options.

Set the issued fields in a dedicated PowerShell process:

proxy-field-mapping
powershellpowershell
$env:SOCKS5IO_HOST = Read-Host 'Issued proxy host'
$env:SOCKS5IO_PORT = Read-Host 'Issued proxy port'
$env:SOCKS5IO_USER = Read-Host 'Issued proxy username'
$markupProxySecret = Read-Host 'Issued proxy password' -AsSecureString
$env:SOCKS5IO_PASSWORD = [System.Net.NetworkCredential]::new('', $markupProxySecret).Password

Then run the already-adapted, approved source with the proxy flag:

powershellpowershell
.\.venv\Scripts\python.exe markup_scraper.py --format xml --url "https://catalog.example.invalid/approved-export.xml" --approved --proxy --out proxy-xml.csv

On macOS/Linux, enter credentials without putting the password in shell history, then run the environment interpreter:

bashbash
read -r -p 'Issued proxy host: ' SOCKS5IO_HOST
read -r -p 'Issued proxy port: ' SOCKS5IO_PORT
read -r -p 'Issued proxy username: ' SOCKS5IO_USER
read -r -s -p 'Issued proxy password: ' SOCKS5IO_PASSWORD; printf '\n'
export SOCKS5IO_HOST SOCKS5IO_PORT SOCKS5IO_USER SOCKS5IO_PASSWORD
.venv/bin/python markup_scraper.py --format xml --url "https://catalog.example.invalid/approved-export.xml" --approved --proxy --out proxy-xml.csv
unset SOCKS5IO_PASSWORD

The complete program uses percent-encoded credentials and socks5h for proxy-side destination DNS, as described in Requests’ SOCKS documentation. The client still needs to resolve and connect to the gateway. TLS verification remains enabled, ambient proxy/trust-bundle variables are ignored, and there is no automatic direct fallback. Organizations needing a custom CA must configure an approved bundle explicitly.

SOCKS5 is not inherently faster, more anonymous or more suitable for XML than HTTP(S). Choose by endpoint and client support; the SOCKS5 versus HTTP proxy guide explains protocol trade-offs. A browser integration must be verified separately rather than assuming Requests’ authentication syntax works unchanged in Chromium.

Verify the route from the same client

Network mode requests the documented ipify public JSON checker, then the approved source, then the checker again within one Session. Review that third-party service’s terms/privacy implications before use. A checker failure stops collection. Adapting to another authorized checker requires its actual endpoint and field contract; do not invent a Socks5.IO detection API.

Inspect before/after observations and same_observed_ip privately in the run log. Compare with an allowed direct baseline and verify geography separately where required. One Session does not pin a rotating exit, and matching bracketed checks cannot prove the exact exit of an intervening per-request rotating request. If continuity is required, configure a supported fixed/sticky policy and enforce that requirement separately.

The CSV product URLs retain the source-supplied product link. The run log removes query/fragment from the source reference and does not save the full requested search URL. If reproducibility needs that approved full URL, maintain a separate restricted record keyed by run_id. Never put reservation tokens, account links, personal data or proxy credentials in public exports. Protect process environment variables and remove the password when finished:

powershellpowershell
Remove-Item Env:SOCKS5IO_PASSWORD
Remove-Variable markupProxySecret

How should you handle failures and scale the collection?

The program emits allowlisted diagnostic categories rather than raw exception text, URLs or credentials. For example, HTTP_STATUS_403 and HTTP_STATUS_429 distinguish access and quota responses; XML_NAMESPACE distinguishes a source vocabulary mismatch. Unknown exceptions receive a generic category. Inspect detailed source material privately, not in published logs.

safe-diagnostics
Observation Correct next action
HTTP_STATUS_401 / HTTP_STATUS_403 Stop and review authorization/access conditions
HTTP_STATUS_429 Stop and review quota, Retry-After and permitted retry guidance
RESPONSE_MIME Inspect consent, challenge, error and redirect behavior before parsing
INPUT_SIZE Reduce document size within the permitted source contract
XML_NAMESPACE / XML_SYNTAX / XML_DTD Check the source vocabulary, syntax and XML security contract
FIELD_AMOUNT / FIELD_CURRENCY / FIELD_IDENTITY / FIELD_CARDINALITY Inspect required values, duplicates and adapter matching
NETWORK / OUTPUT_IO / CONFIG_FIELD Check the connection, write permissions or required configuration privately
No matching elements Review parser, namespace and source contract; do not infer zero inventory
Malformed XML or prohibited DTD Reject the input under this contract and review the source safely
Missing currency or invalid price Reject the record set rather than inventing values
Proxy connection or authentication failure Check issued fields and route independently of parsing
Memory or time growth Reduce approved document size or implement resource-bounded streaming

Do not rotate IPs automatically to work around a refusal or add unbounded threads to “solve” 429. Use per-source budgets across all workers, cache only where allowed, and bound concurrency, retries and total elapsed time. The connect/read timeouts in this program are not an overall job deadline.

Measure valid source responses separately from accepted records. One response may contain many products, so records divided by requests is not a request-success rate. Track structural failures, field-review rates, duplicates, observation age, elapsed time and attributable billing. Our scraping proxy selection guide provides wider route/cost considerations; parsing quality still needs its own tests.

Business example: reconcile an HTML catalog with an XML export

verification

Consider a retailer authorized to compare its own HTML catalog with its supplier’s approved XML export. The team samples twenty mapped SKUs from each source once a day: forty planned source–SKU checks, not forty guaranteed requests or matching records. Currency, price meaning and product/variant identifiers must be reconciled before differences are acted on.

The team maintains separate source adapters and a shared schema. Missing stock status enters review instead of triggering an out-of-stock alert. A changed XML namespace is an adapter failure, not evidence that the supplier removed every item. A successful HTML request containing a consent page is also a source failure, not a valid empty catalog.

Only when the approved collection requires location-specific routing does the team add and verify a suitable proxy configuration. It measures accepted-record coverage and maintenance cost before increasing the workload. This is a planning example, not a customer result, measured saving or source-access benchmark.

Conclusion

Reliable XML and HTML web scraping starts with the source contract: choose the parser, map fields, handle namespaces and validate the output before exporting. Run the paired examples, inspect the shared schema and review flags, then adapt only to an authorized source whose structure and data meaning you can verify.

When that workflow needs a specific network route, add Socks5.IO using issued credentials and verify it separately from parsing. Expand only after source access, record quality, resource limits and attributable cost meet the job’s acceptance criteria.

FAQs

The main difference is how the retrieved document is interpreted and validated. HTML extraction follows page structure using a selected HTML parser, while XML extraction follows an application vocabulary with strict syntax and namespace-aware names. Both formats can contain useful data and both need business checks. A valid XML tree does not prove complete records, and an HTML selector match does not prove the correct product. Network routing is a separate concern.

Latest articles

Global proxy network illustration

Plug into the global proxy network — every connection faster and more stable

Start for Free