Quick summary
Summarize this blog with AI
Beautiful Soup parses HTML; it does not fetch a page or execute browser JavaScript. A reliable small scraper separates retrieval, parsing, validation, and output. This tutorial extracts fictional job cards into a pandas DataFrame and tests the result against HTML we own.
Tested with Python 3.12.14, Beautiful Soup 4.15.0, Requests 2.34.2, and pandas 2.2.3/3.0.6. All demonstrated requests use a temporary local server. Before adapting retrieval to a real site, confirm permission, access rules, rate limits, and whether a supported API is available; this fixture establishes no permission for a third-party site.
Start with an explicit page contract
Each article.job has a unique data-id and a nonempty h2.title. A missing location is allowed and represented as None. A page with no cards is valid only when it explicitly contains p.no-results; otherwise absence is treated as possible layout drift.
from bs4 import BeautifulSoup
import pandas as pd
HTML = """<main>
<article class="job" data-id="j1"><h2 class="title">Data <b>Analyst</b></h2>
<span class="location">Remote</span></article>
<article class="job" data-id="j2"><h2 class="title">Data Engineer</h2></article>
</main>"""
def parse_jobs(html):
soup = BeautifulSoup(html, "html.parser")
cards = soup.select("article.job")
if not cards and not soup.select_one("p.no-results"):
raise ValueError("neither job cards nor explicit empty marker found")
if len(cards) > 100:
raise ValueError("too many job cards")
rows, seen = [], set()
for card in cards:
job_id = card.get("data-id", "").strip()
title_node = card.select_one("h2.title")
title = title_node.get_text(" ", strip=True) if title_node else ""
if not job_id or not title or job_id in seen:
raise ValueError("missing identity/title or duplicate job ID")
location_node = card.select_one("span.location")
location = location_node.get_text(" ", strip=True) if location_node else None
seen.add(job_id)
rows.append({"job_id": job_id, "title": title, "location": location or None})
return rows
jobs = pd.DataFrame(parse_jobs(HTML), columns=["job_id", "title", "location"], dtype=object)
print(jobs.to_dict("records"))
print(parse_jobs('<p class="no-results">No jobs</p>'))
Expected output:
[{'job_id': 'j1', 'title': 'Data Analyst', 'location': 'Remote'}, {'job_id': 'j2', 'title': 'Data Engineer', 'location': None}]
[]
get_text with a separator keeps nested words from becoming DataAnalyst. Optional fields remain visibly missing rather than being invented. The explicit object dtype retains None consistently across the two tested pandas versions. The DataFrame column list preserves its schema even when there are zero rows. Beautiful Soup may repair malformed HTML; that does not prove the repaired structure satisfies your data contract.
Retrieve a bounded UTF-8 HTML response
import requests
from urllib.parse import urlsplit
def fetch_html(url, max_bytes=65536):
parsed = urlsplit(url)
local = parsed.scheme == "http" and parsed.hostname == "127.0.0.1"
if (parsed.scheme != "https" and not local) or not parsed.hostname:
raise ValueError("use HTTPS or the local fixture server")
if parsed.username is not None or parsed.password is not None:
raise ValueError("credentials must not be embedded in the URL")
if type(max_bytes) is not int or max_bytes < 1:
raise ValueError("max_bytes must be a positive integer")
with requests.Session() as session:
session.trust_env = False
with session.get(url, timeout=(5, 15), stream=True,
allow_redirects=False,
headers={"Accept-Encoding": "identity"}) as response:
response.raise_for_status()
if response.status_code != 200:
raise ValueError("expected HTTP 200 without redirects")
media = response.headers.get("Content-Type", "").split(";", 1)[0].strip().lower()
if media != "text/html":
raise ValueError("expected HTML")
if response.headers.get("Content-Encoding", "identity").lower() != "identity":
raise ValueError("compressed content is not accepted")
content = bytearray()
for chunk in response.iter_content(chunk_size=4096):
if len(content) + len(chunk) > max_bytes:
raise ValueError("HTML exceeds byte limit")
content.extend(chunk)
return content.decode("utf-8")
The helper accepts caller-owned HTTPS URLs, checks HTTP status and content type, refuses redirects and compressed content, and caps actual received bytes. UTF-8 is an explicit contract; a different declared encoding needs a deliberate implementation. The timeout pair bounds connection setup and socket read inactivity, not the entire download's elapsed time. There are no automatic retries or environment-provided credentials/proxies.
HTTPS validates the connection under the usual Requests certificate rules. It does not make arbitrary user-submitted URLs safe to fetch from a server. This helper is not an SSRF-protected public fetch service; keep target URLs application-owned and apply deployment-level destination controls where needed.
Bound pagination without chasing arbitrary links
def collect_jobs(urls):
if not 1 <= len(urls) <= 3 or len(set(urls)) != len(urls):
raise ValueError("supply one to three distinct page URLs")
origins = {(urlsplit(url).scheme, urlsplit(url).netloc) for url in urls}
if len(origins) != 1:
raise ValueError("pages must share one configured origin")
rows, seen = [], set()
for url in urls:
for row in parse_jobs(fetch_html(url)):
if row["job_id"] in seen:
raise ValueError("duplicate job ID across pages")
seen.add(row["job_id"])
rows.append(row)
return pd.DataFrame(rows, columns=["job_id", "title", "location"], dtype=object)
The page list is explicit and limited to three pages, each with at most 100 cards and 64 KiB of HTML. Requests run sequentially; a real permitted source may require an additional delay or lower limits. A repeated job across pages is rejected, so pagination drift cannot silently become a duplicated record. This function does not discover the total number of pages; provide the agreed finite list.
Run the entire pipeline against a local fixture
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
class JobPage(BaseHTTPRequestHandler):
def do_GET(self):
if self.path != "/jobs":
self.send_error(404)
return
body = HTML.encode("utf-8")
self.send_response(200)
self.send_header("Content-Type", "text/html; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass
server = ThreadingHTTPServer(("127.0.0.1", 0), JobPage)
thread = Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
url = f"http://127.0.0.1:{server.server_port}/jobs"
print(collect_jobs([url])["job_id"].tolist())
try:
fetch_html(url + "/missing")
except requests.HTTPError:
print("HTTP error propagated")
finally:
server.shutdown()
server.server_close()
thread.join()
Expected output:
['j1', 'j2']
HTTP error propagated
The server binds only to loopback and uses an assigned temporary port. Cleanup runs even if parsing fails. The empty log_message is only suppressing local access-log noise; it is not an exception handler. In an operational scraper, record request outcomes and parser failures without logging credentials or private page contents.
Fail visibly when a required field disappears
for broken in ['<article class="job" data-id="j1"></article>', '<h1>Login</h1>']:
try:
parse_jobs(broken)
except ValueError:
print("page contract rejected")
Expected output:
page contract rejected
page contract rejected
Do not treat a login page or changed selector as “zero jobs.” Store a failed run as failed, preserve the last valid output, and investigate the source. For a JavaScript-only page, a static HTTP response may lack the desired data; prefer a permitted API or a separately tested browser workflow rather than adding outdated Selenium selectors to this parser.
References: Beautiful Soup parsing and selectors, Requests timeouts, and streamed response handling.
