Quick summary
Summarize this blog with AI
A file download should fail visibly if the server returns an error, the connection stops early, or the response exceeds the size you agreed to accept. It should also leave the previous destination intact until a complete replacement is ready. This guide implements that behavior with Requests and a temporary file in the destination directory.
Tested with Python 3.12.14 and Requests 2.34.2 against a local fixture server. No remote files were downloaded for validation. The function is for a caller-selected trusted URL and destination; it is not an arbitrary-URL fetch service for untrusted users.
Stream into a temporary file, then replace the destination
The destination filename is supplied by the caller, not copied from an untrusted URL or Content-Disposition header. Redirects are rejected so the caller must review a changed destination. HTTPS certificate verification remains enabled. Plain HTTP is permitted only for the loopback test fixture.
from pathlib import Path
from tempfile import NamedTemporaryFile
from urllib.parse import urlsplit
import hashlib
import os
import requests
def download_file(url, destination, max_bytes=4_000_000, expected_sha256=None,
timeout=(5, 15)):
parts = urlsplit(url)
local_http = parts.scheme == 'http' and parts.hostname == '127.0.0.1'
if parts.scheme != 'https' and not local_http:
raise ValueError('use HTTPS except for a loopback test fixture')
if parts.username or parts.password or not parts.hostname:
raise ValueError('URL must have a host and no embedded credentials')
if type(max_bytes) is not int or max_bytes <= 0:
raise ValueError('max_bytes must be a positive integer')
if expected_sha256 is not None:
if not isinstance(expected_sha256, str) or len(expected_sha256) != 64:
raise ValueError('expected SHA-256 must be 64 hexadecimal characters')
try:
if len(bytes.fromhex(expected_sha256)) != 32:
raise ValueError('digest must represent exactly 32 bytes')
except ValueError as exc:
raise ValueError('expected SHA-256 must be hexadecimal') from exc
target = Path(destination)
temporary = None
try:
with requests.Session() as session:
session.trust_env = False
with session.get(url, timeout=timeout, stream=True, allow_redirects=False,
headers={'Accept-Encoding': 'identity'}) as response:
response.raise_for_status()
if response.status_code != 200:
raise ValueError('expected a complete HTTP 200 response')
if response.headers.get('Content-Encoding', 'identity').lower() != 'identity':
raise ValueError('compressed responses are not accepted by this helper')
declared = response.headers.get('Content-Length')
declared = int(declared) if declared is not None else None
if declared is not None and not 0 <= declared <= max_bytes:
raise ValueError('declared response size is outside the limit')
received = 0
digest = hashlib.sha256()
with NamedTemporaryFile(dir=target.parent, prefix='.download-', delete=False) as file:
temporary = Path(file.name)
for chunk in response.iter_content(chunk_size=16384):
received += len(chunk)
if received > max_bytes:
raise ValueError('response exceeds the byte limit')
file.write(chunk)
digest.update(chunk)
if declared is not None and received != declared:
raise ValueError('response length does not match Content-Length')
if expected_sha256 is not None and digest.hexdigest() != expected_sha256.lower():
raise ValueError('SHA-256 does not match the trusted expected digest')
file.flush()
os.fsync(file.fileno())
os.replace(temporary, target)
temporary = None
return received
finally:
if temporary is not None:
temporary.unlink(missing_ok=True)
The same-directory temporary file keeps the final replacement on one filesystem. A successful os.replace replaces an existing destination atomically; another reader sees either the old name's file or the completed replacement. This is not a complete power-loss durability protocol, and it does not coordinate multiple writers. The parent directory must already exist and be controlled by the caller.
A trusted expected digest can detect content that differs from the expected file. A digest supplied by the same untrusted response is not independent authentication. This helper deliberately rejects compressed content encodings so its length and digest checks refer to the bytes it saves.
Run a complete local example
The fixture serves fixed synthetic bytes only on your own computer and stops when the with block exits. It does not expose an existing directory.
from contextlib import contextmanager
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from tempfile import TemporaryDirectory
@contextmanager
def local_files():
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
responses = {
'/ok': (200, b'column\nvalue\n'),
'/missing': (404, b'not found'),
'/large': (200, b'x' * 64),
}
status, body = responses.get(self.path, (404, b'not found'))
self.send_response(status)
self.send_header('Content-Length', str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *args):
pass # Keep this deterministic local fixture quiet.
server = ThreadingHTTPServer(('127.0.0.1', 0), Handler)
thread = Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
yield f'http://127.0.0.1:{server.server_port}'
finally:
server.shutdown()
server.server_close()
thread.join()
with local_files() as base_url, TemporaryDirectory() as directory:
target = Path(directory) / 'report.csv'
print(download_file(base_url + '/ok', target))
print(target.read_text().splitlines())
for path in ('/missing', '/large'):
try:
download_file(base_url + path, target, max_bytes=16)
except (requests.HTTPError, ValueError) as exc:
print(type(exc).__name__)
print(target.read_text().splitlines())
print(list(Path(directory).glob('.download-*')))
Expected output:
13
['column', 'value']
HTTPError
ValueError
['column', 'value']
[]
The successful file contains 13 bytes. The 404 and oversized response both fail without replacing it. Temporary partial files are removed by the finally block. Interrupted-stream, digest-mismatch and timeout cases were also checked in the validation fixtures.
State the timeout and retry contract
The timeout tuple specifies connection and read-inactivity limits. It is not a hard deadline for the whole download: a server can send small pieces before each inactivity limit. A strict elapsed-time deadline needs an execution environment that can cancel the entire operation.
This helper does not retry. For an approved read-only GET, a caller can start a new bounded attempt after a transient failure. Do not append to a partial file unless you implement and validate the server's range and content-identity contract. If the remote file can change between attempts, require a versioned URL or a trusted digest.
Verify the result before using it
An HTTP 200 body can still be the wrong file. Apply the expected file parser and schema before processing it, and never execute a downloaded program simply because its transfer completed. For bounded JSON requests and deliberate retry rules, see the Requests guide.
Primary references: Requests streaming and response cleanup, timeout semantics, and os.replace.
