Quick summary
Summarize this blog with AI
Loops, functions, and exceptions become easier to reason about when they share one small data-cleaning task. This lesson reads numeric measurements, preserves rejected rows for review, normalizes valid values, and flags unusually distant observations. Every policy is explicit; no row disappears just because an exception occurred.
Tested with Python 3.12.14 using the standard library. Run the code blocks in order. The measurements are fictional teaching data, not a statistical benchmark.
Parse rows and keep an error trail
This input contract accepts a list of numeric strings. Empty strings, malformed numbers, NaN, and infinity are rejected. A programming error that passes a non-string is raised separately instead of being mistaken for a bad numeric measurement.
import math
def parse_measurements(raw_values):
valid = []
rejected = []
for row_number, raw in enumerate(raw_values, start=1):
if not isinstance(raw, str):
raise TypeError('each measurement must be text')
try:
value = float(raw)
except ValueError:
rejected.append((row_number, raw, 'not a number'))
continue
if not math.isfinite(value):
rejected.append((row_number, raw, 'not finite'))
continue
valid.append(value)
return valid, rejected
values, rejected = parse_measurements(['10', 'bad', '20', '', '30', 'nan'])
print(values)
print(rejected)
Expected output:
[10.0, 20.0, 30.0]
[(2, 'bad', 'not a number'), (4, '', 'not a number'), (6, 'nan', 'not finite')]
enumerate retains a one-based source row number. continue skips only the current invalid value. Use a for loop when traversing a known iterable; reserve while for a condition-driven process with a clear termination rule. Exceptions should explain an exceptional path, not replace ordinary loop control.
Normalize with an explicit constant-data policy
Min-max normalization maps the smallest value to 0 and the largest to 1 using (x - minimum) / (maximum - minimum). Compute the endpoints once. This implementation accepts a list or tuple of finite built-in integers or floats whose magnitudes fit the floating-point range, returns a new list, returns an empty list for empty input, and maps constant data to zeros by deliberate policy.
def finite_values(values):
if not isinstance(values, (list, tuple)):
raise TypeError('values must be a list or tuple')
if any(type(value) not in (int, float) for value in values):
raise TypeError('values must contain numbers, not strings or booleans')
try:
all_finite = all(math.isfinite(value) for value in values)
except OverflowError as exc:
raise ValueError('measurement magnitude exceeds the float range') from exc
if not all_finite:
raise ValueError('values must be finite')
return list(values)
def min_max_scale(values):
data = finite_values(values)
if not data:
return []
low, high = min(data), max(data)
if low == high:
return [0.0] * len(data)
span = high - low
try:
finite_span = math.isfinite(span)
except OverflowError as exc:
raise ValueError('numeric range is too large') from exc
if not finite_span:
raise ValueError('numeric range is too large')
return [(value - low) / span for value in data]
print(min_max_scale(values))
print(min_max_scale([5, 5, 5]))
print(min_max_scale([]))
Expected output:
[0.0, 0.5, 1.0]
[0.0, 0.0, 0.0]
[]
Python integers have arbitrary precision, while these statistical calculations use floating-point arithmetic. The helper turns an overflow during the finite-value check into a specific ValueError with a clear unsupported-magnitude message. Returning zeros for a constant feature is a modeling choice, not a mathematical requirement. Another application may reject it or remove the feature. Training and prediction also need a shared fitted minimum and maximum; recomputing them independently on each batch changes the meaning of the scale.
Make an outlier threshold actually control the result
The following function uses the interquartile range (IQR). It flags values strictly below Q1 - threshold × IQR or above Q3 + threshold × IQR. Quartiles use the standard library's inclusive method; stating that choice matters because small samples can produce different quartiles under other methods. At least four observations are required by this teaching contract.
from statistics import quantiles
def detect_outliers(values, threshold=1.5):
data = finite_values(values)
if len(data) < 4:
raise ValueError('at least four values are required')
if type(threshold) not in (int, float):
raise TypeError('threshold must be a number')
try:
finite_threshold = math.isfinite(threshold)
except OverflowError as exc:
raise ValueError('threshold magnitude exceeds the float range') from exc
if not finite_threshold or threshold < 0:
raise ValueError('threshold must be finite and non-negative')
try:
q1, _, q3 = quantiles(data, n=4, method='inclusive')
except OverflowError as exc:
raise ValueError('numeric range is too large') from exc
iqr = q3 - q1
low = q1 - threshold * iqr
high = q3 + threshold * iqr
if not all(math.isfinite(value) for value in (q1, q3, iqr, low, high)):
raise ValueError('numeric range is too large')
return [value for value in data if value < low or value > high]
sample = [1, 2, 3, 4, 10]
print(detect_outliers(sample))
print(detect_outliers(sample, threshold=4))
print(detect_outliers([5, 5, 5, 5]))
Expected output:
[10]
[]
[]
For this sample Q1 is 2, Q3 is 4, and IQR is 2. The default upper fence is 7, while a threshold of 4 moves it to 12. The result demonstrates that the argument is used. An outlier flag is a request for investigation, not permission to delete an inconvenient observation. With IQR zero, any value different from the shared quartile is outside the fences; consider that behavior before applying the rule to discrete or highly repeated data.
Exercise the failure paths
for operation in (
lambda: min_max_scale([1, float('nan')]),
lambda: min_max_scale([True, 2]),
lambda: min_max_scale([10**1000]),
lambda: detect_outliers([1, 2, 3]),
lambda: detect_outliers([1, 2, 3, 4], threshold=-1),
):
try:
operation()
except (TypeError, ValueError) as exc:
print(type(exc).__name__)
Expected output:
ValueError
TypeError
ValueError
ValueError
ValueError
Keep exception handling close to the boundary where recovery is possible. A batch loader can record a rejected row; a modeling function can reject an invalid dataset and let its caller decide what to do. Avoid broad handlers that silently substitute zero, because zero may be a legitimate measurement.
Keep the lesson reproducible
The normalization step makes a few linear passes over the data instead of recomputing extrema for every element. The quartile calculation sorts internally, so it has different costs. Benchmark representative data before replacing these clear functions with a more complex implementation. For DataFrame operations, continue with NumPy and pandas data cleaning; for traceback diagnosis, use the error-handling guide.
Primary references: Python's quartile methods, finite-number check, and exception-handling tutorial.
