Quick summary
Summarize this blog with AI
Introduction
Treat missing-value handling as a sequence of explicit decisions: inspect, determine the source, select the analysis population, then replace only values with a justified meaning. dplyr and tidyr make these decisions readable; they cannot determine whether a missing measurement is zero or why it went missing.
Tested with R 4.5.1, dplyr 1.1.4 and tidyr 1.3.2. Run the code blocks in order. Install the named packages before starting; inputs are supplied below.
Inspect raw values and keep the original columns
The fictional source contract below says empty or whitespace-only region labels mean unknown. Scores are numeric measurements and may legitimately be zero. na_if() converts only the documented blank sentinel; it does not erase every string that resembles a missing-value label.
library(dplyr)
library(tidyr)
raw <- tibble(id = 1:5, region = c("North", "", " ", "South", "North"),
score = c(10, NA, 0, 20, NA), note = c(NA, "review", "ok", NA, "ok"))
clean <- raw |> mutate(region_raw = region, region = na_if(trimws(region), ""))
audit <- clean |> summarise(across(c(region, score, note), ~ sum(is.na(.x))))
print(audit)
stopifnot(audit$region == 2, audit$score == 2, audit$note == 2,
identical(clean$region_raw, raw$region), clean$score[3] == 0)
There are 2 unknown regions,2 missing scores and 2 missing notes. Preserve the raw strings so the transformation is auditable. Numeric parse failures, source blanks and join nonmatches may all become NA but require different investigations.
Filter only the variables required by the question
drop_na(score) keeps records with an observed score even when a note or region is missing. Requiring both score and region defines a smaller population. dplyr::filter() keeps only TRUE conditions and drops NA conditions, so make that exclusion deliberate.
scored <- clean |> drop_na(score)
scored_by_region <- clean |> drop_na(score, region)
positive <- clean |> filter(score > 0)
print(scored[c("id", "region", "score")])
stopifnot(identical(scored$id, c(1L, 3L, 4L)),
identical(scored_by_region$id, c(1L, 4L)),
identical(positive$id, c(1L, 4L)), mean(scored$score) == 10,
mean(scored_by_region$score) == 15)
The overall observed-score mean is 10 across 3 records. Restricting to known regions gives mean 15 across 2 records. This change is a population-selection effect, not evidence that filtering improved the estimate. Row 3’s zero is observed and must not be confused with missingness.
Separate display labels from measured values
replace_na() can provide a display label in a new column while preserving the source measurement. “Unknown” is a reporting category here, not an inferred geographic region. Do not replace missing scores with zero merely to silence a summary warning.
display <- clean |> mutate(region_display = replace_na(region, "Unknown"))
summary <- display |> group_by(region_display) |>
summarise(rows = n(), observed_scores = sum(!is.na(score)),
missing_scores = sum(is.na(score)),
mean_observed = if (all(is.na(score))) NA_real_ else mean(score, na.rm = TRUE),
.groups = "drop")
print(summary)
stopifnot(identical(display$score, raw$score), sum(summary$rows) == nrow(raw),
sum(summary$observed_scores) == 3)
North has 2 rows with 1 observed score averaging 10; South has 1 observed score of 20; Unknown has 2 rows with 1 observed score of 0. An all-missing group would have meanNA, not 0.
Distinguish absent combinations from stored missing values
Suppose a complete, verified event extract records a row for any positive daily count and no row means zero events. A stored NA instead means an unresolved count. Under that specific contract, complete(..., explicit = FALSE) can fill newly created combinations without overwriting existing missing values. If absent rows could mean ingestion failure, zero-filling is not justified.
daily <- tibble(site = c("A", "A", "B"), day = c(1L, 2L, 1L), count = c(5L, NA_integer_, 2L))
completed <- daily |> complete(site = c("A", "B"), day = 1:2,
fill = list(count = 0L), explicit = FALSE)
print(completed)
stopifnot(nrow(completed) == 4,
is.na(completed$count[completed$site == "A" & completed$day == 2]),
completed$count[completed$site == "B" & completed$day == 2] == 0L)
A/day 2 remains NA because the unknown count was explicitly recorded. B/day 2 becomes 0 because it is the newly created combination covered by the stated contract. Completion changes the row universe; document the requested site/day range.
Mark reference matches before joining
A missing right-side attribute can mean either “no matching reference row” or “a matching row with an unknown attribute.” Add a match marker on the reference table before the join and declare the expected relationship.
reference <- tibble(id = c(1L, 2L, 3L, 5L), plan = c("Basic", NA, "Plus", "Basic"),
reference_matched = TRUE)
enriched <- left_join(clean, reference, by = "id", relationship = "many-to-one") |>
mutate(reference_matched = coalesce(reference_matched, FALSE))
print(enriched[c("id", "plan", "reference_matched")])
stopifnot(is.na(enriched$plan[2]), enriched$reference_matched[2],
is.na(enriched$plan[4]), !enriched$reference_matched[4],
nrow(enriched) == nrow(clean))
ID 2 matched a reference row with an unknown plan. ID 4 did not match at all. Replacing both plan values with the same label before this check would hide the distinction.
Check boundaries and record the resulting population
Keep raw data, count exclusions and report observed denominators. For modeling, learn imputation parameters on training data and assess sensitivity; a fitted value is not an observed truth. Patterns alone cannot identify the missing-data mechanism. For the lower-level mechanics, see NA masks and safe filtering and join diagnostics.
empty <- clean[FALSE, ]
all_missing <- tibble(score = c(NA_real_, NA_real_))
stopifnot(nrow(drop_na(empty, score)) == 0,
nrow(drop_na(all_missing, score)) == 0,
identical(na_if(c("NA", ""), ""), c("NA", NA_character_)),
identical(raw$score, c(10, NA, 0, 20, NA)))
