How to Remove Outliers in R

Introduction

Flag unusual observations before deciding whether to remove them. A point outside an IQR fence may be a real event, an entry error, or evidence that one pooled rule is inappropriate. The example below preserves record identifiers, handles missing values explicitly, and compares a documented correction with automatic deletion.

Tested with R 4.5.1. Run the code blocks in order. Install the named packages before starting; inputs are supplied below.

Define a reproducible IQR flag

This helper uses the 25th and 75th percentiles with R’s type-7 quantiles. The fence is Q1 − 1.5×IQR through Q3 + 1.5×IQR. Missing observations remain unknown.

iqr_flags <- function(x, multiplier = 1.5) {
  if (!is.numeric(x) || is.complex(x) || !is.null(dim(x)) || any(!is.finite(x) & !is.na(x)))
    stop("x must contain finite real numbers or NA")
  if (!is.numeric(multiplier) || length(multiplier) != 1L ||
      !is.finite(multiplier) || multiplier < 0) stop("Invalid multiplier")
  observed <- x[!is.na(x)]
  if (!length(observed)) return(rep(NA, length(x)))
  q <- quantile(observed, c(.25, .75), type = 7, names = FALSE)
  spread <- q[2] - q[1]
  limits <- c(q[1] - multiplier * spread, q[2] + multiplier * spread)
  if (any(!is.finite(limits))) stop("Fence exceeds numeric range")
  x < limits[1] | x > limits[2]
}
measurements <- data.frame(id = 1:8, duration = c(9, 10, 10, 11, 12, 13, 100, NA))
measurements$flag <- iqr_flags(measurements$duration)
print(measurements)
stopifnot(identical(measurements$flag, c(rep(FALSE, 6), TRUE, NA)))

For these seven observed values Q1 is 10, Q3 is 12.5, and the fences are 6.25 and 16.25. The quantile definition matters; base boxplot hinges can differ from these quartiles for small samples.

Investigate by record, then document the decision

In this fictional example, a source record confirms that ID 7 has a transcription error and cannot be recovered. Only that reviewed record is excluded; the missing duration remains visible.

review <- data.frame(id = 7L, reason = "Confirmed transcription error; original unavailable")
exclude <- measurements$id %in% review$id
cleaned <- measurements[!exclude, , drop = FALSE]
stopifnot(identical(cleaned$id, c(1:6, 8L)), is.na(tail(cleaned$duration, 1)))
stopifnot(nrow(cleaned) + sum(exclude) == nrow(measurements))
print(review)

Do not invent a reason from the flag itself. If the value is valid, retain it or choose an analysis that suits the target population. A rare event is not a data error simply because it changes the mean. For a model, learn preprocessing rules on the training partition and apply the agreed rule to evaluation data without refitting it there.

Report the sensitivity of the result

Show the observed sample size, mean and median before and after the confirmed exclusion. This exposes how much the decision matters.

summarize_duration <- function(x) {
  z <- x[!is.na(x)]
  c(observed = length(z), missing = sum(is.na(x)),
    mean = if (length(z)) mean(z) else NA_real_,
    median = if (length(z)) median(z) else NA_real_)
}
sensitivity <- rbind(before = summarize_duration(measurements$duration),
                     after = summarize_duration(cleaned$duration))
print(round(sensitivity, 3))
# before: observed 7 missing 1 mean 23.571 median 11
# after:  observed 6 missing 1 mean 10.833 median 10.5
stopifnot(isTRUE(all.equal(unname(sensitivity[, "mean"]), c(165/7, 65/6))))

The median is less affected here, but it estimates a different summary than the mean. Transformations or robust methods should serve the analytical question; they are not automatic repairs. Avoid deleting stock-price jumps or health measurements solely to make a plot or test look cleaner.

Check constant, empty and invalid inputs

An all-missing input has no estimated fence. With zero IQR, any value different from the common quartile can be flagged; that may be unhelpful for discrete data.

stopifnot(identical(iqr_flags(numeric(0)), logical(0)))
stopifnot(all(is.na(iqr_flags(c(NA_real_, NA_real_)))))
stopifnot(identical(iqr_flags(rep(5, 4)), rep(FALSE, 4)))
stopifnot(identical(iqr_flags(c(rep(0, 8), 1)), c(rep(FALSE, 8), TRUE)))
for (bad in list("10", Inf, c(1, -Inf), matrix(1:4, 2)))
  stopifnot(inherits(tryCatch(iqr_flags(bad), error = identity), "error"))

Document the population, quantile type, multiplier, missing-value policy and each exclusion. Do not present a threshold as proof of an error or as a guarantee of improved statistical accuracy.

Primary references

• R sample quantiles
• R boxplot statistics