Understanding Pearson Correlation Assumptions in R

Quick summary

Summarize this blog with AI

Introduction

Pearson’s sample correlation describes linear association. Computing it does not require normally distributed variables. The assumptions for a p-value or confidence interval are a separate question. This guide distinguishes calculation, interpretation and classical inference, using complete paired observations and visible counterexamples.

Tested with R 4.5.1. Run the code blocks in order. Install the named packages before starting; inputs are supplied below.

Calculate a descriptive coefficient on matched pairs

Both columns must describe the same observational units in the same order. A finite sample coefficient requires at least two usable pairs with nonzero variation in each variable. We explicitly count and exclude nonfinite pairs here; that decision may introduce bias if missingness is related to the variables.

dat <- data.frame(x = c(1:8, NA), y = c(2, 4, 5, 4, 5, 7, 8, 9, 10))
 keep <- is.finite(dat$x) & is.finite(dat$y)
 cat("Excluded pairs:", sum(!keep), "
")
 paired <- dat[keep, ]
 r <- cor(paired$x, paired$y)
 formula_r <- cov(paired$x, paired$y) / (sd(paired$x) * sd(paired$y))
 cat("Pearson r:", round(r, 4), "
")
 stopifnot(abs(r - formula_r) < 1e-12)

One pair is excluded. The coefficient is about 0.9512. It summarizes this sample’s positive linear association, without establishing causality or predicting that every individual follows a line.

See what a single coefficient hides

Pearson correlation can be computed for a curved relationship, but may summarize it poorly. A symmetric U-shaped relation can have r = 0 despite perfect deterministic dependence. Extreme observations can also dominate the coefficient; investigate provenance and report sensitivity rather than automatically deleting them.

curve_x <- -3:3
 curve_y <- curve_x^2
 stopifnot(abs(cor(curve_x, curve_y)) < 1e-12)
 r_with_extreme <- cor(c(paired$x, 30), c(paired$y, -10))
 print(round(c(original = r, with_extreme = r_with_extreme), 4))
 draw_correlation <- function() {
   old <- par(mfrow = c(1, 3), mar = c(4, 4, 3, 1))
   on.exit(par(old))
   plot(paired$x, paired$y, pch = 19, main = "Paired data", xlab = "x", ylab = "y")
   abline(lm(y ~ x, paired), col = "#a63d28")
   plot(curve_x, curve_y, pch = 19, main = "U-shape: r = 0", xlab = "x", ylab = "x squared")
   plot(c(paired$x, 30), c(paired$y, -10), pch = 19, main = "Extreme added pair",
        xlab = "x", ylab = "y", col = c(rep("#222222", 8), "#a63d28"))
 }
 draw_correlation()

Correlation describes linear association and can hide nonlinear dependence or sensitivity to a single pair.

Name the assumptions for classical inference

The usual Pearson cor.test() test of zero population correlation uses independent observations from a bivariate normal population. Its t statistic has n − 2 degrees of freedom under that null; the Fisher-transformed confidence interval is approximate. Separate normality tests of x and y cannot establish joint bivariate normality. Constant conditional variance is a feature of the bivariate-normal model, not a prerequisite for computing sample r.

test <- cor.test(paired$x, paired$y, method = "pearson")
 n <- nrow(paired)
 t_manual <- r * sqrt((n - 2) / (1 - r^2))
 p_manual <- 2 * pt(-abs(t_manual), df = n - 2)
 print(c(r = unname(test$estimate), p = test$p.value,
         lower = test$conf.int[1], upper = test$conf.int[2]))
 stopifnot(abs(unname(test$statistic) - t_manual) < 1e-12,
           abs(test$p.value - p_manual) < 1e-12)

These numbers demonstrate the calculation, not a verified population model for the constructed fixture. Repeated measures, time dependence, selection and influential pairs can invalidate the usual inference. A small p-value is not the probability that the relationship arose “by chance.”

Choose the association you want to measure

Spearman correlation measures rank association and is useful for monotonic relationships, including nonlinear ones. Kendall’s tau measures concordance. Neither automatically detects every nonlinear pattern or repairs dependence. A transformation changes the variables whose association is being measured; justify it using the scientific question rather than a desired p-value.

u <- 1:8
 v <- u^3
 print(c(Pearson = cor(u, v), Spearman = cor(u, v, method = "spearman"),
         Kendall = cor(u, v, method = "kendall")))
 stopifnot(abs(cor(u, v, method = "spearman") - 1) < 1e-12,
           abs(cor(curve_x, curve_y, method = "spearman")) < 1e-12)

The cubic relationship is strictly increasing, so both rank coefficients equal 1. The U-shaped example has Spearman correlation 0 as well. A scatter plot remains essential.

Guard calculation boundaries

The helper below deliberately rejects missing/nonfinite or mismatched vectors. Handle the pairing and exclusions before calling it. Avoid pairwise deletion across a large correlation matrix without checking which observations support each cell; the resulting matrix can fail to be positive semidefinite.

checked_cor <- function(x, y) {
   if (!is.numeric(x) || !is.numeric(y) || !is.null(dim(x)) || !is.null(dim(y)) ||
       length(x) != length(y) || length(x) < 2 ||
       any(!is.finite(x)) || any(!is.finite(y))) stop("Need aligned finite numeric vectors")
   if (!is.finite(sd(x)) || !is.finite(sd(y)) || sd(x) == 0 || sd(y) == 0)
     stop("Need positive finite variation in both vectors")
   cor(x, y)
 }
 fails <- function(expr) inherits(tryCatch(expr, error = identity), "error")
 stopifnot(abs(checked_cor(paired$x, paired$y) - r) < 1e-12,
           fails(checked_cor(numeric(), numeric())), fails(checked_cor(c(1, 1), c(1, 2))),
           fails(checked_cor(c(1, NA), 1:2)), fails(checked_cor(1:3, 1:2)),
           fails(checked_cor(matrix(1:4, 2), 1:4)))