Ultimate Guide: Loading CSV Data into R

Introduction

Import CSV with an explicit schema and inspect parsing failures before calculating results. A CSV field may contain a quoted comma or newline, so splitting text lines manually is not a CSV parser. This lesson creates its own small file, preserves leading-zero identifiers, and demonstrates a bounded-memory aggregate.

Tested with R 4.5.1 and readr 2.2.0. Run the code blocks in order. Install the named packages before starting; inputs are supplied below.

Create and read a complete CSV fixture

The identifiers are text, not quantities. Keep the original file and select missing-value tokens deliberately: a real text value of NA would need a different convention.

path <- tempfile(fileext = ".csv")
writeLines(c('id,note,amount', '001,"red, blue",12.5',
             '002,"two', 'lines",NA', '003,"plain",7.5'), path)
base_data <- read.csv(path, colClasses = c("character", "character", "numeric"),
                      na.strings = "NA", check.names = FALSE)
stopifnot(identical(base_data$id, c("001", "002", "003")))
stopifnot(identical(base_data$note, c("red, blue", "two\nlines", "plain")))
stopifnot(identical(base_data$amount, c(12.5, NA_real_, 7.5)))
cat(nrow(base_data), "records; known amount", sum(base_data$amount, na.rm = TRUE), "\n")
# 3 records; known amount 20

A physical line is not always one record. Specify the encoding and delimiter that the producer actually used; read.csv2 uses a different default separator/decimal convention. Type guessing is convenient exploration, not a stable data contract.

Use readr types and require a clean parse

This reads the same file using an explicit schema. The result should agree with base R for the fields we defined.

schema <- readr::cols(id = readr::col_character(), note = readr::col_character(),
                      amount = readr::col_double())
parsed <- readr::read_csv(path, col_types = schema, na = "NA", show_col_types = FALSE)
readr::stop_for_problems(parsed)
stopifnot(identical(parsed$id, base_data$id), identical(parsed$note, base_data$note),
          identical(parsed$amount, base_data$amount))
empty <- readr::read_csv(I("id,note,amount\n"), col_types = schema, show_col_types = FALSE)
stopifnot(nrow(empty) == 0L, is.double(empty$amount))

Inspect column names and row counts as well as types. A successfully parsed file can still be the wrong file. readr records recoverable parsing problems; an invalid number can become NA unless you check those problems.

Distinguish invalid text from a declared missing value

The warning below is intentional. Capture it only to inspect the known failure; stop_for_problems converts that failure into an error.

bad_csv <- 'id,note,amount\n001,ok,oops\n'
bad <- suppressWarnings(readr::read_csv(I(bad_csv), col_types = schema,
                                       na = "NA", show_col_types = FALSE))
problems <- readr::problems(bad)
stopifnot(nrow(problems) == 1L, is.na(bad$amount))
stopifnot(inherits(tryCatch(readr::stop_for_problems(bad), error = identity), "error"))
print(problems$actual)  # "oops"
base_failure <- tryCatch(read.csv(text = bad_csv,
  colClasses = c("character", "character", "numeric")), error = identity)
stopifnot(inherits(base_failure, "error"))

Do not suppress warnings in a production import and then continue as if missing data were expected. Review the source, report rejected records, and choose whether the import must fail.

Aggregate chunks without retaining every chunk

The callback keeps only running totals. It does not repeatedly append rows, reread overlapping headers, or assume that a newline is a record boundary.

state <- new.env(parent = emptyenv())
state$rows <- 0L; state$missing <- 0L; state$known_total <- 0
callback <- readr::SideEffectChunkCallback$new(function(chunk, position) {
  readr::stop_for_problems(chunk)
  state$rows <- state$rows + nrow(chunk)
  state$missing <- state$missing + sum(is.na(chunk$amount))
  state$known_total <- state$known_total + sum(chunk$amount, na.rm = TRUE)
})
readr::read_csv_chunked(path, callback, chunk_size = 2,
                       col_types = schema, na = "NA", progress = FALSE,
                       show_col_types = FALSE)
stopifnot(state$rows == 3L, state$missing == 1L, state$known_total == 20)
unlink(path)

This bounds the stored result, though parser buffers and an unusually large record still consume memory. A parse error invalidates the partial totals; discard them. Only publish totals after the entire read succeeds. Use object.size for an object estimate and measure your real workload; changing CSV libraries does not guarantee lower memory use.

Primary references

• R read.table/read.csv
• readr parsing problems
• readr chunked reads