Clean Survey Data

Clean Survey Data

A skill for cleaning survey data in R, handling common patterns like missing value codes, variable recoding, and Stata label conversion.

Quick Start

library(data.table)
library(rio)
library(haven)

# Load data
dt  3 levels
dt[, educ3 := fcase(
    education %in% c("Less than HS", "HS diploma"), "HS or less",
    education %in% c("Some college", "Associate"), "Some college",
    education %in% c("Bachelor's", "Graduate"), "Bachelor's+"
)]

# Marital status: 6 levels -> 3 levels
dt[, marital3 := fcase(
    marital_status %in% c("Married", "Living with partner"), "Partnered",
    marital_status %in% c("Widowed", "Divorced", "Separated"), "Prev married",
    marital_status == "Never married", "Never married"
)]

5. Create Composite Scores

Sum or average across multiple items:

# Sum score with missing handling
items = as.Date("2020-04-01"), 1L, 0L)]

Complete Example

library(data.table)
library(rio)

# Load and standardize names
dt <- as.data.table(import("survey_data.dta"))
names(dt) <- tolower(names(dt))

# Define label conversion function
attach_label_to_variable <- function(x, na_exclude = TRUE) {
  var_lab <- attr(x, 'labels')
  if (na_exclude) {
    var_lab <- var_lab[!var_lab %in% c(91, 92, 97, 98)]
  }
  if (!is.null(var_lab)) {
    x <- factor(x, levels = var_lab, labels = names(var_lab))
  }
  return(x)
}

# Clean missing values
for (var in c("income", "health_status", "satisfaction")) {
  dt[get(var) %in% c(91, 92, 97, 98), (var) := NA]
}

# Convert labeled variables
dt[, gender := attach_label_to_variable(gender)]
dt[, education := attach_label_to_variable(education)]
dt[, marital := attach_label_to_variable(marital)]

# Create derived variables
dt[, age := year(interview_date) - year(dob)]
dt[, age_group := cut(age, breaks = c(17, 35, 50, 65, 100),
                      labels = c("18-34", "35-49", "50-64", "65+"))]

# Select and save cleaned variables
keep_vars <- c("id", "gender", "age", "age_group", "education", "marital")
dt_clean <- dt[, ..keep_vars]
saveRDS(dt_clean, "cleaned_data.rds")

Required Packages

install.packages(c("data.table", "rio", "haven"))

Tips

• Always standardize column names to lowercase first
• Document missing value codes specific to your survey
• Create a data dictionary mapping original codes to cleaned values
• Test label conversion on a small sample before applying to full dataset
• Use fcase() for complex recoding (faster than nested ifelse)

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: letitbk
• Source: letitbk/claude-academic-setup
• License: MIT

Install and usage instructions live in the source repository linked above.