Censored Regression

Censored Regression (Censored Normal / Censored Lognormal)

Overview

Many process lab measurements (H2S, COS, mercaptans, DMS, disulfides, unknown sulfur, trace metals) are reported as below-LOD, below-LOQ, above an instrument range, or within an interval. Ordinary least squares on log(S + epsilon) is biased and understates uncertainty; the correct treatment is a likelihood-based censored regression.

This skill provides:

• A censored-normal regression on the log scale by default (censored lognormal

on the original response scale).

• A censored-normal regression on the raw scale with --no-log.
• Left, right, and interval censoring.
• Maximum-likelihood fitting through SciPy.

The implementation uses a hand-rolled SciPy likelihood. It does not calculate standard errors, robust covariance estimates, or the naive substitution fit. Use bootstrap/profile-likelihood checks externally when inference quality matters.

Prerequisites

1. uv available.
2. The script declares numpy, scipy, and pandas in its PEP-723 header.
3. On first use, writes LICENSE_NOTIFICATION.txt.

When to Use

• Lab data has below-LOQ rows that you cannot drop without introducing

selection bias.

• Reported values include intervals, for example between LOD and LOQ.
• Fitting a regression of log(species) on operating variables, where the

species is sometimes censored.

Don't use for

• Fully uncensored regression: use engineering-statistics.
• Time-to-event survival analysis: use a survival-analysis package.
• Compositional response data (sulfur speciation fractions): combine with

compositional-data-analysis.

Utility Scripts

• uv run scripts/censored.py fit --data data.csv --response S_total --predictors temperature,c5_c6plus_balance --lower-col LOQ --output /tmp/fit.json
• uv run scripts/censored.py interval --data data.csv --response S_total --predictors temperature --lower-col LOD --upper-col LOQ --output /tmp/int.json

Input CSV layout:

• One row per observation.
• response column with the measured value when fully observed; NaN or blank

when censored.

• lower-col and/or upper-col columns with the censoring bound for censored

rows.

• Predictor columns referenced by name in --predictors.

Censoring-column semantics used by the script:

• finite response: exact observation; bounds ignored.
• missing response + only lower-col finite: left-censored, meaning

y = upper_col (above-range row).

• missing response + both bounds finite: interval-censored, meaning

`lower_col 80%), report only the censoring fraction and rank/order conclusions; full regression coefficients are unstable.

• If the row order is a time series, use block bootstrap or a dynamic model for

uncertainty rather than i.i.d. inference.

Verification

• Run the listed script with representative inputs and an --output file when a deterministic calculation is available.
• Confirm the JSON result contains ok: true, expected units, and no unhandled warnings.
• Check result magnitudes against the stated assumptions, references, and a hand calculation or known operating range before reporting them.

References

• references/censored_likelihood.md — likelihood derivation.
• Greene, Econometric Analysis, chapter on truncated/censored models.
• Helsel, Statistics for Censored Environmental Data Using MINITAB and R.

Anti-Patterns

• Hiding the censoring fraction in the report.
• Reporting a censored-lognormal coefficient as if it were on the original

response scale; by default, the response model is on log scale.

• Using ordinary bootstrap on autocorrelated censored data without preserving

time dependence.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: jskherman
• Source: jskherman/engg-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.