What is a Residual?
Understanding residuals is fundamental to regression analysis and statistical modeling. Learn the complete definition, types, and applications of residuals.
1. Basic Definition
A residual is the difference between an observed value and the value predicted by a statistical model.
In simple terms, residuals tell us how far off our predictions are from the actual observed data. They are the "leftover" or "remaining" differences that our model couldn't explain.
Key Points:
- Residuals measure prediction errors
- They show how well a model fits the data
- Smaller residuals indicate better model fit
- Residuals are used for model diagnostics
Think of residuals as the "unexplained variance" in your data. If you're trying to predict house prices based on size, and your model predicts $300,000 but the actual price is $320,000, then the residual is $20,000.
Simple Example
Scenario: Predicting test scores
Observed Score: 85
Predicted Score: 82
Residual: 85 - 82 = 3
The model underestimated the score by 3 points.
2. Mathematical Definition
Basic Formula
Residual Formula:
$$e_i = y_i - \hat{y}_i$$
Where:
- $e_i$ = residual for observation $i$
- $y_i$ = observed value for observation $i$
- $\hat{y}_i$ = predicted value for observation $i$
Vector Notation
$$\mathbf{e} = \mathbf{y} - \hat{\mathbf{y}}$$
Where e, y, and ลท are vectors of residuals, observed values, and predicted values respectively.
Mathematical Properties
Sum Property
$$\sum_{i=1}^{n} e_i = 0$$
The sum of residuals equals zero in least squares regression
Mean Property
$$\bar{e} = 0$$
The mean of residuals is zero
Orthogonality
$$\sum_{i=1}^{n} x_i e_i = 0$$
Residuals are orthogonal to predictors
3. Types of Residuals
1. Raw Residuals
$$e_i = y_i - \hat{y}_i$$
The basic residual - simply the difference between observed and predicted values. These are the most straightforward to calculate and interpret.
2. Standardized Residuals
$$r_i = \frac{e_i}{s}$$
Where $s$ is the residual standard error
Raw residuals divided by their standard error. This makes residuals comparable across different scales and helps identify outliers.
3. Studentized Residuals
$$t_i = \frac{e_i}{s\sqrt{1-h_i}}$$
Where $h_i$ is the leverage of observation $i$
Accounts for the varying precision of different fitted values. More reliable for outlier detection than standardized residuals.
4. Deleted Residuals
$$d_i = y_i - \hat{y}_{i(-i)}$$
Prediction made without observation $i$
The residual when the observation is excluded from fitting the model. Useful for detecting influential observations.
4. Why Residuals Matter
Model Assessment
Residuals help evaluate how well your model fits the data. Patterns in residuals reveal model inadequacies.
Assumption Checking
Statistical models make assumptions about residuals (normality, homoscedasticity). Residual analysis verifies these assumptions.
Outlier Detection
Large residuals identify outliers or unusual observations that may need special attention or investigation.
Model Improvement
Patterns in residual plots suggest ways to improve the model, such as adding variables or transforming data.
Validation
Residual analysis is essential for validating that your model is appropriate for the data and research question.
Prediction Intervals
Residuals help estimate the uncertainty in predictions and construct prediction intervals.
5. How to Interpret Residuals
Residual Size Guidelines
| Residual Type | Small | Moderate | Large | Very Large |
|---|---|---|---|---|
| Raw Residuals | Close to 0 | 1-2 ร typical scale | 3-4 ร typical scale | > 4 ร typical scale |
| Standardized | |r| < 1 | 1 โค |r| < 2 | 2 โค |r| < 3 | |r| โฅ 3 |
| Studentized | |t| < 2 | 2 โค |t| < 2.5 | 2.5 โค |t| < 3 | |t| โฅ 3 |
Signs and Meanings
Positive Residual (+)
$e_i > 0$
Meaning: The observed value is greater than predicted. The model underestimated the actual value.
Negative Residual (โ)
$e_i < 0$
Meaning: The observed value is less than predicted. The model overestimated the actual value.
Zero Residual (0)
$e_i = 0$
Meaning: Perfect prediction! The observed value exactly matches the predicted value.
Ready to Calculate Residuals?
Use our online calculator to compute different types of residuals with step-by-step explanations.