Degrees of freedom (statistics)

In statistics, the term degrees of freedom is used in two distinct ways.

Residuals

In fitting statistical models to data, the vectors of residuals are often constrained to lie in a space of smaller dimension than the number of components in the vector. That smaller dimension is the number of degrees of freedom for error.

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

Perhaps the simplest example is this. Suppose

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

:X_1,dots,X_n,

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

are random variables each with expected value μ, and let

Related Topics:
Random variable - Expected value

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

:overline{X}_n={X_1+cdots+X_n over n}

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

be the "sample mean". Then the quantities

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

:X_i-overline{X}_n,

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

are residuals that may be considered estimates of the errors Xi − μ. The sum of the residuals (unlike the sum of the errors) is necessarily 0. That means they are constrained to lie in a space of dimension n − 1. If one knows the values of any n − 1 of the residuals, one can thus find the last one.

Related Topics:
Estimates - Errors

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

One says that "there are n − 1 degrees of freedom for error."

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

An only slightly less simple example is that of least squares estimation of a and b in the model

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

:Y_i=a+bx_i+arepsilon_i mathrm{for} i=1,dots,n

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

where εi, and hence Yi are random. Let widehat{a} and widehat{b} be the least-squares estimates of a and b. Then the residuals

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

:e_i=y_i-(widehat{a}+widehat{b}x_i),

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

are constrained to lie within the space defined by the two equations

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

:e_1+cdots+e_n=0,,

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

:x_1 e_1+cdots+x_n e_n=0.,

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

One says that "there are n − 2 degrees of freedom for error."

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

(The fastidious will note that capital Y is used in specifying the model, and lower-case y in the definition of the residuals. That is because the former are hypothesized random variables and the latter are data.)

~ ~ ~ ~ ~ ~ ~ ~ ~ ~

Another simple and frequently seen example arises in multiple comparisons.

~ ~ ~ ~ ~ ~ ~ ~ ~ ~