Photo By Clint Shelton
Table of Contents
- Context
- Formulas and Derivation
- Useful Resources
Context
Introduction to the Regression F Test
Context
Introduction to the Regression F Test
The F test is a test of the statistical significance of the regression analysis model as a whole. Unlike the T test, which is a test of means, the F test is a test of variances. The linear regression F test determines if the variance of the model is significantly less than that of a baseline model.
Just as in parts 1 and 2 utilizing matrix variables rather than only scalar makes deriving the solutions much easier. In this case this applies to constructing the formula for the degrees of freedom when both terms in the squared error are estimators.
Baseline, Final, and Explained Variance
The F test compares the variance of the final model, also called the “unexplained” variance (as in the deviation of the actual values from the predicted values not explained by the model), to the “explained” variance (the deviation between the predicted values of a baseline model to those of the final model).
\[\large{
\begin{align}
\stackrel{\text{baseline}}{\tilde{e} = (y-\tilde{y})} && \stackrel{\text{final}}{\widehat{e} = (y-\widehat{y})} && \stackrel{\text{explained}}{(\tilde{e} – \widehat{e}) = (\widehat{y}-\tilde{y})}
\end{align}
}\]
The Test Hypotheses
The null hypothesis is that the final model is no more explanatory than the baseline model. Under the null hypothesis the final and explained variance are both equal to the baseline variance (and therefore to each other).
\[\large{
\begin{align}
&\text{H}_0:\sigma_\text{final}^2=\sigma_\text{explained}^2 \\ \\
&\text{H}_a:\sigma_\text{final}^2<\sigma_\text{explained}^2
\end{align}
}\]
Many sources incorrectly use a “not equal to” comparison for the alternative hypothesis. This is incorrect, because the final model includes all the factors from the baseline and then some. If those extra factors are explanatory, then the final variance will be less than the explained variance. If they are not explanatory, then the final and explained variances will be statistically equal.
Implications of the Null Hypothesis
While not actually components of the hypothesis test itself the implications of the null hypothesis below are important to constructing the test formulas (as is show in the derivation section).
\[\large{
\begin{align}
&\text{H}_0\longrightarrow\text{E}[\tilde{y}]=\text{E}[\widehat{y}] \\ \\
&\text{H}_0\longrightarrow\sigma_\text{baseline}^2 = \sigma_\text{final}^2 = \sigma_\text{explained}^2
\end{align}
}\]
The Test
The test statistic, if the standard assumptions of the linear model are met, is considered to be generated from an F distribution (explained in more detail in the formula derivation section). If the probability of the observed F statistic is too unlikely (ie less than some predetermined probability \(0<a<1\)), then the hypothesis that the final model is no more explanatory than the baseline is rejected.
\[\large{
\begin{align}
&f_{df_\text{explained},df_\text{final}} = \frac{\sigma_\text{explained}^2}{\sigma_\text{final}^2} \sim F(df_\text{explained},df_\text{final}) & && &\normalsize \text{(Test Statistic)} \\ \\
&P_F(df_\text{explained},df_\text{final}) \begin{cases}
>= a && \small\text{accept $H_0$} \\
< a && \small\text{reject $H_0$, accept $H_a$}
\end{cases} & && &\normalsize\text{(Test)}
\end{align}
}\]
Baseline Is Subset of Final
The F Test is based on the comparison of a baseline model to a final model. The baseline is the estimate produced from any proper subset of factors from the final model.
\[\large{
\begin{align}
&y = \stackrel{n\times q}{X}b + \varepsilon & && & n > q & && & & && &\small\text{(linear regression model)} \\ \\
&\stackrel{n\times o}{W} = X_{[,1:o]} & && & o<q & && & & && &\small\text{(baseline design matrix)} \\ \\
&\tilde{b} = (W^\text{T}W)^{-1}W^\text{T}y & && &\tilde{y} = W\tilde{b} & && &\tilde{e} = (y-\tilde{y}) & && &\small\text{(baseline residual error)} \\ \\
&\widehat{b} = (X^\text{T}X)^{-1}X^\text{T}y & && &\widehat{y} = X\widehat{b} & && &\widehat{e} = (y-\widehat{y}) & && &\small\text{(final residual error)}
\end{align}
}\]
Most Common Baselines
Almost universally the baseline models of linear regression are one of two varieties: “constant factor” baseline or “Null factor” baseline.
Constant factor baseline is a design matrix, which contains only the constant factor 1. In this special case the coefficient estimator formula reduces to the standard sample mean formula. This results in every model estimate being the mean of the actual values from \(y\). Under the null factor baseline, there is no significant factor, the model is assumed to be only the error term \(\varepsilon\) and every estimated value is 0.
\[\large{
\begin{align}
&\stackrel{\text{Constant Basline}\\}{y = \begin{bmatrix} 1 \\ \vdots \\ 1 \end{bmatrix}b + \varepsilon} & && &\tilde{b} = \frac{1}{n}\sum_{i=1}^n y_i = \widehat{\mu}_Y & && &\tilde{y} = \begin{bmatrix} \widehat{\mu}_Y \\ \vdots \\ \widehat{\mu}_Y \end{bmatrix} \\ \\
&\stackrel{\text{Null Baseline}\\}{y = \varepsilon} & && &\tilde{b} = \small\text{undefined} & && & \tilde{y} = \begin{bmatrix} 0 \\ \vdots \\ 0 \end{bmatrix}
\end{align}
}\]
WARNING! on Changing Baselines
The constant and null factor baselines are so ubiquitous they are built into the functionality of the R. Including the constant factor in the final model automatically uses the constant factor baseline for the F test. CHANGING THE BASELINE BETWEEN TWO MODELS MAKES THEM NOT COMPARABLE.
In order to compare a model with a constant factor to one without it is necessary to include the constant factor as an actual column of data so that the null factor baseline can be specified for both models.
n <- 30
df <- data.frame(
x = runif(n,-100,100)
,e = rnorm(n,0,35)
)
df$y <- 40 + 2.5 * df$x + df$e
df$const <- rep(1,n)
# + 1 automatically uses the constant factor baseline
summary(lm(y ~ x + 1, df))
# model w/o constant factor using null factor baseline
summary(lm(y ~ x + 0, df))
# model w/ constant factor using null factor baseline
summar(lm(y ~ x + Const + 0), df)
F Test Only Valid For Linear Regression
In part I of this series I mentioned that linear regression is only one member of an entire family of linear models (called the generalized linear model). This was in the context of choosing the “best” estimate for the model coefficients.
I detailed a method for estimating the coefficient vector called maximum likelihood estimation. The reason I gave for detailing this method and not the much more common “least squares” method is that MLE produces EXACTLY the same result for linear regression, but unlike least squares it functions for the entire generalized linear model.
Because other members of the family do not seek to minimize squared error, the final estimate may not be the one that produces the lowest squared error. The F test applies lower p-values to models which produce lower variance, which is just a scaled measure of squared error. But if you are working with a model and optimizing for something other than squared error it makes no sense to judge the model accuracy using a statistic which is based on squared error.
Although you can calculate the canonical F statistic for the entire GLM, as it is merely a function of the baseline, final, and explained variance, for the reasons stated above it is invalid to use it to judge the accuracy of any generalized linear model other than linear regression.
Formulas and Derivation
The Regression F Test Formulas
\[\large{
\begin{align}
&y = \stackrel{n\times q}{X}b + \varepsilon & && &\small\text{(regression model)} \\ \\
&\widehat{b} = (X^\text{T}X)X^\text{T}y,\quad\widehat{y}=X\widehat{b} & && &\small\text{(final estimate)} \\ \\
&y = \stackrel{n\times o}{W}b + \varepsilon,\;\; W=X_{[,1:o]},\;\;o<q & && &\small\text{(baseline model)} \\ \\
&\tilde{b} = (W^\text{T}W)W^\text{T}y,\;\;\tilde{y}=W\tilde{b} & && &\small\text{(baseline estimate)} \\ \\
&df_\text{final} = (n\;-\;q) & && &\small\text{(final degrees of freedom)} \\ \\
&df_\text{baseline} = (n\;-\;o) & && &\small\text{(baseline degrees of freedom)}\\ \\
&df_\text{explained} = (q\;-\;o) & && &\small\text{(“explained” degrees of freedom)}\\ \\
&\widehat{\sigma}_\text{final}^2=\frac{1}{df_\text{final}}\sum_{i=1}^n (y_i \;-\; \widehat{y}_i)^2 & && &\small\text{(final variance)}\\
&\widehat{\sigma}_\text{baseline}^2=\frac{1}{df_\text{baseline}}\sum_{i=1}^n (y_i \;-\; \tilde{y}_i)^2 & && &\small\text{(baseline variance)}\\
&\widehat{\sigma}_\text{explained}^2=\frac{1}{df_\text{explained}}\sum_{i=1}^n (\widehat{y}_i \;-\; \tilde{y}_i)^2 & && &\small\text{(explained variance)} \\ \\
&f_{df_\text{explained},\;df_\text{final}} = \frac{\widehat{\sigma}_\text{explained}^2}{\widehat{\sigma}_\text{final}^2} & && &\small\text{(F Test Statistic)} \\ \\
\end{align}
}\]
R Code For Regression F Test
The R code below manually implements the formulas from above, also uses the standard R functionality to achieve the same results, and then compares the two.
If you are new to R I suggest R-Studio as and IDE.
######################################
## Generate Data, Declare Variables ##
######################################
rm(list = ls())
`%+%` <- function(a, b) paste(a, b, sep="")
IsConstFactor <- T # control if constant factor in model
IsSigFactors <- T # control if significant factors in model
IsNonSigFactor <- T # control if non-significant factor in model
n <- 100 # sample size
sigma.model <- 40 # error standard deviation
# independent factors aka design matrix
X <- cbind(
if(IsConstFactor == T){rep(1,n)}else{NULL}
,if(IsSigFactors == T){runif(n,-100,100)}
,if(IsSigFactors == T){rpois(n,10)}
,if(IsNonSigFactor == T){rexp(n,0.1)}else{NULL}
)
# coefficient vector
b <- rbind(
if(IsConstFactor == T){40}else{NULL}
,if(IsSigFactors == T){2.5}
,if(IsSigFactors == T){4}
,if(IsNonSigFactor == T){0}else{NULL}
)
# error, linear regression model, baseline estimate
e <- cbind(rnorm(n,0,sigma.model))
y <- X %*% b + e
baseline <-
if(IsConstFactor == T) {
mean(y)
} else {0}
# QR factorization of X for more
# efficient processing
qr <- qr(X)
Q <- qr.Q(qr)
R <- qr.R(qr)
rm(qr)
# labels
colnames(X) <- c("X" %+% seq(as.numeric(!IsConstFactor),
ncol(X) - as.numeric(IsConstFactor)))
rownames(b) <- c("b" %+% seq(as.numeric(!IsConstFactor),
nrow(b) - as.numeric(IsConstFactor)))
###############################
## Linear Regression Using R ##
###############################
model.formula <- if(IsConstFactor == T) {
"y ~ 1" %+% paste(" + " %+% colnames(X)[2:ncol(X)], collapse='')
} else {"y ~ 0 " %+% paste(" + " %+% colnames(X), collapse='')}
linear.model <- lm(model.formula,as.data.frame(X))
#######################################
## Perform Liner Regression Manually ##
#######################################
b_ <- solve(R) %*% t(Q) %*% y # estimated coefficients
#b_ <- solve(t(X) %*% X) %*% t(X) %*% y
rownames(b_) <- rownames(b)
y_ <- X %*% b_ # estimated model
# degrees of freedom
df.baseline <- if(IsConstFactor == T) {n - 1} else {n}
df.final <- n - nrow(b_)
df.explained <- df.baseline - df.final
# residuals
res <- cbind(
c(y - baseline) # baseline/"total" error
,c(y - y_) # final/"unexplained" error
,c(y_ - baseline) # "explained" error ("total" - "unexplained")
); colnames(res) <- c("baseline","final","explained")
# variances
var_.baseline <- sum(res[,"baseline"]^2) / df.baseline
var_.final <- sum(res[,"final"]^2) / df.final
var_.explained <- sum(res[,"explained"]^2) / df.explained
# F-test
f.stat <- var_.explained / var_.final
pf <- 1 - pf(f.stat,df.explained,df.final)
ret.model <- cbind(sqrt(var_.final),f.stat,df.explained,df.final,pf)
colnames(ret.model) <- c("Std. Error","F-stat","Explained df","Final df","P-value")
rownames(ret.model) <- c("Model")
# residuals
ret.residuals <- cbind(min(res[,"final"]),quantile(res[,"final"],0.25),median(res[,"final"]),quantile(res[,"final"],0.75),max(res[,"final"]))
colnames(ret.residuals) <- c("Min","1Q","Median","3Q","Max")
rownames(ret.residuals) <- c("Residuals")
#############
## Compare ##
#############
summary(linear.model)
ret.residuals
ret.model
Step 1: Degrees of Freedom
Part 2 of this series defined degrees of freedom and showed why it is a necessary component for unbiased variance estimators. It also derived the degrees of freedom formula for an estimator where the sum of squared error has one actual term and one estimated term. See the example of the final model variance estimator below.
\[\large{
\begin{align}
\widehat{y} = \stackrel{n\times q}{X}\widehat{b} && \widehat{\sigma}_\text{final}^2 = \frac{1}{df_\text{final}}\sum_{i=1}^n(y_i\;-\;\widehat{y}_i)^2 && df_\text{final} = n-q
\end{align}
}\]
Unlike the baseline and final model variance estimators in the explained estimator BOTH terms of the squared error are estimators. Both terms being estimators with all factors from the baseline contained in the final model changes the degrees of freedom formula.
Explained Degrees of Freedom
Start with the explained sum of squared error. The null hypothesis implies that the expected difference in between the final model and the baseline model is 0 (said differently the expected value of their respective predicted values is equal).
\[\large{
\begin{align}
\text{E}\Big[SSE_\text{explained}\Big] &= \text{E}\Big[\sum_{i=1}^n(\widehat{y}_i – \tilde{y}_i-0)^2\Big] && \small (1-1) \\ \\
&=\text{E}\Big[\sum_{i=1}^n \widehat{y}_i^2 – 2\widehat{y}_i\tilde{y}_i + \tilde{y}_i^2\Big]
\end{align}
}\]
It is easier to factor out terms by first converting the sum to matrix/vector form. If you need a refresher, see the matrix quick reference.
\[\large{
\begin{align}
&=\text{E}\Big[\widehat{y}^\text{T}\widehat{y} \;-\; 2 \widehat{y}^\text{T}\tilde{y} + \tilde{y}^\text{T}\tilde{y}\Big] && \small (1-2) \\ \\
&= \text{E}\Big[\widehat{y}^\text{T}\widehat{y} \;-\; 2y^\text{T}X(X^\text{T}X)^{-1}X^\text{T}W(W^\text{T}W)^{-1}W^\text{T}y + \tilde{y}^\text{T}\tilde{y}\Big]
\end{align}
}\]
The \(\widehat{y}^\text{T}\tilde{y}\) term can be reduced to \(\tilde{y}^\text{T}\tilde{y}\). In order to understand how, consider the standard solution for the coefficient estimator multiplied by the design matrix.
\[\require{color}\large{
\begin{align}
X\widehat{b} =& {\color{aqua}\stackrel{P_X}{X(X^\text{T}X)^{-1}X^\text{T}}}y = P_Xy = \widehat{y} \\ \\
&{\color{aqua}\stackrel{P_X}{X(X^\text{T}X)^{-1}X^\text{T}}}X = P_XX = X
\end{align}
}\]
\(P_X\) is known as a “projection matrix.” It projects \(X\) onto \(y\), the result of which is the vector of model estimates \(\widehat{y}\). It should be obvious, both intuitively and via the matrix algebra, that projecting \(X\) onto \(X\) would result in itself.
What is easy to overlook is that the projection operates on the right hand matrix columns individually, because that’s simply how matrix multiplication works. Look at the two examples below and keep in mind matrix multiplication is ROW TO COLUMN.
\[\require{color}\large{
\begin{align}
&\stackrel{P_X}{\begin{bmatrix}
P_{1,1} & \ldots & P_{1,n} \\
\ldots & \ldots & \ldots \\
P_{n,1} & \ldots & P_{n,n}
\end{bmatrix}} &
&\stackrel{X}{\begin{bmatrix}
{\color{orange}X_{1,1}} & {\color{aqua}X_{1,2}} & {\color{gold}X_{1,3}} \\
{\color{orange}\vdots} & {\color{aqua}\vdots} & {\color{gold}\vdots} \\
{\color{orange}X_{n,1}} & {\color{aqua}X_{n,2}} & {\color{gold}X_{n,3}} \\
\end{bmatrix}}& &=
\stackrel{X}{\begin{bmatrix}
{\color{orange}X_{1,1}} & {\color{aqua}X_{1,2}} & {\color{gold}X_{1,3}} \\
{\color{orange}\vdots} & {\color{aqua}\vdots} & {\color{gold}\vdots} \\
{\color{orange}X_{n,1}} & {\color{aqua}X_{n,2}} & {\color{gold}X_{n,3}} \\
\end{bmatrix}} \\ \\
&\stackrel{P_X}{\begin{bmatrix}
P_{1,1} & \ldots & P_{1,n} \\
\ldots & \ldots & \ldots \\
P_{n,1} & \ldots & P_{n,n}
\end{bmatrix}} &
&\stackrel{X}{\begin{bmatrix}
{\color{orange}X_{1,1}} \\
{\color{orange}\vdots} \\
{\color{orange}X_{n,1}}
\end{bmatrix}}& &=
\stackrel{X}{\begin{bmatrix}
{\color{orange}X_{1,1}} \\
{\color{orange}\vdots} \\
{\color{orange}X_{n,1}}
\end{bmatrix}}
\end{align}
}\]
Every row in the left hand matrix is multiplied by the first column in the right hand matrix and the results are placed top to bottom in the first column vector of the product. Removing columns from X does not change the projection result for the columns that remain. Recall that \(W\) is made up of columns from \(X\).
\[\large{
P_XX = X \quad\land\quad W=X_{[,1:o]}\quad\longrightarrow\quad P_XW = W
}\]
This equivalency allows us to factor out all the \(X\) terms from the middle term of the explained SSE, which reduces it to \(\tilde{y}^\text{T}\tilde{y}\).
\[\require{cancel}\large{
\begin{align}
&= \text{E}\Big[\widehat{y}^\text{T}\widehat{y} \;-\; 2y^\text{T}\stackrel{P_X}{\cancel{X(X^\text{T}X)^{-1}X^\text{T}}}W(W^\text{T}W)^{-1}W^\text{T}y + \tilde{y}^\text{T}\tilde{y}\Big] && \small (1-3) \\ \\
&= \text{E}\Big[\widehat{y}^\text{T}\widehat{y} \;-\; \tilde{y}^\text{T}\tilde{y}\Big] \\ \\
&= \sum_{i=1}^n \text{E}[\widehat{y}_i^2]\;-\;\text{E}[\tilde{y}_i^2]
\end{align}
}\]
The expected values of squared estimator random variables can be expanded by rearranging the basic variance equivalency \(\text{Var}[Y] = \text{E}[Y^2] – \text{E}[Y]^2\). The assumption of the null hypothesis for the F test implies that the means of both estimator random variables are equal.
\[\require{cancel}\large{
\begin{align}
= \sum_{i=1}^n\text{Var}[\widehat{y}_i] + \cancel{\text{E}[\widehat{y}_i]^2} \;-\; \text{Var}[\tilde{y}_i] \;-\; \cancel{\text{E}[\tilde{y}_i]^2} &&\small (1-4)
\end{align}
}\]
As was shown in part 2 of this series, the sum of the variances of the mean estimator random variables is equal to \(\sigma_Y^2\) multiplied by the number of factors in the estimator.
\[\large{
\begin{align}
\text{E}\Big[SSE_\text{explained}\Big] &= q\sigma_Y^2\;-\;o\sigma_Y^2 &&\small (1-5) \\ \\
&= (q\;-\;o)\sigma_Y^2 \longrightarrow df_\text{explained} = (q\;-\;o)
\end{align}
}\]
Finally, note that the degrees of freedom of the explained variance estimator IS NOT A FUNCTION OF \(n\). This will be important in understanding the mechanics of the F statistic.
Step 2: Variance Estimators
The F statistic functions based on the mechanics of the baseline, final, and explained variance estimators. Under the null hypothesis the final and explained variance estimators will have an expected value equal to that of the baseline estimator.
\[\large{
\text{H}_0\longrightarrow \text{E}[\widehat{\sigma}_\text{baseline}^2] =\text{E}[\widehat{\sigma}_\text{final}^2] =\text{E}[\widehat{\sigma}_\text{explained}^2]
}\]
Under the alternative hypothesis the final model is more explanatory than the baseline and will have a smaller variance while the explained variance estimator will diverge to infinity.
\[\large{
\text{H}_a\longrightarrow \sigma_\varepsilon^2 \le \text{E}[\widehat{\sigma}_\text{final}^2] < \text{E}[\widehat{\sigma}_\text{baseline}^2] < \text{E}[\widehat{\sigma}_\text{explained}^2]< \infty
}\]
Baseline Variance
The baseline variance estimator uses the baseline model estimated values from \(\tilde{y}\).
\[\large{
\begin{align}
&\tilde{y} = \stackrel{n\times o}{W}\tilde{b} \\ \\
&df_\text{baseline} = n-o \\ \\
&\widehat{\sigma}_\text{baseline}^2 = \frac{1}{df_\text{baseline}} \sum_{i=1}^n\big(y_i-\tilde{y}_i\big)^2
\end{align}}
\]
Final Variance
The final variance estimator uses the final model estimated values from \(\widehat{y}\). It is often referred to as the “unexplained” variance or error.
Under the null hypothesis the final model is no more explanatory than the baseline, which means the variance estimator’s expected value is equal to that of the baseline estimator.
Under the alternative the final estimator is more predictive of \(y\) than the baseline and therefore the expected value of the final variance estimator will be lower than the baseline. However, the final variance estimator’s expected value cannot drop below the variance of the linear regression noise/error term \(\varepsilon\) (because that term by definition represent unexplainable variation).
\[\large{
\begin{align}
&\widehat{y} = \stackrel{n\times q}{X}\widehat{b}\\ \\
&df_\text{final} = n-q \\ \\
&\widehat{\sigma}_\text{final}^2 = \frac{1}{df_\text{final}} \sum_{i=1}^n\big(y_i-\widehat{y}_i\big)^2 \\ \\
&\text{H}_0 \longrightarrow \text{E}[\widehat{\sigma}_\text{final}^2] \quad=\quad \text{E}[\widehat{\sigma}_\text{baseline}^2] \\ \\
&\text{H}_a \longrightarrow (\sigma_Y^2=\sigma_\varepsilon^2) \quad\le\quad \text{E}[\widehat{\sigma}_\text{final}^2] \quad<\quad \text{E}[\widehat{\sigma}_\text{baseline}^2]
\end{align}}
\]
Explained Variance
The explained error is the difference between the baseline model error and the final model error. If an actual value is 10, the baseline prediction is 2, and the final prediction is 9, then 7 is what is “explained” by the final model (unexplained error is 1).
\[\large{
\stackrel{\text{baseline error}}{\tilde{e} = (y-\tilde{y})} \quad \stackrel{\text{model error}}{\widehat{e} = (y-\widehat{y})} \quad \stackrel{\text{explained error}}{(\tilde{e}-\widehat{e}) = (\widehat{y}-\tilde{y})}
}\]
Under the null hypothesis the expected difference in error is zero, but the individual final estimated values WILL deviate from the baseline. This is due to the variance of the coefficient estimates for factors present in the final, but absent in the baseline.
As was shown in part 2 the variance of coefficient estimates is expressed in units of actual model variance: \(\text{Var}[\widehat{b}]=(X^\text{T}X)^{-1}\sigma_Y^2\). This allows the explained variance estimator to be adjusted, through its degrees of freedom, to have an expected value equal to baseline variance (which is our best estimate for \(\sigma_Y^2\) under \(\text{H}_0\)).
Under the alternative hypothesis the estimator diverges to infinity, because the degrees of freedom is not a function of \(n\) (see the derivation of the degrees of freedom above). If the final model is more explanatory than the baseline, the sum of squared error is expected to increase with every observation, but the degrees of freedom will remain fixed.
\[\large{
\begin{align}
&df_\text{explained} = q-o\\ \\
&\widehat{\sigma}_\text{explained}^2 = \frac{1}{df_\text{explained}} \sum_{i=1}^n\big(\widehat{y}_i-\tilde{y}_i\big)^2 \\ \\
&\text{H}_0 \longrightarrow \text{E}[\widehat{\sigma}_\text{explained}^2] = \text{E}[\widehat{\sigma}_\text{baseline}^2] \\ \\
&\text{H}_a \longrightarrow \text{E}[\widehat{\sigma}_\text{baseline}^2] < \text{E}[\widehat{\sigma}_\text{explained}^2] < \infty
\end{align}
}\]
Step 3: Construct The Regression F Test Statistic
It can be shown that under the assumptions of the linear regression, most notably normally distributed errors, the ratio of the explained error and final model variance estimators form an F random variable with degrees of freedom taken respectively from the estimators.
Z, Chi-Squared, and F Random Variables
Any normally distributed random variable can be transformed into standard normal, by subtracting its own mean and dividing by its own standard deviation. A chi-squared distribution is the sum of one or more squared Z random variables (aka squared standard normal random variables) and having \(k\) degrees of freedom, which may not be the same as the number of values in the sum.
The F distribution is the ratio of two chi-squared random variables scaled by the inverse of their degrees of freedom.
\[\large{
\begin{align}
&X\sim \text{Nor}(\mu,\sigma) & && & Z \sim \text{Nor}(0,1) = \frac{X-\mu}{\sigma} \\
&\chi^2\sim \text{Chi}(k) = \sum_{i=1}^n Z_i^2 & && & F\sim F(a,b) = \frac{\chi^2_a/a}{\chi^2_b/b}
\end{align}
}\]
Constructing the Statistic
The F test requires the assumption of normally distributed errors. How to validate that assumption is topic for a future post. For now take it as a given that the final model error and the explained error are normally distributed with mean 0 (the coefficient estimator values are required to produce 0 error on average, otherwise they wouldn’t be the most likely values).
\[\large{
\begin{align}
&(y-\widehat{y})\sim\text{Nor}(0,\sigma_\text{final})\\ \\
&(\widehat{y}-\tilde{y})\sim\text{Nor}(0,\sigma_\text{explained})
\end{align}
}\]
Under the null hypothesis the expected value of the final model and explained error variance estimators are equal. This assumption of an expected equal variance can be used to transform the sum of squared errors of the estimators into Chi-squared random variables and therefore their quotient into an F random variable.
Start with the quotient of the variance estimators and factor in the shared expected variance.
\[\large{
\begin{align}
\frac{\widehat{\sigma}_\text{explained}^2}{\widehat{\sigma}_\text{explained}^2} \cdot \frac{\sigma^2}{\sigma^2} = \frac{\frac{1}{df_\text{explained}}\sum_{i=n}^n(\frac{\widehat{y}-\tilde{y}}{\sigma})^2}{\frac{1}{df_\text{final}}\sum_{i=n}^n(\frac{y-\widehat{y}}{\sigma})^2} && \small (2-1)
\end{align}
}\]
The sums are now of normally distributed values, divided by their standard deviation, squared, which is to say they are sums of squared Z random values. That makes the sums themselves Chi-Squared random variables.
\[\large{
\begin{align}
&\frac{\frac{1}{df_\text{explained}}\sum_{i=n}^n(\frac{\widehat{y}-\tilde{y}}{\sigma})^2}{\frac{1}{df_\text{final}}\sum_{i=n}^n(\frac{y-\widehat{y}}{\sigma})^2} = \frac{\frac{1}{df_\text{explained}}\chi_{df_\text{explained}}^2}{\frac{1}{df_\text{final}}\chi_{df_\text{final}}^2} && \small (2-2)
\end{align}
}\]
Remember that the degrees of freedom for the Chi-squared random variable is NOT the number of random variables in the sum, but the degrees of freedom of the sum of squared error component.
\[{
\sum_{i=1}^n Z^2 = \sum_{i=1}^n(\frac{X_i-\widehat{\mu}_X}{\sigma_X})^2 = \frac{\sum_{i=1}^n(X_i-\widehat{\mu}_X)^2}{\sigma_X^2} = \frac{SSE_{\{df=(n-1)\}}}{\sigma_X^2}
}\]
The quotient of two chi-squared random variables scaled by their degrees of freedom is the definition of an F random variable.
\[\large{
\begin{align}
F_{df_\text{explained},\;df_\text{final}} = \frac{\frac{1}{df_\text{explained}}\chi_{df_\text{explained}}^2}{\frac{1}{df_\text{final}}\chi_{df_\text{final}}^2} && \small (2-3)
\end{align}
}\]
The \(\sigma\) can be canceled out from the expanded chi-squared quotient.
\[\require{cancel}\large{
\begin{align}
F_{df_\text{explained},\;df_\text{final}} = \frac{\frac{1}{df_\text{explained}}\sum_{i=n}^n(\frac{\widehat{y}-\tilde{y}}{\cancel{\sigma}})^2}{\frac{1}{df_\text{final}}\sum_{i=n}^n(\frac{y-\widehat{y}}{\cancel{\sigma}})^2}
&& \small (2-4)
\end{align}
}\]
We now have that the the F statistic is equal to the quotient of the explained error variance estimator and the final model variance estimator.
\[\large{
\begin{align}
f_{df_\text{explained},\;df_\text{final}} = \frac{\widehat{\sigma}_\text{explained}^2}{\widehat{\sigma}_\text{final}^2} && \small (2-5)
\end{align}
}\]
tadalafil price online
azithromycin 500 mg generic price
synthroid 112 mcg price
valtrex without prescription
lisinopril average cost
metformin tablet price
top online pharmacy india
metformin hcl 500
rxpharmacycoupons
lisinopril cost 40 mg
indian pharmacy paypal
can i buy valtrex over the counter
tadalafil 20mg canada drug
lisinopril drug
synthroid 0.5 mg
where can i buy prednisone online without a prescription
where can i get prednisone over the counter
synthroid 88 mcg tablet
internet pharmacy mexico
world pharmacy india
how much is valtrex cost
prinivil drug cost
tadalafil from india 5mg
azithromycin 500 mg tablets
synthroid 0.88 mcg
synthroid 25 mcg cost
tadalafil 40 mg uk
pharmacy rx world canada
lisinopril cheap price
valtrex 500 price
zithromax 500mg coupon
pharmacy store
no prescription pharmacy paypal
synthroid 137 mcg price
prinivil online
generic cialis 2.5 mg online
synthroid 175 price
buy valtrex online cheap australia
lisinopril 20mg prices
prednisone where to buy uk
non prescription prednisone 20mg
family care rx pharmacy
cheapest pharmacy for prescriptions
synthroid 112 mg
synthroid 50 mg tablet
prednisone 50 mg
prednisone 20 mg tablet price
synthroid for sale
tadalafil 20mg lowest price
azithromycin online uk
can i buy synthroid online
buy prednisone online from canada
synthroid 75 mg tablets
metformin how to buy
lisinopril 10mg online
prednisone online for sale
buy lisinopril 5mg
how to buy lisinopril
buy prednisone online uk
prednisone canada pharmacy
zestril 5 mg price in india
online order prednisone
lisinopril 40mg
can i buy azithromycin
prednisone generic brand name
online pharmacy pain relief
synthroid coupon brand
buy prinivil
buy cheap tadalafil online
buy tadalafil generic
happy family drugstore
buy synthroid over the counter
prednisone pill cost
azithromycin 250
azithromycin 500 mg daily
secure medical online pharmacy
how to get azithromycin in usa
buy synthroid australia
zithromax capsules buy
synthroid 50 mcg online
cialis pills online canada
valtrex generic no prescription
zithromax 200mg
synthroid 12.5 mcg
synthroid canada price
lisinopril 20 mg no prescription
prednisone price canada
zestril 10mg
synthroid 200 mcg cost
prednisone 10 mg
lisinopril online without prescription
azithromycin 100mg
how much is zithromax 250 mg
synthroid 125 mcg tab
how to get azithromycin without prescription
lisinopril 20 mg price
azithromycin pills 250 mg
tadalafil tablets 10 mg price
buy metformin 500mg
buy valtrex online india
happy family canadian pharmaceuticals online
prinivil 2.5 mg
buy prednisone online india
generic tadalafil safe
lisinopril 2.5 mg
viagra online canadian pharmacy
valtrex 500mg uk
generic prednisone tablets
prednisone where can i buy
where to buy prednisone
buy tadalafil 10mg india
prednisone 20mg online pharmacy
generic zithromax medicine
how can i order prednisone
synthroid 0.125 mg tablet
pharmacy in canada for viagra
cheapest generic tadalafil online
over the counter valtrex
synthroid 50 mcg
tadalafil tablets 20mg
online pharmacy in turkey
lisinopril 20 tablet
cialis singapore pharmacy
medication synthroid
pharmacy online shopping usa
how to get valtrex cheap
metformin 1000 mg for sale
synthroid 75 mcg price in india
azithromycin buy without prescription
medical pharmacy south
where to buy metformin online
valtrex medication for sale
tadalafil 12mg
online pharmacy for sale
prescription free canadian pharmacy
200 mg prednisone daily
lisinopril rx coupon
buy zestoretic
how to get azithromycin 500 mg
rx pharmacy online
cortisol prednisone
cheap lisinopril no prescription
cialis 10mg online india
online pharmacy worldwide shipping
metformin online buy
synthroid 0.1 mcg
prednisone 10mg online
zestoretic coupon
buying prednixone for animals
tadalafil pills online
cheap valtrex
generic cialis cheap
valtrex 2000 mg
online pharmacy delivery dubai
buy cheap prednisone
cheap viagra online canadian pharmacy
synthroid online purchase
order zithromax over the counter
canadian pharmacy viagra
can you purchase azithromycin over the counter
generic tadalafil in usa
tadalafil online in india
tadalafil tablet buy online
discount pharmacy
where to buy valtrex generic
azithromycin rx
azithromycin 250 mg buy online
safe canadian pharmacy
prednisone 216
synthroid 0.05mg tablet
best canadian pharmacy to order from
cialis uk no prescription
canadian world pharmacy
buy tadalafil 5mg online
where can i get valtrex over the counter
synthroid 0.025 mg
canada rx pharmacy world
synthroid 1 mg
online pharmacies that use paypal
generic lisinopril 10 mg
prednisone buy cheap
4 azithromycin pills
purchase prednisone online
buy synthroid canada no prescription
metformin 750 mg
buy metformin online us
pharmacy no prescription required
azithromycin pills for sale
cheapist price for prednisone without prescription
zestril 20 mg cost
synthroid 0.0125 mcg
big pharmacy online
buy prednisone with paypal canada
valtrex without a prescription
tadalafil 5mg online
zithromax price
azithromycin 500mg price in usa
tadalafil prices
zithromax otc
brand name synthroid price
prednisone 10mg buy online
tadalafil price australia
where to buy valtrex online
good online mexican pharmacy
cialis 5 mg discount
azithromycin 500 tabs
lisinopril 18 mg
tulip happy family
how to buy tadalafil online
order metformin online uk
prednisone 20 mg medication
canadian family pharmacy
800 mg cialis
1.5 prednisone
online pharmacy drop shipping
prednisone 1 tablet
synthroid 115 mcg
price of lisinopril
online pharmacy indonesia
how much is valtrex canada
lowest cost generic cialis
cost of lisinopril
azithromycin prescription price
generic tadalafil us
lisinopril 0.5 mg
lisinopril 20g
canadian neighbor pharmacy
prinivil 40 mg
happy family drugstore
buy prinivil online
azithromycin drug
valtrex price
tadalafil online with out prescription
how can i get synthroid cheap
online pharmacy no prescription needed
cheap generic synthroid
viagra online canadian pharmacy
daily prednisone
generic prednisone without prescription
buy lisinopril 40 mg online
pharmacy without prescription
buy azithromycin 500mg online uk
valtrex brand name
cialis from india online pharmacy
cialis online mexico
synthroid 150 mg coupon
synthroid 50 pill
usa pharmacy online
mexican pharmacy what to buy
zithromax 500mg cost
zestoretic 10 12.5 mg
online pharmacy bc
legit non prescription pharmacies
azithromycin online prescription
order lisinopril online
top 10 online pharmacy in india
buy prinivil online
prednisone canada pharmacy
over the counter synthroid
how to get tadalafil
can you buy metformin over the counter in australia
can i buy synthroid online
buy generic tadalafil online
average cost of azithromycin
tadalafil – generic
canadian pharmacy generic tadalafil
online pharmacy cialis
synthroid purchase
metformin discount
over the counter valtrex medication
where to buy prednisone online
prices pharmacy
cialis canada online pharmacy
buy lisinopril canada
where to buy cheap synthroid
how to get cialis online
prinivil drug
valtrex 500 mg daily
valtrex pill cost
best price for synthroid
generic cialis 5mg daily
pharmacy in canada for viagra
azithromycin gel in india
real cialis cheap
synthroid 1mg
lisinopril price
generic cialis tadalafil
valtrex 10mg
tadalafil for sale from india
tadalafil 80mg online pharmacy no prescription
azithromycin cost australia
synthroid brand name price
online pharmacy without prescription
zithromax online usa
metformin 500mg tablets
can you buy azithromycin without a prescription
buy zestril 20 mg online
can i buy valtrex over the counter
10mg cialis online
online pharmacy ed
azithromycin 500mg
generic cialis tablets
buy tadalafil generic
pharmacy in canada for viagra
lisinopril 10 mg canada cost
prednisone medication
canadian pharmacy without prescription
where to get synthroid
tadalafil cheapest price
best online thai pharmacy
order synthroid
best tadalafil tablets in india
valtrex without presciption
synthroid 15 mcg
pharmaceuticals online australia
cialis 2.5 mg price india
10 mg lisinopril tablets
lisinopril cost 5mg
online drugstore metformin
glucophage canada
trustworthy canadian pharmacy
valtrex pills for sale
canadian pharmacies that deliver to the us
metformin usa
synthroid 100 mg prices
order lisinopril online united states
prescription free canadian pharmacy
generic zestoretic
metformin online
cheap generic cialis 5mg
online valtrex
metformin prices
where to buy prednisone online without a script
buy prednisone
cost of cialis 5mg
lisinopril 30 mg price
canadian pharmacy service
the canadian pharmacy
buy valtrex india
happy family store ivermectin
where to buy zithromax over the counter
generic cost of synthroid
pharmacy order online
synthroid rx coupon
zithromax online prescription
online order prednisone 10mg no prescription
azithromycin generic over the counter
mypharmacy
prednisone 1 mg price
online pharmacy discount code 2018
synthroid 0.175 mg
lisinopril 40
where to buy valtrex 1g
lisinopril 10 mg daily
canadian pharmacy no prescription
where can i buy azithromycin 500mg tablets
save on pharmacy
valtrex generic cost
azithromycin 500 mg cost
how to buy valtrex
buy synthroid online without prescription
valtrex best price
online med pharmacy
rate online pharmacies
cheap synthroid
buy synthroid online canada
online pharmacy no prescription needed
prednisone prescription cost
synthroid 0.05 mg daily
metformin for sale online
synthroid without a rx
cheap lisinopril no prescription
purchase azithromycin 1g
prednisone 20mg tab price
where can i buy zithromax in canada
prednisone drugs
zestril 10mg
where can i buy metformin
valtrex tablets uk
how much is metformin 1000 mg
can you buy valtrex over the counter in uk
synthoid
synthroid 88 mcg
canadian prescription pharmacy
hq pharmacy online 365
rx 535 lisinopril 40 mg
rx online pharmacy
metformin er 500mg
prednisone 1 mg tab
prednisone 1 mg cost
where to get metformin
tadalafil 5mg price australia
online pharmacy ordering
pharmacy online 365 discount code
pharmacy website
synthroid 50 mg price
online pharmacy pain
happy family online shop
zestoretic cost
synthroid 150 mcg cost
metformin for sale online
online pharmacy india
how to get azithromycin without prescription
order azithromycin online canada
buy azithromycin 500mg online us
how much is valtrex in canada
internet pharmacy mexico
buy cialis online free shipping
valtrex for sale canada
zithromax antibiotic
antibiotic azithromycin
synthroid 125
zithromax capsules australia
canadian pharmacy viagra
metformin 584
how much is azithromycin over the counter
cost of 10mg cialis
cialis europe
metformin over the counter
cheapest price for lisinopril
synthroid generic india
no prescription pharmacy paypal
cialis canada online pharmacy
how much is valtrex
synthroid for sale without prescription
happy family pharmacy coupon
northern pharmacy
lisinopril generic price
lisinopril 20 mg price
zithromax pack
valtrex drug
zestril 10mg price
international online pharmacy
reputable online pharmacy no prescription
cost of prednisone 5mg tablets
canadian pharmacy india
buy valtrex without prescription
zestril pill
lisinopril generic over the counter
low cost online pharmacy
canadian prescription pharmacy
canadian pharmacy coupon
buy metformin from mexico
azithromycin capsules 250mg
mexican pharmacy what to buy
buy metformin usa
where can i buy prednisone
lisinopril 20 mg uk
lisinopril 20 mg tablet
25 mg tadalafil
price of valtrex generic
metformin canada online
buy azithromycin from mexico
canada happy family pharmacy
lisinopril for sale
zithromax antibiotic
glucophage without prescription
best tadalafil
buying prednisone on line
metformin from india
prednisone 20 mg pill
ordering lisinopril without a prescription
azithromycin 1g tablet
herpes medication valtrex
where to buy valtrex generic
lisinopril 3760
best online pet pharmacy
price of lisinopril in india
10mg tadalafil
can you buy azithromycin over the counter nz
pharmacy website india
order valtrex generic
metformin 250 mg tab
canadian pharmacy 24h com
buy prednisone from india
order valtrex generic
buy tadalafil 100mg
synthroid 0.5 mcg
metformin 500 mg brand name
prednisone 10 mg coupon
lisinopril 5mg tab
tadalafil soft gel capsule
how to get metformin without prescription
sterapred ds
synthroid 75 pill
pharmacy online shopping usa
synthroid 150 mcg
buy generic valtrex online cheap
prednisone 10mg price in india
how can i get metformin
drug cost metformin
price of medicine valtrex
prednisone 10mg no prescription
zithromax 500
cheapest pharmacy to get prescriptions filled
medicine prednisone 10mg
uk pharmacy no prescription
can you buy valtrex over the counter in uk
lisinopril brand name in india
azithromycin brand name in canada
cialis for daily use cost
where to buy valtrex online
cross border pharmacy canada
metformin hcl 1000
online pharmacy australia
buy lisinopril without a prescription
synthroid cheap price
prinivil 10 mg
best canadian pharmacy to order from
safe canadian pharmacy
reputable online pharmacy uk
prednisone online australia
compare pharmacy prices
synthroid 112 mcg price
20 mg prednisone generic
lisinopril 10 12.5 mg
purchase discount cialis online
metformin 584
synthroid 100 pill
tadalafil over the counter australia
generic tadalafil cheap
prednisolone prednisone
valtrex medication
best european online pharmacy
valtrex rx cost
online pharmacy drop shipping
cialis buy online australia
generic tadalafil medication
synthroid 0.5 mg
azithromycin 250 mg
can i buy metformin over the counter in canada
prednisone 10mg no prescription
generic for synthroid
sure save pharmacy
where can i get azithromycin pills
can i buy metformin over the counter in uk
canada discount pharmacy
cialis 2mg
synthroid prescription discount
lisinopril 30 mg
canadapharmacyonline com
2 prinivil
generic valtrex cost
best online pharmacy reddit
BgjiaOGHdv
qCojUmBrLKuDA
best canadian online pharmacy
prednisone 60
azithromycin 500mg lowest price online
online pharmacy dubai
legit pharmacy websites
prednisone 20mg cost
azithromycin in singapore
buy prednisone 20mg without a prescription best price
60 mg tadalafil
average cost of zithromax
cheap valtrex uk
lisinopril in mexico
buy prednisone mexico
lisinopril 3973
tadalafil cheap no prescription
purchase synthroid online
canada pharmacy online legit
lisinopril in mexico
buying valtrex online
tadalafil free shipping
canadian pharmacy 24 com
synthroid rx
cialis 20mg coupon
buy tadalafil 40 mg
synthroid from india
azithromycin cost canada
lisinopril online pharmacy
happy family drugstore
how to get prednisone prescription
tadalafil generic from canada
metformin price australia
prednisone by mail
escrow pharmacy canada
zestril 5mg
valtrex order
buy synthroid canada
lisinopril prescription
synthroid 0.125 mg tablet
synthroid 1.25 mcg
tadalafil uk online
prednisone without a prescription
tadalafil india online
synthroid 125 mcg price
best canadian pharmacy no prescription
prednisone canadian
synthroid 75 mcg price canada
azithromycin for sale uk
mail order pharmacy no prescription
valtrex 500mg where to buy
metformin 1000 tab
online pharmacy no prescription needed
canadian pharmacy mall
azithromycin order canada
canadian pharmacy sildenafil
azithromycin for sale
canada drugstore pharmacy rx
glucophage 500 online
promo code for canadian pharmacy meds
synthroid 0.1
prednisone mexico
prednisone online india
buy synthroid online without a prescription
azithromycin medicine over the counter
where can i buy azithromycin over the counter
happy family store
lisinopril india
where to buy metformin uk
canadian pharmacy generic viagra
buy zithromax usa
valtrex 500mg best price
price of tadalafil