Introduction

• In practice, models typically use multiple variables where it is possible to isolate the unique contribution of each explanatory variable.  A 𝑘-variate regression model enables the coefficients to measure the distinct contribution of each explanatory variable to the variation in the dependent variable.
• Multiple regression is regression analysis with more than one independent variable. The general form of a multiple regression can be written as



where:

Y_i = i^th observation of dependent variable Y

X_ki = i^th observation of k^th independent variable X

α = intercept term

β_k = slope coefficient of k^th independent variable

ϵ_i = error term of i^th observation

n = number of observations

k = total number of independent variables

Additional Assumptions of Multiple Regression

• Extending the model to multiple regressors requires one additional assumption, along with some modifications, to the six assumptions of linear regression with single regressors.
• The additional assumption is –
• Multiple linear regression assumes that the explanatory variables are not perfectly linearly dependent (i.e., each explanatory variable must have some variation that cannot be perfectly explained by the other variables in the model).

If this assumption is violated, then the variables are perfectly collinear.

• The remaining assumptions require simple modifications to account for 𝑘-explanatory variables. These become –
1. All variables must have positive variances so that



2. The error is assumed to have mean zero conditional on the explanatory variables



3. The random variables



4. The probability of large outliers in each explanatory variable should be small so that



5. The constant variance assumption is similarly extended to hold for all explanatory variables



6. The error terms should be uncorrelated across all observations, i.e.

Interpretation of Coefficients

• Slope coefficient (β_k) – It’s the change in the dependent variable from a unit change in the corresponding independent (X_ki) variable keeping all other independent variables constant.

When all explanatory variables are distinct (i.e., no variable is an exact function of the others), then the coefficients are interpreted as holding all other values fixed. For example, β_1 is the effect of a small increase in X₁ holding all other variables constant.

• When the value of the independent variable changes by one unit, the change in the dependent variable is not equal to the slope coefficient but depends on the correlation among the independent variables as well.

Therefore, the slope coefficient are called partial slope coefficients.

• Intercept coefficient (α) – The intercept (or the constant) is the expected value of the dependent variable  Y when all the independent variables X_ks are equal to 0.

Interpretation of coefficients – Indistinct Variables

• If some explanatory variables are functions of the same random variable (e.g., if X₂=X₁²) then:



In this case,

it is not possible to change X₁ while holding the other variables constant.

The interpretation of the coefficients in models with this structure depends on the value of X₁ because a small change of ΔX₁  in X₁ changes 𝑌 by



This effect captures the  direct, linear effect of a change in X₁ through β₁ and its nonlinear effect through β₂.

Ols Estimators for Multiple Regression Parameters

• Estimating the multiple regression parameters can be quite demanding since it involves a lot of calculations. A basic understanding can be developed using the multiple regression model with two independent variables, which can be extended for than 2 independent variables.
• Suppose the two variable model is



The OLS estimator for β₁ can be computed using three single-variable regressions.

1. The first regresses X₁ on X₂ and retains the residuals from this regression.
2. The second regression does the same for 𝑌
3. The final step regresses the residual of 𝑌 on the residual of X₁
• The first two regressions have a single purpose – to remove the direct effect of X₂ from 𝑌 and X₁. They do this by decomposing each variable into two components: one that is perfectly correlated with X₂(i.e., the fitted value) and one that is uncorrelated with X₂ (i.e., the residual). As a result, the two residuals are uncorrelated with X₂ by construction.
• The final regression estimates the linear relationship (i.e., β₁) between the components of 𝑌 and X_₁ that are uncorrelated with (and so cannot be explained by) X_₂.
• Finally, the OLS estimate of β₂ can be computed in the same manner by reversing the roles of X₁ and X₂ (i.e., so that β_2 measures the effect of the component in X₂ that is uncorrelated with X₁).
• This stepwise estimation can be used to estimate models with any number of regressors. In the 𝑘-variable model:



the OLS estimate of β₁ is computed by first regressing each of X₁ and Y on a constant and the remaining k-1 explanatory variables. The residuals from these two regressions are mean zero and uncorrelated with the remaining k-1 explanatory variables. The OLS estimator of β₁ is then estimated by regressing the residuals of 𝑌 on the residuals of X₁.

Measuring Model Fit

• The total variation in the dependent variable is called the total sum of squares (TSS), which is defined as the sum of the squared deviations of Y_i around the sample mean Y ̅  :



• Each dependent variable is decomposed into two components: the fitted value (Y ̂_i ) and the estimated residual (ϵ ̂_i), so that:



• Minimizing the squared residuals decomposes the total variation of the dependent data into two distinct components:
1. RSS – one that captures the unexplained variation (due to the error in the model), and
2. ESS – another that measures explained variation (which depends on both the estimated parameters and the variation in the explanatory variables).
• The residual sum of squares (RSS/SSR) is the sum of squared deviations of the actual (or observed) values of Y_i, from the predicted value of Y_i (i.e. Y ̂_i).



Hence RSS is simply the sum of the squares of the error terms, i.e.



• The explained sum of squares (ESS) is the sum of squared deviations of the predicted values of Y_i (i.e. (Y_i ) ̂  ), from the sample mean the Y_i’s (i.e. Y ̅).



• It is important to note that

TSS = ESS + RSS

Standard Error of Regression

• SER measures the degree of variability of the actual Y-values Y_i, with respect to the estimated Y values (i.e.. (Y_i ) ̂). It is a measure of the spread (or standard deviation) of the observations around the regression line. 



• The SER conveys the “fit” of the regression line, and the fit is better if SER is smaller.

Coefficient of Determination, R²

• R^2 is the proportion of the variance in the dependent variable that is explained by the (variation in) the independent variables. It is calculated as the ratio of the explained sum of squares to the total sum of squares



• Because OLS estimates parameters by finding the values that minimize the 𝑅𝑆𝑆, the OLS estimator also maximizes R².
• In case of linear regression with a single regressor, R²  is defined as the squared correlation between the dependent variable and the explanatory variable in a model with a single explanatory variable.

When a model has multiple explanatory variables, R² is a complicated function of the correlations among the explanatory variables and those between the explanatory variables and the dependent variable.

However, R² in a model with multiple regressors is the squared correlation between Y_i and the fitted value Y ̂_i,



This interpretation of R² provides another interpretation of the OLS estimator: The regression coefficients β ̂₁, … ,β ̂_k are chosen to produce the linear combination of X₁, … , X_k that maximizes the correlation with 𝑌.

• A model that is completely incapable of explaining the observed data has an R² of 0 (because all variation is in the residuals). A model that perfectly explains the data (so that all residuals are 0) has an R² of 1. All other models must produce values that fall between these two bounds so that R² is never negative and always less than 1.

Limitation of R²

• While R^2 is a useful method to assess model fit, it has three important limitations.
1. Adding a new variable to the model always increases the R², even if the new variable has an insignificant effect on the dependent variable. For example, if a regression model with one explanatory variable is modified to have two explanatory variables, the new R² is greater or equal to that of the original model which contained a single explanatory variable. i.e., if the original model is



and the expanded model is



then the R² of the expanded model must be greater than or equal to the R² of the original model. This is because the expanded model always has the same TSS and nearly always has a smaller 𝑅𝑆𝑆, resulting in a higher R². The only situation where adding a variable does not increase R² is if β₂=0. In that case, the RSS remains the same (as does the R²).

2. The coefficient of determination R² cannot be compared across models with different dependent variables. For example, when Y_i is always positive, it is not possible to compare the R² of a model in levels (Y) and logs (ln⁡Y_i  ).

It is also not possible to compare the  R² for two models that are logically equivalent (in the sense that both the fit of the model as measured by 𝑅𝑆𝑆 and predictions from the models are identical). This can occur when the dependent variable is transformed by adding or subtracting one or more of the explanatory variables.

3. There is no general value which can be considered as a “good” value for R². Whether a model provides a good description of the data depends on the nature of the data. For example –

An R² of 5% would be implausibly high for a model that predicts the one-day ahead return on a liquid equity index futures contract using the current value of explanatory variables.

On the other hand, an R² less than 70% would be quite low for a model for predicting the returns on a well-diversified large-cap portfolio using the contemporaneous return on the equity market (i.e., CAPM).

Adjusted R²

• As discussed, R^2  mostly increases with the increase in the number of independent variables, even if those new independent variables may not contribute in explaining the variation in the dependent variable. Hence, a high value of R^2 might be falsely indicative of a high collective explanatory power of the independent variables, but in reality, it might just reflect the impact of a large set of independent variables.
• This limitation is addressed (in a limited way) through another measure known as Adjusted R^2 (written as R ̅²  or R_a²) which adjusts the R² for the degrees of freedom (or number of independent variables). It is defined as



where

𝑛 is the number of observations in the sample, and

𝑘 is the number of explanatory variables included in the model (not including the constant term 𝛼).

• Adjusted R^2 can also be expressed as:



where the adjustment factor



Note that 𝜉 must be greater than 1 because the denominator is less than the numerator.

• Including additional explanatory variables (i.e., increasing 𝑘) always increases 𝜉. The adjusted R^2 captures the tradeoff between increasing 𝜉 and decreasing 𝑅𝑆𝑆 as models become larger.  If a model with additional explanatory variables produces a negligible decrease in the 𝑅𝑆𝑆 when compared to a base model, then the loss of a degree of freedom produces a smaller R ̅^2.
• The adjustment to the R^2 may produce negative values if a model produces an exceptionally poor fit. In most financial data applications, 𝑛 is relatively large and so the loss of a degree of freedom has little effect on R ̅^2. In large samples, the adjustment term 𝜉 is very small and so R ̅^2tends to increase even when an additional variable has little explanatory power.

Testing parameters in regression Models

• Testing a hypothesis about a single coefficient in a model with multiple regressors is identical to testing in a model with a single explanatory variable. Tests of the null hypothesis



are implemented using a 𝑡-test with sample test statistic as



where

(s.e.) ̂(β ̂_j ) is the estimated standard error of β ̂_j  .

However, the 𝑡-test is not directly applicable when testing complex hypotheses that involve more than one parameter, because the parameter estimators can be correlated. This correlation complicates extending the univariate 𝑡-test to tests of multiple parameters.

• Instead, the more common choice is an alternative called the 𝐹-test. This type of test compares the fit of the model (measured using the 𝑅𝑆𝑆) when the null hypothesis is true  relative to the fit of the model without the restriction on the parameters assumed by the null.

The F-Test-Joint hypothesis Testing

• Implementing an 𝐹-test requires estimating two models. The first is the full model that is to be tested. This model is called the unrestricted model and has an 𝑅𝑆𝑆 denoted by RSS_U. The second model, called the restricted model, imposes the null hypothesis on the unrestricted model and its 𝑅𝑆𝑆 is denoted RSS_R. The 𝐹-test compares the fit of these two models:



where

𝑞 is the number of restrictions imposed on the unrestricted model to produce the restricted model,

k_U is the number of explanatory variables in the unrestricted model, and

𝐹-test has an F_q,n-kU-1 distribution.

• 𝐹-tests can be equivalently expressed in terms of the R² from the restricted and unrestricted models. Using this alternative parameterization:



• If the restriction imposed by the null hypothesis does not meaningfully alter the fit of the model, then the two 𝑅𝑆𝑆 measures are similar, and the test statistic is small.

On the other hand, if the unrestricted model fits the data significantly better than the restricted model, then the 𝑅𝑆𝑆 from the two models should differ by a large amount so that the value of the 𝐹-test statistic is large.

A large test statistic indicates that the unrestricted model provides a superior fit and so the null hypothesis is rejected.

• Implementing an 𝐹-test requires imposing the null hypothesis on the model and then estimating the restricted model using OLS. For example, consider a test of whether CAPM, which only includes the market return as a factor, provides as good a fit as a multi-factor model that additionally includes the size and value factors.

The unrestricted model includes all three explanatory variables, so that:



where

𝑚 indicates the market (i.e., so that R_m×i  is the return to the market factor above the risk-free-rate),

𝑠 indicates size, and

𝑣 indicates value.

The null hypothesis is then:



The alternative hypothesis is that at least one of parameters is not equal to zero:



so that the null should be rejected if at least one of the coefficients is different from zero.

• In this hypothesis test, two coefficients are restricted to specific values and so q=2. The 𝐹-test is then computed by estimating both regressions, storing the two 𝑅𝑆𝑆 values, and then computing:



• Finally, if the test statistic 𝐹 is larger than the critical value of an F_2,n-4 distribution using a size of 𝛼 (e.g., 5%), then the null is rejected. If the test statistic is smaller than the critical value, then the null hypothesis is not rejected, and it is concluded that CAPM appears to be adequate in explaining the returns to the portfolio.
• Imposing the null hypothesis requires replacing the parameters with their assumed value if the null is true. Imposing the null hypothesis on the unrestricted model produces the restricted model:



which is the CAPM.

Multivariate Confidence Intervals

• The method for constructing a confidence interval for single coefficients in the multiple regression model is also the same as in the single-regressor model.
• The confidence interval for β_j can be constructed as



or