On the Modified Almost Unbiased Ridge Estimator in Linear Regression Model

In linear regression model, the problem of multicollinearity occurs in the existence of linear dependencies between the explanatory variables. It is well known that ordinary least squares (OLS) estimation is the preferred method for estimating the parameters in linear regression model since it gives an unbiased estimator with minimum variance [Johnson and Wichern (2007)]. However, when the problem of multicollinearity exists, the ordinary least squares estimator (OLSE) will be unstable with high variance [Vinod and Ullah (1981)].

To circumvent the multicollinearity problem in linear regression, many popular estimators of biased estimation methods have been introduced. These estimators include the Stein estimator by Stein (1956), the principal component estimator by Massy (1965), the ridge estimator by Hoerl and Kennard (1970), the Liu estimator by Liu (1993), and the Liu-type estimator by Liu (2003). Also, two versions of the two-parameter estimator by Özkale and Kaçiranlar (2007) and Yang and Chang (2010), the ridge-type estimator by Kibria and Lukman (2020), the modified one-parameter Liu estimator by Lukman et al. (2020), the generalized Kibria-Lukman estimator by Dawoud et al. (2022), and the new two-parameter estimator by Owolabi et al. (2022) were introduced for the linear regression model.

Another set of suggested estimators for dealing with multicollinearity are the almost unbiased estimators. For the linear regression model, the almost unbiased ridge estimator (AURE) by Singh et al. (1986), the almost unbiased Liu estimator by Alheety and Kibria (2009), the almost unbiased ridge-type principal component estimator and the almost unbiased Liu-type principal component estimator by Li and Yang (2014), the modified almost unbiased Liu estimator by Armairajan and Wijekoon (2017), and the almost unbiased Liu principal component estimator by Ahmed et al. (2021) were presented.

In this paper, a new estimator namely modified almost unbiased ridge estimator (MAURE) is proposed for overcoming the effects of multicollinearity in linear regression.

This paper is structured as follows. In Section 2, for linear regression model, the OLSE, RE, and AURE and their statistical characteristics are discussed. In Section 3, the new estimator, MAURE is presented. In Section 4, the superiority of the MAURE over the OLSE, RE, AURE based on the criteria of matrix mean squared error (MMSE) and squared bias (SB) is provided. In Section 5, selection of the biasing parameter is given. In Section 6, a study of Monte Carlo simulation is performed to compare the performance of the new estimator, MAURE with other considered estimators: OLSE, RE, and AURE in terms of scalar mean squared error (SMSE). Also, a real data example is included in Section 7. Finally, in Section 8, the conclusion is given.

1. Statistical Methodology

The linear regression model has the following standard form:

                                                                                             (1)

where  is a  vector of response variable,  is a  full rank matrix of  observations on   explanatory variables,  is a  vector of unknown regression coefficients, and  is a  vector of random error with mean vector  and covariance matrix ,  is an  identity matrix.

By considering the OLS method for estimating the regression coefficients, the OLSE of   can be obtained as follows:

                                                                                     (2)

where .

The  is an unbiased estimator and its covariance matrix is given as

                                           .                                  (3)

The MMSE of  is as follows:

                           (4)

where denotes the bias vector.

Also, the SMSE of  is given as follows:

                                                      (5)

where  are the eigenvalues of V.

When the model (1) is suffering from multicollinearity because of correlated explanatory variables, the OLSE becomes biased and has high variance, which leads to unstable parameters estimates.

For tackling the effect of multicollinearity in linear regression model, Hoerl and Kennard (1970) introduced the RE which is defined as follows:

                                                                      (6)

where , and  is the biasing parameter called the ridge parameter, .

The following statistical characteristics belong to the RE:

                                                                                       (7)

                                          B                                     (8)

                                                       (say)

                                      Cov                                (9)

                                      (10)

and

                                                           (11)

where  is the th element of  and  is an orthogonal matrix defined as .

Based on the following definition, the AURE is proposed by Singh et al. (1986) in linear regression model.

Definition 1. Assume that  is a biased estimator of  and  is the bias vector of . Then, the almost unbiased estimator of  is . [Kadiyala (1984)]

The AURE is defined as follows:

                                                                                 (12)

where .

The statistical characteristics of the AURE are given as follows:

                                                                                  (13)

                                      B                                    (14)

                                                       (say)

                                  Cov                                (15)

                       (16)

and

                                                   (17)

1. The New Estimator

In this section, a new almost unbiased estimator, MAURE is proposed based on the RE and AURE as follows:

                                                                                   (18)

The new estimator, MAURE has the following characteristics:

                                                                          (19)

                    B                                              (20)

                                       (say)

                              Cov                       (21)

             (22)

and

                                              (23)

1. Superiority of the MAURE

Based on the MMSE and SB, the following comparisons are performed.

4.1 MMSE comparisons

      When the comparison between any two estimators and  of  is performed by the criterion of MMSE, the following Lemmas can be used:

Lemma 1. Suppose that  and are  matrices such that , and . Then,  if and only if . [Rao et al. (2008)]

Lemma 2. For two estimators and  of , suppose that  is positive definite. Then,  is non-negative definite if and only if , where  and  denote the bias vectors of and  respectively. [Trenkler and Toutenburg (1990)]

The following comparisons are performed between the new estimator, MAURE and the OLSE, RE, and AURE by the MMSE criterion.

3.1.1        The MMSE comparison between the OLSE and MAURE

Using (4) and (22), the MMSE difference of the OLSE and MAURE is given by

            ( 24)

where  .

Then, according to Lemma 2, Theorem 1 is stated as follows:

Theorem 1. When , the MAURE is superior to OLSE based on  the MMSE criterion if and only if  .

Proof: Since  and  are positive definite matrices, then,  according to Lemma 1 if and only if . Consequently, from Lemma 2,
 is a non-negative definite matrix if and only if .

3.1.2        The MMSE comparison between the RE and MAURE

Using (10) and (22), the MMSE difference of the RE and MAURE is as follows:

                                                               (25)

where  , and .

Then, Theorem 2 is stated as follows:

Theorem 2. When , the MAURE is superior to RE based on the MMSE criterion if and only if  .

Proof: Since  and  are positive definite matrices, then,  according to Lemma 1 if and only if . Consequently, from Lemma 2,  is a non-negative definite matrix if and only if

.

3.1.3        The MMSE comparison between the AURE and MAURE

Using (16) and (22), the MMSE difference of the AURE and MAURE is given as follows:

                                                               (26)

where  .

Then, Theorem 3 is stated as follows:

Theorem 3. When , the MAURE is superior to AURE based on the MMSE criterion if and only if .

Proof: Since  and  are positive definite matrices, then, from Lemma 1,  if and only if . Consequently, by Lemma 2,  is a non-negative definite matrix if and only if

.

3.2   Squared bias comparisons

Based on the SB criterion, the following comparisons are discussed between the MAURE and the RE and AURE.

3.2.1        The SB comparison between the RE and MAURE

From (8) and (20), the difference of SB between the RE and MAURE is given as follows:

                                      .          (27)

Then, Theorem 4 is given as follows:

Theorem 4. Under SB criterion,  for .

Proof: Since, the difference of SB between the RE and MAURE is

where .

Therefore, for , .

3.2.2        The SB comparison between the AURE and MAURE

From (14) and (20), the difference of SB between the AURE and MAURE is given by

                                      .          (28)

Then, Theorem 5 is stated as follows:

Theorem 5. Under SB criterion,  for .

Proof: Since, the difference of SB between the AURE and MAURE is

where .

Therefore, for , .

1. Selection of  Biasing Parameter

For determining the biasing parameter , many methods have been proposed. Some of the more popular of these methods are considered as follows:

Hoerl and Kennard (1970) suggested the following estimator of :

                                                                                            (29)

where , and  is the maximum value of .

Hoerl et al. (1975) suggested an alternative estimator as follows:

                                                     .                                        (30)

In Khalaf and Shukur (2005), the following estimator was proposed:

                                          .                            (31)

In addition, Alkhamisi et al. (2006) proposed an estimator of  as follows:

                                       (32)

Further, Muniz and Kibria (2009) suggested the following estimator:

                                  .                            (33)

1. Monte Carlo Simulation Study

A Monte Carlo simulation study is conducted using the R 4. 0. 3 programme to evaluate the performance of the new estimator, MAURE in the linear regression model against the OLSE, RE, and AURE in the sense of SMSE.

The explanatory variables are generated following McDonald and Galarneau (1975) as follows:

                (34)

where represents the correlation degree between any two explanatory variables, and  is the independent pseudo-random variable.

The response variable is obtained as follows:

                (35)

where , and the coefficient  is chosen so that , and , where  following Kibria (2003).

For  and  explanatory variables, different values of  and , different sample sizes , and , and different values of error variance  and  are considered.

For a combination of the values of , and , the generated data are repeated  times and the SMSE is computed as follows:

                                                             (36)

where  is the estimated value of  by any estimator in the rth replication.

The estimated SMSE values for all estimators with  and the combination of , and  respectively are given in Tables 1-9.

As Tables 1-9 show, the values of the estimated SMSE of all estimators: OLSE, RE, AURE, and MAURE increase as  increases. Also, regarding , it is clear that there is an increase in the estimated values of SMSE for all estimators when  increases. Regarding the number of explanatory variables , there is an increase in the estimated SMSE of all estimators as  increases. While, regarding the sample size , the estimated values of SMSE decrease for all estimators as  increases. Additionally, the new estimator, MAURE has the best performance among the OLSE, RE, and AURE in all cases in the sense of SMSE, and the RE has better performance than the AURE. Furthermore, for different selection formulas of  estimators for RE, AURE, and MAURE, the  outperforms the other estimators as it has the lowest SMSE values. Moreover, the MAURE with  achieves the best performance compared to the RE, AURE, and OLSE in terms of SMSE.

Table 1. Estimated SMSE values when  and

Table 2. Estimated SMSE values when  and

Table 3. Estimated SMSE values when  and

Table 4. Estimated SMSE values when  and

Table 5. Estimated SMSE values when  and

Table 6. Estimated SMSE values when  and

Table 7. Estimated SMSE values when  and

Table 8. Estimated SMSE values when  and

Table 9. Estimated SMSE values when  and

1. Real Data Example

In this section, a real data set of Total National Research and Development Expenditures is considered as a percent of Gross National Product by Country: 1972-1986 according to Gruber (1998) and later analyzed by Li and Yang (2011) and Arumairajan and Wijekoon (2017). This data set involves  observations. The response variable  as well as the four explanatory variables  and  are defined as follows: y is the percentage spent by the United States,  is the percent spent by France, is the percent spent by West Germany,  is the percent spent by Japan, and  is the percent spent by the former Soviet Union.

The statistic value of the Shapiro-Wilk normality test equals to  with , which indicates the normality of  the response variable  at  significance level.

The correlation matrix of the explanatory variables is as follows:

It is obvious that there are correlations greater than  between , , and  which indicates the existence of high relationship between the explanatory variables.

Also, for checking the presence of multicollinearity, the condition number () of the data is computed by

                                     (37)

where  and  are the largest and smallest eigenvalues of  respectively.

Since the eigenvalues of  matrix are obtained as , , , and , the value of is  shows the presence of severe multicollinearity in this data.

The estimated coefficients and SMSE of the OLSE, RE, AURE, and MAURE for  are given in Table 10.

Table 10. Estimated coefficients and SMSE of the estimators

Table 10 shows that the new estimator, MAURE has the smallest SMSE values than other estimators for all values of .

1. Conclusion

In this paper, for overcoming multicollinearity in linear regression model, a new estimator, MAURE was presented with its statistical characteristics. By considering the criteria of MMSE and SB, the comparisons between the new estimator, MAURE and the OLSE, RE, and AURE were provided. Further, a study of Monte Carlo simulation and a real data example were conducted to evaluate the performance of the MAURE versus the other existing estimators under the SMSE criterion. The results showed the superiority of the new estimator, MAURE over all existing estimators in terms of SMSE. So, the MAURE can be safely used when multicollinearity exists in a linear regression model.