CC BY
66
ASYMMETRIC PRICE TRANSMISSION MODELING: THE IMPORTANCE OF MODEL COMPLEXITY AND THE PERFORMANCE OF THE SELECTION CRITERIA
Henry de-Graft Acquah, Researcher Department of Agricultural Economics and Extension, University of Cape Coast, Ghana
E-mail: [email protected]
ABSTRACT
Information Criteria provides an attractive basis for selecting the best model from a set of competing asymmetric price transmission models or theories. However, little is understood about the sensitivity of the model selection methods to model complexity. This study therefore fits competing asymmetric price transmission models that differ in complexity to simulated data and evaluates the ability of the model selection methods to recover the true model. The results of Monte Carlo experimentation suggest that in general BIC, CAIC and DIC were superior to AIC when the true data generating process was the standard error correction model, whereas AIC was more successful when the true model was the complex error correction model. It is also shown that the model selection methods performed better in large samples for a complex asymmetric data generating process than with a standard asymmetric data generating process. Except for complex models, AIC’s performance did not make substantial gains in recovery rates as sample size increased. The research findings demonstrate the influence of model complexity in asymmetric price transmission model comparison and selection.
KEYWORDS
Monte Carlo Simulation; Asymmetric Price Transmission; Model Selection; Model Complexity; Information Criteria; Model Recovery Rate.
Researchers analysing asymmetries in agricultural markets have developed alternative approaches to measure asymmetric price transmission processes. However, the goal of asymmetric price transmission modeling is to select one model from a set of competing models that best captures the underlying asymmetric price transmission process.
Information theory provides an attractive basis for selecting the best model from a set of competing models or theories. For instance, Acquah (2010) applied Monte Carlo simulations to investigate the performance of information criteria in identifying the true asymmetric model across various sample sizes and stochastic variance. The findings suggest that the model selection methods do point to the correct model. Though Acquah (2010) sheds light on the relative performance of the model selection methods in a Monte Carlo Experimentation but did not consider the sensitivity of the model selection methods to model complexity ( i.e. the number of asymmetric adjustment parameters). However, little is understood about the performance of the model selection methods in selecting the correct asymmetric model when the true model is complex.
Furthermore, the practice of selecting the model that best fits a particular set of data is based on the justification that the model providing the best fit is the one that most closely approximates the underlying asymmetric process. However, this justification is baseless since a complex model can provide a good fit without necessarily bearing any interpretable relationship with the underlying process. In other words, the performance of the model selection methods may be sensitive to a complex data generating process.
The purpose of this paper is to support this claim and, in so doing, demonstrate the importance of complexity in asymmetric price transmission model selection. Specifically, models differing in complexity (i.e. a standard and complex error correction data generating process) are used to investigate how the model selection methods vary in their sensitivity to model complexity and sample size.
Measuring Asymmetric Price Transmission. Granger and Lee (1989) propose an Error Correction Model (ECM) which can be specified as follows:
Ayt = &Axt + p2(y - X)t_! + £2 t £2,t~W(0,52) (1)
where y and x are price series of a marketing chain. If y and x are integrated of the order one
process, I( 1) that are cointegrated then there exist an equilibrium relationship between y and
x which is defined by an error correction term. The long run dynamics captured by the error correction term are implicitly symmetric. In order to allow for asymmetric adjustments, the error correction term can be segmented as follows:
f >.+ \(y - x)t if (y - x)t > 0 ...
(y - x)? = ^ \ ", (2)
L zero otherwise
r \_ \(y - x)t if (y - x)t < 0 ...
(y-x)t = w J \ ", (3)
L zero otherwise
The resulting asymmetric model is defined as:
tyt = Pitet + Pi (y - x)t_i +P_(y- x)__1 + e3it £3,t~N(0, S2) (4)
This specification is referred to as the Granger and Lee asymmetric model or the standard asymmetric error correction model. Asymmetry is incorporated by allowing the speed of adjustment to differ for the positive and negative components of the Error Correction Term (ECT) since the long run relationship captured by the ECT was implicitly symmetric. Symmetry in equation (4) is tested by determining whether the coefficients (P2+
and P2-) are identical (that is H0: P2+ = P2-).
Von Cramon-Taubadel and Loy (1996) applied an alternative but a more complex method to test for price asymmetry. In this methodology, asymmetries specified affects the direct impact of price increases and decreases as well as adjustments to the equilibrium level. Where Ax+ and Ax- are the positive and negative changes in xt and the remaining variables are defined as in equation (4 ).
Ayt = PIkxt + P_Ax_ + pi(y - x)t_i + p_(y - x)__i + St; £4it~N(0, S2) (5)
A formal test of the asymmetry hypothesis using the above equation is: H 0 : Pj+ = p j- andP2+=P2". In this case, a joint F-test can be used to determine
symmetry or asymmetry of the price transmission process. The conventional Houck’s model commonly used to model asymmetric price transmission can be specified as:
A y t = Pj+A x+ + P1~A x- + £ £ ~ N(0,0-1) (6),
where Ax+ and Ax- are the positive and negative changes in xt . The explanatory variable xt is generated as independent draws from normal distribution with a constant mean and a variance of one. Asymmetry is incorporated by permitting differing speeds of adjustments for the coefficients of Ax+ and Ax- in equation (6) and £ is generated as i.i.d. draws from the standard normal distribution with a sample size n. Ayt can be obtained using the values for beta, positive and negative changes in xt (i.e. Ax+ and Ax-) and the error term as specified in equation 6. Symmetric price transmission is tested by determining whether the coefficients (and p- ) are identical (i.e. H0: p/ =P1-).
Model Selection. Model selection refers to the problem of choosing the most appropriate and concise model to express a given data in an abstract fashion. It has attracted the attention of many researchers for several decades since the introduction of
Akaike’s Information Criterion (AIC), which had a fundamental effect on model selection research. Subsequently,several model selection criteria have been introduced. It comes as no surprise that many of those model selection techniques have been employed in many asymmetric price transmission empirical applications. Most of these model selection methods adjust for a variation in the number of parameters among models, essentially penalizing models with additional parameters. They include the Akaike information criterion (AIC; Akaike, 1973); Consistent Akaike information criterion (CAIC; Bozdogan, 1987); the Bayesian information criterion (BIC; Schwarz, 1978); and the Draper’s Information Criteria (DIC; Draper, 1995). The model selection methods are defined as follows:
AIC =- 2 lo g ( L ) + 2 p (7),
CAIC = -2 log(L) + p[(log n) + 1] (8),
BIC = - 2 lo g ( L ) + p lo g (n ) (9),
DIC =- 2 lo g ( L ) + p lo g ( n / 2 n ) (10),
where L refers to the likelihood under the fitted model, p is the number of parameters in the model and n is the sample size. Models that minimize the AIC, BIC, CAIC and DIC are selected. AIC differs from CAIC, BIC and DIC in the second term which now takes into account sample size n. Thus CAIC, BIC and DIC allows for asymptotic consistency.
Data Generating Process. The study uses simulated data. This study draws from the experimental design of Acquah (2010) and specifies the Standard ECM (SECM) and complex Von Cramon-Taubadel and Loy ECM (CECM) data generating process as follows and with an error variance of 1.
Ayt = 0.50Axt - 0.25(y - x) +_1 - 0.75(y - x)-_1 + £
Ayt = 0.95Ax+ + 0.20Ax+ - 0.25(y - x)+-1 - 0.75(y - x)--1 + £
The variables in the model remain as defined previously under measuring asymmetric price transmission.
RESULTS AND DISCUSSION
This section evaluates the importance of model complexity and the relative performance of the model selection criteria in recovering the true data generating process by simulating the effect of sample size and complexity ( number of asymmetric adjustment parameters) on model selection. Subsequently, the competing models that differ in complexity are fitted to the simulated data and their ability to recover the true model measured (i.e. Model Recovery Rates). The model recovery rates define the percentage of samples in which each competing model provides a better model fit than the other competing models. In this study, all recovery rates are derived using 1000 Monte Carlo simulations. Impliedly, the amount of samples in which each model fits better than the other competing models is measured out of the 1000 samples and expressed as a percentage. In this context, the values derived from each model by selection methods are derived as the arithmetic mean based on 1000 samples. For the purpose of brevity, the standard asymmetric error correction model, the complex asymmetric error correction model and the Houck’s model are denoted by SECM, CECM and HkD respectively.
In order to simulate the effects of sample size on model selection, this study considers three sample sizes ranging from small to large corresponding to 50, 150 and 500. Using an error size of 1, data is generated from the Standard Error Correction Model (SECM) specified in equation (4). The results of the Monte Carlo simulations comparing the performance of the model selection methods are displayed below in Table 1.
Generally, inspection of the recovery rates for the different model selection criteria illustrates the extent to which the true model (SECM) is recovered by each selection criteria across the different sample sizes. In the small sample size of 50, the true model was recovered at least 78.2 percent across the model selection criteria in the top part of the Table 1. At a sample size of 500, the model selection methods recovered 84.6 to 99.3 percent of the true model.
Table 1. Relative Performance of Model Selection Methods across Sample Size*
Sample Size Model Fitted
Methods CECM HKD SECM(DGP)
50 AIC 118(17%) 126 ( 4.8% ) 117 (78.2%)
BIC 126 (6.3%) 131 (11.9%) 124 (81.8%)
CAIC 129 (3.1%) 131 (16.7%) 125 (80.2%)
DIC 129 (4.6%) 132 (14.1%) 126 (81.3%)
150 AIC 402 (18.3%) 435 (0%) 401 (81.7%)
BIC 416 (2.4%) 444 (0.1%) 412 (97.5%)
CAIC 417 (0.8%) 443 (0.1%) 413 (99.1%)
DIC 419(1.5%) 445 (0.1%) 414(98.4%)
500 AIC 1396 (15.4%) 1517 (0%) 1395 (84.6%)
BIC 1417(1.6%) 1529 (0%) 1411(98.4%)
CAIC 1416 (0.7%) 1527 (0%) 1410 (99.3%)
DIC i 419 (f%) 1531 (0%) "1413 (99%)
* Based on 1000 Replications.
In comparison with the small sample recovery rates, model recoveries of the true model improved significantly when the sample size was large. Despite differences in performance among the model selection criteria, trends holding across the different criteria were evident in the simulation results. In effect, the performance of the model selection methods to select the true model (i.e. recovery rates of SECM) generally increased with increases in sample size from 50 to 500. However, two distinct patterns can also be observed in the Table 1. First, the selection criteria that account for sample size (CAIC, BIC and DIC) performed similarly to one another and their recovery rates varied strongly as a function of sample size. Second, although AIC performed well in the small samples, it did not make substantial gains in recovery rates as the sample size increased. This is not surprising given that AIC does not account for sample size. Additionally, AIC exhibit a slight tendency to select complex models across the various sample size studied, though the true data generating process was the standard error correction model.
The observed trends are consistent with previous studies on model selection. Ichikawa (1988)’s simulation results in a factor analysis indicated that the ability of AIC to select a true model rapidly increased with sample size but at larger sample sizes it continued to exhibit a slight tendency to select complex models. Similarly, Markon and Krueger (2004) reviewed existing work on factor analysis and noted that AIC performs relatively well in small samples, but is inconsistent and does not improve in performance in large samples whilst BIC in contrast appears to perform relatively poorly in small samples, but is consistent and improves in performance with sample size. Fishler et al. (2002) also investigated the performance of BIC in a factor analysis and their results suggest that BIC performs poorly at small sample sizes, but improves with increasing sample size to eventually choose the correct model with perfect probability.
The results further suggest that there was a slight tendency for DIC to outperform BIC across all large sample sizes. Similarly, Markon and Krueger (2004) noted that the DIC outperforms the BIC in a structural equation modeling framework. The tendencies of DIC to outperform BIC in the asymmetric price transmission modeling framework are better explained by the fact that the improved performance of the DIC was the motivation for its development and implementation in Draper (1995). There was also a slight tendency for
CAIC to outperform AIC across all sample sizes. This is because CAIC corrects for sample size whereas AIC fails to take into account sample size.
In order to simulate the effects of sample size and model complexity on model selection this study considers three sample sizes ranging from small to large corresponding to 50, 150 and 500. Using an error size of 1, data is generated from the Complex Asymmetric Error Correction Model (CECM) specified in equation (5). The results of the Monte Carlo simulations comparing the performance of the model selection methods are displayed below in Table 2.
Table 2. Effects of Sample Size on Model Recovery*
Sample Size Model Fitted
Methods CECM (DGP) HKD SECM
50 AIC 118 (54.5%) 126 (7.6%) 120 (37.9%)
BIC 126 (29.3%) 131 (19.6%) 127 (51.1%)
CAIC 129 (17.5%) 131 (28.9%) 128 (53.6%)
DIC 129 (23.3%) 132 (24.2%) 129 (52.5%)
150 AIC 402 (97%) 435 (0%) 412 (3%)
BIC 416 (86%) 444 (0.1%) 423 (13.9%)
CAIC 417(81.3%) 443 (0.5%) 424 (18.2%)
DIC 419 (83.6%) 445 (0.2%) 425 (16.2%)
500 AIC "1396 (100%) 1517 (0%) 1434 (0%)
BIC 1417 (99.9%) 1529 (0%) 1451 (0.1%)
CAIC 1416 (99.9%) 1527 (0%) 1450 (0.1%)
DIC 1419 (99.9%) 1531 (0%) 1453 (0.1%)
* Based on 1000 Replications
In general, trends in performance across the different model selection criteria as the sample size increases are similar to those observed when the data was simulated from the standard asymmetric ECM. The ability of the model selection methods to recover the true model (DGP) generally increased with sample size as illustrated in Table 2.
The relative performance trends of the model selection criteria illustrates that when the true model is complex, AIC persistently outperforms CAIC, BIC and DIC across all sample sizes. This was not the case when the true model was the standard asymmetric ECM. Using a small sample of 50, the top part of Table 2 indicates that AIC recovers 54.5 percent of the true data generating process whilst CAIC, BIC, and DIC recovered between 17.5 to 29.3 percent of the true model. In large samples of 500, AIC achieve full recovery of 100 percent whilst CAIC, BIC and DIC achieve 99.9 percent recovery each when the true data generating process is complex. Similarly, previous studies (Lin and Dayton, 1997) found that AIC was superior to BIC when the true model was complex in mixture models. Gagne and Dayton (2002) also observed that AIC was more successful when the true model was relatively complex in multiple regression analysis.
An important point is that comparatively, the model selection methods performed better when the true asymmetric data generating is relatively complex (CECM) and the sample size is large than when the true data generating process is the standard error correction model (SECM) and the sample size is large. This is noted when the recovery rates of Table 1 are compared with Table 2 under sample size of 500. For instance, under a sample size of 500 the model selection methods achieve at least 99.9 percent recovery of the true model when the data generating process is complex but achieves at least 84.6 percent recovery when the true model is the standard asymmetric error correction model.
The foregoing discussions point to the fact that another factor that may influence the performance of the model selection methods is model complexity or the number of asymmetric adjustment parameters. In large samples, the model selection methods performed better when the true asymmetric data generating process is the complex error correction model.
In summary, larger sample sizes improve the ability to make correct inferences about the true asymmetric price transmission model. This research notes that an important factor
that influences the performance of the model selection criteria in addition to sample size is model complexity (i.e. number of asymmetric adjustment parameters or the number of informative variables in the model). Intuitively, the increase in model recovery of the true model can also be interpreted as due to an increase in asymmetric information provided by the additional variables or additional asymmetric adjustment parameters.
CONCLUSION
The main point of this article is that the performance of model selection methods may be influenced by model complexity in asymmetric price transmission modeling framework. This study therefore fits competing asymmetric price transmission models that differ in complexity to simulated data and evaluates the ability of the model selection methods to recover the true model. Monte Carlo simulation results, suggest that the ability of the model selection methods to select the true model generally increased with increases in sample size from small to large. AIC is superior to CAIC, BIC and DIC when the true model is the complex error correction model. An important point is that in large samples, the model selection methods performed better when the true asymmetric data generating process is the complex Error Correction Model (CECM) as compared to the standard ECM data generating process. In effect, this research suggests that model complexity (i.e. number of asymmetric adjustment parameters or the number of informative variables in the model) influences model recovery in asymmetric price transmission modeling.
REFERENCES
[1] Acquah, H.D.(2010). Comparison of Akaike information criteria (AIC) and Bayesian information criteria (BIC) in selection of asymmetric price relationships. Journal of Development and Agricultural Economics Vol. 2(1) pp.001-006.
[2] Akaike, H. (1973). Information Theory and an Extension of the Maximum Likelihood Principle. B. N. Petrov and F. Csaki (eds.) 2nd International Symposium on Information Theory: 267-81. Budapest: Akademiai Kiado.
[3] Bozdogan, H. (1987). Model Selection and Akaike’s Information Criterion (AIC): The General Theory and Its Analytical Extensions, 52, No. 3, 345-370.
[4] Draper, D. (1995). Assessment and Propagation of Model Uncertainty. Journal of Royal Statistical Society. Series B (Methodological), 57, No. 1 (1995), pp 45-97.
[5] Fishler, E., Grosmann, M., and Messer, H. (2002). Detection of signals by information theoretic criteria: general asymptotic performance analysis. IEEE Trans. Signal Process, 50, pp.1027-1036.
[6] Gagne, P. and Dayton, C.M. (2002). Best Regression model using information criteria. Journal of Modern Applied Statistical Methods, 1, pp.497-488.
[7] Granger, C.W. J. and Lee, T.H. (1989). Investigation of Production, Sales and Inventory Relationships using Multicointegration and non-symmetric Error Correction Models, Journal of Applied Econometrics, 4, pp. 135- 159.
[8] Ichikawa, M. (1988). Empirical assessments of AIC procedure for model selection in factor analysis. Behaviormetrika , 24, pp. 33-40.
[9] Lin, T. H., and Dayton, C. M. (1997). Model selection information criteria for nonnested latent class models. J. Educat. Behav. Stat., 22, pp. 249-264.
[10] Markon, K. E. and Krueger, R. F. (2004). An Empirical Comparison of Information-Theoretic Selection Criteria for Multivariate Behaviour Genetic Models. Behaviour Genetics, 34, (6), pp.593- 609.
[11] Schwarz, G. (1978) “Estimating the Dimension of a Model.” Annals of Statistics, 6, pp. 461- 464.
[12] Von Cramon-Taubadel, S. and Loy, J.-P. (1996). Price Asymmetry in the international Wheat Market: Comment. Canadian Journal of Agricultural Economics, 44, pp. 311317.
