ARIMA modelling in R: minimizing AIC criterion

In earlier posts I talked about the arima() function in R. In particular, I gave some examples showing that it does some funny things when calculating residuals and I also mentioned that three different estimation methods could be specified using the method=() argument. Again, the estimation methods are "CSS-ML", "ML" and "CSS".

One way of specifying an ARIMA model is to choose the specification that minimizes Aikake's information criterion. This process, which I'll refer to as specification search, is data-mining. As far as I am aware, there is an R function called step() that can automatically identify the ARIMA model with the smallest AIC value for a given series of data.

Personally, I am not a fan of this modelling procedure as I have preference for following the Box-Jenkins methodology when modelling univariate time-series. Part of the BJ methodology relies on investigating the estimated autocorrelation function and partial autocorrelation function in order to try to identify possible models worth entertaining as candidate models.

Let me try to justify my preference using a single argument.

One has to remember that there are both producers and consumers of econometric models. The producer builds the model with some purpose, such as for forecasting or to guide policy decisions, and the consumer of the model uses it if he or she believes that the model is valid.

In my opinion, a producer who follows the Box-Jenkins methodology, or some other sound systematic procedure that can be explained to and understood by the consumer, should be in a position with decent chances of convincing the consumer that the final model is valid.

On the other hand, a producer who builds models by simply clicking a button and waiting for the result to pop up, may have trouble convincing consumers that the model is valid. This second kind of producer may find that he or she has very few customers willing to do business (even if the model actually does a good job at representing the data generating mechanism)! I wouldn't argue that an R function like step() is useless or should be avoided, but it probably shouldn't be at the helm during the model building procedure.

Next, I want to discuss a little problem in R that is not related to step().

The arima() function reports an AIC value as part of its output alongside estimated coefficients and standard errors as is standard for any popular statistical package. Now, caution must be taken here. This AIC value is only valid if the estimation method is specified as "ML". So, this AIC value is only useful to the analyst if their model is of the form: arima(x, c(p,d,q), method="ML"). The arima help page tells us the following about the reported AIC value:

aic: the AIC value corresponding to the log-likelihood. Only valid for 'method = "ML"' fits.

There has been some confusion about this and it has been raised in a discussion on the R mailing list, so it's worth pointing it out. I have also come across these University Lecture Notes on Model Selection in R Using AIC which happen to contain errors; the author estimates ARIMA models using the default ("CSS-ML") method and then uses the reported AIC values to select a final model despite the fact that the AIC values are only valid when method="ML"!

There are two key points here. The first is that if a model builder is going to use R for data analysis, he or she should carefully read the documentation. Secondly, model builders should recognize that information about their models, and quite possibly their model building procedure, may have to be communicated to consumers of their models. It's worth noting that consumers - including managers and politicians - may not have a great deal of technical knowledge of model building or even request a technical explanation of the modelling procedure, but they may put more trust in the models if they hear more than "the computer told me... this one".