Average Economic Regression

There's a sort of textbook or stylized view of econometrics that is often taught to students. It's quite simplified and is referred to as average economic regression (AER) by Christopher L. Gilbert in his paper Professor Hendry's Econometric Methodoogy. It goes a little like this (in my own words):

To build an econometric model we first consider what economic theory has to say. Having done that, we throw some variables into a single equation, add an error term, and then all we have to do is obtain estimates for the beta coefficients. Job done. Well, nearly. Before we finish, we must solve some potential problems, such as low Durbin-Watson statistics, wrong signs, and insignificant coefficients. To deal with these problems, we chuck some variables out of the model and possibly apply the Cochrane-Orcutt transformation to help deal with serial correlation in the error term. One eventually emerges from the econometrics basement serving on a golden platter the final model that has all of the correct signs, with t-values greater than 2, and an R-squared that is acceptably high for the model builder.

Gilbert explains what's wrong with this. He says:

The problem with the AER approach is that we are using econometrics to illustrate the theories which we believe independently. The alternative might be to use econometrics to discover which views of the economy (or market) are tenable and to test, scientifically, the rival views. Our normal concept of science tends to be based on the natural, and in particular, the physical sciences, and does not obviously translate to the social sciences. This alternative programme for econometrics raises the question of what is required for economics to be scientific. 

Professor Edward Leamer also views textbook econometrics with some skepticism. He says:

Theoretical econometricians have interpreted scientific objectivity to mean that an economist must identify exactly the variables in the model, the functional form, and the distribution of the errors. Given these assumptions, and given a data set, the econometric method produces an objective inference from a data set, unencumbered by the subjective opinions of the researcher. 

This advice could be treated as ludicrous, except that it fills all the econometric textbooks. Fortunately, it is ignored by applied econometricians. The econometric art as it is practiced at the computer terminal involves fitting many, perhaps thousands, of statistical models. One or several that the researcher finds pleasing are selected for reporting purposes. This searching for a model is often well intentioned, but there can be no doubt that such a specification search invalidates the traditional theories of inference. The concepts of unbiasedness, consistency, efficiency, maximum likelihood estimation, in fact, all the concepts of traditional theory, utterly lose their meaning by the time an applied researcher pulls from the bramble of computer output the one thorn of a model he likes best, the one he chooses to portray as a rose. The consuming public is hardly fooled by this chicanery. The econometrician's shabby art is humorously and disparagingly labelled 'data mining', 'fishing', 'grubbing', 'number crunching'. A joke evokes the Inquisition: 'If you torture the data long enough, Nature will confess' (Coase). Another suggests methodological fickleness: 'Econometricians, like artists, tend to fall in love with their models' (wag unknown). Or how about: 'There are two things you are better off not watching in the making: sausages and econometric estimates'. 

This is a sad and decidedly unscientific state of affairs we find ourselves in. Hardly anyone takes data analyses seriously. Or perhaps more accurately, hardly anyone takes anyone else's data analyses seriously. Like elaborately plumed birds who have long since lost the ability to procreate but not the desire, we preen and strut and display our t-values.

My interpretation of the relationship between economic theory and econometrics tells me that a distinction should be drawn between what economic theory tells us and what econometrics can tell us. To lay the foundations of my argument, let me introduce the notion of a formal implication via a passage from Bertrand Russell's Principles of Mathematics:

[But] since the growth of non-Euclidean Geometry, it has appeared that pure mathematics has no concern with the question whether the axioms and propositions of Euclid hold of actual space or not: this is a question for applied mathematics, to be decided, so far as any decision is possible, by experiment and observation. What pure mathematics asserts is merely that the Euclidean propositions follow from the Euclidean axioms - i.e. it asserts an implication: any space which has such and such properties has also such and such other properties. Thus, as dealt with in pure mathematics, the Euclidean and non-Euclidean Geometries are equally true: in each nothing is affirmed except implications. All propositions as to what actually exists, like the space we live in, belong to experimental or empirical science, not to mathematics; when they belong to applied mathematics, they arise from giving to one or more of the variables in a proposition of pure mathematics some constant value satisfying the hypothesis, and thus enabling us, for that value of the variable, actually to assert both hypothesis and consequent instead of asserting merely the implication. We assert always in mathematics that if a certain assertion p is true of any entity x, or of any set of entities x, y, z, . . ., then some other assertion q is true of those entities; but we do not assert either p or q separately of our entities. We assert a relation between the assertions p and q, which I shall call formal implication. (Chapter 1, p.5)

Although Bertrand Russell is addressing the nature and bounds of pure mathematics and applied mathematics, an analogous story could be told of pure economics and econometrics.

Like pure mathematics, what pure economics deals with are sets of axioms and from those axioms follow propositions. Those familiar with Andreu Mas-Colell et. al's Microeconomic Theory (or Euclid's Elements for that matter!) will no doubt be comfortable with this sort of axiomatic system and the use of deductive logic.

The pure economists' concern is to come up with axioms and ensure that the progression from axioms to propositions will be carried out in a logical manner - in the parlance of formal logic, the inferential steps should be "watertight".

So, with one set of axioms we have a set of accompanying propositions and formal implications. As long as the axiomatic system is developed in a logical manner, it may be said to be true, albeit within its own scope - or space. Another set of axioms will generate another set of accompanying propositions and formal implications. Some of these may intersect the sets associated with the first axiomatic system. Indeed, one axiomatic system may encompass another one. Again, this second axiomatic system may be said to be true within its space.

That's what axiomatic systems in pure economics tell us. They assert implications that hold within a given space, or rather, in a particular model of an economy. Whether or not these implications conform to actual space - or actual economies - is another thing. The question to be asked then is why should an econometric model which aims to help us discover facts about actual economies be constrained to corroborate with an axiomatic system which may not necessarily hold true or provide a satisfactory description of actual economic phenomena? It probably shouldn't be that way because then we're using econometrics to illustrate rather than to discover - a reiteration of Gilbert.

Lawrence R. Klein, puts it nicely in Chapter 1 of his Economic Fluctuations in the United States (1921–1941). He says:

The purpose in building econometric models is to describe the way in which the system actually operates. [...] We want also to discovered the best possible theory or theories which explain the fluctuations that we observe. If we know the quantitative characteristics of the economic system, we shall be able to forecast with a specified level of probability the course of certain economic magnitudes such as employment, output, or income; and we shall also be able to forecast with a specified level of probability the effect upon the system of various economic policies. In the course of the search for models which are suitable for our purposes of forecasting and making policy recommendations, we shall inevitably have to consider several alternative theories as admissible hypotheses. The acceptance or  rejection of these hypotheses in the course of our search for truth will be our contribution to the problem of testing business-cycle theories. This is an important problem for which contributions are needed, because too often writers have made bold statements about the operation of the system without examining the factual data to determine whether or not their statements are true.* As a result we have had  continuing economic fluctuations and continuing publication of business cycle theories.

[*the emphasis is my own and does not appear in the original source]

It seems that an econometrican who builds models that are forced to conform to theories derived from an axiomatic system of pure economics is going astray from finding out about how actual economies work. Think for moment that Platonic solids and polygons from Euclid's Geometry can be described as having such and such properties and that's a useful exercise in itself, but believing that such solids and shapes occur in reality, is perhaps, not so wise. Go find me a perfect triangle!

Non-Euclidean Geometries, which permit shapes such as triangles whose angles sum to less than 180 degrees (hyperbolic geometry) or to greater than 180 degrees (elliptic geometry), may conform more closely to actual space than what Euclid's system tells us. As an example, an implication of Einstein's theory of general relativity is that physical space itself is not Euclidean, and Euclidean space is a good approximation for it only where the gravitational field is weak.* Importantly, each geometry, Euclidean or non-Euclidean, is true within its own framework, but their implications may hold differently in actual space. This is just a restatement of what Bertrand Russell said.

Coming back to the average economic regression and a situation in which a coefficient turns out to have the wrong sign. Well, if that's what the data tells us, maybe that's just the way it is for this particular economy - data and observation assumed to be accurate, i.e. assuming no identification problem. If economic theory suggests the sign of the coefficient should be otherwise, then the implications from that axiomatic system may not hold for the actual economy under scrutiny. If this is the case, it should follow that a new axiomatic system is built (which encompasses the old one) to take into account the new information we've discovered from econometrics. This shouldn't be thought of as a step back for economic theory, but an opportunity to advance the subject! Again, Newton didn't overturn Kepler, and Einstein didn't overturn Newton; each encompassed the other.

Similar to how applied mathematics deals with informing us about actual space, econometrics ought to aim at discovering information about economies that actually exist - freed from any bounds that pure economics may set upon itself. A man of considerable stature in economics once said: when the facts change, I change my mind, what do you do, sir?

Well, it's a bit like that. If revisions (or extensions) need to be made to economic theory, be open to making them. Now, it's not a case of scrap what we've already got. It's more so a case of expanding on what some great minds have already developed and kindly handed down to us. Nor is it a case that we're permitted to run any old nonsensical regressions. Sensible regressions that can be explained in terms of some (possibly hitherto unknown) axiomatic system are what we should be seeking to build.

I'll conclude the argument with four points.

• Pure mathematics illustrates. 
• Pure economics illustrates. 
• Applied mathematics discovers. 
• Econometrics discovers.

ARIMA modelling in R: minimizing AIC criterion

In earlier posts I talked about the arima() function in R. In particular, I gave some examples showing that it does some funny things when calculating residuals and I also mentioned that three different estimation methods could be specified using the method=() argument. Again, the estimation methods are "CSS-ML", "ML" and "CSS".

One way of specifying an ARIMA model is to choose the specification that minimizes Aikake's information criterion. This process, which I'll refer to as specification search, is data-mining. As far as I am aware, there is an R function called step() that can automatically identify the ARIMA model with the smallest AIC value for a given series of data.

Personally, I am not a fan of this modelling procedure as I have preference for following the Box-Jenkins methodology when modelling univariate time-series. Part of the BJ methodology relies on investigating the estimated autocorrelation function and partial autocorrelation function in order to try to identify possible models worth entertaining as candidate models.

Let me try to justify my preference using a single argument.

One has to remember that there are both producers and consumers of econometric models. The producer builds the model with some purpose, such as for forecasting or to guide policy decisions, and the consumer of the model uses it if he or she believes that the model is valid.

In my opinion, a producer who follows the Box-Jenkins methodology, or some other sound systematic procedure that can be explained to and understood by the consumer, should be in a position with decent chances of convincing the consumer that the final model is valid.

On the other hand, a producer who builds models by simply clicking a button and waiting for the result to pop up, may have trouble convincing consumers that the model is valid. This second kind of producer may find that he or she has very few customers willing to do business (even if the model actually does a good job at representing the data generating mechanism)! I wouldn't argue that an R function like step() is useless or should be avoided, but it probably shouldn't be at the helm during the model building procedure.

Next, I want to discuss a little problem in R that is not related to step().

The arima() function reports an AIC value as part of its output alongside estimated coefficients and standard errors as is standard for any popular statistical package. Now, caution must be taken here. This AIC value is only valid if the estimation method is specified as "ML". So, this AIC value is only useful to the analyst if their model is of the form: arima(x, c(p,d,q), method="ML"). The arima help page tells us the following about the reported AIC value:

aic: the AIC value corresponding to the log-likelihood. Only valid for 'method = "ML"' fits.

There has been some confusion about this and it has been raised in a discussion on the R mailing list, so it's worth pointing it out. I have also come across these University Lecture Notes on Model Selection in R Using AIC which happen to contain errors; the author estimates ARIMA models using the default ("CSS-ML") method and then uses the reported AIC values to select a final model despite the fact that the AIC values are only valid when method="ML"!

There are two key points here. The first is that if a model builder is going to use R for data analysis, he or she should carefully read the documentation. Secondly, model builders should recognize that information about their models, and quite possibly their model building procedure, may have to be communicated to consumers of their models. It's worth noting that consumers - including managers and politicians - may not have a great deal of technical knowledge of model building or even request a technical explanation of the modelling procedure, but they may put more trust in the models if they hear more than "the computer told me... this one".