Vector Algebra (complete proofs)

Open up a decent undergraduate Linear Algebra textbook (my favourites are Fraleigh & Beauregard and Strang) and you'll quickly come across definitions of vector addition, vector subtraction, and scalar multiplication. A list of the properties of these vector operations usually follow and while one or two proofs are given, the rest are left up to the reader as exercises. That shouldn't be too much of a problem to any mathematics student.

You will, however, encounter such proofs - again left as exercises - in graduate textbooks for economics and finance; one such example being Harrison & Waldron. They also appear in this Advanced Engineering Mathematics textbook that I happened to pick up at a bargain price in Oxfam!

Familiarity with vector algebra is pretty essential to anyone studying a numerical subject and up to a certain level students can advance in their chosen field without being able to write proofs.

There comes a time, though, when it is necessary for the student to be able to understand and write proofs. That's got to be a nightmare for anyone who has not yet grasped how to do it. Even worse, there's probably no course designed to teach it to someone for the first time. Gone are the days of the trivium.

Mathematics students at Virginia Commonwealth University are quite the lucky ones because they have a whole course designed to teach them how to write mathematical proofs. The course is called Introduction to Mathematical Reasoning and the book that's used for the course can be found here (it's an excellent textbook!).

Students of economics are, perhaps, not so lucky. Some may already lack math skills (or rather, the confidence in their math ability), and I've not really come across any material that aims to teach them how to write proofs. It's kind of like they're left to their own devices to try and work it out for themselves. A tad unfair - remember, some 3rd year math students take a whole course in this art!

Now, to write proofs for the properties of vector algebra is very easy and as basic as you can get, but since someone out there (maybe a student of economics) might be having trouble with writing them, I've written up complete proofs and added comments to them that explain how each step in the proof was made. I think they're quite instructive and definitely more detailed than any I've seen in a variety of textbooks. Feedback welcome.

Note: proofs can be written for the properties of matrix algebra in a similar fashion. Following the tradition, I'll leave them as an exercise. Follow the technique I used for vector algebra and you'll have no problem.

The notion of "encompassing"

Christopher L. Gilbert in his paper Professor Hendry's Econometric Methodology gives an outline of six criteria for model selection. These six criteria originate from Hendry and Richard's paper The Econometric Analysis of Economic Time Series. The six criteria are:

1. Models must be data admissible.
2. Models must be consistent with theory.
3. Satisfactory models should have regressors that are (at least) weakly exogenous. 
4. Satisfactory models must  exhibit parameter constancy.
5. A satisfactory model must be data coherent.
6. A satisfactory model should encompass a wide range of rival models.

A model that satisfies all six of these criteria is said to be congruent with the data.

The sixth criterion about encompassing is the one which I want to focus on in this post. Gilbert explains encompassing in the following paragraph:

One model may be said to encompass a second model if it can explain the second model's results. A good model should not only explain the data, but should also explain the successes and the failures of rival models in accounting for the same data. The Special Theory of Relativity encompasses Newtonian mechanics (which could not explain the Michelson-Morley results), but is itself encompassed by the General Theory of Relativity (as it cannot explain the visibility of the planet Mercury behind the Sun during a solar eclipse).

To bring the reference to physics up to date, the search is now on for theoretical physicists to find a Theory of Everything that encompasses both General Relativity and Quantum Mechanics. One contender for the job is a theory fathered by Professor Leonard Susskind at Stanford University called string theory.

A basic example of encompassing would be taking a model with a variable whose coefficient is set equal to one and then developing a second model in which the coefficient is allowed to take on a range of values, say, between zero and one. In this case, the second model encompasses the first model because when the coefficient in the second model happens to equal to one, it reduces to the first model. The basic idea is that it's possible to introduce parameters into existing equations to give them a little bit more of a kick.

University of Yale Professor Charles Bailyn has spent some of his lecture time discussing scientific revolutions and the notion of encompassing - see this Astrophysics lecture. He does an excellent job at explaining it, so it's worth repeating. In the lecture, he said:

But Einstein didn't say that Newton was wrong. That wasn't what happened. That wasn't how a revolution works. You revolutionize science only if the new theory encompasses what was previously known--the previous theories. And this is this business of the parameter, gamma, where under appropriate circumstances, gamma is one. And under those circumstances, you reproduce the theory that was already in existence. And so, you know, there's been hundreds and hundreds of years of Newtonian physics, all of which is basically right, one way or another. So whatever your new idea is has to encompass all the things that were right about the old idea. And so, you don't overturn Newton. What you do is, you demonstrate that Newton's theory is only a part of a bigger theory. And then, the bigger theory is the new theory and it encompasses new kinds of data and explains new kinds of things, while simultaneously not overturning all the things that you know are right about the old theory. 

And this is the thing that the people who wish they were Einstein, but aren't, don't understand. And so, anytime you see something where you're--where they're going to revolutionize all of science, it's certainly wrong. Because there's a large fraction of science in current existence that is correct. And you have to understand that. And you have to encompass it and go beyond that--but it's not going to get overturned. The planets aren't going to stop going in Keplerian orbits because somebody has a new idea. And so, it has to encompass the old idea.

Note the reference to the parameter gamma which equals one in a Newtonian framework, whereas in General Relativity, gamma can be some other number, a number other than one. The same idea of introducing new parameters into old equations can, of course, be applied to economics.

History shows us that encompassing has played an important role (and will continue to do so) in advancing our understanding of physics and of the world in general. Furthermore, there's little doubt that encompassing is important for the future of economics and it's reassuring to me that it's recognized, voiced, and supported by those at the top of the field.

Unfortunately, however, it's easy to be critical of economics and econometrics and there will always be cranks who'll stand up to say that everything we've done up to now has been wrong!

Average Economic Regression

There's a sort of textbook or stylized view of econometrics that is often taught to students. It's quite simplified and is referred to as average economic regression (AER) by Christopher L. Gilbert in his paper Professor Hendry's Econometric Methodoogy. It goes a little like this (in my own words):

To build an econometric model we first consider what economic theory has to say. Having done that, we throw some variables into a single equation, add an error term, and then all we have to do is obtain estimates for the beta coefficients. Job done. Well, nearly. Before we finish, we must solve some potential problems, such as low Durbin-Watson statistics, wrong signs, and insignificant coefficients. To deal with these problems, we chuck some variables out of the model and possibly apply the Cochrane-Orcutt transformation to help deal with serial correlation in the error term. One eventually emerges from the econometrics basement serving on a golden platter the final model that has all of the correct signs, with t-values greater than 2, and an R-squared that is acceptably high for the model builder.

Gilbert explains what's wrong with this. He says:

The problem with the AER approach is that we are using econometrics to illustrate the theories which we believe independently. The alternative might be to use econometrics to discover which views of the economy (or market) are tenable and to test, scientifically, the rival views. Our normal concept of science tends to be based on the natural, and in particular, the physical sciences, and does not obviously translate to the social sciences. This alternative programme for econometrics raises the question of what is required for economics to be scientific. 

Professor Edward Leamer also views textbook econometrics with some skepticism. He says:

Theoretical econometricians have interpreted scientific objectivity to mean that an economist must identify exactly the variables in the model, the functional form, and the distribution of the errors. Given these assumptions, and given a data set, the econometric method produces an objective inference from a data set, unencumbered by the subjective opinions of the researcher. 

This advice could be treated as ludicrous, except that it fills all the econometric textbooks. Fortunately, it is ignored by applied econometricians. The econometric art as it is practiced at the computer terminal involves fitting many, perhaps thousands, of statistical models. One or several that the researcher finds pleasing are selected for reporting purposes. This searching for a model is often well intentioned, but there can be no doubt that such a specification search invalidates the traditional theories of inference. The concepts of unbiasedness, consistency, efficiency, maximum likelihood estimation, in fact, all the concepts of traditional theory, utterly lose their meaning by the time an applied researcher pulls from the bramble of computer output the one thorn of a model he likes best, the one he chooses to portray as a rose. The consuming public is hardly fooled by this chicanery. The econometrician's shabby art is humorously and disparagingly labelled 'data mining', 'fishing', 'grubbing', 'number crunching'. A joke evokes the Inquisition: 'If you torture the data long enough, Nature will confess' (Coase). Another suggests methodological fickleness: 'Econometricians, like artists, tend to fall in love with their models' (wag unknown). Or how about: 'There are two things you are better off not watching in the making: sausages and econometric estimates'. 

This is a sad and decidedly unscientific state of affairs we find ourselves in. Hardly anyone takes data analyses seriously. Or perhaps more accurately, hardly anyone takes anyone else's data analyses seriously. Like elaborately plumed birds who have long since lost the ability to procreate but not the desire, we preen and strut and display our t-values.

My interpretation of the relationship between economic theory and econometrics tells me that a distinction should be drawn between what economic theory tells us and what econometrics can tell us. To lay the foundations of my argument, let me introduce the notion of a formal implication via a passage from Bertrand Russell's Principles of Mathematics:

[But] since the growth of non-Euclidean Geometry, it has appeared that pure mathematics has no concern with the question whether the axioms and propositions of Euclid hold of actual space or not: this is a question for applied mathematics, to be decided, so far as any decision is possible, by experiment and observation. What pure mathematics asserts is merely that the Euclidean propositions follow from the Euclidean axioms - i.e. it asserts an implication: any space which has such and such properties has also such and such other properties. Thus, as dealt with in pure mathematics, the Euclidean and non-Euclidean Geometries are equally true: in each nothing is affirmed except implications. All propositions as to what actually exists, like the space we live in, belong to experimental or empirical science, not to mathematics; when they belong to applied mathematics, they arise from giving to one or more of the variables in a proposition of pure mathematics some constant value satisfying the hypothesis, and thus enabling us, for that value of the variable, actually to assert both hypothesis and consequent instead of asserting merely the implication. We assert always in mathematics that if a certain assertion p is true of any entity x, or of any set of entities x, y, z, . . ., then some other assertion q is true of those entities; but we do not assert either p or q separately of our entities. We assert a relation between the assertions p and q, which I shall call formal implication. (Chapter 1, p.5)

Although Bertrand Russell is addressing the nature and bounds of pure mathematics and applied mathematics, an analogous story could be told of pure economics and econometrics.

Like pure mathematics, what pure economics deals with are sets of axioms and from those axioms follow propositions. Those familiar with Andreu Mas-Colell et. al's Microeconomic Theory (or Euclid's Elements for that matter!) will no doubt be comfortable with this sort of axiomatic system and the use of deductive logic.

The pure economists' concern is to come up with axioms and ensure that the progression from axioms to propositions will be carried out in a logical manner - in the parlance of formal logic, the inferential steps should be "watertight".

So, with one set of axioms we have a set of accompanying propositions and formal implications. As long as the axiomatic system is developed in a logical manner, it may be said to be true, albeit within its own scope - or space. Another set of axioms will generate another set of accompanying propositions and formal implications. Some of these may intersect the sets associated with the first axiomatic system. Indeed, one axiomatic system may encompass another one. Again, this second axiomatic system may be said to be true within its space.

That's what axiomatic systems in pure economics tell us. They assert implications that hold within a given space, or rather, in a particular model of an economy. Whether or not these implications conform to actual space - or actual economies - is another thing. The question to be asked then is why should an econometric model which aims to help us discover facts about actual economies be constrained to corroborate with an axiomatic system which may not necessarily hold true or provide a satisfactory description of actual economic phenomena? It probably shouldn't be that way because then we're using econometrics to illustrate rather than to discover - a reiteration of Gilbert.

Lawrence R. Klein, puts it nicely in Chapter 1 of his Economic Fluctuations in the United States (1921–1941). He says:

The purpose in building econometric models is to describe the way in which the system actually operates. [...] We want also to discovered the best possible theory or theories which explain the fluctuations that we observe. If we know the quantitative characteristics of the economic system, we shall be able to forecast with a specified level of probability the course of certain economic magnitudes such as employment, output, or income; and we shall also be able to forecast with a specified level of probability the effect upon the system of various economic policies. In the course of the search for models which are suitable for our purposes of forecasting and making policy recommendations, we shall inevitably have to consider several alternative theories as admissible hypotheses. The acceptance or  rejection of these hypotheses in the course of our search for truth will be our contribution to the problem of testing business-cycle theories. This is an important problem for which contributions are needed, because too often writers have made bold statements about the operation of the system without examining the factual data to determine whether or not their statements are true.* As a result we have had  continuing economic fluctuations and continuing publication of business cycle theories.

[*the emphasis is my own and does not appear in the original source]

It seems that an econometrican who builds models that are forced to conform to theories derived from an axiomatic system of pure economics is going astray from finding out about how actual economies work. Think for moment that Platonic solids and polygons from Euclid's Geometry can be described as having such and such properties and that's a useful exercise in itself, but believing that such solids and shapes occur in reality, is perhaps, not so wise. Go find me a perfect triangle!

Non-Euclidean Geometries, which permit shapes such as triangles whose angles sum to less than 180 degrees (hyperbolic geometry) or to greater than 180 degrees (elliptic geometry), may conform more closely to actual space than what Euclid's system tells us. As an example, an implication of Einstein's theory of general relativity is that physical space itself is not Euclidean, and Euclidean space is a good approximation for it only where the gravitational field is weak.* Importantly, each geometry, Euclidean or non-Euclidean, is true within its own framework, but their implications may hold differently in actual space. This is just a restatement of what Bertrand Russell said.

Coming back to the average economic regression and a situation in which a coefficient turns out to have the wrong sign. Well, if that's what the data tells us, maybe that's just the way it is for this particular economy - data and observation assumed to be accurate, i.e. assuming no identification problem. If economic theory suggests the sign of the coefficient should be otherwise, then the implications from that axiomatic system may not hold for the actual economy under scrutiny. If this is the case, it should follow that a new axiomatic system is built (which encompasses the old one) to take into account the new information we've discovered from econometrics. This shouldn't be thought of as a step back for economic theory, but an opportunity to advance the subject! Again, Newton didn't overturn Kepler, and Einstein didn't overturn Newton; each encompassed the other.

Similar to how applied mathematics deals with informing us about actual space, econometrics ought to aim at discovering information about economies that actually exist - freed from any bounds that pure economics may set upon itself. A man of considerable stature in economics once said: when the facts change, I change my mind, what do you do, sir?

Well, it's a bit like that. If revisions (or extensions) need to be made to economic theory, be open to making them. Now, it's not a case of scrap what we've already got. It's more so a case of expanding on what some great minds have already developed and kindly handed down to us. Nor is it a case that we're permitted to run any old nonsensical regressions. Sensible regressions that can be explained in terms of some (possibly hitherto unknown) axiomatic system are what we should be seeking to build.

I'll conclude the argument with four points.

• Pure mathematics illustrates. 
• Pure economics illustrates. 
• Applied mathematics discovers. 
• Econometrics discovers.

ARIMA modelling in R: minimizing AIC criterion

In earlier posts I talked about the arima() function in R. In particular, I gave some examples showing that it does some funny things when calculating residuals and I also mentioned that three different estimation methods could be specified using the method=() argument. Again, the estimation methods are "CSS-ML", "ML" and "CSS".

One way of specifying an ARIMA model is to choose the specification that minimizes Aikake's information criterion. This process, which I'll refer to as specification search, is data-mining. As far as I am aware, there is an R function called step() that can automatically identify the ARIMA model with the smallest AIC value for a given series of data.

Personally, I am not a fan of this modelling procedure as I have preference for following the Box-Jenkins methodology when modelling univariate time-series. Part of the BJ methodology relies on investigating the estimated autocorrelation function and partial autocorrelation function in order to try to identify possible models worth entertaining as candidate models.

Let me try to justify my preference using a single argument.

One has to remember that there are both producers and consumers of econometric models. The producer builds the model with some purpose, such as for forecasting or to guide policy decisions, and the consumer of the model uses it if he or she believes that the model is valid.

In my opinion, a producer who follows the Box-Jenkins methodology, or some other sound systematic procedure that can be explained to and understood by the consumer, should be in a position with decent chances of convincing the consumer that the final model is valid.

On the other hand, a producer who builds models by simply clicking a button and waiting for the result to pop up, may have trouble convincing consumers that the model is valid. This second kind of producer may find that he or she has very few customers willing to do business (even if the model actually does a good job at representing the data generating mechanism)! I wouldn't argue that an R function like step() is useless or should be avoided, but it probably shouldn't be at the helm during the model building procedure.

Next, I want to discuss a little problem in R that is not related to step().

The arima() function reports an AIC value as part of its output alongside estimated coefficients and standard errors as is standard for any popular statistical package. Now, caution must be taken here. This AIC value is only valid if the estimation method is specified as "ML". So, this AIC value is only useful to the analyst if their model is of the form: arima(x, c(p,d,q), method="ML"). The arima help page tells us the following about the reported AIC value:

aic: the AIC value corresponding to the log-likelihood. Only valid for 'method = "ML"' fits.

There has been some confusion about this and it has been raised in a discussion on the R mailing list, so it's worth pointing it out. I have also come across these University Lecture Notes on Model Selection in R Using AIC which happen to contain errors; the author estimates ARIMA models using the default ("CSS-ML") method and then uses the reported AIC values to select a final model despite the fact that the AIC values are only valid when method="ML"!

There are two key points here. The first is that if a model builder is going to use R for data analysis, he or she should carefully read the documentation. Secondly, model builders should recognize that information about their models, and quite possibly their model building procedure, may have to be communicated to consumers of their models. It's worth noting that consumers - including managers and politicians - may not have a great deal of technical knowledge of model building or even request a technical explanation of the modelling procedure, but they may put more trust in the models if they hear more than "the computer told me... this one".

Bayesian VARs

There are various approaches to modelling economic time-series. One very popular univariate method is known as the Box-Jenkins methodology and it's based on the idea that the analyst should "let the data speak for itself". Moving from the single-equation framework to a vector of equations, we have vector autoregressions and these are associated with Christopher Sims which he introduced in his paper Macroeconomics and Reality. Other econometric methodologies are linked to David Hendry and Edward Leamer; both of whom take different approaches.

Although I've listed various methodologies and said that they differ, they do, however, contain some elements which are the same across the board. A paper by Adrian Pagan (Three Econometric Methodologies: A Critical Appraisal) gives an excellent overview of the Sims, Hendry and Leamer methodologies and also outlines their similarities and differences.

Atheoretic models, such as univariate Box-Jenkins models and the Unrestricted VAR, do not rely on economic theory as the basis for specifying the model. They let the data do the talking. On the other hand, big simultaneous equation models, often built by a team of economists, have their foundations based on economic theory. While, in the Bayesian approach, beliefs (also known as priors) are allowed to enter the model.

Richard Todd discusses combining two of the aforementioned approaches, namely, the Bayesian approach and Sim's VAR framework, in his paper Improving Economic Forecasting with Bayesian Vector Autoregression. Mixing these two methods results in an economic forecasting model known as the Bayesian vector autoregression (BVAR).

We can immediately see the difference between the data-driven Box-Jenkins methodology and BVARs (data + beliefs) when Todd says:

The Bayesian approach to statistics, a general method for combining beliefs with data, suggests an objective procedure for blending beliefs and data in economic forecasting models. This procedure provides a framework that forecasters can use to document and discuss their beliefs, which can help make economic forecasting more of a science and less of an art.

So, how does the BVAR improve the basic Unrestricted VAR? Well, for a first, a vector autoregression with $N$ variables and $p$ lags, involves $N^{2}p$ parameters. In other words, adding extra variables and lags to a VAR consumes many degrees of freedom. This is a well known feature of vector autoregressions. Consider an example given by Clive Granger:

If N=10, p=4 there are 400 parameters to be estimated from 2000 pieces of data, say, if 200 terms are available for each series. It is clear why there is a desire for simpler models.

Clive's comment about the desire for simpler models contains an appeal to the principle of parsimony which says that we should build models that use the smallest number of coefficients needed to explain the available data. In other words, we should not include any unnecessary coefficients in our models.

Structural econometric models use exclusion restrictions to reduce the number of coefficients to be estimated in a VAR model.

So, how does one go about BVAR modelling in practice? Well, according to these slides by Patrick Brandt at the University of Texas, RATS, Eviews, Stata, and Matlab, are not very conducive to modelling BVARs. However, we're in luck because there is a package that has been developed for modelling BVARs in R. The package is called BMR. Strangely enough, this package was developed by one of my old UCD classmates! Better yet, there appears to be a second package MSBVAR that should also do the trick.

Impulse Response Functions

I bought a copy of Ed Leamer's book Macroeconomic Patterns and Stories after hearing him speaking about it on EconTalk. Actually, Professor Leamer is probably my favourite guest on EconTalk, so purchasing his book was a no-brainer for me. If you've heard of him before it's probably due to his famous paper Let's Take the Con Out of Econometrics. He is also well known for advocating "extreme bounds analysis", which, in basic terms, involves finding a possible range of values for an estimate and investigating the fragility of the model by seeing if the value of the estimate is sensitive to small changes in model specification.

The example of EBA given in Let's Take the Con Out of Econometrics is to do with the effect of capital punishment on the murder rate. Using a number of regressions, with different, but somewhat similar specifications, Leamer shows that at one end of the spectrum, the estimate is negative, but at the other end, the estimate is positive!

In his book, Macroeconomic Patterns and Stories, Professor Leamer talks about four key macro variables in detail; namely, Growth, Unemployment, Inflation, and Interest Rates. In this post, I want to focus on these four variables and in particular to the topic of Chapter 6 of the book which is Extrapolative Forecasting.

In Chapter 6, a table is shown and it contains regression results. There are four separate regressions and each of them are autoregressive. Growth is modeled as an AR(2) process, so too is the unemployment rate and 3-month treasuries, while inflation is modeled as an AR(4) process. Admittedly, Professor Leamer says that a more careful data analysis could be done on the data, but the results from these regressions still permit us to get an insight into the dynamic nature of the four variables.

In R, I've estimated how each of the four variables respond to a one unit shock. I've plotted the results in the Figure 1 below. A similar graph appears on page x in Leamer's book and this gives me confidence that the code I wrote in R is correct.

Figure 1

To explain the dynamic behaviour of each of the four variables in response to a shock, Leamer introduces two concepts to the reader: persistence and momentum.

R Time Series Issue: Residuals (Revisited)

In the previous post, I investigated the arima() function and its method of calculating residuals. The issue that I raised was that the first residual associated with arima(x, c(1,0,0)) happens to be non-zero when it should probably either be treated as zero or classified as n.a. (I prefer the latter) when doing time series analysis.

Last time, I specified only two arguments in the arima() function; namely, x and order. For more information type ?arima in R.

In this post, I'd like to point out some more details about calculating residuals in R via arima() and to do so, I'll talk about a third argument of arima() - the method argument - which is used to specify which estimation method is to be used to estimate the parameters of the model. I briefly mentioned last time that maximum-likelihood and non-linear least squares were two possible estimation methods for ARIMA models.

The method argument has three options: "CSS-ML", "ML", and "CSS". The default is set to "CSS-ML", so that was the method used in the previous post. The help page for the arima() function states the following about the method argument:

Fitting method: maximum likelihood or minimize conditional sum-of-squares. The default (unless there are missing values) is to use conditional-sum-of-squares to find starting values, then maximum likelihood. 

I cannot confirm it yet, but I'm almost certain that "CSS-ML" (the default) refers to Marquardt's compromise, "ML" to maximum likelihood, and "CSS" to what I (and the documentation that I've read) call non-linear least squares.

To begin my investigation, I estimated an AR(1) model for the same data in the previous post using, in turn, each of the three estimation methods. That is, I estimated these three models in R:

1. arima(x, c(1,0,0), method = c("CSS-ML")),
2. arima(x, c(1,0,0), method = c("ML")),
3. arima(x, c(1,0,0), method = c("CSS")).

Now, here is the problem...

Upon inspection of the residuals associated with each model, I found that, in two of the three cases ("CSS-ML", "ML") the first residual is non-zero, while in the third case ("CSS"), the first residual is precisely equal to zero.

I've tabulated the residuals for each of the three models in Table 1 below...

If the analyst accepts these first residuals as appropriate then they can fire away and use x$\$$residuals as they please. On the other hand, if the analyst is of the opinion that the first residual should be omitted from their diagnostic checking procedures, they can do this in R by specifying x$residuals[-1]. This command will return all but the first residual as a vector object.