There are a lot of market questions circulating right now whether the risk models worked properly given the current credit crisis. However, in assessing the models, we need to first decide which market we’re trying to analyze. The most visible losses came in subprime-backed ABS and later in quantitative equity funds. It seems the story in equity funds is somewhat mode-related, but it pertains to the specifics of the models the funds were using to trade. Given this, let’s focus on the current subprime story

To frame this discussion, let’s first define what we mean by Value-at-Risk (VaR) models and stress tests. Broadly, let’s say that VaR models do two things: 1) they look at the historical data on various market factors and try to apply statistical methods to forecast how much those factors might move in the future and; 2) they relate the moves in market factors to the moves in specific investment holdings. For example, for a portfolio of government bonds, the model first forecasts where the interest rate curve might go, and then it relates moves in the curve to price changes for specific bonds. The output of a VaR-like model is a statistical distribution that we forecast for our portfolio.

We can think of stress testing in the same framework, but for the first part (forecasting where the market factors will go), we replace statistical models with subjective scenarios. So rather than using a model to forecast how much the interest rate curve might move, we come up with specific scenarios we are particularly concerned with. How one decides on scenarios is much more art than science, and varies significantly across institutions. The second piece – relating market factor moves to specific holdings – is exactly the same as above. The output of stress testing is a series of answers to questions like “What would happen to my portfolio if …?”

So did stress tests work? If by that we mean did institutions test scenarios that were representative of what ultimately happened, there are probably a lot of answers. The events were hardly a complete surprise, so some institutions probably had a pretty good sense of what was about to occur.

Did VaR models work? The answer does depend on which model you mean, but forecasts of market factors were actually reasonably good, given the nature of the events. One specific case we tested was the spread on the ABX (an index of subprime-backed ABS securities). We ran a simple backtest applying one of our statistical models to the ABX spreads. In the test, we forecast on each day how large we think the 99% worst case move might be for the following day. We then observe what that actual move is. On average, if the model is working well, you would expect that the actual move is larger than our forecast 1% of the time. Call these days excession days. Over a year (250 trading days), this means on average 2.5 excession days. For the three of the five ABX contracts, we observed between one and five excession days, consistent with the model performing well. For the other two contracts, we observed five and seven excession days, a level that raises some questions, but does not call for dismissing the model.

The piece I have not addressed yet is the second piece of both models: the connection between market factors and specific investments. This is where there were significant problems. The bottom line is that many ABS securities do not trade at all, so there is no way to observe how they relate to market factors, as we can do with things like government bonds. So typically, the simple act of assessing the value of an ABS contract cannot be done by looking at where it trades, but rather relies on modeling the fundamental building blocks of the security and making assumptions about things like the market’s appetite for such securities. This poses two big problems: the fundamental modeling is operationally difficult, due to the complexity of the structures; and we wind up only guessing at the market’s appetite for the security, and certainly not factoring in the possibility that this appetite might change. The consequence here is that institutions either had no valuation capability at all, or else had to rely on models that ignored supply and demand effects going on in the market. In the end, we went through a period where the market turned against all subprime ABS, regardless of their fundamentals, which was something none of the valuation models even attempted to account for.

So in the end, the big lesson from this summer was about dealing with securities which do not trade and for which even coming up with a valuation is difficult. In a sense, the modeling problems are even more basic than stress tests or VaR.