Broadly speaking there are three engines to finance. One is profit: finance exists to find ways to deploy assets in a way that will make a return on investment, or a profit on invested capital. The second is risk management: diversification, syndication, and derivatives exist to spread risk around so that one can make bets to get a profit while minimizing the possibility of losing everything by getting it wrong. And the third is, in a way, the guiding force that balances the first two: forecasting.

Like weathermen, psychics, and sports bookies, financiers are in the job of forecasting what will happen in the future and making decisions based on those forecasts. An investment banker will forecast future cash flows for an emerging company in order to create a valuation for a merger, acquisition, or IPO. A hedge fund analyst will do much the same to make buy/sell targets for debt or equity in a firm, private or public. And an economist will forecast future economic growth trends to produce reports that will be read by other people, who will use those reports to inform their own forecasting and decision making.

In each case, there is uncertainty. No model, however perfect, can be expected to get things right 100% of the time, so much so that some hedge funds have made billions of dollars with models that get things right just 53% of the time. That extra 3% above chance is enough that, if well diversified, bets using that model can create a huge profit. For students, it’s perhaps frustrating to learn that billion-dollar fortunes have been made with models that would effectively get an F grade at school.

Such situations demand robust models that can push predictions closer and closer to reality, and since predicting the future is hard (if not impossible), very complicated math is often required to improve models and get them to fit closer to reality. And with the emergence of machine learning and A.I., the models have gotten that much more complicated.

But before we get to that, let’s start with basics: the simple linear regression. This is the simplest forecasting model and, in a lot of cases, is actually pretty good. A basic concept in any elementary statistics course, linear regression will take one data series, find coefficients in changes between instances of the data to predict values. In short, if we have variable a and b, a linear regression will tell us what a is when b is 0 and then extrapolate what a is when b is 0+x. This creates a diagonal line on a dot plot chart, like this:



Here we have a number of daily data points, with a trendline derived from the coefficiency of determination (r2) in the data. That can then be used to project what those numbers will be in the future—that’s the vertical line you see going upwards.

The simple linear regression is, as the name suggests, simple, and there are a lot of aspects of datasets that it doesn’t take into account. Note the bit outlier at the end of this dataset—a simple linear regression doesn’t consider outliers like it. Also note the exponential rate of change in the growth rate from the start in March to July—that would suggest a more complex model would give us a more accurate prediction of the future. And what of the randomness of the numbers between June and August—or is that random?

To account for these issues, there are other regression models that are typically used with increasing complexity. Most recently, machine learning has been used to pile on more and more complex math in an attempt to fit predictions to the data much better. In cases with one data series, this might not yield much, but when one is trying to find patterns between multiple data series, these complex algorithms are not only helpful, but necessary. One can’t simply take, for instance, retail sales, GDP, consumer confidence, miles driven, hospital capacity, political polling, weather patterns, oil production and analyze them all together in a simple linear regression!

For these kinds of multiple data sets, machine learning and A.I. come to the rescue.

The difference between the two terms is debatable, and that’s not really important for us. What is important is understanding the basic concept: putting together various data sets and creating an algorithm with several Boolean logic statements to analyzing data. A good example is the k-nearest neighbors algorithm (kNN), where k = the number of nearest neighbors to an object.

This stuff can sound really complicated, so let’s use an example to make sense of it.



Imagine someone shows you this chart—there are four dots pretty low and two dots very high where the axis goes from 0 to 100. There’s an unidentified data point at around 65—what is that data point?

Well, a kNN analysis would say that F and G are in a class of their own and a-d are in a separate class—here denoted with capital and lower case letters, respectively. A kNN analysis would assume that the unknown data point is in the capital letter class.

Sound too simple? In this example, it certainly is—but when you have many, many data sets intersecting and thousands upon thousands of data points (or millions as the case increasingly has become), this kind of analysis becomes very important. And potentially profitable, if used to analyze a data set that can predict an economic or financial outcome.

Of course, this isn’t to say that the predictiveness of finance and economics has gotten to the level of prophecy—we’re not there yet and probably never will be. But the predictiveness of models has improved significantly thanks to these complex mathematical tools, and an expert financier will understand how they are used and what their benefits and limitations are—even if you might not always know exactly how to use them.