The maths behind the model, from additive assumptions to pseudoinverse matrices

Technical disclaimer: It is feasible to derive a model without normality assumptions. We’ll go down this route since it’s straightforward enough to grasp and by assuming normality of the model’s output, we are able to reason concerning the uncertainty of our predictions.

This post is meant for people who find themselves already aware of what linear regression is (and perhaps have used it a few times) and wish a more principled understanding of the maths behind it.

Some background in basic probability (probability distributions, joint probability, mutually exclusive events), linear algebra, and stats might be required to profit from what follows. Without further ado, here we go:

The machine learning world is filled with amazing connections: the exponential family, regularization and prior beliefs, KNN and SVMs, Maximum Likelihood and Information Theory — it’s all connected! (I like Dark). This time we’ll discuss the best way to derive one other one among the members of the exponential family: the Linear Regression model, and in the method we’ll see that the Mean Squared Error loss is theoretically well motivated. As with all regression model, we’ll have the opportunity to make use of it to predict numerical, continuous targets. It’s an easy yet powerful model that happens to be one among the workhorses of statistical inference and experimental design. Nevertheless we shall be concerned only with its usage as a predictive tool. No pesky inference (and God forbid, causal) stuff here.

Alright, allow us to begin. We would like to predict something based on something else. We’ll call the predicted thing y and the something else x. As a concrete example, I offer the next toy situation: You might be a credit analyst working in a bank and also you’re all for mechanically checking out the suitable credit limit for a bank customer. You furthermore mght occur to have a dataset pertaining to past clients and what credit limit (the predicted thing) was approved for them, along with a few of their features corresponding to demographic info, past credit performance, income, etc. (the something else).

So now we have a fantastic idea and write down a model that explains the credit limit by way of those features available to you, with the model’s predominant assumption being that every feature contributes something to the observed output in an additive manner. Because the credit stuff was only a motivating (and contrived) example, let’s return to our pure math world of spherical cows, with our model turning into something like this:

We still have the anticipated stuff (y) and the something else we use to predict it (x). We concede that some kind of noise is unavoidable (be it by virtue of imperfect measuring or our own blindness) and the very best we are able to do is to assume that the model behind the info we observe is stochastic. The consequence of that is that we’d see barely different outputs for a similar input, so as an alternative of neat point estimates we’re “stuck” with a probability distribution over the outputs (y) conditioned on the inputs (x):

Every data point in y is replaced by a bit of bell curve, whose mean lies within the observed values of y, and has some variance which we don’t care about in the mean time. Then our little model will take the place of the distribution mean.

Assuming all those bell curves are literally normal distributions and their means (data points in y) are independent from one another, the (joint) probability of observing the dataset is

Logarithms and a few algebra to the rescue:

Logarithms are cool, aren’t they? Logs transform multiplication into sum, division into subtraction, and powers into multiplication. Quite handy from each algebraic and numerical standpoints. Eliminating constant stuff, which is irrelevant on this case, we arrive to the next maximum likelihood problem:

Well, that’s the identical as

The expression we’re about to attenuate is something very near the famous Mean Square Error loss. In actual fact, for optimization purposes they’re equivalent.

So what now? This minimization problem might be solved exactly using derivatives. We’ll make the most of the incontrovertible fact that the loss is quadratic, which implies convex, which implies one global minima; allowing us to take its derivative, set it to zero and solve for theta. Doing this we’ll find the worth of the parameters theta that makes the derivative of the loss zero. And why? since it is precisely at the purpose where the derivative is zero, that the loss is at its minimum.

To make every part somewhat simpler, let’s express the loss in vector notation:

Here, X is an NxM matrix representing our whole dataset of N examples and M features and y is a vector containing the expected responses per training example. Taking the derivative and setting it to zero we get

There you may have it, the answer to the optimization problem now we have solid our original machine learning problem into. In the event you go ahead and plug those parameter values into your model, you’ll have a trained ML model able to be evaluated using some holdout dataset (or perhaps through cross-validation).

In the event you think that final expression looks an awful lot like the answer of a linear system,

it’s since it does. The additional stuff comes from the incontrovertible fact that for our problem to be reminiscent of a vanilla linear system, we’d need an equal variety of features and training examples so we are able to invert X. Since that’s seldom the case we are able to only hope for a “best fit” solution — in some sense of best — resorting to the Moore-Penrose Pseudoinverse of X, which is a generalization of the great ol’ inverse matrix. The associated wikipedia entry makes for a fun reading.