An accessible walkthrough of fundamental properties of this popular, yet often misunderstood metric from a predictive modeling perspective

R² (R-squared), also generally known as the coefficient of determination, is widely used as a metric to judge the performance of regression models. It is often used to quantify goodness of fit in statistical modeling, and it’s a default scoring metric for regression models each in popular statistical modeling and machine learning frameworks, from statsmodels to scikit-learn.

Despite its omnipresence, there’s a surprising amount of confusion on what R² truly means, and it just isn’t unusual to come across conflicting information (for instance, in regards to the upper or lower bounds of this metric, and its interpretation). At the foundation of this confusion is a “culture clash” between the explanatory and predictive modeling tradition. Actually, in predictive modeling — where evaluation is conducted out-of-sample and any modeling approach that increases performance is desirable — many properties of R² that do apply within the narrow context of explanation-oriented linear modeling now not hold.

To assist navigate this confusing landscape, this post provides an accessible narrative primer to some basic properties of R² from a predictive modeling perspective, highlighting and dispelling common confusions and misconceptions about this metric. With this, I hope to assist the reader to converge on a unified intuition of what R² truly captures as a measure of slot in predictive modeling and machine learning, and to focus on a few of this metric’s strengths and limitations. Aiming for a broad audience which incorporates Stats 101 students and predictive modellers alike, I’ll keep the language easy and ground my arguments into concrete visualizations.

Ready? Let’s start!

What’s R²?

Let’s start from a working verbal definition of R². To maintain things easy, let’s take the primary high-level definition given by Wikipedia, which is a superb reflection of definitions present in many pedagogical resources on statistics, including authoritative textbooks:

the proportion of the variation within the dependent variable that’s predictable from the independent variable(s)

Anecdotally, this can also be what the overwhelming majority of scholars trained in using statistics for inferential purposes would probably say, if you happen to asked them to define R². But, as we’ll see in a moment, this common way of defining R² is the source of lots of the misconceptions and confusions related to R². Let’s dive deeper into it.

Calling R² a proportion implies that R² shall be a number between 0 and 1, where 1 corresponds to a model that explains all of the variation within the consequence variable, and 0 corresponds to a model that explains no variation within the consequence variable. Note: your model may also include no predictors (e.g., an intercept-only model continues to be a model), that’s why I’m specializing in variation predicted by a model somewhat than by independent variables.

Let’s confirm if this intuition on the range of possible values is correct. To accomplish that, let’s recall the mathematical definition of R²:

Here, RSS is the residual sum of squares, which is defined as:

This is solely the sum of squared errors of the model, that’s the sum of squared differences between true values y and corresponding model predictions ŷ.

Then again, TSS, the full sum of squares, is defined as follows:

As you may notice, this term has an analogous “form” than the residual sum of squares, but this time, we’re the squared differences between the true values of the consequence variables y and the mean of the consequence variable ȳ. That is technically the variance of the consequence variable. But a more intuitive strategy to take a look at this in a predictive modeling context is the next: this term is the residual sum of squares of a model that at all times predicts the mean of the consequence variable. Hence, the ratio of RSS and TSS is a ratio between the sum of squared errors of your model, and the sum of squared errors of a “reference” model predicting the mean of the consequence variable.

With this in mind, let’s go on to analyse what the range of possible values for this metric is, and to confirm our intuition that these should, indeed, range between 0 and 1.

What’s the most effective possible R²?

As we’ve seen up to now, R² is computed by subtracting the ratio of RSS and TSS from 1. Can this ever be higher than 1? Or, in other words, is it true that 1 is the biggest possible value of R²? Let’s think this through by looking back on the formula.

The one scenario through which 1 minus something will be higher than 1 is that if that something is a negative number. But here, RSS and TSS are each sums of squared values, that’s, sums of positive values. The ratio of RSS and TSS will thus at all times be positive. The biggest possible R² must subsequently be 1.

Now that we’ve established that R² can’t be higher than 1, let’s try to visualise what must occur for our model to have the utmost possible R². For R² to be 1, RSS / TSS should be zero. This will occur if RSS = 0, that’s, if the model predicts all data points perfectly.

In practice, this can never occur, unless you’re wildly overfitting your data with an excessively complex model, or you’re computing R² on a ridiculously low number of information points that your model can fit perfectly. All datasets could have some amount of noise that cannot be accounted for by the info. In practice, the biggest possible R² shall be defined by the quantity of unexplainable noise in your consequence variable.

What’s the worst possible R²?

Up to now so good. If the biggest possible value of R² is 1, we will still consider R² because the proportion of variation within the consequence variable explained by the model. But let’s now move on to the bottom possible value. If we buy into the definition of R² we presented above, then we must assume that the bottom possible R² is 0.

When is R² = 0? For R² to be null, RSS/TSS should be equal to 1. That is the case if RSS = TSS, that’s, if the sum of squared errors of our model is the same as the sum of squared errors of a model predicting the mean. For those who are higher off just predicting the mean, then your model is actually not doing a really good job. There are infinitely many the explanation why this will occur, one among these being a difficulty together with your alternative of model — if, for instance, if you happen to try to model really non-linear data with a linear model. Or it may be a consequence of your data. In case your consequence variable could be very noisy, then a model predicting the mean could be the most effective you’ll be able to do.

But is R² = 0 truly the bottom possible R²? Or, in other words, can R² ever be negative? Let’s look back on the formula. R² < 0 is only possible if RSS/TSS > 1, that’s, if RSS > TSS. Can this ever be the case?

That is where things start getting interesting, as the reply to this query depends very much on contextual information that we’ve not yet specified, namely which kind of models we’re considering, and which data we’re computing R² on. As we’ll see, whether our interpretation of R² because the proportion of variance explained holds is determined by our answer to those questions.

The bottomless pit of negative R²

Let’s looks at a concrete case. Let’s generate some data using the next model y = 3 + 2x, and added Gaussian noise.

import numpy as np
x = np.arange(0, 1000, 10)
y = [3 + 2*i for i in x] 
noise = np.random.normal(loc=0, scale=600, size=x.shape[0])
true_y = noise + y

The figure below displays three models that make predictions for y based on values of x for various, randomly sampled subsets of this data. These models are usually not made-up models, as we’ll see in a moment, but let’s ignore this straight away. Let’s focus simply on the sign of their R².

Let’s start from the primary model, a straightforward model that predicts a continuing, which on this case is lower than the mean of the consequence variable. Here, our RSS shall be the sum of squared distances between each of the dots and the orange line, while TSS shall be the sum of squared distances between each of the dots and the blue line (the mean model). It is straightforward to see that for many of the data points, the gap between the dots and the orange line shall be higher than the gap between the dots and the blue line. Hence, our RSS shall be higher than our TSS. If that is so, we could have RSS/TSS > 1, and, subsequently: 1 — RSS/TSS < 0, that's, R²<0.

Actually, if we compute R² for this model on this data, we obtain R² = -2.263. If you wish to check that it’s actually realistic, you’ll be able to run the code below (as a result of randomness, you’ll likely get a similarly negative value, but not the exact same value):

from sklearn.metrics import r2_score
# get a subset of the info
x_tr, x_ts, y_tr, y_ts = train_test_split(x, true_y, train_size=.5)
# compute the mean of one among the subsets 
model = np.mean(y_tr)
# evaluate on the subset of information that's plotted
print(r2_score(y_ts, [model]*y_ts.shape[0]))

Let’s now move on to the second model. Here, too, it is straightforward to see that distances between the info points and the red line (our goal model) shall be larger than distances between data points and the blue line (the mean model). Actually, here: R²= -3.341. Note that our goal model is different from the true model (the orange line) because we’ve fitted it on a subset of the info that also includes noise. We’ll return to this in the following paragraph.

Finally, let’s take a look at the last model. Here, we fit a 5-degree polynomial model to a subset of the info generated above. The space between data points and the fitted function, here, is dramatically higher than the gap between the info points and the mean model. Actually, our fitted model yields R² = -1540919.225.

Clearly, as this instance shows, models can have a negative R². Actually, there isn’t a limit to how low R² will be. Make the model bad enough, and your R² can approach minus infinity. This may occur with a straightforward linear model: further increase the worth of the slope of the linear model within the second example, and your R² will keep taking place. So, where does this leave us with respect to our initial query, namely whether R² is in undeniable fact that proportion of variance within the consequence variable that will be accounted for by the model?

Well, we don’t tend to think about proportions as arbitrarily large negative values. If are really attached to the unique definition, we could, with a creative leap of imagination, extend this definition to covering scenarios where arbitrarily bad models can add variance to your consequence variable. The inverse proportion of variance added by your model (e.g., as a consequence of poor model decisions, or overfitting to different data) is what’s reflected in arbitrarily low negative values.

But that is more of a metaphor than a definition. Literary pondering aside, probably the most literal and most efficient way of eager about R² is as a comparative metric, which says something about how significantly better (on a scale from 0 to 1) or worse (on a scale from 0 to infinity) your model is at predicting the info in comparison with a model which at all times predicts the mean of the consequence variable.

Importantly, what this implies, is that while R² generally is a tempting strategy to evaluate your model in a scale-independent fashion, and while it’d is sensible to make use of it as a comparative metric, it’s a removed from transparent metric. The worth of R² won’t provide explicit information of how mistaken your model is in absolute terms; the most effective possible value will at all times be depending on the quantity of noise present in the info; and good or bad R² can come about from a wide range of reasons that will be hard to disambiguate without the help of additional metrics.

Alright, R² will be negative. But does this ever occur, in practice?

A really legitimate objection, here, is whether or not any of the scenarios displayed above is definitely plausible. I mean, which modeller of their right mind would actually fit such poor models to such easy data? These might just appear like ad hoc models, made up for the aim of this instance and never actually fit to any data.

This is a superb point, and one which brings us to a different crucial point related to R² and its interpretation. As we highlighted above, all these models have, actually, been fit to data that are generated from the identical true underlying function as the info within the figures. This corresponds to the practice, foundational to predictive modeling, of splitting data intro a training set and a test set, where the previous is used to estimate the model, and the latter for evaluation on unseen data — which is a “fairer” proxy for a way well the model generally performs in its prediction task.

Actually, if we display the models introduced within the previous section against the info used to estimate them, we see that they are usually not unreasonable models in relation to their training data. Actually, R² values for the training set are, no less than, non-negative (and, within the case of the linear model, very near the R² of the true model on the test data).

Why, then, is there such an enormous difference between the previous data and this data? What we’re observing are cases of overfitting. The model is mistaking sample-specific noise within the training data for signal and modeling that — which just isn’t in any respect an unusual scenario. Because of this, models’ predictions on recent data samples shall be poor.

Avoiding overfitting is maybe the largest challenge in predictive modeling. Thus, it just isn’t in any respect unusual to look at negative R² values when (as one should at all times do to be sure that the model is generalizable and robust ) R² is computed out-of-sample, that’s, on data that differ “randomly” from those on which the model was estimated.

Thus, the reply to the query posed within the title of this section is, actually, a powerful yes: negative R² do occur in common modeling scenarios, even when models have been properly estimated. Actually, they occur on a regular basis.

So, is everyone just mistaken?

If R² is not a proportion, and its interpretation as variance explained clashes with some basic facts about its behavior, do we’ve to conclude that our initial definition is mistaken? Are Wikipedia and all those textbooks presenting an analogous definition mistaken? Was my Stats 101 teacher mistaken? Well. Yes, and no. It depends hugely on the context through which R² is presented, and on the modeling tradition we’re embracing.

If we simply analyse the definition of R² and take a look at to explain its general behavior, regardless of which kind of model we’re using to make predictions, and assuming we’ll wish to compute this metrics out-of-sample, then yes, they’re all mistaken. Interpreting R² because the proportion of variance explained is misleading, and it conflicts with basic facts on the behavior of this metric.

Yet, the reply changes barely if we constrain ourselves to a narrower set of scenarios, namely linear models, and particularly linear models estimated with least squares methods. Here, R² will behave as a proportion. Actually, it may be shown that, as a result of properties of least squares estimation, a linear model can never do worse than a model predicting the mean of the consequence variable. Which suggests, that a linear model can never have a negative R² — or no less than, it cannot have a negative R² on the identical data on which it was estimated (a debatable practice if you happen to are inquisitive about a generalizable model). For a linear regression scenario with in-sample evaluation, the definition discussed can subsequently be considered correct. Additional fun fact: this can also be the one scenario where R² is reminiscent of the squared correlation between model predictions and the true outcomes.

The rationale why many misconceptions about R² arise is that this metric is commonly first introduced within the context of linear regression and with a concentrate on inference somewhat than prediction. But in predictive modeling, where in-sample evaluation is a no-go and linear models are only one among many possible models, interpreting R² because the proportion of variation explained by the model is at best unproductive, and at worst deeply misleading.

Should I still use R²?

We now have touched upon quite just a few points, so let’s sum them up. We now have observed that:

• R² can’t be interpreted as a proportion, as its values can range from -∞ to 1
• Its interpretation as “variance explained” can also be misleading (you’ll be able to imagine models that add variance to your data, or that combined explained existing variance and variance “hallucinated” by a model)
• On the whole, R² is a “relative” metric, which compares the errors of your model with those of a straightforward model at all times predicting the mean
• It’s, nevertheless, accurate to explain R² because the proportion of variance explained within the context of linear modeling with least squares estimation and when the R² of a least-squares linear model is computed in-sample.

Given all these caveats, should we still use R²? Or should we quit?

Here, we enter the territory of more subjective observations. On the whole, if you happen to are doing predictive modeling and you wish to get a concrete sense for how mistaken your predictions are in absolute terms, R² is not a useful metric. Metrics like MAE or RMSE will certainly do a greater job in providing information on the magnitude of errors your model makes. This is beneficial in absolute terms but in addition in a model comparison context, where it is advisable to know by how much, concretely, the precision of your predictions differs across models. If knowing something about precision matters (it hardly doesn’t), you may no less than want to enhance R² with metrics that claims something meaningful about how mistaken each of your individual predictions is more likely to be.

More generally, as we’ve highlighted, there are plenty of caveats to take note if you happen to resolve to make use of R². A few of these concern the “practical” upper bounds for R² (your noise ceiling), and its literal interpretation as a relative, somewhat than absolute measure of fit in comparison with the mean model. Moreover, good or bad R² values, as we’ve observed, will be driven by many aspects, from overfitting to the quantity of noise in your data.

Then again, while there are only a few predictive modeling contexts where I even have found R² particularly informative in isolation, having a measure of fit relative to a “dummy” model (the mean model) generally is a productive strategy to think critically about your model. Unrealistically high R² in your training set, or a negative R² in your test set might, respectively, make it easier to entertain the chance that you simply could be going for an excessively complex model or for an inappropriate modeling approach (e.g., a linear model for non-linear data), or that your consequence variable might contain, mostly, noise. That is, again, more of a “pragmatic” personal take here, but while I’d resist fully discarding R² (there aren’t many good global and scale-independent measures of fit), in a predictive modeling context I’d consider it most useful as a complement to scale-dependent metrics corresponding to RMSE/MAE, or as a “diagnostic” tool, somewhat than a goal itself.

Concluding remarks

R² is in every single place. Yet, especially in fields which can be biased towards explanatory, somewhat than predictive modelling traditions, many misconceptions about its interpretation as a model evaluation tool flourish and persist.

On this post, I even have tried to offer a narrative primer to some basic properties of R² as a way to dispel common misconceptions, and help the reader get a grasp of what R² generally measures beyond the narrow context of in-sample evaluation of linear models.

Removed from being a whole and definitive guide, I hope this generally is a pragmatic and agile resource to make clear some very justified confusion. Cheers!

Unless otherwise states within the caption, images in this text are by the writer