After a bank card? An insurance policy? Ever wondered concerning the three-digit number that shapes these decisions?

Introduction

Scores are utilized by numerous industries to make decisions. Financial institutions and insurance providers are using scores to find out whether someone is true for credit or a policy. Some nations are even using social scoring to find out a person’s trustworthiness and judge their behaviour.

For instance, before a rating was used to make an automatic decision, a customer would go right into a bank and speak to an individual regarding how much they need to borrow and why they need a loan. The bank worker may impose their very own thoughts and biases into their decision-making process. Where is that this person from? What are they wearing? Even, how do I feel today?

A rating levels the playing field and allows everyone to be assessed on the identical basis.

Recently, I actually have been collaborating in several Kaggle competitions and analyses of featured datasets. The primary playground competition of 2024 aimed to find out the likelihood of a customer leaving a bank. It is a common task that is beneficial for marketing departments. For this competition, I believed I might put aside the tree-based and ensemble modelling techniques normally required to be competitive in these tasks, and return to the fundamentals: a logistic regression.

Here, I’ll guide you thru the event of the logistic regression model, its conversion right into a rating, and its presentation as a scorecard. The aim of doing that is to indicate how this may reveal insights about your data and its relationship to a binary goal. The advantage of such a model is that it is less complicated and easier to clarify, even to non-technical audiences.

My Kaggle notebook with all my code and maths will be found here. This text will deal with the highlights.

What’s a Rating?

The rating we’re describing here relies on a logistic regression model. The model assigns weights to our input features and can output a probability that we are able to convert through a calibration step right into a rating. Once we’ve this, we are able to represent it with a scorecard: showing how a person is scoring based on their available data.

Let’s undergo a straightforward example.

Mr X walks right into a bank on the lookout for loan for a brand new business. The bank uses a straightforward rating based on income and age to find out whether the person must be approved.

Mr X is a young individual with a comparatively low income. He’s penalised for his age, but scores well (second best) within the income band. In total, he scores 24 points on this scorecard, which is a mid-range rating (the utmost variety of points being 52).

A rating cut-off would often be utilized by the bank to say what number of points are needed to be accepted based on internal policy. A rating relies on a logistic regression which is built on some binary definition, using a set of features to predict the log odds.

Within the case of a bank, the logistic regression could also be attempting to predict people who have missed payments. For an insurance provider, those that have made a claim before. For a social rating, people who have ever attended an anarchist gathering (not likely sure what these scores can be predicting but I can be fascinated to know!).

We is not going to undergo all the pieces required for a full model development, but a few of the key steps that can be explored are:

• Weights of Evidence Transformation: Making our continuous features discrete by banding them up as with the Mr X example.
• Calibrating our Logistic Regression Outputs to Generate a Rating: Making our probability right into a more user-friendly number by converting it right into a rating.
• Representing Our Rating as a Scorecard: Showing how each feature contributes to the ultimate rating.

Weights of Evidence Transformation

Within the Mr X example, we saw that the model had two features which were based on numeric values: the age and income of Mr X. These variables were banded into groups to make it easier to know the model and what drives a person’s rating. Using these continuous variables directly (as oppose to inside a bunch) could mean significantly different scores for small differences in values. Within the context of credit or insurance risk, this comes to a decision harder to justify and explain.

There are a number of how to approach the banding, but normally an initial automated approach is taken, before fine-tuning the groupings manually to make qualitative sense. Here, I fed each continuous feature individually into a call tree to get an initial set of groupings.

Once the groupings were available, I calculated the weights of evidence for every band. The formula for that is shown below:

It is a commonly used transformation technique in scorecard modelling where a logistic regression is used given its linear relationship to the log odds, the thing that the logistic regression is aimed to predict. I is not going to go into the maths of this here as this is roofed in full detail in my Kaggle notebook.

Once we’ve the weights of evidence for every banded feature, we are able to visualise the trend. From the Kaggle data used for bank churn prediction, I actually have included a few features for instance the transformations.

The red bars surrounding each weights of evidence show a 95% confidence interval, implying we’re 95% sure that the weights of evidence would fall inside this range. Narrow intervals are related to robust groups which have sufficient volume to be confident within the weights of evidence.

For instance, categories 16 and 22 of the grouped balance have low volumes of consumers leaving the bank (19 and 53 cases in each group respectively) and have the widest confidence intervals.

The patterns reveal insights concerning the feature relationship and the prospect of a customer leaving the bank. The age feature is barely simpler to know so we are going to tackle that first.

As a customer gets older they usually tend to leave the bank.

The trend is fairly clear and mostly monotonic except some groups, for instance 25–34 12 months old individuals are less prone to leave than 18–24 12 months old cases. Unless there’s a robust argument to support why that is the case (domain knowledge comes into play!), we may consider grouping these two categories to make sure a monotonic trend.

A monotonic trend is essential when making decisions to grant credit or an insurance policy as this is commonly a regulatory requirement to make the models interpretable and not only accurate.

This brings us on to the balance feature. The pattern isn’t clear and we don’t have an actual argument to make here. It does seem that customers with lower balances have less likelihood to go away the bank but you would wish to band several of the groups to make this trend make any sense.

By grouping categories 2–9, 13–21 and leaving 22 by itself (into bins 1, 2 and three respectively) we are able to begin to see the trend. Nevertheless, the down side of that is losing granularity in our features and certain impacting downstream model performance.

For the Kaggle competition, my model didn’t must be explainable, so I didn’t regroup any of the features and just focused on producing essentially the most predictive rating based on the automated groupings I applied. In an industry setting, I might imagine twice about doing this.

It’s value noting that our insights are limited to the features we’ve available and there could also be other underlying causes for the observed behaviour. For instance, the age trend could have been driven by policy changes over time equivalent to the move to online banking, but there is no such thing as a feasible technique to capture this within the model without additional data being available.

If you would like to perform auto groupings to numeric features, apply this transformation and make these associated graphs for yourselves, they will be created for any binary classification task using the Python repository I put together here.

Once these features can be found, we are able to fit a logistic regression. The fitted logistic regression can have an intercept and every feature within the model can have a coefficient assigned to it. From this, we are able to output the probability that somebody goes to go away the bank. I won’t spend time here discussing how I fit the regression, but as before, all the main points can be found in my Kaggle notebook.

The fitted logistic regression can output a probability, nonetheless this isn’t particularly useful for non-technical users of the rating. As such, we want to calibrate these probabilities and transform them into something neater and more interpretable.

Do not forget that the logistic regression is geared toward predicting the log odds. We are able to create the rating by performing a linear transformation to those odds in the next way:

In credit risk, the points to double the chances and 1:1 odds are typically set to twenty and 500 respectively, nonetheless this isn’t at all times the case and the values may differ. For the needs of my evaluation, I stuck to those values.

We are able to visualise the calibrated rating by plotting its distribution.

I split the distribution by the goal variable (whether a customer leaves the bank), this provides a useful validation that every one the previous steps have been done accurately. Those more prone to leave the bank rating lower and those that stay rating higher. There’s an overlap, but a rating is never perfect!

Based on this rating, a marketing department may set a rating cut-off to find out which customers must be targeted with a specific marketing campaign. This cut-off will be set by taking a look at this distribution and converting a rating back to a probability.

Translating a rating of 500 would give a probability of fifty% (keep in mind that our 1:1 odds are equal to 500 for the calibration step). This may imply that half of our customers below a rating of 500 would go away the bank. If we wish to focus on more of those customers, we might just need to lift the rating cut-off.

Representing Our Rating as a Scorecard

We already know that the logistic regression is made up of an intercept and a set of weights for every of the used features. We also know that the weights of evidence have a direct linear relationship with the log odds. Knowing this, we are able to convert the weights of evidence for every feature to know its contribution to the general rating.

I actually have displayed this for all features within the model in my Kaggle notebook, but below are examples we’ve already seen when transforming the variables into their weights of evidence form.

Age

Balance

The advantage of this representation, versus the weights of evidence form, is it should make sense to anyone with no need to know the underlying maths. I can tell a marketing colleague that customers age 48 to 63 years old are scoring lower than other customers. A customer with no balance of their account is more prone to leave than someone with a high balance.

You’ll have noticed that within the scorecard the balance trend is the alternative to what was observed on the weights of evidence stage. Now, low balances are scoring lower. That is as a result of the coefficient attached to this feature within the model. It’s negative and so is flipping the initial trend. This may occur as there are numerous interactions happening between the features through the fitting of the model. A call should be made whether these varieties of interactions are acceptable or whether you’ll need to drop the feature if the trend becomes unintuitive.

Supporting documentation can explain the complete detail of any rating and the way it’s developed (or at the very least should!), but with just the scorecard, anyone should give you the option to get immediate insights!

Conclusion

We have now explored a few of the key steps in developing a rating based on a logistic regression and the insights that it will possibly bring. The simplicity of the ultimate output is why such a rating continues to be used to this present day within the face of more advanced classification techniques.

The rating I developed for this competition had an area under the curve of 87.4%, while the highest solutions based on ensemble techniques were around 90%. This shows that the straightforward model continues to be competitive, although not perfect when you are only on the lookout for accuracy. Nevertheless, if on your next classification task you’re on the lookout for something easy and simply explainable, what about considering a scorecard to achieve insights into your data?

Reference

[1] Walter Reade, Ashley Chow, Binary Classification with a Bank Churn Dataset (2024), Kaggle.