Estimating your possibilities of winning the lottery with sampling

11 min read

·

10 hours ago

Statistical estimates may be fascinating, can’t they? By just sampling a number of instances from a population, you may infer properties of that population akin to the mean value or the variance. Likewise, under the appropriate circumstances, it is feasible to estimate the total size of the population, as I would like to indicate you in this text.

I’ll use the instance of drawing lottery tickets to estimate what number of tickets there are in total, and hence calculate the likelihood of winning. More formally, this implies to estimate the population size given a discrete uniform distribution. We’ll see different estimates and discuss their differences and weaknesses. As well as, I’ll point you to another use cases this approach may be utilized in.

Playing the lottery

Let’s imagine I am going to a state fair and buy some tickets within the lottery. As a knowledge scientist, I would like to know the probability of winning the primary prize, in fact. Let’s assume there’s only a single ticket that wins the primary prize. So, to estimate the likelihood of winning, I want to know the full variety of lottery tickets N, as my probability of winning is 1/N then (or k/N, if I purchase k tickets). But how can I estimate that N by just buying a number of tickets (that are, as I saw, all losers)?

I’ll make use of the actual fact, that the lottery tickets have numbers on them, and I assume, that these are consecutive running numbers (which implies, that I assume a discrete uniform distribution). Say I even have bought some tickets and their numbers so as are [242,412,823,1429,1702]. What do I learn about the full variety of tickets now? Well, obviously there are a minimum of 1702 tickets (as that’s the very best number I even have seen to this point). That provides me a primary lower certain of the variety of tickets, but how accurate is it for the actual variety of tickets? Simply because the very best number I even have drawn is 1702, that doesn’t mean that there are any numbers higher than that. It is extremely unlikely, that I caught the lottery ticket with the very best number in my sample.

Nonetheless, we are able to make more out of the info. Allow us to think as follows: If we knew the center variety of all of the tickets, we could easily derive the full number from that: If the center number is m, then there are m-1 tickets below that middle number, and there are m-1 tickets above that. That’s, the full variety of tickets can be (m-1) + (m-1) + 1, (with the +1 being the ticket of number m itself), which is the same as 2m-1. We don’t know that middle number m, but we are able to estimate it by the mean or the median of our sample. My sample above has the (rounded) average of 922, which yields 2*922-1 = 1843. That’s, from that calculation the estimated variety of tickets is 1843.

That was quite interesting to this point, as just from a number of lottery ticket numbers, I used to be able to present an estimate of the full variety of tickets. Nonetheless, chances are you’ll wonder if that’s one of the best estimate we are able to get. Let me spoil you instantly: It shouldn’t be.

The tactic we used has some drawbacks. Let me display that to you with one other example: Say we now have the numbers [12,30,88], which leads us to 2*43–1 = 85. Which means, the formula suggests there are 85 tickets in total. Nonetheless, we now have ticket number 88 in our sample, so this will not be true in any respect! There may be a general problem with this method: The estimated N may be lower than the very best number within the sample. In that case, the estimate has no meaning in any respect, as we already know, that the very best number within the sample is a natural lower certain of the general N.

A greater approach: Using even spacing

Okay, so what can we do? Allow us to think in a unique direction. The lottery tickets I purchased have been sampled randomly from the distribution that goes from 1 to unknown N. My ticket with the very best number is number 1702, and I ponder, how far-off is that this from being the very best ticket in any respect. In other words, what’s the gap between 1702 and N? If I knew that gap, I could easily calculate N from that. What do I learn about that gap, though? Well, I even have reason to assume that this gap is predicted to be as big as all the opposite gaps between two consecutive tickets in my sample. The gap between the primary and the second ticket should, on average, be as big because the gap between the second and the third ticket, and so forth. There is no such thing as a reason why any of those gaps must be greater or smaller than the others, apart from random deviation, in fact. I sampled my lottery tickets independently, in order that they must be evenly spaced on the range of all possible ticket numbers. On average, the numbers within the range of 0 to N would appear to be birds on an influence line, all having the identical gap between them.

Which means I expect N-1702 to equal the typical of all the opposite gaps. The opposite gaps are 242–0=242, 412–242=170, 823–412=411, 1429–823=606, 1702–1429=273, which supplies the typical 340. Hence I estimate N to be 1702+340=2042. Briefly, this may be denoted by the next formula:

Here x is the largest number observed (1702, in our case), and k is the variety of samples (5, in our case). That is just a brief type of calculating the typical as we just did.

Let’s do a simulation

We just saw two estimates of the full variety of lottery tickets. First, we calculated 2*m-1, which gave us 1843, after which we used the more sophisticated approach of x + (x-k)/k and obtained 2042. I ponder which estimation is more correct now? Are my possibilities of winning the lottery 1/1843 or 1/2042?

To indicate some properties of the estimates we just used, I did a simulation. I drew samples of various sizes k from a distribution, where the very best number is 2000, and that I did a number of hundred times each. Hence we’d expect that our estimates also return 2000, a minimum of on average. That is the consequence of the simulation:

What will we see here? On the x-axis, we see the k, i.e. the variety of samples we take. For every k, we see the distribution of the estimates based on a number of hundred simulations for the 2 formulas we just got to know. The dark point indicates the mean value of the simulations each, which is at all times 2000, independent of the k. That may be a very interesting point: Each estimates converge to the proper value in the event that they are repeated an infinite variety of times.

Nonetheless, besides the common average, the distributions differ quite lots. We see, that the formula 2*m-1 has higher variance, i.e. its estimates are far-off from the true value more often than for the opposite formula. The variance has an inclination to diminish with higher k though. This decrease doesn’t at all times hold perfectly, as that is just as simulation and remains to be subject to random influences. Nonetheless, it is kind of comprehensible and expected: The more samples I take, the more precise is my estimation. That may be a quite common property of statistical estimates.

We also see that the deviations are symmetrical, i.e. underestimating the true value is as likely as overestimating it. For the second approach, this symmetry doesn’t hold: While a lot of the density is above the true mean, there are more and bigger outliers below. How does that come? Let’s retrace how we computed that estimate. We took the largest number in our sample and added the typical gap size to that. Naturally, the largest number in our sample can only be as big as the largest number in total (the N that we would like to estimate). In that case, we add the typical gap size to N, but we are able to’t get any higher than that with our estimate. In the opposite direction, the largest number may be very low. If we’re unlucky, we could draw the sample [1,2,3,4,5], during which case the largest number in our sample (5) could be very far-off from the actual N. That’s the reason larger deviations are possible in underestimating the true value than in overestimating it.

Which is best?

From what we just saw, which estimate is best now? Well, each give the proper value on average. Nonetheless, the formula x + (x-k)/k has lower variance, and that could be a big advantage. It means, that you just are closer to the true value more often. Let me display that to you. In the next, you see the probability density plots of the 2 estimates for a sample size of k=5.

I highlighted the purpose N=2000 (the true value for N) with a dotted line. To begin with, we still see the symmetry that we now have seen before already. Within the left plot, the density is distributed symmetrically around N=2000, but in the appropriate plot, it’s shifted to the appropriate and has an extended tail to the left. Now let’s take a take a look at the grey area under the curves each. In each cases, it goes from N=1750 to N=2250. Nonetheless, within the left plot, this area accounts for 42% of the full area under the curve, while for the appropriate plot, it accounts for 73%. In other words, within the left plot, you might have a probability of 42% that your estimate is not deviating greater than 250 points in either direction. In the appropriate plot, that probability is 73%. Which means, you might be far more more likely to be that near the true value. Nonetheless, you usually tend to barely overestimate than underestimate.

I can inform you, that x+ (x-k)/k is the so-called uniformly minimum variance unbiased estimator, i.e. it’s the estimator with the smallest variance. You won’t find any estimate with lower variance, so that is one of the best you should use, usually.

Use cases

We just saw learn how to estimate the full variety of elements in a pool, if these elements are indicated by consecutive numbers. Formally, it is a discrete uniform distribution. This problem is usually referred to as the German tank problem. Within the Second World War, the Allies used this approach to estimate what number of tanks the German forces had, just through the use of the serial numbers of the tanks that they had destroyed or captured to this point.

We will now consider more examples where we are able to use this approach. Some are:

• You’ll be able to estimate what number of instances of a product have been produced in the event that they are labeled with a running serial number.
• You’ll be able to estimate the variety of users or customers for those who are capable of sample a few of their IDs.
• You’ll be able to estimate what number of students are (or have been) at your university for those who sample students’ matriculation numbers (on condition that the university has not yet used the primary numbers again after reaching the utmost number already).

Nonetheless, bear in mind that some requirements have to be fulfilled to make use of that approach. A very powerful one is, that you just indeed draw your samples randomly and independently of one another. Should you ask your pals, who’ve all enrolled in the identical yr, for his or her matriculation numbers, they won’t be evenly spaced on the entire range of matriculation numbers but will probably be quite clustered. Likewise, for those who buy articles with running numbers from a store, it is advisable to make certain, that this store got these articles in a random fashion. If it was delivered with the products of numbers 1000 to 1050, you don’t draw randomly from the entire pool.

Conclusion

We just saw other ways of estimating the full variety of instances in a pool under discrete uniform distribution. Although each estimates give the identical expected value in the long term, they differ by way of their variance, with one being superior to the opposite. That is interesting because neither of the approaches is fallacious or right. Each are backed by reasonable theoretical considerations and estimate the true population size appropriately (in frequentist statistical terms).

I now know that my probability of winning the state fair lottery is estimated to be 1/2042 = 0.041% (or 0.24% with the 5 tickets I purchased). Possibly I should reasonably invest my money in cotton candy; that may be a save win.

References & Literature

Mathematical background on the estimates discussed in this text may be found here:

• Johnson, R. W. (1994). Estimating the scale of a population. Teaching Statistics, 16(2), 50–52.

Also be happy to ascertain out the Wikipedia articles on the German tank problem and related topics, that are quite explanatory:

That is the script to do the simulation and create the plots shown within the article:

import numpy as np
import random
from scipy.stats import gaussian_kde
import matplotlib.pyplot as plt
if __name__ == "__main__":
N = 2000
n_simulations = 500
estimate_1 = lambda sample: 2 * round(np.mean(sample)) - 1
estimate_2 = lambda sample: round(max(sample) + ((max(sample) - k) / k))
estimate_1_per_k, estimate_2_per_k = [],[]
k_range = range(2,10)
for k in k_range:
diffs_1, diffs_2 = [],[]
# sample without duplicates:
samples = [random.sample(range(N), k) for _ in range(n_simulations)]
estimate_1_per_k.append([estimate_1(sample) for sample in samples])
estimate_2_per_k.append([estimate_2(sample) for sample in samples])
fig,axs = plt.subplots(1,2, sharey=True, sharex=True)
axs[0].violinplot(estimate_1_per_k, positions=k_range, showextrema=True)
axs[0].scatter(k_range, [np.mean(d) for d in estimate_1_per_k], color="purple")
axs[1].violinplot(estimate_2_per_k, positions=k_range, showextrema=True)
axs[1].scatter(k_range, [np.mean(d) for d in estimate_2_per_k], color="purple")
axs[0].set_xlabel("k")
axs[1].set_xlabel("k")
axs[0].set_ylabel("Estimated N")
axs[0].set_title(r"$2times m-1$")
axs[1].set_title(r"$x+frac{x-k}{k}$")
plt.show()
plt.gcf().clf()
k = 5
xs = np.linspace(500,3500, 500)
fig, axs = plt.subplots(1,2, sharey=True)
density_1 = gaussian_kde(estimate_1_per_k[k])
axs[0].plot(xs, density_1(xs))
density_2 = gaussian_kde(estimate_2_per_k[k])
axs[1].plot(xs, density_2(xs))
axs[0].vlines(2000, ymin=0, ymax=0.003, color="grey", linestyles="dotted")
axs[1].vlines(2000, ymin=0, ymax=0.003, color="grey", linestyles="dotted")
axs[0].set_ylim(0,0.0025)
a,b = 1750, 2250
ix = np.linspace(a,b)
verts = [(a, 0), *zip(ix, density_1(ix)), (b, 0)]
poly = plt.Polygon(verts, facecolor='0.9', edgecolor='0.5')
axs[0].add_patch(poly)
print("Integral for estimate 1: ", density_1.integrate_box(a,b))
verts = [(a, 0), *zip(ix, density_2(ix)), (b, 0)]
poly = plt.Polygon(verts, facecolor='0.9', edgecolor='0.5')
axs[1].add_patch(poly)
print("Integral for estimate 2: ", density_2.integrate_box(a,b))
axs[0].set_ylabel("Probability Density")
axs[0].set_xlabel("N")
axs[1].set_xlabel("N")
axs[0].set_title(r"$2times m-1$")
axs[1].set_title(r"$x+frac{x-k}{k}$")
plt.show()

Like this text? Follow me to be notified of my future posts.