Introducing useful climate datasets and validating a worldwide warming prediction

During a dreary stretch of June and July in Boston, it appeared to rain each time my family had planned to do something fun. We began to wonder if we were stuck in a rainy pattern and asked, “Does the indisputable fact that it has rained loads for 3 days straight, make it more likely to rain tomorrow?” I spotted that this query is straightforward to reply using available weather data.

This text presents the US weather datasets I used, the Python/pandas program I wrote to research the information, and the outcomes. Briefly, yes, a stretch of rainy days strongly predicts more rain. And, surprisingly, the longer the stretch of rain, the more likely it’s to rain the following day. The outcomes also prove a prediction from global warming models — there may be more rain now than in previous years.

There are two key datasets about rainfall from the US Oceanic and Atmospheric Administration (NOAA). I used Hourly Precipitation Data (HPD). The outline page is mostly helpful, however the link to the dataset is mistaken and points to an older version. The brand new dataset is here and covers the period 1940 to 2022. (2023 data might be added.) HPD has effective granularity with hourly precipitation amounts from greater than 2000 collection stations across the US. The info files contain all years for one station in each CSV file. I used only the day by day totals, however the hourly information is perhaps useful for future evaluation.

What about when it snows moderately than rains? Snow accumulation is melted to search out the equivalent rain amount. So all data in HPD includes liquid rain, melted snow, and the whole lot in between resembling slush and hail.

There’s one other priceless dataset named Local Climatological Data (LCD) that may very well be used for similar evaluation. LCD accommodates way more information than simply precipitation and in addition includes temperature, sunrise/sunset, air pressure, visibility, wind speed, fog, smoke, monthly summaries, and more. LCD is updated day by day, so it accommodates yesterday’s data. To make use of it it is advisable to decode the Integrated Surface Dataset (ISD) station numbers.

The rain evaluation program is written in Python/pandas. I wrote the code to be readable as is, nevertheless it is price exploring some specific features.

This system can read the complete list of HPD stations or a particular list of stations from a text file. This feature is used to re-run this system with various parameters while ensuring to make use of the very same stations as a previous run.

from rain_helpers import ALL_STATION_FILES
STATION_LIST_INPUT = "/Users/chuck/Desktop/Articles/hpd_stations_used_list_1940-1950.txt"
ALL_STATIONS = True  # use every station, or a particular list ?
# Select either all stations we find out about, or a particular list of stations (often from a previous run of this program)
if (ALL_STATIONS):
station_files = ALL_STATION_FILES
else:
with open(STATION_LIST_INPUT, 'r') as fp:  
data = fp.read()
station_files = data.split("n")
fp.close()

One other handy feature is the power to decide on a subset of the station files. You’ll be able to run the code in a short time with 1/100 of the stations for debugging, or something like 1/3 for an accurate approximation of the outcomes. By the Law Of Large Numbers, my tests with 1/3 (about 600 stations) produced results which are nearly an identical to the complete dataset.

SKIP_COUNT = 3 # 1 = don't skip any.
for i in range (0, len(station_files), SKIP_COUNT):    
station_url = HPD_LOCAL_DIR + station_files[i] 
stationDF = pd.read_csv(station_url, sep=',', header='infer', dtype=str)

One other speed improvement is to download the entire stations files to your local machine, so that you don’t must fetch them from NOAA every time. The total set is about 20GB. In the event you don’t have this extra space, the code runs just effective while reading from the cloud.

HPD_CLOUD_DIR = "https://www.ncei.noaa.gov/data/coop-hourly-precipitation/v2/access/"  # Hourly Precipitation Data (HPD)
HPD_LOCAL_DIR = "/Users/chuck/Desktop/Articles/NOAA/HPD/"
station_url = HPD_LOCAL_DIR + station_files[i]  # toggle between local and cloud

The trickiest a part of the code is the look-back on each date to see if it has been rainy for a stretch of previous days. The issue is that the information to be looked up is inside the exact same DataFrame, a self-join. It’s tempting to make use of a loop over the DataFrame and, for every row, look up previous dates as you go. But loops over large data structures are bad style in any programming language, especially pandas. My code solves this problem by taking a snapshot of the DataFrame, creating fields on each row that contain the nine previous dates (and one for tomorrow), after which using those fields to hitch with the snapshot.

    # Grab a snapshot for a self-join later. Adjust fields names to avoid confusion after the join.
stationCopyDF = stationDF
stationCopyDF = stationCopyDF[["STATION","DATE","DlySumToday"]]  # keep just what we want
stationCopyDF = stationCopyDF.rename({"DlySumToday":"DlySumOther", "DATE":"DATEother"}, axis='columns')  
# Add in another dates, for which we'll pull in rainfall.
stationDF["DATE_minus9"] = stationDF["DATE"] - pd.offsets.Day(9)
stationDF["DATE_minus8"] = stationDF["DATE"] - pd.offsets.Day(8)
...
stationDF["DATE_minus1"] = stationDF["DATE"] - pd.offsets.Day(1)
stationDF["DATE_plus1"] = stationDF["DATE"] + pd.offsets.Day(1)
# Join other rainfall onto base record. Adjust column names to clarify what we did.
stationDF = stationDF.merge(stationCopyDF, how='inner', left_on=["STATION","DATE_minus9"], right_on = ["STATION","DATEother"])
stationDF = stationDF.rename({"DlySumOther":"DlySum9DaysAgo"}, axis='columns')  
stationDF = stationDF.drop(columns=["DATEother"])
stationDF = stationDF.merge(stationCopyDF, how='inner', left_on=["STATION","DATE_minus8"], right_on = ["STATION","DATEother"])
stationDF = stationDF.rename({"DlySumOther":"DlySum8DaysAgo"}, axis='columns')  
stationDF = stationDF.drop(columns=["DATEother"])
....
stationDF = stationDF.merge(stationCopyDF, how='inner', left_on=["STATION","DATE_minus1"], right_on = ["STATION","DATEother"])
stationDF = stationDF.rename({"DlySumOther":"DlySum1DayAgo"}, axis='columns')  
stationDF = stationDF.drop(columns=["DATEother"])
stationDF = stationDF.merge(stationCopyDF, how='inner', left_on=["STATION","DATE_plus1"], right_on = ["STATION","DATEother"])
stationDF = stationDF.rename({"DlySumOther":"DlySumTomorrow"}, axis='columns')  
stationDF = stationDF.drop(columns=["DATEother"])

After getting the previous days’ rainfalls on each row, the code easily finds the length of every rainy period. Note that when calculating what number of days it has been raining, today counts as a day.

    stationDF["DaysOfRain"] = 0   
stationDF.loc[(stationDF["DlySumToday"] >= RAINY), "DaysOfRain"] = 1
stationDF.loc[(stationDF['DlySumToday'] >= RAINY) & (stationDF['DlySum1DayAgo'] >= RAINY), 'DaysOfRain'] = 2
stationDF.loc[(stationDF['DlySumToday'] >= RAINY) & (stationDF['DlySum1DayAgo'] >= RAINY) & (stationDF['DlySum2DaysAgo'] >= RAINY), 'DaysOfRain'] = 3
... etc

Using the years 2000 to 2021 inclusive, there are 1808 stations with valid data, with 8,967,394 data points (a date, location, and rain amount).

• The common rainfall over all data points was 0.0983 inches, or about 1/10 of an inch.
• The fraction of days it was rainy (≥ 0.5 inches) was 6.2%.
• The fraction of dry days (≤ 0.05 inches) was 78.0%.

The reply to the unique query that prompted this project is,

Yes, rainy days predict rain tomorrow. The longer it has been raining (as much as 8 days), the more likely it’s to rain again.

And a related result…

Rainy days predict how much rain is anticipated tomorrow. The longer it has been raining (as much as 7 days), the more rain is probably going tomorrow.

Two charts show this result.

I experimented with different settings for “rainy day”, changing it from 0.5 inches to 0.75 and 1.0. These changes show the identical general phenomena of rainy days predicting more rain, but without perfect correlation through eight days. The definition of “rainy” as 0.5 inches appears to be the sweet spot for predicting tomorrow’s rain.

You would possibly wonder where it could rain for 10 days straight. Over 22 years across your entire US, with almost nine million data points, there have been only 118 runs of such weather. Among the places were: Boca Raton, FL; San Juan, PR; Kahuna Falls, HI; Kaumana, HI; Kihalani, HI; Paakea, HI; Pascagoula, MS; Quinault, WA; and Quilcene, WA.

Sequential days of dryness (< 0.05 inches of rain) also correlate well with dryness the following day, however the prediction shouldn't be as strong because the possibilities of a dry day tomorrow are very close to one another. The prospect of a dry day tomorrow is all the time near the general mean of 78%.

Sequential days of dryness are somewhat higher at predicting the quantity of rain expected the following day.

An obvious related query is whether or not the outcomes described here have modified because the earth’s temperature rises from climate change. I ran the identical evaluation of US data from 1940 to 1960, 1960 to 1980, and 1980 to 2000.

The central result is similar — rainy days predict more rain. The precise numbers are barely different in every time span, but they don’t change the strong correlation. For instance, from 1960 to 1980, there have been 1388 stations with valid data and 6,807,917 data points, with these results:

Fraction of days it's rainy after 1 rainy day = 17.3%
Fraction of days it's rainy after 2 rainy days = 19.6%
Fraction of days it's rainy after 3 rainy days = 27.4%
Fraction of days it's rainy after 4 rainy days = 37.1%
Fraction of days it's rainy after 5 rainy days = 43.8%
Fraction of days it's rainy after 6 rainy days = 51.5%
Fraction of days it's rainy after 7 rainy days = 52.4%

A more essential corollary, predicted by climate change models, is that because the earth warms up, it’ll rain more. The HPD dataset can confirm this, at the least over the past 80 years.

The straightforward approach could be to take the entire current climate stations (about 2000) and take a look at rainfall data during each decade. But there may be a possible bias with doing that. There are more weather stations now than there have been in 1940, since stations have step by step been added over the past 80 years. It is feasible that newer stations have been inbuilt rainy places. If that’s the case, newer data would show more rain, but only because the general set of stations is wetter than the 1940 set.

A more accurate approach is to search out the set of stations that had rain data within the Nineteen Forties, after which use the identical stations for each decade. My program can do that since it emits the list of stations actually used on each run. So first I discovered data for the years 1940 to 1950, after which re-used the emitted station list again for 1950 to 1960, then 1960 to 1970, etc. That is about 840 stations with between 400K and a pair of.5M data points in the assorted a long time.

The common rainfall in each decade must be very close — by the Law Of Large Numbers again. However the chart below shows significantly increased rain over the identical collection stations. It is a remarkable result supporting a key predication of worldwide warming models.

https://www.weather.gov/rah/virtualtourlist — How the US National Weather Service makes forecasts.

https://www.ncdc.noaa.gov/cdo-web/datasets — Overview of US Oceanic and Atmospheric Administration datasets.

https://www.epa.gov/climate-indicators/climate-change-indicators-us-and-global-precipitation — US Environmental Protection Agency report on precipitation increase from climate change.