When working with time-series data it might be vital to use filtering to remove noise. This story shows easy methods to implement a low-pass filter in SQL / BigQuery that may turn out to be useful when improving ML features.
Filtering of time-series data is probably the most useful preprocessing tools in Data Science. In point of fact, data is nearly all the time a mixture of signal and noise where the noise is just not only defined by the dearth of periodicity but additionally by not representing the knowledge of interest. For instance, imagine every day visitation to a retail store. In case you are serious about how seasonal changes impact visitation, you may not be serious about short-term patterns attributable to weekday changes (there could be an overall higher visitation on Saturdays in comparison with Mondays, but that is just not what you might be serious about).
time-series filtering is a cleansing tool on your data
Though this might appear like a small issue in the information, noise or irrelevant information (just like the short-term visitation pattern) definitely increases your feature complexity and, thus, impacts your model. If not removing that noise, your model complexity and volume of coaching data must be adjusted accordingly to avoid overfitting.
That is where filtering involves the rescue. Much like how one would filter outliers from a training set or less vital metrics from a feature set, time-series filtering removes noise from a time-series feature. To place it short: time-series filtering is a cleansing tool on your data. Applying time-series filtering will restrict your data to reflect only the frequencies (or timely patterns) you might be serious about and, thus, ends in a cleaner signal that may enhance your subsequent statistical or machine-learning model (see Figure 1 for an artificial example).
An in depth walkthrough of what a filter is and the way it really works is beyond the scope of this story (and a really complex topic usually). Nevertheless, on a high level, filtering could be seen as a modification of an input signal by applying one other signal (also called kernel or filter…