N-of-1 Trials in Practice
To point out you an example of this technique in practice, I’ll conduct my very own evaluation on a number of data collected from my Whoop strap from April twenty seventh, 2018 to October fifth, 2019. Our research query for this N-of-1 study is:
Does drinking alcohol result in poor sleep?
As an athlete and epidemiologist, I’m very aware of how detrimental alcohol could be in your sleep, athletic performance and general wellbeing. I’ve continuously been told how athletes shouldn’t drink, nonetheless its one thing to be told, but one other to see the evidence for yourself. Once I began wearing my Whoop I noticed how my sleep rating (a metric calculated by the Whoop app) would suffer after drinking alcohol. Sometimes even a day later, I assumed I could still see the effect. These observations made me wish to do my very own evaluation, which I can finally complete now.
Notes on the Data
The 2 variables of interest in our evaluation is sleep performance rating and alcohol consumption. Sleep performance rating ranges from 0 to 100 and is a metric calculated by the Whoop app from biometric data like respiratory rate, light sleep duration, slow wave sleep duration, and REM sleep duration.
The alcohol consumption variable is the response to the query “Did you could have any alcoholic drinks yesterday?” that’s responded to by Whoop users every day upon waking up. I at all times answered these questions truthfully and consistently, although we’re limited in our data in that the app doesn’t ask questions on how much alcohol was consumed. Which means that all levels of alcohol consumption are treated equally, which eliminates the chance to investigate the connection on a deeper level. There was some missing data in our alcohol feature, but this missing information was imputed with ‘No’s as I do know from personal experience that if I had drunk the night before I used to be sure to mark it within the app.
Exploratory Data Evaluation
Step one in any evaluation is to do some exploratory data evaluation (EDA). That is simply to get a general idea of what our data looks like, and to create a visible that can help direct our investigation.
From the above box-plots, we see that average sleep rating appears to be higher when no alcohol was consumed, and to have a narrower distribution. Curiously, there appears to be more outliers in sleep performance rating when alcohol isn’t consumed. Perhaps travel days and jet-lag can account for these outliers, as I traveled overseas 5 times during this sample period.
Now that we’ve got gotten first have a look at the info of interest, its time to dig into the statistical evaluation.
Hypothesis Testing
To reply our research query, I shall be conducting hypothesis testing. Hypothesis testing is a statistical technique that permits us to make inferences a couple of population based on some sample data. On this case, we are trying to infer if me drinking alcohol is related to having poor sleep that night. We don’t have data on alcohol consumption and sleep for each night I’ve been alive, so we study our sample data as a proxy.
Step one in hypothesis testing is to formulate my hypotheses. A ‘null hypothesis’ is the idea that nothing interesting is going on or that there is no such thing as a relationship or effect. In our case the null hypothesis is: There isn’t any difference in mean sleep performance between nights through which alcohol was consumed and was not consumed.
An ‘alternative hypothesis’ is the hypothesis that contradicts the null, and claims that actually there’s something interesting happening. In our example the alternative hypothesis is: There may be a difference in mean sleep performance between nights through which alcohol was consumed and was not consumed.
Selecting a Statistical Test
To evaluate which of those hypotheses is true, we’ve got to selected a statistical test. We’re curious if the typical sleep rating for nights through which I drank alcohol is different from the typical sleep rating for nights through which I didn’t drink alcohol, and so shall be using a difference in means to check this. Specifically, our test statistic is: Mean sleep performance with no alcohol — Mean sleep performance with alcohol
Now that we’ve got defined our framework, we will use R to calculate our test statistic and evaluate our hypotheses.
Conducting our Evaluation in R
From our sample data we will calculate our observed test statistic. The code in R is included below.
test_stat <- data |>
specify(formula = sleep_performance ~ alcohol) |>
calculate(
stat = "diff in means",
order = c("No", "Yes")
)
Our test statistic is 8.01. This number implies that the typical sleep rating for nights through which I consumed no alcohol is 8.01 points higher than nights through which I did eat alcohol.
The following step within the evaluation is to generate a null distribution from our sample data. A null distribution represents all different values of test statistic we might observe if samples were drawn repeatedly from the population. The distribution is supposed to reflect the variation within the test statistic purely as a consequence of random sampling. The null distribution is created in R below:
set.seed(42) #Setting seed for reproducibilitynull_distribution <- data |>
specify(formula = sleep_performance ~ alcohol) |>
hypothesize(null = "independence") |>
generate(reps = 1000, type = "permute") |>
calculate(
stat = "diff in means",
order = c("No", "Yes")
)
What we’re doing above is taking samples with substitute from our data, and calculating the difference in means from those samples. We do that 1000 times to generate a big enough distribution in order that we will determine if our observed test statistic is critical.
After we’ve got our null distribution and test statistic, we will calculate a two-sided p-value for an alpha of 0.05. The p-value could be considered the probability of getting a test statistic that’s as extreme or greater than our observed test statistic if the null hypothesis is true. Put into plain words; it represents how likely it could be to see this result if there was no true association. We calculate a two-sided p-value in R below, as we’re eager about the opportunity of the test statistic being greater or lesser than expected.
p_value <- null_distribution|>
get_p_value(test_stat, direction = "each")
Our p-value is 0.017 which implies that our finding is critical on the alpha=0.05 level, which is a commonly accepted level of significance in statistics. It implies that the difference in sleep rating we found was significant! Now we have the evidence to reject the null hypothesis and accept the choice; there is a difference in mean sleep performance between nights through which alcohol was consumed and was not consumed.
I’ve included a helpful visualization of the null distribution, test statistic, and 95% quantile range below. The grey bars are the various possible test statistics calculated from our 1000 samples, and the orange line represents the density of those values. The blue dashed lines represent the 97.fifth and a couple of.fifth quantiles of this distribution, beyond which our test statistic (in red) is shown to be significant.
Final Conclusions
Well, it seems my coaches were right all along! Our evaluation found that my average sleep rating after I didn’t eat alcohol was 8.01 points higher than my average sleep rating after I did eat alcohol. This difference was found to be statistically significant, with a p-value of 0.017, meaning that we reject the null hypothesis in favor of the choice. This statistical result backs up my personal experience, giving me a quantitative result that I can have faith in.
Going Further
Now that I even have this initial evaluation under my belt, I can explore more associations in my data, and even use more complicated methods like forecasting and machine learning models.
This evaluation is a really basic example of an N-of-1 study, and isn’t without limitations. My study was observational reasonably than experimental, and we cannot declare causality, as there are numerous other confounding variables not measured by my Whoop. If I wanted to search out a causal relationship, I might must fastidiously design a study, record data on all possible confounders, and discover a technique to blind myself to the treatment. N-of-1 studies are hard to do outside of a clinical setting, nonetheless we will still find meaningful associations and relationships by asking easy questions of our data.
I hope that after this tutorial you’re taking the initiative to download your individual data from whatever fitness tracker you may get your hands on, and mess around with it. I do know everyone can provide you with a hypothesis about how some variable affects their health, but what most individuals don’t realize, is that you just’re closer to getting a quantifiable answer to that query than you’re thinking that.
References and Further Reading
[1] Davidson, K., Cheung, K., Friel, C., & Suls, J. (2022). Introducing Data Sciences to N-of-1 Designs, Statistics, Use-Cases, the Future, and the Moniker ‘N-of-1’ Trial. Harvard Data Science Review, (Special Issue 3). https://doi.org/10.1162/99608f92.116c43fe
[2] Lillie EO, Patay B, Diamant J, Issell B, Topol EJ, Schork NJ. The n-of-1 clinical trial: the last word strategy for individualizing medicine? Per Med. 2011 Mar;8(2):161–173. doi: 10.2217/pme.11.7. PMID: 21695041; PMCID: PMC3118090.
[3] Daza EJ. Causal Evaluation of Self-tracked Time Series Data Using a Counterfactual Framework for N-of-1 Trials. Methods Inf Med. 2018 Feb;57(1):e10-e21. doi: 10.3414/ME16–02–0044. Epub 2018 Apr 5. PMID: 29621835; PMCID: PMC6087468.
[4] Schork, N. Personalized medicine: Time for one-person trials. Nature 520, 609–611 (2015). https://doi.org/10.1038/520609a