What to do when the unit of remark differs from the unit of randomization

A/B tests are the golden standard of causal inference because they permit us to make valid causal statements under minimal assumptions, due to randomization. Actually, by randomly assigning a treatment (a drug, ad, product, …), we’re capable of compare the consequence of interest (a disease, firm revenue, customer satisfaction, …) across subjects (patients, users, customers, …) and attribute the typical difference in outcomes to the causal effect of the treatment.

Sometimes it happens that the unit of treatment task differs from the unit of remark. In other words, we don’t take the choice on whether to treat each remark independently, but relatively in groups. For instance, we’d determine to treat all customers in a certain region while observing outcomes at the client level, or treat all articles of a certain brand, while observing outcomes on the article level. Normally this happens due to practical constraints. In the primary example, the so-called geo-experiments, it happens because we’re unable to trace users due to cookie deprecations.

When this happens, treatment effects are not independent across observations anymore. Actually, if a customer in a region is treated, also other customers in the identical region will probably be treated. If an article of a brand just isn’t treated, also other articles of the identical brand is not going to be treated. When doing inference, we’ve to take this dependence under consideration: standard errors, confidence intervals, and p-values ought to be adjusted. In this text, we’ll explore tips on how to do this using cluster-robust standard errors.

Imagine you were a web-based platform and also you were excited by increasing sales. You simply had a fantastic idea: showing a carousel of related articles at checkout to incentivize customers so as to add other articles to their basket. With the intention to understand whether the carousel increases sales, you select to AB test it. In principle, you might just determine for each order whether to display the carousel or not, at random. Nonetheless, this could give…