{"title":"Precision-based designs for sequential randomized experiments","authors":"Mattias Nordin, Mårten Schultzberg","doi":"arxiv-2405.03487","DOIUrl":null,"url":null,"abstract":"In this paper, we consider an experimental setting where units enter the\nexperiment sequentially. Our goal is to form stopping rules which lead to\nestimators of treatment effects with a given precision. We propose a\nfixed-width confidence interval design (FWCID) where the experiment terminates\nonce a pre-specified confidence interval width is achieved. We show that under\nthis design, the difference-in-means estimator is a consistent estimator of the\naverage treatment effect and standard confidence intervals have asymptotic\nguarantees of coverage and efficiency for several versions of the design. In\naddition, we propose a version of the design that we call fixed power design\n(FPD) where a given power is asymptotically guaranteed for a given treatment\neffect, without the need to specify the variances of the outcomes under\ntreatment or control. In addition, this design also gives a consistent\ndifference-in-means estimator with correct coverage of the corresponding\nstandard confidence interval. We complement our theoretical findings with Monte\nCarlo simulations where we compare our proposed designs with standard designs\nin the sequential experiments literature, showing that our designs outperform\nthese designs in several important aspects. We believe our results to be\nrelevant for many experimental settings where units enter sequentially, such as\nin clinical trials, as well as in online A/B tests used by the tech and\ne-commerce industry.","PeriodicalId":501330,"journal":{"name":"arXiv - MATH - Statistics Theory","volume":"31 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-05-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - MATH - Statistics Theory","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2405.03487","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
In this paper, we consider an experimental setting where units enter the
experiment sequentially. Our goal is to form stopping rules which lead to
estimators of treatment effects with a given precision. We propose a
fixed-width confidence interval design (FWCID) where the experiment terminates
once a pre-specified confidence interval width is achieved. We show that under
this design, the difference-in-means estimator is a consistent estimator of the
average treatment effect and standard confidence intervals have asymptotic
guarantees of coverage and efficiency for several versions of the design. In
addition, we propose a version of the design that we call fixed power design
(FPD) where a given power is asymptotically guaranteed for a given treatment
effect, without the need to specify the variances of the outcomes under
treatment or control. In addition, this design also gives a consistent
difference-in-means estimator with correct coverage of the corresponding
standard confidence interval. We complement our theoretical findings with Monte
Carlo simulations where we compare our proposed designs with standard designs
in the sequential experiments literature, showing that our designs outperform
these designs in several important aspects. We believe our results to be
relevant for many experimental settings where units enter sequentially, such as
in clinical trials, as well as in online A/B tests used by the tech and
e-commerce industry.