"Automating the statistician": Tight and rigorous proof-via-simulation
We show how to convert simulations into rigorous Type I error bounds for general statistical procedures. We require a well-specified exponential family model for observations, a pre-specified plan for sampling and hypothesis rejection, a gridding of a compact region of interest in the null hypothesis space, and (many) simulations taken over the grid. By tuning critical values, we can achieve a Type I error guarantee at a fixed level, with conservatism that can be reduced with computing power. Applications include problems with highly adaptive sampling, frequentist inference for Bayesian procedures, and validation of black-box design and analysis plans. The approach can be extended to work with confidence intervals, FDR, and properties of bounded estimators. In this talk we will discuss the key ideas: Monte Carlo simulation, Taylor expansions, and bounding arguments. Our team has released a prototype software implementation at GitHub.