Abstract

Recently developed safe tests based on e-values, and anytime-valid confidence intervals form a suite of statistical methods that simplify and optimise the design, conduct, and inferential process for both single-lab experiments and large-scale multi-lab (replication) studies. Safe tests combine the interpretability of Bayes factors (i.e. measuring evidence for and against a null hypothesis) with frequentist power and type I error guarantees. These guarantees are maintained even if the safe test is conducted after each observation and used to determine whether the experiment should be (prematurely) stopped or continued. Similarly, unlike 95% (Bayesian) credible intervals and 95% (frequentist) confidence intervals, a 95% anytime-valid confidence interval will, with at least 95% chance, cover the true effect size regardless of whether or how data collection is stopped. In this workshop we will provide an introduction to this novel framework of statistical inference, and show how it can be exploited to yield more generalisable conclusions with less data. We will alternate between short theoretical lectures and hands-on practical sessions that focus on designing and making inference for practical problems with R/RStudio.

Grünwald, P. D., de Heide, R., Boehm, U., Turner, R. J., & Ly, A.

Schedule

We alternate between theory and practical sessions using R. For the practical sessions R Markdown documents are made, and the reader is invited to adapt the code.

Time	Title	Presenter
09.00 – 09.30	Introduction: p-values, Bayes factors and e-values, Rianne’s slides	Rianne de Heide
09.30 – 10.30	Practical 1: Testing the equality of two normal means. R Markdown: 0. Installation guide, I. Testing, II. Design	Udo Boehm
10.30 – 11.00	Confidence/credible intervals, and anytime-valid confidence sequences, Alexander’s slides.	Alexander Ly
11.00 – 11.30	Practical 2: Interval estimation of a normal mean. R Markdown: III. Interval estimation	Rosanne J. Turner
11.30 – 12.00	Extensions to other inference problems	Peter Grünwald

Programme

Introduction: p-values, Bayes factors and e-values – Rianne de Heide (Vrije Universiteit Amsterdam) [Slides.]

A general overview is provided on p-values, Bayes factors and e-values in the context of sequential testing, and optional stopping in particular.

Practical 1: Testing the equality of two normal means – Udo Boehm (CWI Amsterdam)

In this R tutorial we work through the following R Markdown documents:
0. Installing the safestats package
I. Testing: P-values, subjective Bayes factor, and e-variables
II. Design of experiments: Behaviour of e-variables under alternatives

Confidence/credible intervals, and anytime-valid confidence sequences – Alexander Ly (CWI Amsterdam) [Slides.]

A general overview is given on the conversion between safe tests/e-variables into anytime-valid confidence sequences.

Practical 2: Interval estimation of a normal mean – Rosanne J Turner (CWI Amsterdam/University Medical Center Utrecht)

In this R tutorial we work through the following R Markdown documents:
III. Interval estimation: Classical confidence intervals, (Bayesian) credible interval, and anytime-valid confidence sequences

Extensions to other inference problems – Peter Grünwald (CWI Amsterdam and Leiden University)

We brush on general purpose algorithms that allow for the development of e-variables yielding e-values for general testing problems.

Additional notes and software

– safestats github repository
– In case remote installation doesn’t work: Package tar.gz The download button can be found on top left.