19 July 2018
on course website
Addressing the Replication Crisis Using (Bayesian) Model Selection
This 4-day course will highlight the main causes of the replication crisis, such as questionable research practices and publication bias. Subsequently, it will be elaborated and practiced [this is an applied course] how these causes can be addressed by the evaluation of (informative) hypotheses employing the Bayes factor and a generalization of the AIC. There will be attention for replication research and combining evidence from multiple studies.
The first day of the course will start with an elaboration of null-hypothesis significance testing. Subsequently, it will be elaborated why the omnipresent alpha level of .05 leads to questionable research practices, publication bias, and the replication crisis. This day will be concluded with a hand-on introduction to R and the packages Bain and Goric.
The second day of the course will start with an introduction of informative hypotheses like, for example, H1: mu1 > mu2 > mu3, that is, an ordering of three means. Subsequently, it will be elaborated how the Bayes factor can be used to evaluate null, alternative, and informative hypotheses. There will be attention for Bayesian error probabilities and Bayesian updating. This day will be concluded with examples of replication research and combining evidence from multiple studies in which informative hypotheses and Bayes factor play an important role.
The third day of the course will start with an introduction of information criteria (i.e., AIC and the generalization called the GORIC) and a brief repetition of informative hypotheses. Subsequently, it will be elaborated how the GORIC can be used to evaluate null, alternative, and informative hypotheses. There will be attention for GORIC weights, which have a nice interpretation. This day will, like Day 2, be concluded with examples of replication research and combining evidence from multiple studies in which informative hypotheses play an important role.
The fourth day is a lab-meeting. This day you will analyse data using i) null-hypothesis significance testing and the evaluation of informative hypotheses using ii) Bayes factor and iii) a generalization of the AIC (i.e. GORIC). You will make comparisons between these three approaches to understand their strengths and weaknesses. You will also evaluate a replication study and combine the evidence from multiple data sets using informative hypotheses. This day will be concluded with a discussion.
1) Kuiper, R.M. & Hoijtink, H.J.A. (2010). Comparisons of Means Using Exploratory and Confirmatory Approaches. Psychological Methods, 15 (1), (pp. 69-86).
2) Kuiper, R.M., Buskens, V.W., Raub, W. & Hoijtink, H. (2013). Combining statistical evidence from several studies: A method using bayesian updating and an example from research on trust problems in social and economic exchange. Sociological Methods and Research, 42 (1), (pp. 60-81).
3) Bain Tutorial, which is part of the bain zip file downloadable from https://informative-hypotheses.sites.uu.nl/software/bain/
4) goric/restriktor tutorial: http://www.restriktor.ugent.be/tutorial/example6.html
Dr. Rebecca Kuiper
Participants (Research Master Students, PhD students, or post-graduate researchers) from a variety of fields, including sociology, psychology, education, human development, marketing, business, biology, medicine, political science, and communication sciences, may benefit from the course. Participants should have an interest in statistical analyses of their own data without contributing to the replication crisis.
Three important notes:
- A maximum of 24 participants will be allowed in this course.
- After registration, we will ask you to write a short motivation letter after which we will determine whether this is the right course for you.
- You must bring your own laptop to this course.
After attending the course, you will understand the main causes of the replication crisis and how these can be addressed by evaluating of informative hypotheses making use of the Bayes factor and a generalization of the AIC. You will obtain the practical skills necessary to evaluate informative hypotheses. Moreover, you will learn to use informative hypotheses to evaluate replication studies and to combine the evidence from multiple studies.
Certificate of Attendance
EUR 550: Course + course materials + lunch
Tuition fee for PhD candidates from the Faculty of Social and Behavioural Sciences from Utrecht University will be funded by the Graduate School of Social and Behavioural Sciences.
EUR 200: Housing
on course website