Crafting Online Surveys and AI Survey Assistance

About

Workshop content and objectives

In this workshop we will explain and put into practice the steps necessary to design and collect online surveys as arranged in three major steps: pre-fielding, fielding and post-fielding. The lectures will present evidence-based studies for each of the steps in light of the most recent literature (e.g. Biffignandi & Bethlehem, 2021), journal special issues (e.g. Maslovskaya et al., 2022), and meta-analyses (e.g. Daikeler et al., 2020).

In the pre-field stage the participants will be getting familiar with the concepts of sampling (Chapter 4 in Biffignandi & Bethlehem, 2021), mode elaboration, questionnaire preparation, question types & design (e.g. Pielot & Callegaro, 2024), questionnaire logic, branching and randomization (Feeney & Feeney, 2019), use of vignettes (McInroy & Beer, 2022), questionnaire pre-testing, technical preparation, and survey paradata (Callegaro, 2013; McClain et al., 2019), nonresponse strategy, and general management of the survey project (Callegaro et al., 2015). During the fielding stage the participant will recruit, measure (collect data) and learn how to monitor the fieldwork. In the post-fielding phase, the participants will get familiar with preparing the dataset, conduct some simple preliminary analysis and document the whole project getting it ready for publication and archiving (Callegaro et al., 2015). The last step consists of weighting the survey (Chapter 12 in Biffignandi & Bethlehem, 2021), advanced analytics, writing the final survey report/ paper and creating different social media assets to advertise the study. While we won’t have time to conduct such steps during the class, we will highlight strategies to tackle them.

Other topics related to online surveys are part of the workshop: sample sources for web surveys such as probability and non-probability online panels (Callegaro et al., 2014), plus crowdsourcing platforms. Big Data initially promised to reduce or even replace surveys, but that did not happen (Callegaro & Yang, 2018). Concepts of inattentive and fraudulent respondents will also be debated in light of the recent literature on the topic (Johnson et al., 2024). Mobile surveys and their advantages (Maxl, 2009) cannot be left out, as well as the nascent literature on wearables, apps & sensors (WAS) (Keusch et al., Forthcoming) The last part of the workshop will be dedicated to show how and how well (or not) the usage of AI can assist in all the steps of the survey project (Callegaro & Yang, 2025).

Organizational structure of the workshop

The course will be extremely practical and taught in a workshop style where the lectures will be using presentation slides as a roadmap. After each concept has been introduced and discussed, the participants will be invited to work individually and in small groups on each of the steps of the survey process.

The class will be highly practical. For example, participants will need to draw a sample, craft the survey email invitations and reminders, program the questionnaire, test it, and work with paradata. The participants are invited to work on their own survey project, if they have one already, if not they will be working on an existing online survey that is very well documented.

We will use Qualtrics as survey platform. If a participant already has an account via their institution, they can use that one, if not, they will be given a Qualtrics temporary one. If participants want to use another platform, they should contact the instructor beforehand to make arrangements.

Participants will be strongly encouraged to ask questions during lectures, especially on how the content relates to their own research.

The emphasis of the class will be on data collection more than data analysis as we won’t have time in the five days to discuss survey statistics and data analysis concepts in-depth.

Learning objectives

By the end of the course, you will:

Learn how carry out an online survey from start to finish
How to design, program, test, and field an online questionnaire
Strategies to be more efficient, save time when working on a web survey and commit less errors
How to maximise response rates and establish a nonresponse strategy
How to evaluate the quality of different online samples
If using non-probability online samples and or crowdsourcing platforms, strategies to mitigate survey fraud
Learn the latest research on best practices for each of the steps and will return to your institution with the lecture slides and the readings for each class
Learn what AI tools can help in each of these steps and how to evaluate their quality

Software and hardware requirements

As a participant you are required to come to class with your laptop in order to be able to work on the exercises. Qualtrics will be used for the survey programming & fielding stage. The participants will work on Google docs, presentations and spreadsheets during the class practical assignments. Class material will be shared via Google drive so participants are invited to provide their Gmail address before the start of the class so we could share a folder with all the materials and readings.

Daily schedule

Day 1
Introduction
Pre-fielding: Coverage, probability sampling & sample sizes & survey platforms
Pre-fielding: Non-probability samples & fraudulent respondents
Pre-fielding: Survey platforms & survey triggering options
Day 2
Pre-fielding: questionnaire preparation, question types & programming in Qualtrics
Pre-fielding: Accessibility for online surveys, mobile optimization & paradata
Day 3
Mobile only surveys & WAS
Pre-fielding: Questionnaire testing
Pre-fielding: Nonresponse strategy
Pre-fielding: management of data collection
Day 4
Fielding & monitoring
Post-fielding: data preparation, preliminary results
Day 5
Post-fielding: Documentation and archiving
AI assisted surveys in light of the most recent literature

**The Summer School cannot grant credits. We only deliver a Certificate of Participation, i.e. we certify your attendance.**

If you consider using Summer School workshops to obtain credits (ECTS), you will have to investigate at your home institution (contact the person/institute responsible for your degree) to find out whether they recognise the Summer School, how many credits can be earned from a workshop/course with roughly 35 hours of teaching, no graded work, and no exams.

Make sure to investigate this matter before registering if this is important to you

Course leader

Mario Callegaro is an independent consultant with over 35 years of experience on survey research methods. Mario worked for 15 years at Google as survey research scientist in the marketing organization first, and then as user experience researcher in the Cloud organization. Mario obtained a Master and Ph.D. in Survey Research and Methodology from the University of Nebraska, Lincoln

Target group

You will find the course useful if you are a master, PhD student, postdoctoral researcher, or other academic researcher working in social sciences.

Prerequisites

The participants are required to have good knowledge of the key survey methodology concepts such as the Total Survey Error (TSE) framework, questionnaire design, and basic knowledge about sampling and weighting

Fee info