ESRA 2025 Preliminary Program
All time references are in CEST
Complex measurements in online self-completion surveys 2 |
| Session Organisers |
Dr Cristian Domarchi (University of Southampton)
Professor Lisa Calderwood (UCL Centre for Longitudinal Studies)
Mr Curtis Jessop (National Centre for Social Research)
|
| Time | Thursday 17 July, 14:00 - 15:00 |
| Room |
Ruppert D - 0.22 |
Accurately capturing complex phenomena is critical for the success of social surveys. As the field increasingly shifts towards online data collection, a key challenge emerges: how to administer complex measures without compromising data quality or comparability with other survey modes.
The session is proposed by Research Strand 5 of the Survey Futures* project, “Complex measurement in self-completion surveys” which focuses on how to collect measures of i) industry and occupation ii) event histories and retrospective data iii) consent for data linkage and re-contact and iv) cognitive assessments in self-completion surveys.
Each of these measures presents unique challenges. Ensuring that online participants provide sufficient detail to allow for accurate industry and occupation coding in the absence of interviewer probing can be difficult. Retrospective life history data collected in self-completion surveys may be less complete due to lack of interviewer support. Consent rates for data linkage are consistently lower in online-administered surveys. Adapting cognitive assessments designed for in-person administration for use online may not be feasible or may result in significant mode effects.
This session invites researchers to share insights into how to improve the collection of these and other types of complex measures in online self-completion surveys. We welcome submissions that present evidence on mode and measurement effects in these and other complex measures, as well as findings from trials aimed at enhancing data quality when collecting complex measures in online surveys. Submissions employing experimental designs or other innovative methodologies that can inform future survey strategies are especially encouraged.
*Survey Futures is a UKRI-ESRC funded research programme dedicated to ensuring that large-scale social surveys in the UK can innovate and adapt in an evolving landscape. The programme is a multi-institution collaboration between universities and survey practice organisations.
Keywords: self-administered surveys, complex measurements, occupation coding, consent, retrospective data, cognitive function
Papers
Meaningful and Measurable? Redesigning the Crime Survey for England & Wales (CSEW) to obtain complex crime data online
Ms Sophie Gurr (Office for National Statistics) - Presenting Author
The Centre for Crime and Justice at the Office for National Statistics (ONS) is redesigning the Crime Survey for England and Wales (CSEW) as part of a wider transformation programme.
Following a public consultation in 2022, key changes were made to the CSEW including redesign from a cross-sectional, face-to-face survey to a multi-modal, longitudinal panel design incorporating an online survey mode.
Prior to this, the CSEW remained largely unchanged since its introduction in 1981 due to the importance of maintaining the time-series. The risk of introducing new modes was mitigated by retaining face-to-face interviews at wave 1. Respondents are interviewed again 12 months later (wave 2), currently by telephone and following further development, online.
One of the critical requirements of the CSEW relates to the collection of accurate incidence data from victims of multiple or ‘complex’ crimes. Our Discovery research concluded that whilst an online mode would likely be effective in obtaining data from respondents with no or simple experiences, it is more challenging for victims of multiple, ‘complex’ or series crimes. Research employing a Respondent Centred Design framework generated insight into how respondents conceptualise and articulate their experience of crime. We found variation in the mental models of victims including order of event recall, for example by forward or reverse chronology, severity or impact of incidents.
The structure and wording of the CSEW screener questions have been redesigned accordingly, to encourage respondents to report their experience in a way that aims to increase the accuracy of incidence data. Respondents report their whole experience upfront in the screener module and further questions are asked to establish whether multiple crimes occurred within one incident. The highest priority crime is subsequently determined and recorded, aligning with the relevant code from the Home Office Counting Rules.
Comparing Scenario-Based and Self-Reported Measures of Smartphone Skills
Mr Wai Tak Tung (University of Mannheim) - Presenting Author
Dr Alexander Wenz (University of Mannheim)
Digital skills have become important for navigating in today’s information society but are still unevenly distributed in the population. While prior research has mostly focused on studying general Internet use, research on smartphone-specific digital inequalities is still scarce. To date, we have a limited understanding of the distribution of smartphone skills in the population as well as the determinants and consequences of inequalities in smartphone skills. In addition, existing measurement instruments of digital skills mostly rely on survey-based self-reports or small-scale laboratory-based performance tests that are potentially subject to measurement and representation errors.
In this paper, we report the results from a survey experiment comparing a scenario-based measure as an innovative method for measuring smartphone skills with a self-reported skills measure. In the scenario-based measure, respondents are presented with hypothetical scenarios of activities that they would perform on their smartphone, such as buying a train ticket with an app that is not yet installed on their device. They are then asked to correctly order a set of steps to carry out these activities, such as downloading an app from the app store, entering login details, and searching for train connections. In the self-reported measure, respondents are asked to rate their smartphone skills on a scale from 1=Beginner to 5=Advanced. Data were collected in the German Internet Panel, a probability-based online panel of the general population aged 16-75 in Germany, in March 2022. First, we will examine to what extent both measures capture the same construct by conducting an Exploratory Factor Analysis. Second, we will assess whether predictors of smartphone skills vary by how skills are being measured by fitting OLS regression models.
