Applying a Total Survey Error approach to business surveys |
|
| Session Organiser | Dr Gijs van Houten (Eurofound) |
| Time | Friday 2 July, 15:00 - 16:30 |
Business surveys differ in nature from population surveys in many ways. Business surveys are often not limited to a single respondent, respondents are acting in a professional capacity, and are often reporting on characteristics of the business (or part of the business) rather than on their personal behaviour or attitudes, respondents will need to be identified using some sort of screening process, and even when respondents are identified they might fully or partially delegate the questionnaire administration if the survey mode and design allows this. The evaluations and trade-offs of different errors therefore tend to be different than in population surveys. This session includes a variety of papers in which a TSE perspective is applied to issues such as sampling, respondent selection, incentive and reminder strategies, and survey mode switches in business surveys, showing how a Total Survey Error approach can benefit their design and implementation.
Keywords: Total Survey Error, Business surveys, Respondent selection, Mode effects, Incentives, Sampling frames
Ms Marieke Volkert (IAB) - Presenting Author
Ms Corrina Koenig (IAB)
Professor Joseph W. Sakshaug (IAB)
Dr Susanne Kohaut (IAB)
Mr Peter Ellguth (IAB)
The IAB Establishment Panel is an annual employer survey in Germany that collects longitudinal data to provide deep insights into macroeconomic developments since 1993. Until 2018, the primary mode of data collection was face-to-face with self-administration available upon request. From 2018 onward, a Web starting mode was introduced as part of a sequential mixed-mode design with nonresponse follow-ups conducted via the traditional face-to-face procedure. However, during the COVID-19 pandemic wave face-to-face interviewing was not feasible and therefore a new sequential mixed-mode design was implemented combining self-administered (Web and/or mail) modes with telephone interviewing. Among the establishments already belonging to the panel, interviewers contacted those that were interviewed face-to-face in the previous wave by telephone while establishments previously interviewed in a self-administered mode were initially invited to the Web mode with telephone follow-up. Among the refreshment sample cases, an experimental mode design was implemented in order to investigate mode effects associated with introducing the telephone mode into the survey. Specifically, the refreshment sample was randomly divided into three groups. The first group was directly contacted via telephone and could participate via self-administration only upon request. The second group was invited to participate via the Web with nonresponse follow-ups conducted via telephone. Finally, the last group was initially invited to participate online with nonresponse follow-ups conducted via paper questionnaire. The experiment should identify differences in the response rates of the different modes and mode designs. In particular, the nonresponse behavior of the establishments and possible selectivity with respect to establishment characteristics are of special interest.
Mr Michele Consolini (Eurofound) - Presenting Author
Dr Gijs van Houten (Eurofound)
Dr Franz Eiffe (Eurofound)
The European Company Survey (ECS) 2019 is a push-to-web survey targeting managers and employee representatives in private sector establishments in all EU Member States. Around a quarter of respondents to the ECS 2019 consented to being re-contacted for follow-up research. In November 2020 Eurofound and Cedefop approached these respondents, inviting them to complete a 10-15-minute follow-up questionnaire on the impact of COVID-19 on workplace practices.
To entice respondents to participate, a short, customised report based on the establishment’s responses to the ECS 2019 was attached to a random subsample of 80% of the invitation emails. While these customised reports were expected to improve response rates, there was concern about selection effects (i.e. the impact of offering such incentive on the sample composition) as respondents with a greater interest in the survey topic might respond better to the incentive than less interested respondents. Another concern regarded the double impact on the substantive responses of those receiving the report: i) being reminded of their previous answers might refresh their memory and increase consistency in the answers between the two waves; ii) being confronted with the relative positioning of the practices in their establishments compared to other similar establishments might increase socially desirable responses.
In this presentation we will show the impact of offering these customised reports had on the response rate and the effect on sample composition and substantive responses. We conclude that the effectiveness of this type of incentive depends on the objectives of the survey. Future research should also look at the content of the customised reports, as it can be expected that this determines whether the impact of the report on substantive answers is mainly negative (increased desirability bias) or mainly positive (increased consistency between waves).
Dr Ahu Alanya (Ipsos)
Dr Andrew Cleary (Ipsos)
Dr Femke Dekeulenaer (Ipsos) - Presenting Author
Dr Gijs van Houten (Eurofound)
The European Company Survey 2019 (ECS) used a push to web survey design, with telephone initial contact and short CATI screener questionnaire, and web interviewing for the full substantive sample. The three previous iterations of the survey have been conducted via CATI. The survey is undertaken by Eurofound, with this fourth edition taking place in partnership with Cedefop and fieldwork delivery commissioned to Ipsos. The survey informant is the manager responsible for human resources in a randomly sampled establishment and when possible an employee representative. Main survey fieldwork took place in the first half of 2019, with over 20,000 interviews across 28 European countries.
With the move to an online mode of administration, it was expected that ECS 2019 would obtain lower response rates than in previous waves, when the survey was administered via CATI. In order to get insight into differences between respondents and non-respondents in the web interview sample, additional questions were included in the CATI screener. Given that both respondents and non-respondents in the online survey completed the CATI screener survey, the data collected at the CATI screener stage can be used to explore the correlates of non-response in the web interview sample. In the first part of this paper, we examine response rates and non-response bias in the web interview sample, overall and across countries, drawing conclusions relevant to others interested in conducting push to web business surveys.
In the second part of the paper, we use an alternative approach to study non-response by comparing early and late respondents in the online survey. A number of measures were introduced in ECS 2019 to encourage participation in the online survey: (1) up to four email reminders, spaced apart by five working days, were sent to managers who had not (yet) completed the online interview, and (2) one of the reminders was conducted by telephone, scheduled after the second email reminder. We investigate whether significant differences can be detected between “early” respondents and “late” respondents, distinguishing between managers who completed the reminder immediately after the survey invitation, after one or more email reminders or after the CATI reminder. Managers who required more reminders before they participated would have been non-respondents if the data collection had stopped earlier. As such, the repeated reminders in ECS 2019 were effective in obtaining a greater sample size, but it remains to be investigated whether they were also effective in improving the representativeness of the final sample. In addition to the question about representativeness, we also assess whether significant differences can be detected in the quality of the answers provided by “early” respondents and “late” respondents.
Mr Nikola Jovanovski (Sample Solutions) - Presenting Author
Mr Carsten Broich (Sample Solutions)
In the last years, more and more countries in Europe started to publish company register data online and freely accessible. While industry classifications are harmonized under NACE Rev.2 (statistical classification of economic activities), many differences still exist with regard to data accuracy and data comprehensiveness. Different levels of contact data (address based, phone based or email based), issues in convertibility between national Industry classification and that of NACE Rev.2, or sizes (definitions for employee level or turnover) co-exist which creates challenges for Pan-European surveys.
Such errors affect the estimations and verification of B2B sampling frames of each country's Business Universe in a 27-countries establishments survey - drawing extreme contrasts among one country to another,with applications of custom rules and workarounds for many countries that stand as outliers from the rest.
In this paper an outline is provided for overlaying multiple business data sources by collecting, comparing and validating business information, to increase coverage but also reduce the sampling error by using Big Data enrichment efforts. Next to the approach, pros and cons are showcased, discussed in detail and experience from the actual fieldwork is being shared.