Quality Improvement and Increase of Cost Efficiency in Telephone Surveys for Researchers and Agencies: Metadata and Model based Optimization of Response Rates and Production Rates |
|
| Chair | Ms Teresa Silvestre (Statistics Portugal ) |
Due to the Mobile-Only-Population, CATI Surveys that target the general German population should consider dual-frame approaches that consist of an independent landline and mobile sample. To construct a sampling frame for landline numbers, a method developed by Gabler and Häder (1996) is usually employed. The method allows to sample listed and non-listed landline numbers with equal probabilities and little effort based on banks of numbers that are listed in a telephone book. However, this method cannot be applied when constructing a sampling frame for mobile phone numbers in Germany, since the proportion of listed mobile phone numbers is comparatively small. E.g. in 2015 only 2 million of the 112 million mobile accesses had been listed. Hence, a sampling frame for German mobile samples is usually generated by a modified RDD-approach that employs numerous data sources to eliminate banks of numbers that lead to non-existing and business contacts. Nevertheless, the hitrate of such samples is still comparatively low.
Fortunately, HLR-Lookup-Procedures can be employed to verify existing mobile accesses in advance of a survey. Such a procedure generally checks, whether a particular number exists within a mobile network by technical means and without contacting any numbers and while returning a validated list of actual mobile accesses within a sample. Struminskaya et al. (2011) and Kunz and Fuchs (2011) already demonstrated that such procedures might yield higher
response rates while lowering the costs of a survey. On the other hand, they also argue, that particular lookups might proof to be false negative or false positive.
To examine the potential of such procedure, the project Verification of Mobile Infomation (VermIn)) has been conducted by GESIS, recently. This project tested 30,000 mobile numbers on four distinct occasions via three different HLR-Lookup-Provider to compare the validity of such procedures. Furthermore, it investigated the volatility of such results and the status of mobile accesses. Another application of such procedures might also be the validation of a sampling frame in order to increase the hitrate of mobile phone samples while reducing the amount of numbers of unknown eligibility. Nevertheless, to use these procedures for such purposes, the status of a mobile phone number should not be too volatile, in order to avoid frequent lookups of the whole sampling frame. Therefore, this paper examines the results of these HLR-Lookups over the period of one year (and four occasions) to recommend particular applications of HLR-Lookups for mobile samples and the construction of sampling frames.
Increase efficiency in mobile phone samples using HLR-lookup: results from an experiment for (German) mobile phone numbers
Tobias Gramlich, Matthias Sand, GESIS Mannheim
Despite being conducted by the same mode of data collection ("telephone, "CATI", ...), surveys via landline and mobile phone (dual-frame) share a set of characteristics that are related to the merits and problems of mixed-mode surveys (e.g. differential coverage, nonresponse, data quality and weighting issues). First, dual-frame designs (telephone) often are used to overcome coverage problems by combining two (otherwise often identically administered) surveys using different sampling frames, each having strengths and weaknesses in covering a part of the target population.
Typically, these frames differ not only in coverage of the overall target population or parts of it, but depending on the way they are constructed in efficiency or hit rate.
At the cost of increased coverage (and hopefully decreased coverage bias) comes decreased efficiency of a dual-frame RDD sample: whereas techniques have been established to ensure a high hit rate in landline sampling frames (e.g. list-based RDD samples), a list-based procedure is not useful for general population samples of mobile phone numbers (since the fraction of listed mobile phone numbers is typically very low, possibly resulting in a large coverage error).
For mobile phone numbers, there is a feature within the architecture for mobile communication (GSM and successors) that allows to check a generated "mobile phone" number for being actually in use by a mobile device.
Since several years, survey agencies use the feature of this "Home Location Registry-lookup" (HLR-lookup) to clean a sample of generated mobile phone numbers. Whereas this increases efficiency (or hit rate) of the sample (and also has other implications, e.g. for the calculations of a response rate), still little is known about reliability and validity of the procedure itself or different service providers.
The talk will shed light on practicability, reliability and validity of this HLR-lookup feature in order to clean a sample of mobile phone numbers to increase frame efficiency (or "hit rate") in a RDD mobile phone survey (or as a part of a dual-frame telephone survey). It will describe a part of a one-year experiment of querying a typical RDD sample of (German) "mobile phone" numbers (30.000 numbers) comparing results from up to four different HLR-lookup providers at the same time. The talk will discuss merits, problems, pitfalls as well as technical and legal questions using the HLR-lookup feature within GSM and successors.
(Note: there will be a second --different-- talk that will emphasize on the longitudinal aspect of this experiment by comparing HLR-lookup results of the same number and HLR-lookup service provider over time: after two weeks, three months and after one year.)
Due to the increasing mobile-only rates in many European countries, dual frame sampling techniques in telephone surveys are considered state of art and are well developed and widely accepted by survey organizations. Due to the negligible no-phone population and thus high overall telephone coverage it is estimated that dual frame sampling method can improve the quality of survey estimates with respect to coverage bias. On the downside, dual frame samples come at a considerably higher cost. Regardless of the specific approach chosen (screening for mobile-onlys or conducting interviews using mobile phone numbers even in the overlap), the cost of dual frame sample is considerably higher as compared to a single frame sample. In addition, dual frame samples require differential design weighting according to unequal selection probabilities in both frames due to household size and telephone equipment which results in design effects that have the potential to reduce the effective sample size. In order to compensate for complex and expensive dual frame approaches, mobile phone sampling can be considered a possible alternative. Due to the continuing increase of mobile phone penetration rates in most European countries, mobile sampling methods can be employed in aiming to substitute the dual frame approach. Considering Eurobarometer data of 2015 about 93 percent of the European population reported to have an own mobile phone.
In this study we aim to simulate, whether the mobile phone population differs significantly from the general population and the dual frame telephone population with respect to socio-demographic (gender, age, marital status, age of education, type of community, internet use, difficulties paying bills) and substantive variables (life satisfaction, political discussion, voice counts, direction things are going). Using Eurobarometer data from 2014 and 2015 across 28 European countries we estimate the magnitude of the coverage bias in the mobile phone population in contrast to the coverage bias induced by dual frame sample.
Results of the simulation reveal slight differences in the estimated size and direction of the coverage bias for socio-demographic and substantive variables in the mobile population compared with the respective bias existing in dual frame population across Europe. A more detailed analysis suggests that in certain European countries mobile phone survey achieve smaller coverage bias estimates than dual frame survey. We discuss findings concerning mobile phone samples in light of potential other sources of error (non-response and measurement error).
The successful use of CATI labour force survey, demands the optimization of the available resources in order to attain an acceptable minimum of contact attempts, maximum response rate and minimum average productivity in each shift for each interviewer.
To reach the above goals Statistics Portugal decided to design and use a case management approach and monitoring tools during data collection cycle. Among several we developed an adaptive tool using available information prior to and during data collection cycle. This allows to hierarchize by initial priority position, after each attempt, as result of call’s history analysis and to act on specific sets to maximize response rate.
This approach allowed us to increase the response rates in about 7% , to reduce the data collection cycle (only in specific situations it is needed the third week after reference week) and to get better team and individual performance. However this was a hard process: years to achieve actual strategy; many adjustments needed and the same data collection approach does not work effectively throughout entire data collection cycle. Additionally there are some external events leading to temporary unsuccessful contact attempts, like football games or some important holidays – meaning that we have always to online supervise the performance indicators and the contact queue in order to act time and accurately.