Measurement issues in household surveys |
|
| Chair | Dr Tobias Schmidt (Deutsche Bundesbank ) |
| Coordinator 1 | Dr Junyi Zhu (Deutsche Bundesbank) |
| Coordinator 2 | Dr Alessandra Gaia (ISER, University of Essex) |
| Coordinator 3 | Dr Jonathan Burton (ISER, University of Essex) |
Income data collected as part of household surveys is critical for the study of material living standards, however, it is known to suffer from misreporting. We present findings from a series of experiments implemented as part of a UK household panel survey – the Understanding Society Innovation Panel - designed to reduce misreporting.
Under-reporting of receipt of state transfers (welfare benefits) is common in household surveys. This can be due to recall error or social-desirability bias. Where a respondent does report the receipt of a state transfer, they may fail to remember the correct amount or period or make an error during the interview. Our first innovation is to use an end-of-module “summary screen” to allow respondents to review and edit their reports in-interview. We assess the effects of the “summary screen” on under-reporting of benefit receipts and the accuracy of reported amounts (through examining the presence of outliers, rounding, and comparing them to the official benefit rates).
Misclassification of state transfers is also possible. This occurs where a respondent knows they receive a benefit, but they do not know the name, or confuse it with a similar sounding one. eg. in the UK, respondents may confuse “carers allowance” with “attendance allowance”. Our second innovation is to experiment with a series of soft-checks that follow-up where a respondent reports receiving a state transfer but where they are unlikely to meet the eligibility criteria for that transfer.
Third, we also experiment with two questionnaire designs in the collection of state transfers. The first is characterised by the use of filter questions, which are commonly relied on in household surveys to reduce respondent burden. A weakness of filter questions is that reporting errors on the filter question mean that eligible respondents do not pass the filter. In the case of state transfers, this would lead to under-reporting. We compare the design with filter questions to a second simpler design. In the later, respondents are sequentially presented with a series of showcards listing specific types of state transfer without the use of filters.
In addition to assessing the effectiveness of each innovation independently, we analyse the effects of them on the total individual income distribution. We analyse various distributional measures including measures of poverty and inequality.
Finally, we also perform experiments which assess the reliability of before vs. after tax income measures. We conclude by providing guidelines on best practice for survey designers.
Income and wealth concentration is increasingly viewed as a potential source of political and macroeconomic instability. A flurry of research using income tax data has shown that income is becoming more concentrated (Piketty, 1999, Piketty and Saez, 2003). Though the same forces that increase income concentration may also increase wealth concentration—for example, through saved income (Saez and Zucman, 2016)—the available wealth data are inconclusive about some of the basic facts, such as how fast wealth concentration is growing.
In the U.S. there are no micro wealth data with coverage comparable to income tax data, and wealth concentration estimates are based on three other data sources. First, household surveys measure wealth, and those with a wealthy oversample—such as the Survey of Consumer Finances (SCF) —can credibly represent the top of the wealth distribution (Bricker et al, 2016, Kennickell, 2011). Second, wealth concentration can be inferred by capitalizing income tax data into a wealth estimate (Saez and Zucman, 2016, Greenwood, 1983). Third, data from so-called “rich lists” (for example, the Forbes 400) are used to estimate parameters from a Pareto distribution and interpolate the top of the wealth distribution with the aid of survey data (Vermeulen, 2015).
But how fast is wealth concentration growing in these data? Purportedly, the level and growth of concentration differs between methods in recent years (Saez and Zucman, 2016). The top 1 percent in 2013 hold 43 percent of wealth when wealth is inferred from income tax data, but hold just 36 percent when wealth is measured in the SCF survey. While much of the difference in levels is due to measurement choices (Bricker et al 2016), there is a clear difference in growth: wealth concentration has grown from 34 to 43 percent over the past decade when it is inferred from income tax data, but has grown much more modestly---from 34 to 36 percent---when measured in the SCF (figure 1).
However, we show in this paper that under reasonable assumptions there may be no difference in wealth concentration growth between these methods. We begin by exploring the degree of uncertainty in wealth concentration estimates from each method and show that concentration measures derived from capitalized income tax data are sensitive to small changes in assumed rates of return, especially when the return is low (Kopczuk, 2015), and using annual rather than permanent income. The ostensible benefit of inferring wealth from income tax records is the strong coverage at the top of the distribution that the tax data provide. But, this benefit is overwhelmed by the costs of using a model used to predict wealth (figure 2).
Over the past decade, in fact, the variability in SCF wealth concentration estimates---measured by sampling and non-sampling variability---is smaller than the variability from inferred wealth, even when including estimation variability from correcting for under-coverage through a Pareto distribution assumption. Overall, a survey with strong oversampling gives more precise estimates of wealth concentration than wealth inferred from income tax data.
We use the marginal and bivariate Pareto distributions to extrapolate both income and wealth distributions. Evaluation is performed by comparing the top marginal distributions using either marginal or joint approaches. We further introduce the observed top distributions from income tax statistics and manager magazines as the evaluation information as well as the tail distributions in estimating the Pareto distribution. Two aspects of questions can be answered by using the extrapolated distributions: simulation studies on taxation policies (interaction between capital and labor income taxes as demanded by inflation driven bargaining, capital income or wealth taxation); whether differences in saving rate or rates of return are shaping the wealth inequality.