Tuesday 16th July
Wednesday 17th July
Thursday 18th July
Friday 19th July
Download the conference book
Hierarchical data, what to do? Comparing multi-level modelling, cluster-robust standard errors, and two-step approaches |
|
| Convenor | Dr Merlin Schaeffer (Social Science Research Center Berlin) |
| Coordinator 1 | Professor Johannes Giesecke (University of Bamberg) |
| Coordinator 2 | Mr Jan Paul Heisig (Social Science Research Center Berlin) |
Social scientists are frequently interested in the importance of social context for people's actions, attitudes, interests and so on. For example, researchers want to know whether pupils in small classes perform better than those in large ones, whether people living in socio-economically deprived neighbourhoods are less satisfied with their lives, or whether the relationship between socio-economic status and health depends on a country's health care system. Generally speaking, these relationships have a multi-level structure in that outcomes and explanatory factors are situated at two different levels: individual and contextual.
To test hypotheses about contextual effects, sociologists and political scientists are usually taught to use random intercept and slope models. Most economists, by contrast, seem to rely on ordinary least squares estimation with cluster-robust standard errors. Finally, some researchers use a two-step approach: They first obtain context-specific estimates of parameters of interest which are then modelled as a function of contextual characteristics in a second step. Although they address similar problems, these different approaches have largely developed in isolation from each other and thorough comparative discussion is lacking. As a consequence, little is known about the relative advantages and pitfalls of the different approaches: Are they approximately equivalent, is one more reliable in general, or are different approaches appropriate in different situations, e.g., is one better-suited for comparisons of a small number of large and highly distinct clusters such as countries, while another works best when studying a large number of relatively similar smaller clusters such as school classes?
We invite submissions that aim to answer these questions by comparing different approaches to analyse hierarchical data. We particularly welcome simulations as well as analytical studies. However, empirical analyses that demonstrate differences between approaches for a particular case are also of interest.
When testing hypotheses about contextual effects, social scientists rely on three broad classes of modelling strategies: random intercept and slope models, pooled OLS estimation with cluster-robust standard errors, and two-step approaches. We conduct Monte-Carlo simulations to better understand the relative strengths and weaknesses of these different methods. We are particularly interested in whether the three different strategies are appropriate for different situations, e.g. is one better-suited for comparisons of a small number of large and highly distinct clusters (such as countries), while another works best when studying a large number of relatively similar smaller clusters (such as school classes)?
We investigate how four crucial dimensions affect bias and efficiency: First, we vary the number of level one units and how balanced these are distributed over the level two units. Second, we manipulate the degree to which level one effects are heterogeneous. How well do the different approaches handle slopes that vary over the level two units in random or systematic ways? Third, we explore the consequences of heteroskedastic errors at both levels one and two. Finally, we alter the number of level two units, as any simulation study on hierarchical data should.
We generally focus on linear models with continuous outcomes and initially rely on standard set ups as they are typically implemented in applied research papers. However, we also plan to investigate whether more refined versions of the three modelling approaches, e.g., bootstrapping or jackknife techniques, improve their performance in certain settings.
In this talk, I will present the following problem: I want to estimate the change of occupational mean wages over time with data from the German Socio Economic Panel. Observations over time are clustered in individuals and individuals are clustered in occupations. Because of the panel structure the growth of the occupational mean wage is confounded with the individual wage growth. The typical wage growth is assumed to be dependent on occupational properties. Therefore, individual growth parameters should be allowed to vary across occupations. If the clustering of individuals in occupations were stable over time, a multilevel growth model might be appropriate. However, complications arise because occupational changes are common. If a multilevel growth model was estimated with these data, occupational changes would result in "new and independent" observations which would result in incorrect standard errors. But even more serious consequences are likely: If individuals change occupations, their new wage will in many cases dependent on their past occupational wage. Therefore, occupational changes might bias occupational means.
The literature offers cross-classified models as a solution to the first problem. The problem of biased coefficients is not part of these discussions. I will present estimates of multilevel growth models, cross-classified models and models without the assumption of a hierarchical structure. I will compare them and discuss advantages and disadvantages.
Complex survey data are usually collected based on list frames using stratified multi-stage sampling. Analyzing such data requires dealing appropriately with the effects of stratification and clustering on the properties of estimators and test statistics, i.e., the different levels at which sample units have been selected have to be regarded. For this purpose different methods exist such as multi-level approaches and replication techniques like bootstrapping and balanced repeated replication. However, little is known about their performance in different settings. In this talk, we compare two methods to estimate sampling variances for stratified and clustered data. We use the method of balanced repeated replication and a multi-level approach. We conduct our analysis using data of the fifth and the ninth grader cohort sample of the German National Educational Panel Study. Both samples have been established on the basis of stratified multistage sampling designs. For the considered application we find that a multi-level modeling approach clearly outweighs the method of balanced repeated replication.