Variance estimation in complex sample surveys |
|
| Coordinator 1 | Mr Umut Atasever (IEA (International Association for the Evaluation of Educational Achievement) ) |
| Coordinator 2 | Dr Sabine Meinck (IEA (International Association for the Evaluation of Educational Achievement) ) |
Complex sample survey designs, such as those used in Large-Scale Assessments in Education (LSAs), typically involve multi-stage cluster sampling and stratification. The complex nature of the data generated through this process makes it unsuitable to rely on simple random sampling assumptions for estimating sampling variance. Specific “replication methods” such as Balanced Repeated Replication (BRR) and Jackknife Repeated Replication (JRR) have been developed to approximate the sampling distribution of estimators based on such data. These replication methods rely on resampling principles by systematically manipulating estimation weights. In ILSAs, BRR and JRR are the most commonly employed methodologies for estimating sampling variance. Even though these methods are applied for decades, they are not without challenges as they are often made “fit for purpose” and cover a multitude of different design particularities given the large-scale and cross-national nature of the studies. They also have various modifications; for instance, Robert Fay in 1984 developed a modification of BRR to perturb the weights by a factor, and the number of resampling replicate weights in BRR and JRR can vary depending on the number of variance strata. Additionally, other techniques employed in complex survey designs include JK1, bootstrapping, and Taylor series linearization. Empirical evidence on the performance of these methods varies depending on survey settings, such as the intraclass correlation coefficient and sample sizes.
This session invites researchers to examine the performance of sampling variance estimators within the context of complex survey sampling designs. We encourage discussions around the application of these methods in educational assessments, social surveys, and other fields. Submissions may cover estimation of variances of both smooth statistics (e.g., means, percentages, correlations, linear regression) and non-smooth statistics (e.g., percentiles) under varying conditions. Submissions using simulated data mirroring survey designs or real survey data are also encouraged.