All time references are in CEST
Variance estimation in complex sample surveys |
|
| Session Organisers | Mr Umut Atasever (IEA (International Association for the Evaluation of Educational Achievement) ) Dr Sabine Meinck (IEA (International Association for the Evaluation of Educational Achievement) ) |
| Time | Tuesday 18 July, 09:00 - 10:30 |
| Room | |
Complex sample survey designs, such as those used in Large-Scale Assessments in Education (LSAs), typically involve multi-stage cluster sampling and stratification. The complex nature of the data generated through this process makes it unsuitable to rely on simple random sampling assumptions for estimating sampling variance. Specific “replication methods” such as Balanced Repeated Replication (BRR) and Jackknife Repeated Replication (JRR) have been developed to approximate the sampling distribution of estimators based on such data. These replication methods rely on resampling principles by systematically manipulating estimation weights. In ILSAs, BRR and JRR are the most commonly employed methodologies for estimating sampling variance. Even though these methods are applied for decades, they are not without challenges as they are often made “fit for purpose” and cover a multitude of different design particularities given the large-scale and cross-national nature of the studies. They also have various modifications; for instance, Robert Fay in 1984 developed a modification of BRR to perturb the weights by a factor, and the number of resampling replicate weights in BRR and JRR can vary depending on the number of variance strata. Additionally, other techniques employed in complex survey designs include JK1, bootstrapping, and Taylor series linearization. Empirical evidence on the performance of these methods varies depending on survey settings, such as the intraclass correlation coefficient and sample sizes.
This session invites researchers to examine the performance of sampling variance estimators within the context of complex survey sampling designs. We encourage discussions around the application of these methods in educational assessments, social surveys, and other fields. Submissions may cover estimation of variances of both smooth statistics (e.g., means, percentages, correlations, linear regression) and non-smooth statistics (e.g., percentiles) under varying conditions. Submissions using simulated data mirroring survey designs or real survey data are also encouraged.
Keywords: Complex Sample Survey Design, Sampling Variance Estimation, Replication Methods, Large-Scale Surveys, Simulation Studies
Mr Umut Atasever (IEA Hamburg) - Presenting Author
The primary objective of this study is to compare the relative performance of the most widely used sampling variance estimators in international large-scale assessments (ILSAs): the Balanced Repeated Replication (BRR) method and the (paired) Jackknife Repeated Replication (JK2) method. Additionally, this study examines the impact of a Fay modification factor on BRR and JK2, assessing its effect on the precision of sampling variance estimation. A Monte Carlo simulation approach is employed, simulating a TIMSS student population with realistic distributions of achievement scores, standard deviation, intra-class correlation coefficient (ICC), and background characteristics. Probability samples are repeatedly drawn following a two-stage stratified cluster sampling design, where schools are primary sampling units (PSUs) and classes are secondary sampling units. For each sample, the population parameter of interest and its sampling variance are estimated, with the sampling variance approximated from the variability of estimates across samples. The performance of each variance estimator is evaluated by comparing estimated sampling variance to the approximated variance. Results suggest that JK2 and BRR without Fay modification yield the most precise variance estimates for smooth statistics such as means, with JK2 demonstrating the highest stability across conditions. Fay-modified JK2 does not significantly impact variance precision, while BRR with a Fay factor enhances performance for non-smooth statistics, particularly at 10% and 30%, compared to the 50% Fay factor and BRR without Fay. Under non-response conditions, JK2 Original Strata (OrgStr) underestimates variance, while BRR Original Strata (OrgStr) inflates variance estimates. The OrgStr approach retains the initial variance strata structure, even when some PSUs do not respond, rather than reconstructing strata based only on participating units. This study advances discussions on variance estimation in ILSAs, providing insights into the optimal application of resampling methods, including the trade-offs of Bootstrapping, BRR, and JK2 under varying statistical