Relative Bias in Health Estimates from Probability-Based Online Panels: Systematic Review and Meta-Analysis

Andrea Ivanovska 

Faculty of Social Sciences, University of Ljubljana
https://orcid.org/0009-0008-2653-2503

Michael Bosnjak
Department of Psychology, University of Trier, Germany
https://orcid.org/0000-0002-1431-8461

Vasja Vehovar
Faculty of Social Sciences, University of Ljubljana
https://orcid.org/0000-0002-3253-7959

https://doi.org/10.33700/jhrs.4.1.137

Abstract

Introduction: Health surveys require the highest data quality, especially when they inform public health policies. With recent technological developments, probability-based online panels (PBOPs) are becoming an attractive cost-effective alternative to traditional surveys. They are also beginning to be used for official health statistics. However, PBOPs still face concerns about bias, especially for health-related estimates.

Method: Using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) approach, we conducted a systematic review and meta-analysis of PBOP PBOP health survey data quality, with relative bias (RB) of the estimates as the effect size. We analysed 137 health-related survey items from 14 studies and used a linear regression model to examine factors that moderate RB.

Results: The RB varied considerably across the subjects, and its overall median was 12.7%. The highest RBs were exhibited by disabilities (23.6%), mental illnesses (23.2%), personal mental health conditions (20.8%) and drug use (20.7%), and the lowest, by doctor’s treatment (2.24%). The measurement levels with ordinal scales (25.8%) showed higher RB, and certain country effects were also observed.

Conclusion: This moderate bias of the health estimates raises concerns about the accuracy of PBOP estimates regarding sensitive health topics. Therefore, PBOP should be used cautiously for official health statistics; and when designing PBOP surveys for health subjects, the item and study characteristics should be included as methodological considerations.

Keywords: Probability-based online panels, Data quality, Relative bias, Meta-analysis, Health research

Full Text Article