LLM-generated responses (synthetic respondents) and social science research |
|
| Coordinator 1 | Dr Yongwei Yang (Google) |
| Coordinator 2 | Mr Joseph M. Paxton (Google) |
| Coordinator 3 | Dr Mario Callegaro (Callegaro Research Methods Consulting) |
Generative AI developments have led to great interest in replacing human survey responses with LLM-generated ones. Between October 2022 and July 2024 we have seen at least 60 papers on this topic, often posted on preview platforms (arXiv, SSRN, etc.). The enthusiasm stems from studies suggesting LLM responses (synthetic responses) can resemble human responses in public opinion and organization surveys, psychological experiments, and consumer studies. The excitement is amplified by the perceived potential for faster and cheaper data collection. However, concerns are also raised:
(1) LLM-generated data often produce smaller variance.
(2) They may not represent the full spectrum of human thoughts.
(3) They may reflect stereotypes entrenched in training data.
(4) They might not recover multivariate relationships or human mental processes.
This session will advance discussions about the utility and appropriateness of LLM-generated responses to research. We invite BOTH empirical and didactic works about:
(1) Supporting and refuting arguments on the use of LLM-generated responses
(2)Good and bad use cases of LLM-generated data
(3) Methodological challenges and solutions with LLM-vs-Human comparative studies, including research design, software and model choices, data generation process, data analysis and inferences, transparency and reproducibility
We welcome diverging or even provocative viewpoints as well as those that connect with the proliferation of other data collection practices (e.g., opt-in panels, paradata, social media data). At the same time, we stress the critical value of research rigor and expect these viewpoints to be supported by sound theory and evidence. Specifically:
(1) With didactic work, we expect depth in theoretical arguments and thoroughness in literature review.
(2) With empirical work, we expect thoughtful design and clarity about implementation (e.g., models, hyperparameters). Where applicable, we expect designs to include a temporal component that addresses changes in relevant. LLMs.