编辑论文 - Lucher

中文标题/译名英文标题/原题状态

研究方向关键词

作者通讯作者摘要

Large language models (LLMs) are increasingly used to annotate text in computational content analysis, and their scores are entered as indicators in structural equation models after agreement with human codes has been checked construct by construct. When one model–prompt combination scores both constructs of a structural relation, its systematic tendencies enter both, and this shared method variance biases standardized paths by an amount that does not shrink with sample size. In five Monte Carlo studies, relative bias reached 77%–88%, confidence intervals excluded the true value, and averaging repeated scores, adding indicators, and ordinal estimation did not help. Independent human codes for a random subsample of texts can correct the bias in two ways. An anchored multitrait–multimethod model was the most efficient correction but was biased when the LLM's errors did not follow its measurement model, whereas prediction-powered inference was robust but less precise. A bootstrap comparison of the two corrections held its nominal size and detected strong misspecification. An application to online discussion posts from course forums, scored by open-weight LLMs from two model families, illustrates the diagnostics and shows how strongly anchored estimates can depend on specification. We close with recommendations for design and reporting.

基金信息备注

期刊与投稿/发表信息

这里的信息会保存为这篇论文的关联投稿记录。

选择期刊/会议

原地址/投稿系统地址检索情况

投稿日期发表日期

附件

还没有附件。