Article

Model shopping: how answer selection across AI models inflates scientific error rates

2026-08-20

Abstract excerpt

<p>Researchers increasingly consult several large language models on the same question and adopt whichever answer best supports their prior conclusion — a practice we call model shopping. We formalize model shopping as the AI-era analogue of p-hacking: the error-generating mechanism is selection over outcomes, not any individual model's quality. For a researcher with a binary hypothesis of prior probability q, con...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: c40cdc0f-56f3-5c67-be4c-79938c9ea505
DOI: 10.31222/osf.io/5sz8u_v1

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Model shopping: how answer selection across AI models inflates scientific error ratesDOI 10.31222/osf.io/5sz8u_v1Read

AI Cannot Be Reproduced: Medical AI Needs a Culture of Logging, Not RerunsSemantic proximity Read

fAIr science — How AI can support a transparent biopsychological research cycleSemantic proximity Read

fAIr science — How AI can support a transparent biopsychological research cycleSemantic proximity Read

Agentic AI for Streamlining Title and Abstract Screening: Addressing Precision and evaluating calibration of AI guardrailsSemantic proximity Read

Benchmarking Large Language Model Rationality Using Measurement AxiomsSemantic proximity Read

When do people rely on LLMs as they should? Measuring trust and trustworthiness on a single scaleSemantic proximity Read

Are LLMs better expert judges? Rethinking content validity assessment in the age of AISemantic proximity Read

Are LLMs better expert judges? Rethinking content validity assessment in the age of AISemantic proximity Read

Epistemological Fault Lines Between Human and Artificial IntelligenceSemantic proximity Read

When Science Loses Its Immune System: Epistemic Immunodepression in the Age of AISemantic proximity Read

From Silicon to Solutions: AI's Impending Impact on Research and DiscoverySemantic proximity Read

An AI system to help scientists write expert-level empirical softwareSemantic proximity Read