Article

What Makes a Programming Problem Hard for a Language Model? An Empirical Study of Item Difficulty Across Code LLMs on Two Benchmarks

2026-07-13

Abstract excerpt

<title>Abstract</title> <p>Large language models (LLMs) for code are now evaluated with benchmarks such as Human Evaland MBPP, and they are increasingly used to generate, grade, and explain exercises in introductory programming courses. Yet the standard reporting practice, a single aggregate pass@kscore per model, treats every problem as interchangeable and tells us nothing about which problems are hard or why. W...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: 90691a38-a21e-57d0-bb90-2f37d8e26e29
DOI: 10.21203/rs.3.rs-10241058/v1

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

What Makes a Programming Problem Hard for a Language Model? An Empirical Study of Item Difficulty Across Code LLMs on Two BenchmarksDOI 10.21203/rs.3.rs-10241058/v1Read

A Reference Framework for Empirical Studies on LLM-Based Code Generation: A Review-Grounded SynthesisSemantic proximity Read

Investigating Reproducibility Challenges in LLM Bugfixing on the HumanEvalFix BenchmarkSemantic proximity Read

Unit Test Case Generation using Large Language Models for Students in Computer ProgrammingSemantic proximity Read

Looks good on paper: LLM-generated exams are face-valid but psychometrically weaker than human assessmentsSemantic proximity Read

Efficient Program Optimization through Knowledge-Enhanced LoRA Fine-Tuning of Large Language ModelsSemantic proximity Read

AiDER: Auditing and Document Evaluation via Rule Compilation and Small Language Models - An Education Case StudySemantic proximity Read

Optimizing GPT-Based Distractor Generation for the Korean CSAT English ExamSemantic proximity Read

Execution Grounded Multiagent Systems for Reliable Backend Code Generation with Large Language Models'Semantic proximity Read

Evaluating Open-Source LLMs for Automated Essay Scoring: The Critical Role of Prompt DesignSemantic proximity Read

AI-Driven Code Documentation: Comparative Evaluation of LLMs for Commit Message GenerationSemantic proximity Read

ChatGPT for Education and Research: Opportunities, Threats, and StrategiesSemantic proximity Read

Predicting Item-Level Characteristics of Knowledge Tests Using Large-Language ModelsSemantic proximity Read