Article

Investigating Reproducibility Challenges in LLM Bugfixing on the HumanEvalFix Benchmark

2025-05-29

Abstract excerpt

Benchmark results for Large Language Models often show inconsistencies across different studies. This paper investigates the challenges of reproducing these results in automatic bugfixing using LLMs, on the HumanEvalFix benchmark. To determine the cause of the differing results in the literature, we attempted to reproduce a subset of them by evaluating 11 models in the DeepSeekCoder, CodeGemma, and CodeLlama model...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: 05683a2f-0354-5c60-83c7-2f82a2f94d39
DOI: 10.20944/preprints202505.2321.v1

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Investigating Reproducibility Challenges in LLM Bugfixing on the HumanEvalFix BenchmarkDOI 10.20944/preprints202505.2321.v1Read

Evaluation and Improvement of Test Selection for Large Language ModelsSemantic proximity Read

Efficient Program Optimization through Knowledge-Enhanced LoRA Fine-Tuning of Large Language ModelsSemantic proximity Read

A Reference Framework for Empirical Studies on LLM-Based Code Generation: A Review-Grounded SynthesisSemantic proximity Read

AI-Driven Code Documentation: Comparative Evaluation of LLMs for Commit Message GenerationSemantic proximity Read

What Makes a Programming Problem Hard for a Language Model? An Empirical Study of Item Difficulty Across Code LLMs on Two BenchmarksSemantic proximity Read

Self-Debugging AI: A Comprehensive Analysis of Claude 4.1 Sonnet's Code Generation and Error Resolution CapabilitiesSemantic proximity Read

Emergent Numeric Bias in Large Language Models: An Empirical Study on the Anomalous Recurrence of the Number 27 Across Independent SessionsSemantic proximity Read

Harm-Conditioned Computational Friction as a Contrastive Diagnostic of Safety Representations in Language Models: A Proof-of-Concept and Control AuditSemantic proximity Read

Bash Comment Generation via Data Augmentation and Semantic-Aware CodeBERTSemantic proximity Read

Large Language Models Robustness Against PerturbationSemantic proximity Read

RePCS: Diagnosing Data Memorization in LLM-Powered Retrieval-Augmented GenerationSemantic proximity Read

Basics of Large Language Models - transformers to LLMsSemantic proximity Read