Article

Benchmarking Large Language Models for Data Pipeline Code Generation and Execution

2025-07-02

Abstract excerpt

<title>Abstract</title> <p>In today’s data-driven landscape, organizations face mounting pressure to accelerate data processing and analysis while minimizing manual engineering efforts. This paper investigates the application of Large Language Models (LLMs) to automate the creation of data pipelines, evaluating their efficacy across code-based (Apache Airflow), low-code (Azure Data Factory), and hybrid (Databrick...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: bfd9caab-abeb-5faf-b0d2-fef9f06ec8b0
DOI: 10.21203/rs.3.rs-6786102/v1

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Benchmarking Large Language Models for Data Pipeline Code Generation and ExecutionDOI 10.21203/rs.3.rs-6786102/v1Read

Self-Debugging AI: A Comprehensive Analysis of Claude 4.1 Sonnet's Code Generation and Error Resolution CapabilitiesSemantic proximity Read

Intent-Driven Code Synthesis: Redefining Software Development with TransformersSemantic proximity Read

Leveraging large language models for data analysis automationSemantic proximity Read

Investigating the Refactoring Capabilities of Small Open-Weight Language ModelsSemantic proximity Read

Human-Machine Cooperative Program Translation Based on Abstract Syntax TreeSemantic proximity Read

Efficient Program Optimization through Knowledge-Enhanced LoRA Fine-Tuning of Large Language ModelsSemantic proximity Read

Variable Naming Impact on AI Code Completion: An Empirical StudySemantic proximity Read

From Prompt to Production: A Case Study of Human-AI Collaborative Software Development Using Claude Code Multi-Agent OrchestrationSemantic proximity Read

A Reference Framework for Empirical Studies on LLM-Based Code Generation: A Review-Grounded SynthesisSemantic proximity Read

r2py: AI-Assisted Conversion of R Statistical Packages to PythonSemantic proximity Read

Balancing Autonomy and Oversight in Language-Agent-Guided Physical ExperimentationSemantic proximity Read

AiDER: Auditing and Document Evaluation via Rule Compilation and Small Language Models - An Education Case StudySemantic proximity Read