Article

NucEL: Single-Nucleotide ELECTRA-Style Genomic Pre-training for Efficient and Interpretable Representations

2025-08-17

Abstract excerpt

Pre-training large language models on genomic sequences has become a powerful approach for learning biologically meaningful representations. While masked language modeling (MLM)-based approaches, such as DNABERT and Nucleotide Transformer (NT), achieve strong performance, they are hindered by inefficiencies due to partial token supervision, pre-training/fine-tuning mismatches, and high computational costs. We intr...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: fae2acdf-3279-561c-84ce-e72b47b517b5
DOI: 10.1101/2025.08.17.670700

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

NucEL: Single-Nucleotide ELECTRA-Style Genomic Pre-training for Efficient and Interpretable RepresentationsDOI 10.1101/2025.08.17.670700Read

Tokenization to Transfer: Do Genomic Foundation Models Learn Good Representations?Semantic proximity Read

Benchmarking DNA Foundation Models for Genomic and Genetic TasksSemantic proximity Read

Genomic Touchstone: Benchmarking Genomic Language Models in the Context of the Central DogmaSemantic proximity Read

Annotating the genome at single-nucleotide resolution with DNA foundation modelsSemantic proximity Read

BioToken and BioFM – Biologically-Informed Tokenization Enables Accurate and Efficient Genomic Foundation ModelsSemantic proximity Read

Decoding Gene Enhancers with DNA Language Models: A Survey of Methods, Limitations, and ProspectsSemantic proximity Read

Fine-Tuning Transformers For Genomic TasksSemantic proximity Read

DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genomeSemantic proximity Read

Findings from Sparse Autoencoders for DNA Sequence Models: Motif Detectors, Reading-Frame Features, and the Scarcity of Regulatory LogicSemantic proximity Read

The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human GenomicsSemantic proximity Read

The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human GenomicsSemantic proximity Read

The human genome’s vocabulary as proposed by the DNA language model GROVERSemantic proximity Read