Article

Tokenization to Transfer: Do Genomic Foundation Models Learn Good Representations?

2024-12-20

Abstract excerpt

<h4> A bstract </h4> The success of Large Language Models has inspired the development of Genomic Foundation Models (GFMs) through similar pretraining techniques. However, the relationship between pretraining performance and effectiveness in downstream genomic tasks remains unclear. Additionally, the high computational cost of pretraining raises questions about its cost-efficiency. To assess the usefulness of p...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: 30b18df6-a1d9-5f00-b002-d9c304999e74
DOI: 10.1101/2024.12.18.628606

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Tokenization to Transfer: Do Genomic Foundation Models Learn Good Representations?DOI 10.1101/2024.12.18.628606Read

NucEL: Single-Nucleotide ELECTRA-Style Genomic Pre-training for Efficient and Interpretable RepresentationsSemantic proximity Read

Genomic Touchstone: Benchmarking Genomic Language Models in the Context of the Central DogmaSemantic proximity Read

BioToken and BioFM – Biologically-Informed Tokenization Enables Accurate and Efficient Genomic Foundation ModelsSemantic proximity Read

Benchmarking DNA Foundation Models for Genomic and Genetic TasksSemantic proximity Read

Distinguishing word identity and sequence context in DNA language modelsSemantic proximity Read

Towards a universal foundation model for biobank-scale human genome variationSemantic proximity Read

The human genome’s vocabulary as proposed by the DNA language model GROVERSemantic proximity Read

MutBERT: Probabilistic Genome Representation Improves Genomics Foundation ModelsSemantic proximity Read

DNA Compression with Genomic Language Models: Tokenization, Benchmarking, and an Information-Content MapSemantic proximity Read

Towards a universal foundation model for biobank-scale human genome variationSemantic proximity Read

The Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human GenomicsSemantic proximity Read

Mut-BPE: A Modified BPE Strategy Improves Variant Effect PredictionSemantic proximity Read