Article

Distinguishing word identity and sequence context in DNA language models

2023-07-12

Abstract excerpt

Transformer-based large language models (LLMs) are very suited for biological sequence data, because the structure of protein and nucleic acid sequences show many analogies to natural language. Complex relationships in biological sequence can be learned, although there may not be a clear concept of words, because they can be generated through tokenization. Training is subsequently performed for masked token predic...

Topics

Open a Topic to create a Post that cites this publication.

Identifiers and source

Literature Corpus work: 7e89e1c3-b503-5829-9d96-8517ea98f923
DOI: 10.1101/2023.07.11.548593

Open publication

Related research

Semantic proximity does not establish scientific evidence.

Click a neighbor to travelStep 1 · 12 closest

Distinguishing word identity and sequence context in DNA language modelsDOI 10.1101/2023.07.11.548593Read

The human genome’s vocabulary as proposed by the DNA language model GROVERSemantic proximity Read

<i>genomicBERT</i> : A Light-weight Foundation Model for Genome Analysis using Unigram Tokenization and Specialized DNA VocabularySemantic proximity Read

Genomic Touchstone: Benchmarking Genomic Language Models in the Context of the Central DogmaSemantic proximity Read

NucEL: Single-Nucleotide ELECTRA-Style Genomic Pre-training for Efficient and Interpretable RepresentationsSemantic proximity Read

DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genomeSemantic proximity Read

Benchmarking DNA Foundation Models for Genomic and Genetic TasksSemantic proximity Read

Tokenization to Transfer: Do Genomic Foundation Models Learn Good Representations?Semantic proximity Read

LncRNA-BERT: An RNA Language Model for Classifying Coding and Long Non-Coding RNASemantic proximity Read

ProkBERT Family: Genomic Language Models for Microbiome ApplicationsSemantic proximity Read

LAMBDA: A Prophage Detection Benchmark for Genomic Language ModelsSemantic proximity Read

Fine-Tuning Transformers For Genomic TasksSemantic proximity Read

Understanding Pathways in Bioinformatics, Genomics, and Health ApplicationsSemantic proximity Read