DNA sequence analysis using masked language modeling

Abstract

This thesis investigates the application of transformer-based deep learning models for DNA sequence prediction and contextual genomic analysis under varying temperature conditions. Inspired by advances in natural language processing, DNA sequences are modeled as structured symbolic patterns in which nucleotide relationships can be learned through contextual attention mechanisms. The work primarily focuses on masked nucleotide prediction, sequence surprisal analysis, and comparative genomic behavior across different environmental conditions. A masked language modeling framework based on the BERT architecture was developed to predict masked nucleotides from surrounding sequence context. The model was trained on genomic datasets corresponding to different temperature conditions, enabling the study of how environmental variations influence sequence predictability and contextual structure. The training pipeline incorporated nucleotide tokenization, dynamic masking, positional encoding, and attention-based learning to capture long range dependencies within genomic sequences. To analyze contextual uncertainty, surprisal values were computed using the negative logarithm of predicted nucleotide probabilities. Lower surprisal values corresponded to highly predictable and conserved regions, while higher values indicated contextual variability. Comparative analysis revealed temperature-dependent variations in prediction accuracy, surprisal distributions, and conserved token structures across datasets. Additional investigation of correctly predicted positions and common sequence regions further highlighted contextual similarities and differences between genomic samples. The results demonstrate that transformer architectures can effectively capture hidden contextual patterns and long-range dependencies in DNA sequences. The proposed framework provides a computational approach for genomic sequence analysis using attention-driven models and contributes toward the integration of machine learning techniques with computational genomics.