Scripts for the paper: "Life as a function: Why Transformer architectures struggle to gain foundational capabilities at the genome level."
Python
3
8 commits
updated Apr 23, 2026
Recent advances in generative models for nucleotide sequences have shown promise, but their practical utility remains limited. In this study, we explore DNA as a complex functional representation of evolutionary processes and assess the ability of transformer-based models to capture this complexity. Through experiments with both synthetic and real DNA sequences, we demonstrate that current transformer architectures, particularly auto-regressive models relying on next-token prediction, struggle to effectively learn the underlying biological functions. Our findings suggest that these models face inherent limitations, that cannot be overcome with scale, highlighting the need for alternative approaches that incorporate evolutionary constraints and structural information. We propose potential future directions, including the integration of topological methods or the switch of modeling paradigms, to enhance the generation of genomic sequences.
At this stage the models and code for training on viral sequences is being held back due to biosecurity concerns.
Paper will be reviewed and models retrained on non-pathogenic data.
Scripts for the paper: "Life as a function: Why Transformer architectures struggle to gain foundational capabilities at the genome level."
Python
3
8 commits
updated Apr 23, 2026
Recent advances in generative models for nucleotide sequences have shown promise, but their practical utility remains limited. In this study, we explore DNA as a complex functional representation of evolutionary processes and assess the ability of transformer-based models to capture this complexity. Through experiments with both synthetic and real DNA sequences, we demonstrate that current transformer architectures, particularly auto-regressive models relying on next-token prediction, struggle to effectively learn the underlying biological functions. Our findings suggest that these models face inherent limitations, that cannot be overcome with scale, highlighting the need for alternative approaches that incorporate evolutionary constraints and structural information. We propose potential future directions, including the integration of topological methods or the switch of modeling paradigms, to enhance the generation of genomic sequences.
At this stage the models and code for training on viral sequences is being held back due to biosecurity concerns.
Paper will be reviewed and models retrained on non-pathogenic data.