Draft:Hyena (technology)

• Comment: Please replace all Arxiv.org references. That source is unreliable. We can review it once suitable sources have been chosen 🇺🇦 Fiddle^Timtrent Faddle^Talk to me 🇺🇦 20:52, 2 August 2024 (UTC)

---

Hyena is an architecture for creating large language models (LLMs), a realm of generative artificial intelligence, developed by researchers at Stanford University and released in March 2023. It was designed as a computationally more efficient alternative to the transformer architecture that underlies popular LLMs such as ChatGPT from OpenAI, Gemini from Google, Claude from Anthropic, and Llama from Meta. Hyena replaces the attention mechanism of transformer architecture with a combination of long convolutions and data-controlled gating mechanisms. This allows Hyena to achieve sub-quadratic time complexity, thereby reducing computational costs compared to transformer-based models, while demonstrating similar performance.

Large Language Models

Large language models work by first being trained with massive amounts of text gathered from a variety of sources. This "training data" is ingested into the model and processed as "tokens"—i.e., pieces of text, each as short as a single character or as long as a word or part of a word—during the training phase. Once trained, the model can be given a prompt or query ("input") in the inference phase and then generate a response ("output"), by predicting the next token in a sequence. Generally, LLMs trained with larger data sets perform better than those trained with smaller data sets, and LLMs able to process larger amounts of input data (a larger "context window") can handle more complex inference tasks. However, computational demands generally increase with the size of the input data and context window the model can handle.

Quadratic Time Complexity of Transformer Architecture

Since its introduction by Google in 2017, transformer architecture has become the predominant paradigm for LLMs such as ChatGPT. A foundational component of the transformer architecture is its attention mechanism, which operates by evaluating the dependencies between every pair of elements (tokens) in an input sequence. Thus, as the sequence increases in size, the computation—and associated cost—required to process these pairwise operations increases in proportion to the square of the number of elements in the sequence.

In computer science, this is known as "quadratic time complexity," which limits the scalability of the models due to the high computational costs. Researchers therefore have attempted to develop alternative approaches, such as Hyena, that significantly reduce computational costs by achieving "sub-quadratic time complexity."

Long Convolutions and Gating

To achieve sub-quadratic time complexity, Hyena replaces the attention mechanism of transformer architectures "by interleaving implicitly parametrized long convolutions and data-controlled gating." The long convolutions in Hyena use a feed-forward network to dynamically define the convolutional filters. This allows the model to handle very long sequences more efficiently than attention mechanisms.

Data-controlled gating in the Hyena architecture is designed to control the influence of different parts of the input data on the model's output, dynamically adjusting based on the input data. This is achieved through multiplicative element-wise gating, where the input data is modulated by a gate that determines how much of the input should be passed through. The gate values are computed as a function of the input, allowing the model to adaptively focus on different parts of the input sequence, further adding to Hyena's efficiency.

Applications

The Hyena architecture has been employed in various applications, including:

• HyenaDNA: HyenaDNA is an advanced genomic model designed to handle extremely long DNA sequences, up to 1 million tokens. It is designed to make genomic predictions, such as identifying regulatory elements and chromatin profiles, and supports in-context learning.
• StripedHyena: This model is designed for both natural language and biological sequence processing. It combines "rotary attention"—a technique to more efficiently handle longer text sequences—and gated convolutions arranged in Hyena blocks, offering improved scaling and efficiency over traditional transformer models.
• Evo-1-7B: This is a biological foundation model designed for long-context modeling and generative design across various biological modalities, including DNA, RNA, and proteins. Trained on a large prokaryotic whole-genome dataset, it can handle sequences at a single-nucleotide, byte-level resolution and is able to generate entire genomes. Evo-1-7B uses the StripedHyena architecture to achieve efficient computation and memory scaling.

Performance

Hyena matched attention-based models in recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens. Hyena "set a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile)." It reached "transformer quality with a 20% reduction in training compute required at sequence length 2K." At sequence lengths of 8K, Hyena operators are "twice as fast as highly optimized attention, and 100x faster at sequence length 64K."