An alternative approach to the transformer model for text generation

Since the release of ChatGPT in late November 2022, LLMs (Large Language Models) have become, almost, a household name.

There is good reason for this; Their success lies in their architecture, in particular Mechanism of attention This allows the model to compare each word they process. every other The word

This gives LLMs extraordinary abilities to understand and create the human-like text we are all familiar with.

However, these models are not without flaws. They demand enormous computational resources for training. For example, Meta’s Llama 3 model took 7.7 million GPU hours of training[1]. Additionally, their reliance on enormous data sets—covering trillions of tokens—raises questions about scalability, accessibility, and environmental impact.

Despite these challenges, since the paper ‘Attention is all you need’ in mid-2017, recent advances in AI have mainly focused on enhancing attentional mechanisms rather than exploring new architectures. Who is