Transformer
Transformer is an AI structure announced by Google in 2017, and is the foundation technology for most modern generative AI, including ChatGPT.
Transformer is a deep learning structure announced by Google researchers in 2017, and its core is a method called attention that calculates how related all words in a sentence are to each other at once. Unlike the previous method of reading a book line by line in order, it can be likened to the reading method of unfolding the entire page and connecting the important parts. The existing structure of processing words in order had limitations such as forgetting previous content in long sentences and making parallel calculations difficult. Transformer solved this problem and enabled large-scale parallel learning, and as a result, most modern LLMs, including the GPT series, and even image and voice generation AI were built on this structure. The T in GPT stands for transformer.
However, there is a structural burden in which the calculation amount increases rapidly as the input becomes longer, so research on modifications and alternative structures to improve this is actively ongoing.
✅ Why it matters
- It is the core structure that forms the basis of most modern generative AI
- It is advantageous for parallel processing, enabling large-scale learning and scaling competition
- It has expanded beyond language to various fields such as image, voice, and video.
⚠️ Limits and debates
- As the input becomes longer, the amount of computation and memory usage increases rapidly
- It needs to be grown on a large scale to be truly valuable and the learning cost is very high
- Research on new structures to replace them in the long term continues to challenge the challenge.