{"id":37441,"date":"2021-11-08T09:00:00","date_gmt":"2021-11-08T07:00:00","guid":{"rendered":"https:\/\/forklog.com\/en\/?p=37441"},"modified":"2025-08-29T17:07:06","modified_gmt":"2025-08-29T14:07:06","slug":"what-are-transformers-machine-learning","status":"publish","type":"post","link":"https:\/\/forklog.com\/en\/what-are-transformers-machine-learning\/","title":{"rendered":"What are transformers? (machine learning)"},"content":{"rendered":"
\n

What are transformers?<\/strong><\/h2>\n

Transformers are a relatively new type of neural network designed to handle sequences while coping easily with long-range dependencies. Today they are the most advanced technique in natural-language processing (NLP).<\/p>\n

They can translate text, write poems and articles, and even generate computer code. Unlike recurrent neural networks (RNNs), transformers do not process sequences strictly in order. For example, if the input is text, they do not need to process the end of a sentence only after the beginning. This makes such networks easy to parallelise and much faster to train.<\/p>\n<\/div>\n

\n

When did they appear?<\/strong><\/h2>\n

Transformers were first described by engineers at Google Brain in the 2017 paper \u201cAttention Is All You Need\u201d<\/a>.<\/p>\n

One key difference from earlier methods is that the input sequence can be fed in parallel, allowing efficient use of graphics processors and higher training speeds.<\/p>\n<\/div>\n

\n

Why use transformers?<\/strong><\/h2>\n

Until 2017, engineers used deep learning to understand text via recurrent neural networks.<\/p>\n

\"What<\/figure>\n

Suppose we are translating a sentence from English into Russian. An RNN would take the English sentence as input, process its words one by one, and then output their Russian counterparts sequentially. The crucial word here is \u201csequential\u201d. Word order matters in language; you cannot simply shuffle words.<\/p>\n

RNNs run into several problems. First, they try to process long sequences of text. By the time they reach the end of a paragraph, they can \u201cforget\u201d the beginning. For instance, a translation model based on RNNs may struggle to remember the gender of an entity in a long passage.<\/p>\n

Second, RNNs are hard to train. They suffer from the so-called vanishing\/exploding gradient<\/a> problem.<\/p>\n

Third, because they process words sequentially, they are difficult to parallelise. That means you cannot easily speed up training by using more graphics processors. Consequently, training on very large datasets is impractical.<\/p>\n<\/div>\n

\n

How do transformers work?<\/strong><\/h2>\n

The main components of transformers are an encoder and a decoder.<\/p>\n

\"What
Transformer architecture. Source: \u201cAttention Is All You Need\u201d<\/a>.<\/figcaption><\/figure>\n

The encoder transforms the input (for example, text) into a vector (a set of numbers). The decoder then renders it as a new sequence (for example, an answer) or as words in another language, depending on the model\u2019s purpose.<\/p>\n

Other innovations behind transformers boil down to three core ideas:<\/p>\n