Natural Language Processing with Transformers

"Natural Language Processing with Transformers, Revised Edition" by Lewis Tunstall, Leandro von Werra, and Thomas Wolf is a practical guide for data scientists and coders on using transformer models in NLP. Since their introduction in 2017, transformers have become the dominant architecture for achieving state-of-the-art results across various NLP tasks. This revised edition, authored by some of the creators of Hugging Face Transformers, a Python-based deep learning library, employs a hands-on approach to teach how transformers function and how to integrate them into applications. The book covers how transformers have been used for tasks such as writing realistic news stories, improving Google Search queries, and creating chatbots.

Readers will learn to build, debug, and optimize transformer models for core NLP tasks like text classification, named entity recognition, and question answering. The guide also delves into applying transformers for cross-lingual transfer learning, using them in scenarios with scarce labeled data, and making them efficient for deployment through techniques like distillation, pruning, and quantization. It further explains how to train transformers from scratch and scale them to multiple GPUs and distributed environments, equipping practitioners to solve a variety of real-world NLP challenges.

Key concepts

• Transformers — The dominant architecture for achieving state-of-the-art results on a variety of natural language processing tasks since 2017.
• Hugging Face Transformers — A Python-based deep learning library used in the book to train and scale transformer models.
• Core NLP Tasks — The book teaches how to build and optimize models for text classification, named entity recognition, and question answering.
• Cross-lingual Transfer Learning — A technique for applying transformers in multilingual contexts.
• Model Efficiency for Deployment — Techniques like distillation, pruning, and quantization are covered to optimize transformer models for practical deployment.
• Training and Scaling Transformers — Readers learn to train transformers from scratch and scale them across multiple GPUs and distributed environments.

Popular questions readers ask

• The text states transformers have become the "dominant architecture" since 2017. What fundamental limitations or inherent challenges of prior NLP models did the transformer architecture likely overcome to achieve such rapid and widespread dominance across diverse tasks?
• How does the ability to perform "cross-lingual transfer learning" and operate effectively in "scenarios where labeled data is scarce" fundamentally change the accessibility and application of state-of-the-art NLP, especially for languages or domains previously underserved?
• Imagine you are tasked with deploying a transformer model to a low-resource edge device. Explain the specific trade-offs a data scientist must consider when applying "distillation, pruning, and quantization," and how these techniques might alter the model's core functionality or performance.
• The guide emphasizes learning "how transformers work" alongside "how to integrate them in your applications." Why is it critical for a data scientist or coder to understand the underlying mechanics rather than just treating the Hugging Face Transformers library as a black box?
• When would a data scientist choose to "train transformers from scratch" instead of fine-tuning a pre-trained model, and what distinct challenges or benefits would this decision present when attempting to "scale to multiple GPUs and distributed environments"?