Resource-efficient use of large language models

With this chapter, we introduce the techniques for efficient inference and training of large language models (LLMs). These techniques allow for deployment or fine-tuning on consumer-level hardware at an acceptable speed while retaining as many capabilities as possible. In particular, we are going to discuss about the impact of the memory footprint of LLMs on the requirements for inference and training, the differences between available numerical precision on the existing hardware and the limitations of this hardware; the techniques adopted with LMs before their transformation into LLMs, like the use of smaller scale models fine-tuned on task specific data, knowledge distillation and mixed precision training; resource efficient algorithms and quantization approaches for inference, like Flash-Attention and GPT-Q quantization; resource efficient tools and approaches for training, which include optimizers with better memory management, checkpointing of gradients and use of adapters (possibly in combination with quantization).