Part 2 Patterns of distributed machine learning systems

Now that you know the basic concepts and background of distributed machine learning systems, you should be able to proceed to this part of the book. We will explore some of the challenges involved in various components of a machine learning system and introduce a few established patterns adopted heavily in industries to address those challenges.

Chapter 2 introduces the batching pattern, used to handle and prepare large datasets for model training; the sharding pattern, used to split huge datasets into multiple data shards that spread among multiple worker machines; and the caching pattern, which could greatly speed the data ingestion process when a previously used dataset is re-accessed for model training.

In chapter 3, we will explore the challenges of the distributed model training process. We’ll cover the challenges of training large machine learning models that tag main themes in new YouTube videos but cannot fit on a single machine. The chapter also covers how to overcome the difficulty of using the parameter server pattern. In addition, we see how to use the collective communication pattern to speed distributed training for smaller models and avoid unnecessary communication overhead among parameter servers and workers. At the end of this chapter, we talk about some of the vulnerabilities of distributed machine learning systems due to corrupted datasets, unstable networks, and preemptive worker machines, and we see how we can address those issues.

Chapter 4 focuses on the model serving component, which needs to be scalable and reliable to handle the growing number of user requests and the growing size of individual requests. We will go through the tradeoffs of making design decisions to build a distributed model serving system. We will use the replicated services to handle the growing number of model serving requests. We will also learn how to assess model serving systems and determine whether the event-driven design would be beneficial in real-world scenarios.

In chapter 5, we’ll see how to build a system that executes complex machine learning workflows to train multiple machine learning models and pick the most performant models to provide good entity tagging results in the model serving system, using the fan-in and fan-out patterns. We’ll also incorporate the synchronous and asynchronous patterns to make machine learning workflows more efficient and avoid delays due to the long-running model training steps that block consecutive steps.

Chapter 6, the last chapter in this part of the book, covers some operational efforts and patterns that can greatly accelerate the end-to-end workflow, as well as reduce the maintenance and communication efforts that arise when engineering and data science teams collaborate. We’ll introduce a couple of scheduling techniques that prevent resource starvation and deadlocks when many team members work in the same cluster with limited computational resources. We will also discuss the benefits of the metadata pattern, which we could use to gain insights from the individual steps in machine learning workflows and handle failures more appropriately to reduce the negative effect on users.