1 Introduction to distributed machine learning systems

This chapter covers

Machine learning systems are becoming more important nowadays. Recommendation systems learn to generate recommendations of potential interest with the right context according to user feedback and interactions, anomalous event detection systems help monitor assets to prevent downtime due to extreme conditions, and fraud detection systems protect financial institutions from security attacks and malicious fraud behaviors.

There is increasing demand for building large-scale distributed machine learning systems. If a data analyst, data scientist, or software engineer has basic knowledge of and hands-on experience in building machine learning models in Python and wants to take things a step further by learning how to build something more robust, scalable, and reliable, this book is the right one to read. Although experience in production environments or distributed systems is not a requirement, I expect readers in this position to have at least some exposure to machine learning applications running in production and should have written Python and Bash scripts for at least one year.

Being able to handle large-scale problems and take what’s developed on your laptop to large distributed clusters is exciting. This book introduces best practices in various patterns that help you speed up the development and deployment of machine learning models, use automations from different tools, and benefit from hardware acceleration. After reading this book, you will be able to choose and apply the correct patterns for building and deploying distributed machine learning systems; use common tooling such as TensorFlow (https://www.tensorflow.org), Kubernetes (https://kubernetes.io), Kubeflow (https://www.kubeflow.org), and Argo Workflows appropriately within a machine learning workflow; and gain practical experience in managing and automating machine learning tasks in Kubernetes. A comprehensive, hands-on project in chapter 9 provides an opportunity to build a real-life distributed machine learning system that uses many of the patterns we learn in the second part of the book. In addition, supplemental exercises at the end of some sections in the following chapters recap what we’ve learned.

1.1 Large-scale machine learning

The scale of machine learning applications has become unprecedentedly large. Users are demanding faster responses to meet real-life requirements, and machine learning pipelines and model architectures are getting more complex. In this section, we’ll talk about the growing scale in more detail and what we can do to address the challenges.

1.1.1 The growing scale

As the demand for machine learning grows, the complexity involved in building machine learning systems is increasing as well. Machine learning researchers and data analysts are no longer satisfied with building simple machine learning models on their laptops on gigabytes of Microsoft Excel sheets. Due to the growing demand and complexity, machine learning systems have to be built with the ability to handle the growing scale, including the increasing volume of historical data; frequent batches of incoming data; complex machine learning architectures; heavy model serving traffic; and complicated end-to-end machine learning pipelines.

Let’s consider two scenarios. First, imagine that you have a small machine learning model that has been trained on a small dataset (less than 1 GB). This approach might work well for your analysis at hand because you have a laptop with sufficient computational resources. But you realize that the dataset grows by 1 GB every hour, so the original model is no longer useful and predictive in real life. Suppose that you want to build a time-series model that predicts whether a component of a train will fail in the next hour to prevent failures and downtime. In this case, we have to build a machine learning model that uses the knowledge gained from the original data and the most recent data that arrives every hour to generate more accurate predictions. Unfortunately, your laptop has a fixed amount of computational resources and is no longer sufficient for building a new model that uses the entire dataset.

Second, suppose that you have successfully trained a model and developed a simple web application that uses the trained model to make predictions based on the user’s input. The web application may have worked well in the beginning, generating accurate predictions, and the user was quite happy with the results. This user’s friends heard about the good experience and decided to try it as well, so they sat in the same room and opened the website. Ironically, they started seeing longer delays when they tried to see the prediction results. The reason for the delays is that the single server used to run the web application can’t handle the increasing number of user requests as the application gets more popular. This scenario is a common challenge that many machine learning applications will encounter as they grow from beta products to popular applications. These applications need to be built on scalable machine learning system patterns to handle the growing scale of throughput.

1.1.2 What can we do?

When the dataset is too large to fit in a single machine, as in the first scenario in section 1.1.1, how can we store the large dataset? Perhaps we can store different parts of the dataset on different machines and then train the machine learning model by sequentially looping through the various parts of the dataset on different machines.

If we have a 30 GB dataset like the one in figure 1.1, we can divide it into three partitions of 10 GB data, with each partition sitting on a separate machine that has enough disk storage. Then, we can consume the partitions one by one without having to train the machine learning model by using the entire dataset at the same time.

Figure 1.1 An example of dividing a large dataset into three partitions on three separate machines that have sufficient disk storage

Then, we might ask what will happen if looping through different parts of the dataset is quite time-consuming. Assume that the dataset at hand has been divided into three partitions. As illustrated in figure 1.2, first, we initialize the machine learning model on the first machine, and then we train it, using all the data in the first data partition. Next, we transfer the trained model to the second machine, which continues training by using the second data partition. If each partition is large and time-consuming, we’ll spend a significant amount of time waiting.

Figure 1.2 An example of training the model sequentially on each data partition

In this case, we can think about adding workers. Each worker is responsible for consuming each of the data partitions, and all workers train the same model in parallel without waiting for others. This approach is definitely good for speeding up the model training process. But what if some workers finish consuming the data partitions that they are responsible for and want to update the model at the same time? Which of the worker’s results (gradients) should we use to update the model first? Then, we must consider the conflicts and tradeoffs between performance and model quality. In figure 1.2, if the data partition that the first worker uses has better quality due to a more rigorous data collection process than the one that the second worker uses, using the first worker’s results first would produce a more accurate model. On the other hand, if the second worker has a smaller partition, it could finish training faster, so we could start using that worker’s computational resources to train a new data partition. When more workers are added, such as the three workers shown in figure 1.2, the conflicts in completion time for data consumption by different workers become even more obvious.

Similarly, if the application that uses the trained machine learning model to make predictions observes much heavier traffic, can we simply add servers, with each new server handling a certain percentage of the traffic? Unfortunately, the answer is not that simple. This naive solution would need to take other things into consideration, such as deciding the best load balancer strategy and processing duplicate requests in different servers.

We will learn more about handling these types of problems in the second part of the book. For now, the main takeaway is that we have established patterns and best practices to deal with certain situations, and we will use those patterns to make the most of our limited computational resources.

1.2 Distributed systems

A single machine or laptop can’t satisfy the requirements for training a large machine learning model with a large amount of data. We need to write programs that can run on multiple machines and be accessed by people all over the world. In this section, we’ll talk about what a distributed system is and discuss one concrete example pattern that’s often used in distributed systems.

1.2.1 What is a distributed system?

Computer programs have evolved from being able to run on only one machine to working with multiple machines. The increasing demand for computing power and the pursuit of higher efficiency, reliability, and scalability have boosted the advancement of large-scale data centers that consist of hundreds or thousands of computers communicating via the shared network, which have resulted in the development of distributed systems. A distributed system is one in which components are located on different networked computers and can communicate with one another to coordinate workloads and work together via message passing.

Figure 1.3 illustrates a small distributed system consisting of two machines communicating with each other via message passing. One machine contains two CPUs, and the other machine contains three CPUs. Obviously, a machine contains computational resources other than the CPUs; we use only CPUs here for illustration purposes. In real-world distributed systems, the number of machines can be extremely large–tens of thousands, depending on the use case. Machines with more computational resources can handle larger workloads and share the results with other machines.

Figure 1.3 An example of a small distributed system consisting of two machines with different amounts of computational resources communicating with each other via message passing

1.2.2 The complexity and patterns

These distributed systems can run on multiple machines and be accessed by users all over the world. They are often complex and need to be designed carefully to be more reliable and scalable. Bad architectural considerations can lead to problems, often on a large scale, and result in unnecessary costs.

Lots of good patterns and reusable components are available for distributed systems. The work-queue pattern in a batch processing system, for example, makes sure that each piece of work is independent of the others and can be processed without any interventions within a certain amount of time. In addition, workers can be scaled up and down to ensure that the workload can be handled properly.

Figure 1.4 depicts seven work items, each of which might be an image that needs to be modified to grayscale by the system in the processing queue. Each of the three existing workers takes two to three work items from the processing queue, ensuring that no worker is idle to avoid waste of computational resources and maximizing the performance by processing multiple images at the same time. This performance is possible because each work item is independent of the others.

Figure 1.4 An example of a batch processing system using the work-queue pattern to modify images to grayscale

1.3 Distributed machine learning systems

Distributed systems are useful not only for general computing tasks but also for machine learning applications. Imagine that we could use multiple machines with large amounts of computational resources in a distributed system to consume parts of the large dataset, store different partitions of a large machine learning model, and so on. Distributed systems can greatly speed up machine learning applications with scalability and reliability in mind. In this section, we’ll introduce distributed machine learning systems, present a few patterns that are often used in those systems, and talk about some real-life scenarios.

1.3.1 What is a distributed machine learning system?

A distributed machine learning system is a distributed system consisting of a pipeline of steps and components that are responsible for different steps in machine learning applications, such as data ingestion, model training, and model serving. It uses patterns and best practices similar to those of a distributed system, as well as patterns designed specifically to benefit machine learning applications. Through careful design, a distributed machine learning system is more scalable and reliable for handling large-scale problems, such as large datasets, large models, heavy model serving traffic, and complicated model selection or architecture optimization.

1.3.2 Are there similar patterns?

To handle the increasing demand for and scale of machine learning systems that will be deployed in real-life applications, we need to design the components in a distributed machine learning pipeline carefully. Design is often nontrivial, but using good patterns and best practices allows us to speed the development and deployment of machine learning models, use automations from different tools, and benefit from hardware accelerations.

There are similar patterns in distributed machine learning systems. As an example, multiple workers can be used to train the machine learning model asynchronously, with each worker being responsible for consuming certain partitions of the dataset. This approach, which is similar to the work-queue pattern used in distributed systems, can speed up the model training process significantly. Figure 1.5 illustrates how we can apply this pattern to distributed machine learning systems by replacing the work items with data partitions. Each worker takes some data partitions from the original data stored in a database and then uses them to train a centralized machine learning model.

Figure 1.5 An example of applying the work-queue pattern in distributed machine learning systems

Another example pattern commonly used in machine learning systems instead of general distributed systems is the parameter server pattern for distributed model training. As shown in figure 1.6, the parameter servers are responsible for storing and updating a particular part of the trained model. Each worker node is responsible for taking a particular part of the dataset that will be used to update a certain part of the model parameters. This pattern is useful when the model is too large to fit in a single server and dedicated parameter servers for storing model partitions without allocating unnecessary computational resources.

Figure 1.6 An example of applying the parameter server pattern in a distributed machine learning system

Part 2 of this book illustrates patterns like these. For now, keep in mind that some patterns in distributed machine learning systems also appear in general-purpose distributed systems, as well as patterns specially designed to handle machine learning workloads at large scale.

1.3.3 When should we use a distributed machine learning system?

If the dataset is too large to fit on our local laptops, as illustrated in figures 1.1 and 1.2, we can use patterns such as data partitioning or introduce additional workers to speed up model training. We should start thinking about designing a distributed machine learning system when any of the following scenarios occurs:

  • The model is large, consisting of millions of parameters that a single machine cannot store and that must be partitioned on different machines.

  • The machine learning application needs to handle increasing amounts of heavy traffic that a single server can no longer manage.

  • The task at hand involves many parts of the model’s life cycle, such as data ingestion, model serving, data/model versioning, and performance monitoring.

  • We want to use many computing resources for acceleration, such as dozens of servers that have many GPUs each.

If any of these scenarios occur, it’s usually a sign that a well-designed distributed machine learning system will be needed in the near future.

1.3.4 When should we not use a distributed machine learning system?

Although a distributed machine learning system is helpful in many situations, it is usually harder to design and requires experience to operate efficiently. Additional overhead and tradeoffs are involved in developing and maintaining such a complicated system. If you encounter any of the following cases, stick with a simple approach that already works well:

  • The dataset is small, such as a CSV file smaller than 10 GBs.

  • The model is simple and doesn’t require heavy computation, such as linear regression.

  • Computing resources are limited but sufficient for the tasks at hand.

1.4 What we will learn in this book

In this book, we’ll learn to choose and apply the correct patterns for building and deploying distributed machine learning systems to gain practical experience in managing and automating machine learning tasks. We’ll use several popular frameworks and cutting-edge technologies to build components of a distributed machine learning workflow, including the following:

A comprehensive hands-on project in the last part of the book consists of an end-to-end distributed machine learning pipeline system. Figure 1.7 is the architecture diagram of the system that we will be building. We will gain hands-on experience implementing many of the patterns covered in the following chapters. Handling large-scale problems and taking what we’ve developed on our personal laptops to large distributed clusters should be exciting.

Figure 1.7 An architecture diagram of the end-to-end machine learning system that we will be building in the last part of the book

We’ll be using TensorFlow with Python to build machine learning and deep learning models for various tasks, such as building useful features based on a real-life dataset, training predictive models, and making real-time predictions. We’ll also use Kubeflow to run distributed machine learning tasks in a Kubernetes cluster. Furthermore, we will use Argo Workflows to build a machine learning pipeline that consists of many important components of a distributed machine learning system. The basics of these technologies are introduced in chapter 2, and we’ll gain hands-on experience with them in part 2. Table 1.1 shows the key technologies that will be covered in this book and example uses.

Table 1.1 The technologies covered in this book and their uses
Technology Use
TensorFlow Building machine learning and deep learning models
Kubernetes Managing distributed environments and resources
Kubeflow Submitting and managing distributed training jobs easily on Kubernetes clusters
Argo Workflows Defining, orchestrating, and managing workflows
Docker Building and managing images to be used for starting containerized environments

Before we dive into details in chapter 2, I recommend that readers have basic knowledge of and hands-on experience in building machine learning models in Python. Although experience in production environments or distributed systems is not a requirement, I expect readers in this position to have at least some exposure to machine learning applications running in production and to have written Python and Bash scripts for at least one year. In addition, understanding the basics of Docker and being able to manage images/containers by using the Docker command-line interface is required. Familiarity with basic YAML syntax is helpful but not required; the syntax is intuitive and should be easy to pick up along the way. If most of these topics are new to you, I suggest that you learn more about them from other resources before reading further.

Summary

  • Machine learning systems deployed in real-life applications usually need to handle the growing scale of larger datasets and heavier model serving traffic.

  • It’s nontrivial to design large-scale distributed machine learning systems.

  • A distributed machine learning system is usually a pipeline of many components, such as data ingestion, model training, serving, and monitoring.

  • Using good patterns to design the components of a machine learning system can speed up the development and deployment of machine learning models, enable the use of automations from different tools, and benefit from hardware acceleration.