OCPREG19 - Facebook Challenges and Opportunities of Architecting AI Systems at Facebook Datacenter

[00:00:00] Speaker 1: Hello, everybody. It's a pleasure having you here. Thank you so much for coming. You're a great community. But our community is only as great as the passion and motivation of its members. So today, I'm going to talk about my passion, which is the challenges and opportunities of architecting AI systems at data center scale. And I hope by sharing this to motivate you to all come together and tackle these hard problems. So first of all, I want to admit my guilt. And my guilt is that I'm using the term AI a little bit too frivolously. When I'm, in fact, referring to deep learning. What is AI? What is machine learning? What is deep learning? AI is effectively a program's ability to learn and reason like human. It's been paraded in all sorts of sci-fi movies and stories. And its north star is artificial general intelligence. What is machine learning? Machine learning is a set of statistical techniques that enable machines to improve with experience. intelligence. It's actually a subset of AI. And there comes even smaller subset called deep learning. Which is a multi-layer neural networks which adapt and learn from vast amounts of data. Deep learning has been tremendously popular and tremendously effective and impactful in the last decay. Which is the reason why some people equate deep learning to AI. How exactly popular is deep learning? Let's look at some statistics. Here, I'm showing the increase of the term deep learning in the research. Starting from '86 all the way to 2017. What is interesting is that you can clearly see exponential growth in the last five, six, seven years. In fact, the growth is not exponential. It's worse than that. The growth is double exponential. But what's really curious about this figure is not the growth, but the little spike that you can see in the year 2012. What do you think happened here? 2012? AlexNet happened here. AlexNet was the first deep learning network that won ImageNet competition in 2012 that showed performance improvement which were head and shoulder above anything else that was known prior to that. That year generated a lot of excitement in deep learning which resulted in a spike, I think. That spike tapered off and then it took off. I want to ask the question. What exactly makes deep learning so unique? What makes it work? And it's really simple. What makes deep learning work is the fact that deep learning just keeps thriving, keeps improving as the data and model complexity grows compared to other techniques. As long as you keep those two things growing, the party will continue. I want to demonstrate that point with an example from Facebook. This figure shows the computer vision model used at Facebook called ResNext. And the model allows different configurations with different complexity. As we vary those configuration along x-axis, we can see that the number of flaps, which is a representation of model complexity, also keeps improving, increasing. So here we are showing two curves for each, a bottom curve and a top curve. The bottom curve shows the improvement in accuracy as the model gets trained on a few millions of images of ImageNet. Not a lot, but they were all carefully labeled. Now, the top figure shows the improvement that comes from training the same model on Instagram with one billion images, three orders of magnitude more. And the miracle happens. All of a sudden, you start seeing a drastic improvement over the bottom curve. And the improvement continues as we increase model complexity. This is where the power of deep learning comes from. So that unreasonable success of deep learning also resulted in tremendous growth and scale. I want to demonstrate the growth and scale using some of the statistics that we have here at Facebook. Let's look at the growth of the data. A fraction of machine learning data that was used for training models as a fraction of the total data warehouse at Facebook was about 30% in 2018. And today, we are actually using 50% of our data warehouse for training deep learning model. There is an interesting catch here. The amount of data just doubled in one year. So those of you doing the math, you can clearly see that the growth in one year is 3x. Again, it's exponential. Also, in one year period, the training growth has also been staggering. The number of ranking engineers doubled. The number of workflows trained and compute consumed has tripled. Those are huge numbers. But what's really interesting, what happens with inference, the whole reason we are doing deep learning training and we really care about the amount of data and the growth and training pipeline is in order to support inference and bring value to the Facebook community. Look, the number of predictions is 200 trillion. What's also interesting, the number of translation is 6.5 billion. And those are narrow machine translations. They're really important. They allow people that are part of the Facebook community to speak one language because otherwise people speak different languages. And it enables people to communicate. Even more impressively, we can automatically remove up to 99% of fake accounts to make Facebook users have a safe and feel safe presence online. So, all those great things come at a cost. Doing deep learning at this scale places a huge strain on the infrastructure, in particular on key components of the data center, storage, network, compute, and memory. At the same time, the speed of innovation requires high performance and flexibility. Machine learning engineers work to experiment fast. They want to try new models, new algorithms, new operators. They want to prototype quickly and ship models to production. At the same time, deep learning evolves very quickly. So, to make matter worse, as I mentioned, deep learning gets better as we increase the data and model complexity. The implication of this is that in order to meet the demands, it requires commensurate increase in the hardware resources. But we have a problem. As Gordon Moore once said, no exponential is forever. His own law, which was supposed to be exponential, has declined. We can no longer meet the requirements of deep learning with the traditional commodity hardware within our data center. In fact, OpenAI pointed out that if you look at the largest training jobs, their growth is 3.5x every four months. This is staggering. How do we deal with those problems? We cannot address them with the traditional commodity hardware. The solution is specialization via hardware, software, co-design. We need to co-design. We need to co-design both special purpose hardware as well as the applications that this hardware is going to run. With any co-design, let's understand what our workloads are. What are those workloads? I'm going to give some examples of workloads used at Facebook. They come from three categories, ranking and recommendation, computer vision, and language, which include scenario of machine translation, speech recognition, content understanding, among other things. Recommendation models are the ones among the most popular models that we have. I want to talk a little bit about recommendation models. Deep learning recommendation models are the models used to recommend a small subset of items to the user from a large set of items. That's why they are called recommendation models. Those models take some features which characterize the user and the item, and they produce some probability that the user is going to interact with that item, like news, for example. Interestingly, many of the features that we have are categorical features. They belong to a finite, small, discrete set of categories. Those features are not good by themselves for machine learning. Machine learning doesn't really deal with discrete. Machine learning deals with continuous, with numerical values. So to map these features to numerical values, we perform so-called embedding table lookup that takes a sparse feature, maps it to a table, performs a lookup of the table, and brings a small embedding vector of a few hundred bytes that encodes or embeds that particular sparse feature. So one of the key operations is embedding table lookup. This is interesting. The reason it's interesting is that this operation is actually a regular unstructured operation which randomly fetches short embeddings from a large table. It's nothing like what we're all used to when we think about deep learning. It's all about matrix, matrix, multiplication. This is about sparsity. And sparsity is a whole different new conversation, and it deserves its own presentation. I'm not going to talk about it in more detail. But what's interesting is that today, deep learning hardware does not tackle, does not deal with sparsity very well. I really would like you to think more about sparsity. So I'm going to talk in more details about some of the challenges. Let's start with matrix, matrix, multiplication. As I mentioned, it's not all about matrix, matrix, multiplication. First, I want to show the execution breakdown as a percentage of time spent in different operators, computer averaged over inference and training. Remarkably, we see that only 40% of the time is spent in matrix, matrix, multiplication. So the first point that I want to drive across is that we should not overdesign our hardware for matrix multiplication and convolution. Even the matrix multiplications that we have, you know, those matrix multiplications are not your regular nice, big, square, dense matrix multiplication, which we typically see in HPC like high-performance LIMPAC or dense linear algebra. Those are sometimes skinny matrix multiplications. Skinny means that one of their dimension is small for different reasons. Depth-wise convolutions, group-wise convolutions, small batches, beam search, small bits, so on and so forth. What does it mean? It means that you want to have fewer number of cores, of tensor units, because you want to have fewer -- actually, you want to have more smaller tensor units, because it's going to give you a higher efficiency than fewer number of big tensor units, because the bigger tensor units are not going to match the dimensions. So, I want to -- I want to talk about memory and storage next. So, when I think about memory, I imagine -- a memory challenge. When I think about memory challenge, I imagine a two-dimensional plot, where we have size of models and neurons along x-axis, and we have arithmetic intensity along y-axis. So, what is arithmetic intensity? Arithmetic intensity is a measure of compute-friendliness of your workload. Arithmetic intensity is the number of operations, floating-point operations, to be precise, performed per each memory fetch. They're important. The higher arithmetic intensity, the easier and the faster your workloads will run, because it's not going to be bound by memory. So, what you really want, if you look at this plot, what are the characteristics of the workload? You want those workloads to be located somewhere along the main diagonal, right? Because, you know, if models are big, you want high arithmetic intensity. If the models are small, they fit into maybe undie caches. Maybe the undie caches have much higher bandwidth. So, risk-meeting intensity can be lower. It's not that critical. Let's actually look where our workloads are. [00:14:50] Speaker ?: What do we see? [00:14:51] Speaker 1: We see exactly the opposite of what we would like to see. The workloads are not along the main diagonal. They're along the opposite diagonal. This is the bad news. Let's look at the example of a recommendation system. Recommendation system sizes are huge. They are tens of gigabytes. And they happen to have low arithmetic intensity because of those sparse accesses and structured accesses that do embedding table lookup. This is bad because we need high capacity and we need high bandwidth memory. You know, HBM is not big enough. NVM is not fast enough. Those two things, high capacity and high bandwidth, are at odds with each other. Also, if you look at computer vision and language models, they're actually smaller. Definitely much smaller, maybe under a gigabyte, than recommendation models. But those models, they need to fit into the last level cache. Because if they don't, they can also suffer from the memory bandwidth bottleneck. And so what I want, ask those of you who are building AI chips, please add larger amount of Undy caches. Fitting model Undy is actually makes life much easier, both for the programmer as well as for the compiler. This provides the flexibility and programmability. I also want to talk about the network. Interconnect matters. What we have seen is that, as I mentioned in the beginning, that models have been growing. And in order to accommodate larger and larger models requires exploring model-level parallelism, where we take a model and we partition the model among multiple devices. Like, for example, here, we have embedding tables. There are huge tens of gigabytes. You need to partition it. That creates model parallelism. But it also, unfortunately, creates different communication patterns. For example, in our recommendation systems, those communication patterns are all-to-all personalized. This is the worst imaginable communication pattern for any fabric. Everybody has to send messages to everybody else. So to build the right fabric, it actually requires high bisection bandwidths. To make matters worse, there is another emerging workload called graph learning, which incorporates structural connectivity information into the learning process. This results in very unstructured communication patterns. And in addition, the messages that are exchanged are very small. Thus, you actually need low latency, low diameter in the network. And all of those creates challenges for the interconnect. I want to talk about a very important area, area of programmability. It's not all about big flaps. Let's look at the example here. If you have two systems, and if a system 2 has much higher flaps than system 1, but takes much longer to program and get to that performance, then the system 2 is less programmable. And it will block software programmer, ML engineers, from innovating and prototyping faster. One thing that I'd like to convey here is those of us who build ML hardware really need to think about software at scale. And here is my subjective view of some of the hardware features which make programmability easier or harder. Everything about programmability is usually very subjective because people differ in their ability to write code and write high performance code. For example, programming few cores is actually simpler than programming many cores. Hardly anybody will disagree with that. Programming CMD and scalar units is simpler than programming tensor units. Having caches is easier to manage because you don't have to manage them than managing software-controlled SRAM. Specialization improves energy efficiency but limits programmability. So the question and the challenge for those of you who are building AI systems is how do we get the best of world worlds? Both energy efficiency, performance, and programmability. At this point, it's clear that deep learning stresses all parts of the system. Memory, compute, fabric. It's not possible to build one system that addresses just small subset of those challenges. We need to address all of them. So I want to share how Facebook addresses those challenges when co-designing its inference and training hardware. I'm particularly excited that some of the most critical parts of those designs have been co-designed together with many of you. And some of the specs were already contributed and more will be contributed shortly. So I'm going to start with Yosemite V2 inference platform. Earlier, we contributed Yosemite V2 to OCP. And we leveraged this platform to build the inference accelerator platform. Our idea was very simple. In order to get high compute, high memory capacity, high SRAM bandwidth, let's combine together multiple accelerator and tightly couple them via high performance PCAE switch. We cannot build one system that will give us a lot of everything, but we can build a system very tightly interconnected that will give us just that. Recall we talked about flexibility. For Yosemite V2, flexibility comes from the common M2 module and from the common compute factor that we defined and work together with some of you to build the ASIC form. In addition to this, we have also been engaged with the community to invest into the programming tools called Glow. Glow is our compiler for programming those accelerators. Next, I want to talk about Xion training platform. Similarly, just like with inference, to bring a training platform that will have a lot of everything to address and accommodate diverse set of workloads, we scaled up multiple CPUs and accelerator together via high bandwidth, low latency interconnect. And those interconnects are desegregators. Here we put together eight CPUs and eight GPUs, each with their own interconnect. As a whole, as a system, let's call it supernova. It has hundreds of teraflops of Bflow 16, high capacity DDR memory in order to accommodate large models. As I mentioned, our recommendation system are very large. And it also has hundreds of gigabytes of high bandwidth memory, HBM, in order to accommodate some of the embeddings, for example, which are more frequently accessed, require higher performance. And as I said, we have desegregated fabric, we have UPI for CPU, and we have different types of fabric for ASIC. So where does the flexibility of the system come from? The flexibility comes from OAM module. In the beginning, we didn't really know which ASIC to use. There is quite a lot of ASICs out there. So we didn't want to build a different system for a different ASIC. So here comes the OCP accelerator module, OAM. So it was an effort laid by Facebook and several other companies where we, together, defined a vendor agnostic, common four-factor, defined the characteristics, compute, memory, bandwidth, and interconnect. And we contributed that to the OCP. And the other aspect of the system, flexibility is incremental software enablement. For example, people, the ML engineers, who want to use a system, they can start with CPU. Between all the CPUs here, there is quite a bit of compute and memory bandwidth, and especially capacity, and they can get going right away. As their software matures, they can offload some of the high-demanding parts of the model, like, for example, matrix-matrix multiplications or embedding tables to the accelerators. That ability to gradually increase the complexity of your software while not meeting a lot of performance clips is really, really important. And I'd like to encourage everybody to see the Zion talk this Friday that will deep dive into a lot more details. I want to end my talk with a call to action. We talk about more slowdown, specialization, and the co-design being very important remedies. We need to tackle problems holistically, memory compute, network storage. The only constant is change. Everything is moving fast. New developments in sparsity, graph learning, and supervised learning. Performance is the start of the conversation. To keep the conversation going, we need programmability. Our journey is only 1% finished. And the door is open for all of you to come and bring forth the ideas and innovations and contribute to open design. It's only through this contribution and working together we'll be able to move the needle of deep learning. And to end, I just want to mention as Richard Feynman once said in his famous now talk, there is plenty of room at the bottom. So to paraphrase it, there is plenty of room at all levels of hardware and software stack for all of us to work together. I'm going to conclude my talk. Thank you very much. Thank you very much. And I'm truly looking forward to catching up with some of you during the break. Thank you.