Over the last decade artifical intelligence (AI) methods have significantly advanced the state-of-the-art in science and engineering. One of the most prominent trends is an ever increasing amount of analyed (training) data, necessitating the usage of parallel and distributed computational resources. A well-known example for this is the machine translation algorithm Generative Pre-trained Transformer 3 (GPT-3) [1]. With a total of 175 billion parameters trained on 285.000 processor cores as well as 10.000 GPUs, this model exceeds the capabilities of traditional AI hardware. In this lecture students will learn about parallelization and scaling approaches for different AI algorithms. An emphasis is put on the advantages of parallel computing for AI, available software packages for implementation, and, majorly, the algorithmic design challenges. In line with this, examples from the following algorithmic classes will illustrates the potential use for scalable AI:

* unsupervised learning
* supervised learning
* neural networks
* ensemble method
* search

In conjuction with the course topic, students will also learn about supporting data formats, machine models and the use of novel hardware, such as quantum computer or neuromorphic devices.

[1] Ben-Nun, Tal, and Torsten Hoefler. "Demystifying parallel and distributed deep learning: An in-depth concurrency analysis." ACM Computing Surveys (CSUR) 52.4 (2019): 1-43.

[2] Brown, Tom B., et al. "Language models are few-shot learners." arXiv preprint arXiv:2005.14165 (2020).