Welcome-breakfast and introduction to resource-aware machine learning

Schedule: Monday, 25.09., 09.00-10.30

Lecturer: Katharina Morik

Start the summer school on resource-aware machine learning together with your fellow participants and our welcome-breakfast.

During the breakfast, listen to the introduction about hot topics on resource-awareness: Where are the limitations, being it due to huge amounts of data, high-dimensionality or restrictions imposed by small devices? What about the four Vs of data analysis: Velocity, Variety, Volume and Value.

Field-programmable technology and machine learning

Schedule: Monday, 25.09, 11.00-12.30

Lecturer: Wayne Luk

This lecture covers recent research on field-programmable technology and machine learning. It describes how field-programmable technology can accelerate machine learning applications, and how machine learning techniques can enhance the performance of systems based on field-programmable technology. The proposed approaches will be illustrated by several examples including hyperspectral image classification, and Sequential Monte Carlo designs for air traffic management. The potential of the proposed approaches will also be presented.

Deep Learning

Schedule: Monday, 25.09, 16.00-17.30

Lecturer: Sebastian Buschjäger

With the rise of Big Data and cheaply available computation power, Neural Networks have exceeded state-of-the art results in many practical areas such as speech recognition or image classification. Even though Neural Networks have been around for over 70 years, this success is quite recent and somewhat sudden. Reasons for this success lie not only in more data and in more computation power available, but also in a more engineering-style approach towards Neural Networks in general. In this lecture, we want to discuss this recent rise of Neural Networks as part of the Deep Learning approach. The first half of the talk will cover the basics about (Convolutional) Neural Networks, such as Backpropagation, the effect of different activation functions and general model architecture. The second half of the talk will focus on recent results in stochastic gradient descent optimization and their impact in Neural Network training.

Deep Learning on FPGAs

Schedule: Monday, 25.09, 18.00-19.00

Lecturer: Sebastian Buschjäger

Deep Learning is one of today's state-of-the art tool for image classification and one of the driving force for autonomous driving (no pun intended). For image classification we usually see deep models with up to 128 layers and more than billions of parameters. Today's approaches for training and deploying these large models are mostly based on energy hungry GPUs and thus are not suitable for embedded devices found in e.g. cars. In this talk we will explore the possibilities of learning and using large Neural Networks on small, embedded devices. A specific focus will be laid on Field-Programmable Gate Arrays (FPGAs) which offer a fast, but less energy demanding computation architecture for Deep Learning. I will cover the basics of FPGAs and discuss some specialized Deep Learning approaches designed for FPGAs. Additionally, I will lay out some recent results in model pre-processing which help to further reduce computational need during model execution.

Generalised LDPC architectures for high-throughput FPGA realisation

Schedule: Monday, 25.09., 14.00-15.30

Lecturer: Rob Maunder

In this lecture, I will provide an tutorial on LDPC codes and their FPGA implementation. I will present a survey of 140 published designs and use this to draw conclusions about the fundamental trade-offs in the FPGA implementation of LDPC decoders. This survey will highlight the need for new designs having much greater flexibility than ever before. Finally, I will present a generalised LDPC architecture, which can flexibly support a wide variety of LDPC code designs with a high throughput.

A Tale of a Quest for Business Intelligence from Social Data

Schedule: Tuesday, 26.09., 09.00-10.30

Lecturer: Rakesh Agrawal

I present the story of a research expedition (code-named WaveFour) into building an enterprise-scale, real-time business intelligence system over social data. I discuss what drove us to undertake this journey and the system prototype we built. I also describe the investigation we carried out to assess the overlap between Google and Bing search results and whether including social data in the mix can produce different and useful results. I conclude with lessons learned and future directions.

Probabilistic Graphical Models

Schedule: Tuesday, 26.09., 11.00-15.30

Lecturer: Nico Piatkowski

Small, wearable, and resource-constrained devices are ubiquitous---they accompany and support us in many personal and professional settings. Uncertainty is inherent in almost all real-world situations since many important aspects of a system may only be partially observed or cannot be observed at all. Probabilistic graphical models allow us to analyze such uncertainties via stochastic dependencies between specific parts of a system, they handle rare and unexpected cases, and may be estimated from incomplete data.

In this lecture, we focus on probabilistic models on ultra-low-power microcontroller units. We will see why a certain class of probability distributions, namely exponential family models, is especially well-suited for resource-constrained systems. We derive ab initio a sub-class of exponential family models, which is restricted to integer parameters and allows for probabilistic inference with integer-only arithmetic. Theoretical and empirical results show, that this technique strongly reduces the memory and computation time requirements for weak computational systems.

On the one hand, restricting the parameter space to the natural numbers, phrased as a special kind of regularization, excludes an uncountable infinite set of models from our consideration. On the other hand, overflows in the underlying data types force us to propose a new approximate inference technique, which is based on the bit-lengths of intermediate results, and is related to the sum-product algorithm. By analyzing these different types of error, we draw conclusions about which kinds of probability mass functions admit a lossless representation by this sub-class of exponential family models.

Finally, we investigate how the integer-restriction increases the size of models that can be applied on real-world ultra-low-power systems.

Graph Streaming

Schedule: Tuesday, 26.09., 16.00-17.30

Lecturer: Chris Schwiegelshohn

Consider a sequence of edge insertions (and possibly deletions) to a graph. In the streaming model we try to (approximately) solve some problem on the graph while storing as little space as possible. In this talk, we will give an overview on the basic aims, challenges, and limitations of the graph streaming model. We will further present two crown jewels of the field:

1. A 2+ε approximation algorithm for weighted matching in insertion-only streams (Paz and Schwartzman, SODA 2017)

2. Connectivity testing in dynamic streams (Ahn, Guha, and McGregor, SODA 2012)

Presentation of student grant awardees

Schedule: Tuesday, 26.09., 18.00-19.00

Lecturer: Jihed Khiari

Presentation: Machine Learning and Public Transport

Urbanization has created massive opportunities and challenges for public transport. Most vehicles collect now data that is valuable for analysis, planning and real-time control. We aim in this short talk to present some applications in this area ranging from bus scheduling to taxi demand estimation which rely on various machine learning techniques.

Lecturer: Mina Niknafs

Presentation: Interarrival Time Prediction for Runtime Resource Management

In a modern heterogeneous architecture, multiple CPUs and GPUs are integrated together in order to obtain performance benefits; however, this can only be achieved if the platform based on this architecture is equipped with an intelligent resource manager (RM). We argue that there exist patterns in the interarrival times of resource requests, and that these patterns can be used for modeling and prediction of the future arrivals by the RM. To this end, we have developed a time series and machine-learning-based framework.

Lecturer: Pantelis Rafail Vlachas

Presentation: ADMM and AMA for MPC - Comparison and Parallels

The alternating direction method of multipliers (ADMM) and the alternating minimization algorithm (AMA) are first order optimization methods that gained significant attention in the research community lately. In the context of model predictive control (MPC) the efficient implementation of these methods for the solution of quadratic programs, requires a proper problem formulation and step-size parameter selection, preconditioning and scaling, as well as acceleration techniques. In this work, we examine the similarities and differences of the methods and a parallelism is established, as the optimal step-sizes of both ADMM and AMA depend on the eigenvalues of the same matrix. The convergence speed of the methods in practice is examined in quadratic programs stemming from three MPC applications (quadruple tank, quadrotor and buffer management system problems).

Lecturer: Sebastian Giraldo

Presentation: Digital Implementation of an always-on Keyword Spotting based on LSTM neural networks

With the ubiquitous importance of speech recognition for different applications in mobile devices, the need for low power embedded execution has increased because of privacy, real-time and power consumption issues. One example of these speech recognition tasks is Keyword Spotting (KWS), where the objective is to detect specific words spoken by the user. However, the KWS system must as such be always active, constantly listening to the environment, hence imposing strict power constraints. Even more challenging is that these systems ideally are capable of local learning, to customize to the specific user. In this project we are developing an always-on keyword spotting system based on LSTMs. Approximate computing techniques are applied with the objective of reducing the power consumption footprint while maintaining high accuracy and reliability.

Lecturer: Vladimer Kobayashi

Presentation: Deep Learning for Automatic Extraction of Job Type Information from Online Vacancies

I deal with the problem of automatic extraction of role requirements in over 25 million job vacancies. After preprocessing and feature extraction a convolutional neural network and word2vec to classify and recover the semantics of words/phrases is used. In the evaluation step a similarity measure and clustering is used to group highly similar vacancies together and use the job titles in those vacancies as synonyms or alternatives of each other. In this way, it is possible to identify transferrable skills and competencies across job professions and industries.

Programming Energy-Harvesting-Based Sensor Nodes

Schedule: Wednesday, 27.09., 09.00-10.30

Lecturer: Olaf Spinczyk

Energy-harvesting based computer systems will enable various interesting use cases, especially in the context of the Industry 4.0 and Internet of Things visions. They turn almost arbitrary objects into smart networked sensors with minimal maintenance effort and at moderate costs.

However, programming these ultra-low-power systems is a challenge. Besides harsh resource constraints in terms of RAM, FLASH, communication capabilities, and CPU power, developers have to deal with a varying amount of available energy. In fact energy management dominates the whole hardware/software design process.

Taking the PhyNode platform and its operating system KratOS as a running example, the lecture will address energy models and their creation, energy management, energy-aware operating systems, and energy-aware communication.

In doing so, the lecture will also prepare the participants for the practical exercises in the following sessions.

Ultra low power learning

Schedule: Wednesday, 27.09., 11.00-19.00 and Thursday, 28.09. 09.00-10.30

Lecturer: Motjaba Masoudinejad

A world full of sensors leads to two main developments: Gigantic centralized services, gathering and analyzing data, and highly distributed data generation. The latter demands respecting a multitude of restricted resources: Computational power, limited communication speed and reliablity, often limited energy. The PhyNodes are embedded computing and sensing platforms developed at the collaborative research center for as a large scale testbed for the Internet of Things.

Tractable Probabilistic Graphical Models

Schedule: Thursday, 28.09., 11.00-12.30

Lecturer: Kristian Kersting

Probabilistic models have had broad impact in machine learning, both in research and industry. Unfortunately, inference in unrestricted probabilistic models is often intractable. Motivated by the importance of efficient inference for large-scale applications, the research focus has shifted to very large probabilistic models that provide enough symmetries so that inference is feasible or even to model that are tractable despite having high treewidth. Example are relational models with lifted inference and sum-product networks. In this tutorial I shall introduce and review some of these developments with an emphasis on distributions beyond multinomials and Gaussian.

Mobility Models

Schedule: Thursday, 28.09., 14.00-15.30

Presenter: Thomas Liebig

In street-based transportation science, vehicle volume estimation receives lots of attention, as it provides important insights to several applications, such as situation-aware trip planning, attraction ranking and traffic control systems. In practice, empirical measurements are sparse due to budget limitations and constrained mounting options. Therefore, estimation of vehicle quantity is required to perform mobility analysis at unobserved locations. Accurate vehicle mobility analysis is difficult to achieve due to non-random path selection of the individual persons (resulting from motivated movement behaviour). This causes the vehicle volumes to distribute non-uniformly among the traffic network. Existing approaches (vehicle simulations and data mining methods) are hard to adjust to sensor measurements or require more expensive input data (e.g. high fidelity street networks plans or total number of vehicles in a city) and are, thus, unfeasible. In order to achieve a mobility model that encodes vehicle volumes accurately, we discuss two methods which overcome the limitations of existing methods. These two methods incorporate topological information and episodic sensor readings, as well as prior knowledge on movement preferences and movement patterns.