Video, audio (multimodal) mobile and edge use cases that utilize machine learning models (e.g. Tiktok, Shazam, Google Home Hub) are becoming more common. However, creating these multimodal ML applications is challenging as developers need to deal with real-time synchronization of time series data during model inference and doing it cross-platform on mobile and edge devices.
Google open sourced MediaPipe in June 2019, a cross-platform applied machine learning pipeline framework that simplifies the development process. My talk will introduce the open source MediaPipe framework, walking through mobile and edge (EdgeTPU coral) demos and getting developers started on building multimodal ML applications.