Scaling Big Data Interactive Workloads across Kubernetes Cluster
The Jupyter Notebook Stack has become the "de facto" platform used by data scientists to interactively work on big data problems. With the popularity of deep learning, there is also an increasing need for resources to make deep learning effective. In this session, we will discuss how we brought support for Kubernetes into Jupyter Enterprise Gateway and touch on some best practices on how to scale an interactive big data workloads across a Kubernets managed cluster.
Luciano Resende is an STSM and Open Source Data Science/AI Platform Architect at IBM CODAIT (formerly Spark Technology Center). He has been contributing to open source at The ASF for over 10 years, he is a member of ASF and is currently contributing to various big data related Apache projects around the Apache Spark ecosystem. Currently, Luciano is contributing to Jupyter Ecosystem projects building scalable, secure and flexible Enterprise Data Science platform.