Cloud Native event: Kubernetes failure stories and more – June 3rd, 2019


Interested in Kubernetes? We have a great meetup lined up for you. We have the great pleasure of welcoming Henning Jacobs from Zalando who is visiting Aarhus. Henning is an experienced speaker, and recently created the Kubernetes Failure Stories repository at Github which features a comprehensive list of Kubernetes post-mortems

Furthermore, Henrik Høegh will provide you with an introduction to how to handle secrets in a Kubernetes environment using HashiCorp Vault.


16:30: Doors Open
17:00: Welcome
17:15: “Manage your Cloud Native secrets with Vault”, Henrik Høegh, Praqma
18:00: Break and Food sponsored by Tradeshift
18:30: “Kubernetes Failure Stories and How to Crash Your Clusters”, Henning Jacobs, Zalando
19:15: Networking
20:00: Thank you and goodnight

TALK 1: “Manage your Cloud Native secrets with Vault”

In this talk, we will look at the method used to integrate Vault with Kubernetes, and how to authenticate, write and read data from it. But running Vault de-coupled from Kubernetes is not “the Cloud Native way”. In a Cloud Native world, we want things to integrate with the orchestrator. We also have sensitive information in our cluster, which we want to keep secret. Kuberentes secrets are not really suitable for storing information like database passwords, usernames or the likes, as they are simply stored with a base64 encoding. Instead, we want to keep our “stuff” in a secure place, where we can rotate access tokens. This is where Vault comes into the picture. Vault can store and control access to sensitive information and it integrates with Kubernetes, allowing pods to authenticate with their service account.

About Henrik Høegh

As a DevOps consultant, Henrik regularly works with build servers, container technologies like Kubernetes, version control, and agile task management systems but also spends a great deal of time working with general organizational, architectural and process related challenges for customers. And yes, he drives a panda.

TALK 2: “Kubernetes Failure Stories and How to Crash Your Clusters”

Bootstrapping a Kubernetes cluster is easy, whereas rolling it out to nearly 200 engineering teams and operating it at scale is a challenge. In this talk, we are presenting our approach to Kubernetes provisioning on AWS, operations and developer experience for our growing Zalando developer base. We will walk you through our horror stories of operating 100+ clusters and share the insights we gained from incidents, failures, user reports and general observations. Our failure stories will be sourced from recent and past incidents, so the talk will be up-to-date with our latest experiences.

Most of our learnings apply to other Kubernetes infrastructures (EKS, GKE, ..) as well. This talk strives to reduce the audience’s unknown unknowns about running Kubernetes in production.

About Henning Jacobs

Henning joined Zalando at the beginning of 2010 and accompanied the transformation of Zalando’s technology department through the eras of PHP/MySQL and Java/PostgreSQL to the new world of “Radical Agility”. He helped to build the STUPS cloud infrastructure to make innovation scale across autonomous teams. His five teams help streamline the developer experience by providing a Cloud Native application runtime to 200+ engineering teams.