Kubecon Europe 2020 - Day 3

Third day of the KubeCon CloudNativeCon Europe 2020 virtual event.

Again lot of interesting sessions !

Check out other days:

Kubecon Europe 2020 - Day 1

Kubecon Europe 2020 - Day 2

Kubecon Europe 2020 - Day 4


Keynote: Kubernetes Project Update - Vicki Cheung, KubeCon + CloudNativeCon Europe 2020 Co-Chair & Engineering Manager, Lyft

Vicky Cheung presented the release notes of Kubernetes 1.18.

  • 38 enhancements

  • Release team: 34 persons

  • 40 000 individual contributors to date !

Key features:

  • Storage Enhancement:

  • Raw block device support graduates to Stable

  • alpha version of CSI Proxy for Windows (to perform privileged storage operations in Windows)

  • Scheduling Enhancements:

  • Run multiple scheduling profiles

  • Taint based Eviction graduation to Stable

  • PodTopologySpred graduation to Beta

  • HPA (Horizontal Pod Autoscaler)

  • Feature in Alpha

  • Finer grain control on autoscaling rates

  • Avoid flapping of replicas

  • Adjust scaling behavior based on application profile

  • Kubectl alpha debug

  • Ephemeral containers were added in 1.16

  • When “kubectl exec” isn’t enough

  • 1 kubectl alpha debug -it demo --image=debian --target=demo

  • Priority and Fairness for API Server Requests

  • Protect API servers from being overloaded while ensuring critical requests go through

  • Prevent loss of user access should anything run amok

  • Node Topology Manager graduates to Beta:

  • Useful for high performance computing and machine learning workloads

  • Other Notable features:

  • IPv6 Beta

  • Certificate API

  • APIServer DryRun and “kubectl diff”

In the next Kubernetes 1.19:

  • Generic ephemeral volumes

  • Kube-proxy IPV6DualStack support on Windows

  • Initial support for cgroup v2 (YES !!!)

Sponsored Keynote: The Kubernetes Effect - Igniting Transformation in Your Team - Briana Frank, Director of Product, IBM Cloud

Briana Franck explained how to transform your IT projects into the Cloud Native world.

New projects allow teams to re-evaluate the current path of transformation using Design thinking workshop. Take advantage of the start of new projects to catalyze innovation and enforce the culture of automation to gain efficiency. Human and communication is key. An exemple for operate and improve communication is ChatOps.

Automate the deployment of thousands of clusters with ease: exemple with Razee.io

Keynote: How to Love K8s and Not Wreck the Planet - Holly Cummins, Worldwide IBM Garage Developer Lead, IBM

This talk could have been renamed “Zombie land” !

Excellent keynote of Holly Cummins explaining the impacts of bad resources usage on the Planet.

The consumption of resources between data centers and aviation is not that far! (1 to 2% vs. 2.5%)

The Kubesprawls trend (pattern of “many clusters” rather than “one big shared” cluster) is not uncommon for multiple teams in the same organization. Cluster are less elastic than applications and have overhead.

There are several reasons to get your own cluster: isolation, security, perf, name collisions, …

Consolidation with multi-tenancy could be a good approach.

In all cases don’t forget your resources : Zombie workloads ! You know the things you forget in your cluster and that stay “alive” but not really and that consume resources for a while !

  • manual solutions exists: meeting, tags on manifest objects, …

  • The solution is to do the right thing

  • GitOps helps a lot with the infra-as-code: “disposable infrastructure” you can deploy, redeploy and delete in a single operation.

To avoid Zombie workload, See the talk “Sharing Clusters: Learnings From Building a Namespace On-Demand Platform“ especially the part “Monitor cost and identify idle namespaces”

Sponsored Keynote: Keep It Simple - A Human Approach to Coping with Complexity - Hannah Foxwell, Director – Platform Services, VMware Pivotal Labs

In this presentation, Hannah Foxwell explains what is the path of least resistance to a successful Cloud Native project:

  • Start by identify Early Adopters

  • Build a Minimum Viable Product

  • Don’t think like building the platform as a silo and more as service

  • Scaling the success

  • Communication and relation is key

  • Small steps are easier

  • Adopt the KISS principle: Keep It Short and Simple

And remember we are human not super human !

Containerd Deep Dive

By Akihiro Suda (Software Engineer, NTT) and Wei Fu (Software Engineer, Alibaba)

This talk was about Containerd, a container runtime implemented the CRI runtime specification.

Containerd is not only used by Docker but also by many distributions: K3s, kubespray, micro8s, Charmed Kubernetes, Kind, minikube, Alibaba ACK, Amazon EKS, Azure AKS, Google GKE, … And also by libs/frameworks like BuildKit, LinuxKit, Fassd, Vmware Fusion Nautilus

A lots of nice features were introduced with version 1.4:

  • Lazy pulling of images: run containers before completion of downloading the images -> improve start speed: it is based on Stargz & eStargz. With a tar format you cannot navigate to a specific offset until all the file is downloaded

  • Support for cgroup v2 and improved support for rootless mode (resource limitation support)

  • Windows CRI support

  • and much more (see release notes: https://github.com/containerd/containerd/releases/tag/v1.4.0)

Containerd is highly Customizable throught its V2 Runtime:

  • Runtime v2 with a shim API that allows for instance integration with low level runtime like gVisor, KataContainer, Firecracker.

  • Binary: specific naming convention

  • Support pluggable logging via STDIO URIs

Containerd is definitely a good CRI runtime with lots of nice features in v1.4. We especially appreciate the lazy pulling of images and expect a lot from cgroup V2.

What You Didn’t Know About Ingress Controllers’ Performance

By Mikko Ylinen (Senior Software Engineer, Intel) and Ismo Puustinen (Cloud Software Engineer, Intel)

Basically Ingress exposes HTTP(s) routes to services within the cluster. The Ingress controller follows the Kubernetes objects and creates the route configuration. The Ingress Proxy reads the configuration and handles the actual traffic routing

There are several implementations like Nginx, HAProxy, Envoy…

Performance bottleneck are mostly bandwidth and latency.

Tuning areas concern:

A focus on TLS Handshake because most of the time of a request is spent on the TLS handshake.

How to improve it:

  • TLS 1.3 or a faster cypher…

  • Sync vs async TLS

  • Async TLS is currently being added in Ingress

  • Async TLS Offloading

When to offloading depends on CPU-bound of the controller, if lot of new HTTPs connections, …

Exemple with HAProxy Hardware Acceleration, HAProxy RSA Multibuffer

Call to action:

  • Check where are your bottlenecks

  • Ingress Controllers: check config of crypto offload, non native resources, node affinity labels

  • Ingress Proxy devs: switch to async TLS and allow custom TLS handshake

A good and very technical presentation on how to tune your Ingress if you need to get the best performances.

Managing Multi-Cluster/Multi-Tenant Kubernetes with GitOps

By Chris Carty (Customer Engineer, Independent)

The author presented what is GitOps (You will also find a presentation of GitOps on the Sokube Blog)

2 main GitOps projects:

Project Structure contains your deployment yaml files.

Some useful tools in the GitOps context :

  • OPA (Open Policy Agent)

  • Conftest (helps you write tests against structured configuration data)

  • Kubeval (tool for validating a Kubernetes YAML or JSON configuration file)

  • Kind (tool for running local Kubernetes clusters using Docker container "nodes" like k3d/k3s):

Git Single vs Multi Repo in case of multi-tenancy

Single Git repo:

  • Containing the CI, Cluster-admins (monitoring, networking, security), teams-1 (app resources), teams-2 (App resources), … :

  • each cluster (Dev, QA, Prod, …) Sync on a specific branch or tag


  • Several Git repos can target the same cluster but in different namespaces

  • Branch/Tag = Grouping/Environnement of cluster

A nice presentation for this very important topic. The speaker did not have time to go into details and to present the pros and cons of each approach, but it gives important pointers and a way to achieve multi-tenancy using Flux in Kubernetes .

Sharing Clusters: Learnings From Building a Namespace On-Demand Platform

By Lukas Gentele (CEO, DevSpace Technologies Inc.)

2 Approaches

  • Single-Tenant k8s: 1 Team/App per Cluster => too expensive

  • Multi-tenant k8s: Sharing large Cluster => less expensive but more complex

Several learning:

  • Centralize user management and authentification

  • SSO for k8s via Dex for instance

  • Restrict user but use smart defaults (UX matters)

  • Pod Security Policy

  • Resource Quotas: Set default via LimitRange (Mutating Admission Controller)

  • Network Policies; Default to deny all (allow traffic from inside the namespace)

  • Automate as much as possible

  • template for RBAC, Quotas, Net Pol

  • OPA for dynamic admission control (ex: Hostname validation for ingress resource or for certificate resource or block certain storage and network conf…)

  • Store everything in kubernetes + git

  • Use annotation, labels, secrets config map to store info about owner, tenants, …

  • GitOps for History, audit, rollback and Approval process vial PR and Code Owner

  • CRDs for even more control & automation

  • Kiosk:

  • Do not hide kubernetes but make it easier to use

  • Engineers need direct access to Kubernetes (to verify, debug, ….)

  • Kubectl is an API not a dev tool

  • Simplifying Local Development in Kubernetes with Telepresence

  • File-Sync-based Dev Experience: Skaffold, DevSpace, Tilt

  • Monitor cost and identify idle namespaces

  • automate the shutdown of idle namespaces with:

  • Cluster-turndown by Kubecost

  • idling on Openshift

  • Sleep mode in loft

  • Sometimes users need more than just namespaces

  • Namespaces based multi tenancy has limitations: CRD needed by users or specific versions of k8s

  • Virtual Cluster can solve this problem: vCluster

A very interesting talk on how to address multi-tenancy providing best practices and tools like Telepresence, Cluster-turndown, Sleep-mode, VCluster and Kiosk…

Multi-Tenant Clusters with Hierarchical Namespaces

By Adrian Ludwin (Senior Software Engineer, GKE, Google)

Concept of hierarchical namespaces is new in Kubernetes.

Multitenancy to care about cost and velocity.

One tenant per cluster was for small team

Kubesprawl : it does scale very well to get 1 cluster per tenant or team

A Kubernetes multi-tenancy group exists to adresse those issues.

Namespaces are the primary unit of tenancy in kubernetes but some security features require namespaces:

  • RBAC works best at the namespace level

  • Also applies to most other policies : resource Quota, NetworkPolicy, …

Policies across namespaces

  • need a tool and source of truth outside k8s: Flux, ArgoCD, ….

  • Alternatively, some in-cluster solution add accounts or tenants: Kiosk or Tenant CRD

Hierarchical Namespace Controller (HNC):

Hierarchical namespaces make it easier to share your cluster by making namespaces more powerful. For example, you can create additional namespaces under your team's namespace, even if you don't have cluster-level permission to create namespaces, and easily apply policies like RBAC and Network Policies across all namespaces in your team (e.g. a set of related microservices).
  • Entirely Kube native

  • Builds on regular kube namespaces

  • Delegate subnamespace creation without cluster privileges !

  • policy propagation

  • subnamespace hierarchy

  • It cannot be moved

  • Trusted label with tree suffix in the child namespace

  • Easy to extend

Other features:

  • Authorization checks before modifying the hierarchy

  • Cascading deletion of subnamespaces

  • Monitoring options (Metrics via OpenCensus)

  • Uninstallation support (to avoid data deletion)

Hierarchical namespaces is in alpha but can simply be added like an addon on Kubernetes 1.15+ (Well done !)

It is a really cool feature that will simplify the multi-tenancy approach in a K8S cluster (especially with a GitOps approach) and give teams control over their own namespace hierarchy.

Automating Load Balancing and Fault Tolerance via Predictive Analysis - Steven Rosenberg, Red Hat

By Steven Rosenberg (Software Engineer, Red Hat)

Load Balancing → Type of solution Fault Tolerance → Live Migration Scheduling → Predictive Analysis

Load Balancing

  • Priority based upon urgency

  • Even Distribution within categories :

  • Urgent priority - Mission Critical - Real Time Processing

  • High Priority - High Importance - near Real Time Processing

  • Neutral Priority - Medium importance - Normal Processing

  • Low Priority - Low importance - Not Time Critical Processing

  • No Priority - Unimportant - Unimportant Proecesses

Fault Tolerance Redundancy Example


  • Ability to launch processes based upon needed resources

  • Monitor the amount of resource each process utilizes

  • Type of Launching/Migration Scenarios :

  • Initial Launch

  • Migration for maintenance

  • Re-balancing - Migration to another host

  • Fault recovery - Migrating to mitigate system/process failure

Policy units - Attributes of scheduling Migrations

  • Filters

  • Weights/Scoring

  • Balancers :

  • Even distribution

  • Power saving

  • Prioritizing

  • Affinity

  • CPU/Non-Unimform Memory Access Pinning for optimal performance

Live migration process to migrate from source host toward destination host :

  • Network connectivity

  • Remote disk availability

  • Migration data on local disk(s)

  • Copying memory state in phases

  • All of the Curren memory contents

  • Current differences before VM pausing

  • Minimal difference during VM pausing

  • Copy CPU State

  • The goal is to limit pausing of the VM

  • Restarting the VM on the destination host

  • Clean up on the source host

Predictive Analysis Topics

  • Predicting future occurrences via analysis of past performance

  • Techniques for predictive analysis

  • Process for developing a prediction model

Predictive analytics methodology :

Get historical data → Create a training Set → Create an algorithm and a model → Get result → Restart process

Process fo developing a prediction model :

Applying Predictive Analytics to Schedulers

  • Criteria for Data

  • Processing time / Iteration - Adjusted for resource capacity and priority

  • Percentage of resources used - Adjusted for capacity and priority

  • Adjust for anomalies when calculating averages

  • Ideas - DSelective techniques applied for other scheduling applications :

  • Comining regression-like modeling and functional approximation using the sum of exponential functions to produce probability estimates

  • Machine Learning and advanced mathematical models

Predictive analysis architecture :

Tracking historical data :

  • The time each process starts and terminates

  • The resources used by each process

  • The time each process uses to migrate

  • The time/iteration that memory/disk transfer occurs per size

Consideration based upon analysis

  • If early can proceed

  • When early migration shall start

  • Error correction/anomaly detection for accurate results

This topic was interesting but is a complex subject. It demonstrates how to be proactive about load-balancing and fault-tolerance, based on collected historical data and mathematical compute to prevent future faults. There are interesting models to improve infrastructure reliability.

Simplify Your Cloud Native Application Packaging and Deployments - Chris Crone, Docker

By Chris Crone (Engineering Manager, Docker)

What is a cloud native application ?

“A program or piece of software designed to fulfill a particular purpose” - Oxford English Dictionary

A cloud native application is :

  • Compute :

  • Containers

  • Function (AWS Lambda, Azure Functions…)

  • Virtual Machines

  • Storage :

  • Databases

  • Object storage

  • Volumes

  • Networking

The CNCF Cloud Native Landscape map represents a view of many applications, tools , runtime and more recommended by a CNCF. On this interactive map you can directly visit the product website by clicking on the product logo.

Deploying application

Are you encountering the following errors ? Probably yes !

  • Often need more than one tool to deploy an application

  • Is the ReadMe up to date ?

  • Which version of the tools ?

  • What if I'm using Windows or Mac and not Linux ?

  • Difficult coordination problem between team members, CI, users …

Ideal deployment tooling ?

  • Defined as code : tools, versions options → what the solution ?

  • Same deployment environment everywhere → what the solution ?

Packaging application

Different parts, different places :

Ideal application packaging ?

  • Immutable application artefact → what's the solution ?

  • Store the whole application in a registry → what's the solution ?

  • Ability to store application artefact online → what's the solution ?

There are many questions without answers!

Cloud Native Application Bundles

→ Here the Cloud Native Application Bundles (CNAB) enter in the picture.

CNAB is a package format specification that describes a technology for building, installing and managing distributed applications, that are by design, cloud agnostic.

CNAB specification

  • Target is tolling developers

  • Packaging specification (bundle)

  • Bundle runtime (action)

  • Install, upgrade, uninstall

  • Optionally

  • Lifecycle tracking

  • Registry storage

  • Security

  • Dependency

Bundle structure :

CNAB runtime

  • Standard actions : install, upgrade, uninstall

  • Custom actions :

  • status, logs …

  • Stateful, stateless

  • Application lifecycle tracked by claims

  • Keep track of state of installation

  • keep record of parameters, outputs…

  • Only data structure defined in specification

Finally CNAB can respond to previous questions :

→ Ideal deployment tooling ?

  • Defined as code : tools, versions options → solution :

  • Porter tool : Package everything you need to do a deployment (command-line tools, configuration files, secrets and bash scripts to glue it together). Package that into a versions bundle distributed over standard Docker registries or tgz files.

  • porter.yaml

  • Stored in CNAB invocation image

  • Same deployment environment everywhere → solution :

  • Containers

→ Ideal application packaging ?

  • Immutable application artifact → solution :

  • Hashes for components

  • Leverage OCI image specification

  • Store the whole application in a registry → solution :

  • Any OCI compliant container registry

  • Ability to store application artifact online → solution :

  • OCI image layout

CNAB in registries :

With CNAB : Different parts, same place :

CNAB Security

  • Leverage same mechanisms as containers

  • TUF

  • In-toto

  • Notary

  • Signy → Is a tool that implements CNAB security specification. It implements signing and verifying for CNAB bundles in the cancel formats.

CNAB website : cnab.io Demo code : chris-crone/kubecon-eu-20 Porter : porter.sh

This topic is again interesting. In fact, it allows true multi platform applications and makes application deployment very easy . Cloud Native Bundle Application can be very useful and accelerate new features deployment. It will be very interesting to test this new standardized bundle in toolchain integration. Something to keep an eye on...

How to Work in Cloud Native Security: Demystifying the Security Role

By Justin Cormack Engineer, Docker

Resource : https://static.sched.com/hosted_files/kccnceu20/06/How to work in cloud native security.pdf

What to take away from this conference (high level of abstraction) :

  • Security must be transverse and present everywhere (Development, Operate = DevSecOps)

  • There's no secret, you have to: Play, understand, break, fix.

  • Security is unimportant most of the time and it is often hard to hire people. (Estimated that there will be 3.5 million unfilled security jobs in 2021)

  • You don't need :

  • Formal qualifications in security

  • To have hacked high profile sites

  • To be a great developer

  • Understand the threat model

  • Security is quality :

  • Handle errors and the unexpected.

  • Understand the issues in domain.

  • Write security tests.

  • A security engineer is a human like any other :

  • You cannot tell anyone about what you do most of the time.

  • Not enough people, often overloaded.

  • Live away from the happy path.

Interesting conference, the speaker depicts the routine of security jobs through his own experience, the difficulties he may have encountered, the complexity of recruiting this type of profile or the negative effects of a job market under stress (burnout, etc..)

Prometheus Deep Dive

By Goutham Veeramachaneni (Software Engineer, Grafana Labs) and Bartlomiej Plotka (Principal Software Engineer, Red Hat)

Resource : https://static.sched.com/hosted_files/kccnceu20/85/KubeCon 2020_ Prometheus Deep Dive Bartek _ Goutham (1).pdf

Favour the federation approach (architecture example) :

  • A server Prometheus (high retention) scraping the client Prometheus (low retention)

  • Federation (/federate) updates every (~15s)

  • The source of your Prometheus clients must be specific (e.g. metrics infrastructure, metrics apps, metrics middleware, etc..)

  • Avoid the Russian doll effect (Never go beyond 4 subsets)

Don't believe everything you're told :

  • Prometheus can work over long term retentions of time (2 Years, easy), if you meet a few prerequisites :

  • Older data use marginal resources when not queried.

  • Get large SSD and plan data size head of time.

Caveats (Be careful!) :

  • Hard to plan disk space for future: Uncontrollable cardinality.

  • Persistent disk backup are not always easy

  • No downsampling

Remote Write ( So what if I want to send to a different DB ?) :

  • Remote Endpoints and Storage :

  • Manually (CLI import) :

This conference was interesting, as it addressed many of Prometheus's often unrecognized functions/aspects.

We would have appreciated a more detailed presentation on the part with Cortex and the demo (using CLI) was not of good enough quality (1000 Kbps) isn't readable.

Deep Dive: Harbor - Enterprise Cloud Native Artifact Registry - Steven Zou & Daniel Jiang, VMware

By Steven Zou (Staff Engineer, VMware) and Daniel Jiang (Software Engineer, VMware)

Harbor is cloud native registry with a solid community graduated this year by CNCF ! This presentation was mainly on new functionalities delivered by v2.0 and v2.1 so there is a little summary:


  • v2 api to manage other than container (helm, opa bundles, cnab)

  • aggregate view of artifacts by project/repository

  • consistent management workflow

  • tls communication across internal components

  • trivy as default scanner


  • Improvement of GC, now it’s non-blocking

  • Possibility to run GC in dry run mode

  • Proxy cache, proxied artifacts are considered as local

  • Management policies can be applied on proxied artifacts

  • P2P , distribute the deploying content to the P2P network

  • Extended artifact processor to handle extra annotations

And here is the roadmap:

Why We Are Choosing Cloud Native Buildpacks at GitLab - Abubakar Siddiq, GitLab

What is a buildpack ? Buildpacks are responsible for transforming code in something executable. Buildpacks are composed of a set of scripts and depend on language. Buildpacks are able to retrieve necessary dependencies for a type of language.

Heroku supports some of most popular languages like Java, Ruby, Node.js, Php…

An open source utility (Herokuish) emulates Heroku build and runtime tasks in containers. It depends on Heroku supported buildpacks.

Cloud Native Buildpacks are modular tools that transform source code in an OCI image. (https://buildpacks.io). CNB is opt-in for Auto-Build jobs in AutoDevOps. Here is an example of using it

We think it’s a very interesting way to create and share a builder across projects and easily maintain it. While it’s not fully mature yet (for example there is no equivalent to a herokuish buildpack test) we will watch its evolution very closely.

Optimized Resource Allocation in Kubernetes? Topology Manager is Here - Conor Nolan, Intel & Victor Pickard, Red Hat

By Victor Pickard (Principal Sofware Engineer, Red Hat) and Conor Nolan (Software Engineer, Intel)

For a better understanding of this topic it’s nice to remember what is NUMA (http://www.techplayon.com/what-is-numa-non-uniform-memory-access/) . In Kubernetes we have CPU managers which permits to allocate exclusive CPUs to pods and Device managers which allocate hardware resources.

This presentation introduces a new manager: Topology Manager. This manager is a beta feature in Kubernetes 1.18. Topology manager provides an interface to coordinate resource assignment because CPU and Device manager assign resources independently which can cause some suboptimal allocation. So now Topology manager offers the possibility to assign resources from the same NUMA node and optimize allocation.

This concept introduces 4 manager policies:

  • None: by default.

  • Best-effort: attempts to align resources on NUMA nodes

  • Restricted: attempts to align resources on NUMA nodes, pod fails if not possible to align resources

  • Single-numa-node:  attempts to align resources on a single NUMA node, pod fails if not possible to align resources

 Here is an example of allocation:

And some numbers for performance improvement:


This conference was informative. It allows to know how to schedule workload for better performance. It introduce of Topology Manger a new beta feature of Kubernetes 1.18. This new feature enables NUMA alignment of CPUs and peripheral devices, allowing your world to run in an environment optimized for low-latency. Today performance is one of the most important key for everything.

Advanced Logging with Fluent Bit

By Eduardo Silva (Principal Engineer, Arm Treasure Data) and Wesley Pettit (Amazon)

Github : fluent/fluent-bit-kubernetes-logging

  • Lightweight and more efficient than Fluentd.

  • Focused in low CPU/Memory usage (~650KB). (It's written in C language, NOT Ruby)

  • Zero dependencies, unless some special plugin requires them.

  • Perfect for :

  • Embedded Linux.

  • Containers.

  • Servers.

  • Already more than 70 plugins.

  • Easily scrapable by a Prometheus for monitoring.

  • Soon : OpenTelemetry, KafkaConnector in the roadmap v1.6.

5 views0 comments