Data Platform Infrastructure

Kind (originally "Kubernetes IN Docker") is software for deploying a Kubernetes cluster using containers. Each node is represented by a container, with a virtual network dedicated to the cluster. Kind was initially developed to test Kubernetes project developments itself but can now be used to test your own applications in a local Kubernetes environment.

There are two ways to install it. Using go, simply run the following command:

Or you can download the binary and place it in a folder included in your $PATH.

The demo cluster consists of one control-plane node and two worker nodes, one of which will be used to receive external connections through the MetalLB load balancer.

Kubectl is the command-line tool used to interact with Kubernetes clusters. It allows you to deploy applications, manage cluster resources, and obtain information about the state of resources.

To install it, you need to download the binary. There are also apt and rpm repositories available.

Deploying applications in a Kubernetes cluster can be tedious. You need to define the various necessary components (pods, services, pv …​), their types (Deployments/DaemonSets, ClusterIP/LoadBalancer, …​), and network access (Ingress, HTTPRoute, …​). This is where Helm comes in, a package manager for Kubernetes that simplifies the deployment and management of applications. Application developers, or the community, can create "Helm charts", which are equivalent to APT/RPM packages for Kubernetes. The difference is that all Helm application configurations must (and can) be defined before installation, via "values".

A Helm repository is made available, and each repository can contain charts for multiple applications. When you install an application in the cluster, it becomes a "release".

Helm stores the application configurations in the Kubernetes cluster itself, using Kubernetes objects such as ConfigMaps and Secrets.

To install it, several methods are available. Here are two of them:

A platform typically consists of multiple applications, and therefore multiple Helm releases. To simplify and centralize the values and configurations, we can use Helmfile. This tool allows you to define all Helm releases in a YAML file and manage them consistently.

You can choose between a single helmfile.yaml per platform or split it with one file per application. By diving into the helmfile configuration details, you could even manage multiple environments. Like Helm, you can use Golang templates, which further multiplies the possibilities.

To install it, once again refer to the documentation page, and here are two possibilities:

You can check if the cluster is up and running with the kubectl get nodes command.

While the cluster is starting, let’s customize the platform a bit. In the .env file, I suggest defining some parameters via environment variables.

The most important is the primary domain name, which will be used to access the various applications. Each application with a web interface or SSL certificate will have its own subdomain, for example, minio.GLOBAL_DOMAIN for Minio, or airbyte.GLOBAL_DOMAIN for Airbyte.

Once modified, let’s load the configuration into memory.

Now that the cluster is up, we’ll retrieve some network information, which will be used to configure the network for MetalLB (see below in the MetalLB section).

We start by installing the resources for the Gateway API, and finally, we install the entire infrastructure with a single command.

This will take several minutes, as it fetches the Docker images and starts the applications. If a timeout occurs, simply re-run the helmfile apply command.

In the meantime, let’s read about what we’re deploying 👇

As the name suggests, MetalLB is a load balancer for Kubernetes. There are 4 types of services to manage networking in Kubernetes. The LoadBalancer type service is the only one that allows network connections from outside the cluster to a "usual" port (80, 443, 5432, …​), through an external load balancer. In a cloud environment, the external LoadBalancer is automatically provisioned, configured, and managed by the cloud provider (AWS ELB, GCP LB, Azure LB, …​). In an on-premise or local environment, there is no native load balancer. MetalLB fills this gap by providing a software implementation of a load balancer, for example, by allocating a virtual IP address on one of the nodes.

Cert-manager is a SSL certificate manager for Kubernetes. It automates the management and renewal of TLS certificates, integrating with certificate authorities like Let’s Encrypt or even private ones. Cert-manager monitors certificate resources in the cluster and ensures that the certificates are valid and up-to-date.

When I evolve the platform to a "production-ready" mode, Cert-manager will be very useful to encrypt all internal communications within the cluster. Right now, Cert-Manager is used to secure your access to the web interfaces.

When Cert-manager manages the PKI by itself, Trust-manager is very useful to distribute the public certificate, so that different applications can trust the certificates issued by Cert-manager.

When web applications are hosted inside a Kubernetes cluster, they need to be accessible from the outside. As already seen with MetalLb, LoadBalancer type services allow this, regardless of the communication protocol used. But this provides direct access to the applications, without managing domain names, SSL certificates, or security policies.

For HTTP(S) protocol, the best method at the moment is to use Ingress with an Ingress Controller (to simplify, it’s Kubernetes' reverse-proxy). In my platform, I use Traefik as the Ingress Controller.

Traefik is also compatible with the Gateway API, which will complement the Ingress for applications requiring more complex configurations.

Minio is an object storage solution compatible with Amazon’s S3 API. It is designed to be highly available, scalable, and performant, and can be deployed on-premise or in the cloud. Minio is often used to store unstructured data such as images, videos, backups, and log files.

Object storage has become predominant in the data world, with solutions like Lakehouses.

Lakehouses rely on the principle of separating storage and computation. Storage is provided by a distributed file system or object storage, accessible by multiple clients and query engines concurrently.

Among the installed tools, there is at least Dagster that will require creating its own Docker images. And to include the public certificate of the root authority of the private PKI, it may also be necessary to create images.

This is the PostgreSQL instance that will provide some source data to Airbyte, Dagster or Olake.