# What if I built my own data platform?

> Why and how to build your own data platform

- URL : https://blog.xreveillon.eu/en-US/blog/data-platform
- Date : 2025-07-23T16:00:00.000Z
- Langue : en-US

---

## Why?

For several reasons, professionally personal, as well as technical.

- To apply all the Data skills I acquired at Oktopus S.A. and then at [Edda Luxembourg S.A.](https://edda.lu) after our merger.
- To demonstrate my Infra/DevOps skills that I've never applied professionally in a project.
- To serve as a demonstration and training material for all the tools presented.

As mentioned, the platform in this series of articles serves as a demonstration, with minimal configuration, to show that you can develop on your machine before deploying to a dev/test/quality/prod environment.

Another post, following this series, will showcase a secure data platform with centralized access rights management.

## How?

Speaking of tools, I will present and use the following technologies:

**Kubernetes:**

- [Kind](https://kind.sigs.k8s.io/), to set up a k8s dev/demo cluster
- [Helm](https://helm.sh/), to deploy pre-packaged applications into this cluster
- [Helmfile](https://helmfile.readthedocs.io/en/latest/), to store the configuration of Helm releases

**Infra:**

- [PostgreSQL](https://www.postgresql.org/), a classic and performant database
- [Minio](https://www.min.io/), an on-premise object storage that replaces Amazon S3
- [Cert-manager](https://cert-manager.io/) to automatically issue SSL certificates, [Trust-manager](https://cert-manager.io/docs/trust/trust-manager/) to distribute the trust chain automatically
- [Traefik](https://traefik.io/), a reverse-proxy with dynamic configuration
- A container image registry, useful for some of the tools

**Data:**

- [Airbyte](https://airbyte.com/), for ingesting data from [disparate sources](https://airbyte.com/connectors), for testing this tool
- [Nessie](https://projectnessie.org/) as a metastore of Iceberg data
- [Trino](https://trino.io/) as a query engine
- [Dagster](https://dagster.io/), for orchestrating dataset pipelines
- [OpenMetadata](https://open-metadata.org/), to have a data catalog
- [Superset](https://superset.apache.org/), to create dashboards


It is evident that the choice has been made in favor of

- the use of containers and Kubernetes clusters, to facilitate deployment and scalability of applications
- open-source or semi open-source tools, to ensure flexibility and scalability
- the use of lakehouse, to separate storage and computing resources, improving performances and scalability of analytic workload

## Use cases

For demonstrating this platform, I'll use two use cases. First one will be to take movies data from [IMDb](https://www.imdb.com/), with [csv files freely available](https://developer.imdb.com/non-commercial-datasets/).
Second will be using structured data storing [status of EV stations](https://data.public.lu/en/datasets/bornes-de-chargement-publiques-pour-voitures-electriques/) of [Chargy](https://chargy.lu/) brand in Luxembourg. I'm storing snapshots each 5 minutes since a year and a half. The Postgres data I'm providing is only for one month, as the whole dataset is too big to store in a docker image open to the public.


<figure>
  <a href="/images/blog/data-platform/data-platform-en.drawio.svg"><img src="/images/blog/data-platform/data-platform-en.drawio.svg" alt="Overview of my data platform" /></a>
  <figcaption>My data platform</figcaption>
</figure>


Let's get started!

**Links to the series' posts:**

- 1st part, xref:../01-infra/index.adoc[the infra]