Airflow 2.0 is a big thing as it implements many new features. Like the high available scheduler or overall improvements in scheduling performance, some of them are real deal-breakers. But apart from deep, core-related features, Airflow 2.0 comes with new ways of defining Airflow DAGs. Let’s take a look at…

Apache Airflow is already a commonly used tool for scheduling data pipelines. But the upcoming Airflow 2.0 is going to be a bigger thing as it implements many new features.

Today, most tech companies process more data per second than ever. Such amount of data is not only a result of the number of served users, but it’s also related to the tendency to “measure” everything so we can make better decisions in the future (for example, recommendations). To make…

It is quite easy to jump to statements like “it’s up to developers to learn how to open source.” But such remarks also mean that we need no schools, right? If we expect that everyone will learn on their own, then there’s no reason to have schools. Of course, everyone…

In the first part and the second part or the series, we’ve learned about types of scaling and what we get out of the box if we use Kubernetes to deploy our apps. When discussing Kubernetes horizontal autoscaling capabilities, I’ve mentioned that HPA (Horizontal Pod Autoscaler) has two drawbacks: a…

These days, for many, Kubernetes is nearly a synonym of cloud-native. In many cases, this means abandoning the on-premise infrastructure to gain all advantages of infrastructure-as-a-service. One of the most interesting advantages of such an approach is the ease of scalability and accompanying cost optimization. In the previous article, I…

The open-source contributors live in basements, they avoid the sun and are afraid to talk to people. And when they speak, they are opinionated and are far from being conciliatory. Well, if that was true, open source would have died a long time ago. The story I want to share…

Apache Airflow is an open-source tool for creating and managing complex workflows. More recently, Airflow has been gaining a lot of traction and popularity among data scientists for managing machine learning workflows. With increasing usage, we can see user expectations increase too. Like all users, we also expect tools to…

In January, together with a group of four friends, we participated in Campus App Challenge — 24h hackathon organized by Indoorway at our university. Our task was to create an app that uses Indoorway’s indoor positioning system that tackles a real problem our alma mater struggles with. We decided to…