Recent Articles

All JSON Snowflake XML
Data Pipeline,

Data Orchestration Deep Dive Snowflake Tasks. An Airflow replacement? 

What are Snowflake Tasks Snowflake introduced Tasks for scheduling and orchestrating data pipelines and workflows in late 2019. The first release of Tasks only offered the option to schedule SQL…
Uli Bethke May 3, 2024
Data Pipeline,

To hell and back with ETL. The unstoppable rise of data warehouse automation.

A brief history of ETL The rise of ETL and the rise of the data warehouse are tightly coupled. It’s like the chicken and the egg. There would be no…
Uli Bethke May 22, 2020
Data Pipeline,

The future of ETL and the limitations of data virtualisation and NoETL.

Data has limited value if we don’t transform, integrate, model (either data modelling and building predictive models) or cleanse it. Collecting raw data for no apparent reason or business case…
Uli Bethke October 11, 2018
Data Pipeline, JSON,

JSON. To ETL or to NoETL? The big data question.

NoETL. The little brother of NoSQL You have probably come across the term NoSQL. It was coined a few years back to describe a class of database systems that can…
Uli Bethke February 19, 2018
Data Pipeline,

Using Apache Airflow to build reusable ETL on AWS Redshift

Building a data pipeline on Apache Airflow to populate AWS RedshiftIn this post we will introduce you to the most popular workflow management tool - Apache Airflow. Using Python as…
Uli Bethke January 1, 2018