Streaming Tweets to Snowflake Data Warehouse with Spark Structured Streaming and Kafka

Dorian Beganovic Kafka, Snowflake, Spark

Streaming Tweets to Snowflake Data Warehouse with Spark Structured Streaming and Kafka Streaming architecture In this post we will build a system that ingests real time data from Twitter, packages it as JSON objects and sends it through a Kafka Producer to a Kafka Cluster. A Spark Streaming application will then consume those tweets in JSON format and stream them ...

Comparing Window Function Features by Database Vendors

Jiří Mauritz Data Warehouse, Redshift, SQL for Analysis, Window Functions

We will round off the series on window functions with comparison of what database vendors offer. There are various mutations of window functions and every vendor supports a different subset or feature. Some also add extra window functions or features beyond standard ANSI SQL. One of the most powerful features is user-defined aggregate functions (UDAF), which some databases allow using ...

Convert ESMA XML to Snowflake

Anvesh Gali XML

Introduction to this walkthrough In this walkthrough, we will demonstrate the process of loading XML data into Snowflake – a cloud based data warehousing service. We have downloaded the XML files from ESMA and then converted them into TSV files using Flexter, a powerful XML parser from Sonra. We then load the TSV files into Snowflake and execute queries. XML ...