Streaming Tweets to Snowflake Data Warehouse with Spark Structured Streaming and Kafka

Dorian Beganovic Kafka, Snowflake, Spark

Streaming Tweets to Snowflake Data Warehouse with Spark Structured Streaming and Kafka Streaming architecture In this post we will build a system that ingests real time data from Twitter, packages it as JSON objects and sends it through a Kafka Producer to a Kafka Cluster. A Spark Streaming application will then consume those tweets in JSON format and stream them ...

Successfully Transitioning your Team from Data Warehousing to Big Data

Uli Bethke Big Data, Data Warehouse

You are planning to complement your traditional data warehouse architecture with big data technologies. Now what? Should you upskill your existing data warehouse team? Or do Big Data technologies require a completely different set of skills? What do we mean by big data technologies anyway? For the purpose of this article, I define big data as any distributed technology that ...

Visualising XMLs and XSDs of common data standards using Flexter

Anvesh Gali Uncategorized

Extensible Markup Language has become the norm for data transmission and other data related activities in a majority of organizations. XMLs provide developers with a flexible platform to configure and correlate data attributes based upon the product structure and requirement. For different domains, there are various types of XML file formats that are in use. The OTC Derivatives industry uses ...

A Library for XML Data Standards

Uli Bethke XML, XSD

First things first. Before we dive into the various data standards let's first explain what XML is. XML is short for Extensible Markup Language. It is used to describe data. The XML standard is a flexible way to create information formats and electronically share structured data via the Internet, as well as via corporate networks. As in life in general, ...