[Event] Building a realtime cloud data warehouse in 4 simple steps
Building a realtime cloud data warehouse in 4 simple steps
Join us for a breakfast session on cloud data warehousing! Learn how to model, load, and discover insights from a real-time cloud data warehouse in 4 simple steps.
Table of Contents
Step 1: Using bike sharing data we will design a data warehouse data model via the Data Vault modeling pattern – the next generation data modelling methodology. This design is agile friendly, supports massively parallel ETL, and by its nature leads to a fast turnaround on requirements. 80% of data warehouses in the Netherlands are implemented using the Data Vault approach.
Step 2: We will deploy the Data Vault data model on Snowflake and build a virtual data mart for reporting and dashboards . Snowflake is a true data warehouse as a service – it was purpose built for the cloud. With its scalability, decoupling of storage & compute, and a pay per use billing model, Snowflake is the #1 cloud data platform.
Step 3: We will stream data from our SQL Server source database into the new Data Vault on Snowflake using Fivetran. Fivetran is a cloud based heterogeneous data pipeline supporting batch processing or streaming, can adjust to source schema changes automatically, and provides a plethora of connectors.
Step 4: We will load the data into the DataChemist Knowledge Graph to find and visualise hidden connections and derive new insights. The DataChemist Knowledge Graph acquires and integrates information and applies a reasoner to infer new knowledge. With DataChemist you can trivially traverse vast datasets and answer the connection oriented questions that matter. DataChemist’s Knowledge Graph perfectly dovetails with the Data Vault data model.
Modeling the Agile data warehouse with Data Vault, Remco Broekmans, Genesee Academy
This is an interactive session on data vault modeling, the next generation modeling approach for data warehouses. It provides an overview of the topics required to understand the Data Vault modeling approach and how this supports an agile data warehouse. In this session we will start with the concept behind data vault – ensemble modeling – and the data warehousing modeling requirements that drive this approach followed by interactive modelling. We will cover the building blocks of the Data Vault: Hubs, Links and Satellites. Together we will model Bikes data into a complete Data Vault model starting with the Core Business Concepts (Hubs), Units of Work and relationships (Links) and context and attributes including history (going to be the satellites).
At the end of this session we will have a complete model which can be deployed to Snowflake. You will understand the basics of how to create your own, simple, Data Vault model and discuss Data Vault models designed by other modelers.
Remco will run a 3 day data vault modeling course on 27-29 March in Dublin. For more information and registration visit the data vault training page.
Genesee Academy, LLC (GA) is an international training, certification and advising organization specializing in data modeling, data warehousing and big data. Genesee Academy is the certifying body for data vault modeling and manages the Certified Data Vault Data Modeler (CDVDM) program.
Loading a data vault on Snowflake cloud data warehouse with Fivetran. Dan Galavan and Uli Bethke, Sonra
In this presentation we will introduce you to Snowflake (a cloud data warehouse as a service) and Fivetran (a cloud based heterogeneous data pipeline). We deploy the data vault model generated in the first session to Snowflake. We will then trickle feed our bikes data from SQL Server to our data warehouse with Fivetran in real-time. Along the way we will show you some unique features of Snowflake such as data sharing, auto-scaling, physical workload separation and more.
Snowflake is the #1 cloud data warehouse platform. It has been purpose built for the cloud with metered access via a pay per use model (just like your electricity bill). Even better: The data warehouse manages itself. You won’t need a DBA to tune a multitude of knobs and switches. Snowflake is a column store data warehouse that doesn’t create hard limits to scaling, and its elasticity, data sharing, and cloning features ensure greater flexibility.
Fivetran is a zero configuration, zero maintenance data pipeline. Fivetran’s fully automated connectors sync data from cloud applications, databases, event logs and more into Snowflake in minutes. Our integrations are built for analysts who need their data centralised, but don’t want to spend time maintaining their own pipelines or ETL systems.
Sonra is a product and consulting company based in Dublin. We are partners of Fivetran and Snowflake.
Visualising connectivity of data in DataChemist, Gavin Medel-Gleason.
In this interactive and visual session, we show you how to use the DataChemist Knowledge Graph to derive new insights from data. We will take the data from the Snowflake cloud data warehouse and load it into the Knowledge Graph.
To incorporate real world complexity, data management should prioritise relationships and connections. A graph database is a next generation database designed to treat the relationships between data as equally important to the data itself. The knowledge part comes when you add ontologies – the meaning of the data is encoded alongside the data in the graph, in the form of an ontology. The ontology is based on logical formalisms which support inference, allowing implicit information to be derived from explicitly asserted data.
DataChemist uses unique mathematical techniques to enable our platform to quickly traverse massive datasets and go beyond 3 or 4 connections to a dozen and beyond – even when dealing with multi-billion node densely connected graphs.
DataChemist’s Knowledge Graph has a well-defined logical schema and mathematically precise recursive query language, allowing the database management system to deal with the details of implementation on modern hardware – including both parallelism and distribution, unburdening the developer and creating modular, composable and maintainable systems.
From loading through modelling to visualisation, we will demonstrate how easy it is to get a new view of relationship oriented data, and how easy it is to pivot to focusing on different characteristics.
8 – 8.30 am Breakfast & Networking
8.30 – 9.30 Interactive and agile data vault modeling session with Remco Broekmans
9.30 – 9.40 Break
9.40 – 10.40 Loading a data vault on Snowflake cloud data warehouse with Fivetran in realtime
10.40 – 10.50 Break
10.50 – 11.50 Using the DataChemist Knowledge Graph to derive new insights from data