Benchmarking Snowflake Native Apps
As a long time user of Snowflake we are impressed with the near unlimited scalability of the Snowflake Data Cloud.
We were wondering if the same scalability and performance magic also applies to Snowflake Native Apps.
In this blog post we give a brief overview on Snowflake Native Apps, benefits and use cases.
We dive deeper into scalability and performance of Native Apps.
What are Snowflake Native Apps?
A Snowflake Native App is a self-contained data application that runs directly within the Snowflake Data Cloud.
What are the benefits of Snowflake Native Apps?
Snowflake Native Apps offer a number of advantages over traditional approaches to building and deploying data-driven applications, including:
Quick App Access via Snowflake marketplace
Explore and buy Snowflake Native Apps on the Snowflake Marketplace. Try a full version before purchase. Once bought, the app is instantly in your Snowflake account, bypassing long vendor wait times. Snowflake Native Apps can be installed and used with just a few clicks.
Data safety assured
Install apps with precise role-based controls. Your data stays put, avoiding extra data storage. While the software can access your data, the provider can’t. Plus, Snowflake vets each public app release for security.
Varied billing methods
Pay for Snowflake Native Apps through integrated billing, including credit cards and ACH. There is also the Marketplace Capacity Drawdown Program where you can use your Snowflake credits to pay for Native App usage.
Performance and scalability:
Snowflake Native Apps are built on top of Snowflake’s massively scalable architecture. Just like any other workloads Native Apps scale linearly with your requirements. We will dive deeper into this later on in this post.
IP
The IP of Native App providers is protected. The source code is not exposed to consumers.
What are Snowflake Native App use cases?
The use cases of Native Apps are only limited by your imagination. Here are some common example use cases of Native Apps
- Apps for Snowflake cost management, cost monitoring, and cost optimisation
- Apps for lookups, e.g. geocoding and reverse geocoding.
- Apps for data profiling and data quality
Brian Hess has put together some less obvious use cases.
Scalability and performance testing of Snowflake Native Apps
As a long time user of Snowflake we are impressed with the near unlimited scalability of the Snowflake Data Cloud.
We were wondering if the same scalability and performance also applies to Snowflake Native Apps.
The short answer is yes!
Test setup
We installed the FlowHigh SQL Analyser Native App from the Snowflake App Marketplace.
FlowHigh SQL Analyser identifies issues in SQL code. It scans the Snowflake query history and detects suboptimal SQL and anti-patterns affecting Snowflake’s performance and query speed. It also finds SQL code that’s hard to read and maintain. Most importantly it identifies SQL code that may yield incorrect or unintended results. Register for FlowHigh to detect common anti patterns in your SQL Code.
For the performance test of Snowflake Native Apps we used the following FlowHigh UDTF to scan the Snowflake query history. We passed in a different number of records for each run.
We used the following query:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
WITH history_subset AS ( SELECT * FROM snowflake.account_usage.query_history LIMIT 1000 -- Specify the umber of records ) SELECT QUERY_ID ,QUERY_TEXT ,AP_TYPE ,POS ,STATUS ,STATUS_MSG ,AP_QUERY_PART FROM history_subset ,TABLE(CODE_SCHEMA.get_ap(QUERY_TEXT)) |
Results
We used three different Virtual Warehouse (VWH) sizes: Small (S), Medium (M), and Large (L).
Performance results are presented in detail for each Virtual Warehouse size. The evaluation spans from 1,000 to 2,000,000 queries that we passed to the FlowHigh SQL Analyser app.
No. of Queries | Time Elapsed in seconds | ||
|---|---|---|---|
VWH Small | VWH Medium | VWH Large | |
1000 | 6.5 | 6.4 | 1.9 |
10000 | 3.5 | 6.7 | 3.1 |
100000 | 12 | 8.2 | 9.3 |
1000000 | 67 | 40 | 26 |
2000000 | 128 | 67 | 42 |
As we can see the execution time is pretty similar for anything up to 100K records. A Small or even X-Small VWH would have been sufficient to run these low volume queries.
Once we hit 1M records we see that Virtual Warehouse Sizing makes a significant difference. We now see near linear scalability across Virtual Warehouses. We can cut processing time in half at the same price point. Scalability of Native Apps is near linear. The same level of scalability we are used to from other Snowflake use cases are also available for Snowflake Native Apps. Q.E.D.
The big advantage of the Snowflake Native Apps framework is that you don’t have to worry about the nitty gritty details of building the infrastructure for scaling your app. It is taken care of for you by the Snowflake Data Cloud.
