Today, we’re announcing an official partnership between Great Expectations (GX) and Snowflake, enabling anyone to improve data quality directly with open-source technology.
While the GX connector to Snowflake is already being used by many in the analytics community, this partnership is the next step in creating a true production-grade standard for data in the warehouse.
Data both at rest and in motion can be stored in various ways within the Snowflake ecosystem (internal tables, external storage, blog storage, etc.)—and the use cases are only expanding. Snowflake is often part of the last mile before data gets in front of users both internal and external.
Now that data is being increasingly activated, quality is of utmost importance.
Data quality standard in a single source of truth
Any organization that adopts Snowflake uses that data across teams. And the existence of the reverse-ETL category, spearheaded by tools like Hightouch and Census, demonstrates that the need to put data from the warehouse into cross-team tools isn’t specific to Snowflake users.
With so many of those new tools being warehouse-native, the data in cloud applications like Snowflake is now directly backing the most important experiences: user-facing ones built by marketing, sales, and customer success teams alike.
As an example: in warehouse-native style, Snowflake recently announced an official connector in partnership with Salesforce. Any team using Salesforce can have data backed by Snowflake without involving any third party; they only need to loop in their internal data team.
The core purpose of sales teams is to drive revenue—their compensation usually depends on it. If issues arise with Salesforce, the core tool most sales teams spend their time in, then what? Well, that’s revenue left on the table. Any data issues affecting the sales team impact their ability to do their best work.
In the B2B world, unconverted prospects are often identified by either website visitors or trial signups. With either definition, properties about these prospects are helpful for sales teams when prioritizing who to reach out to first.
For instance, people don’t convert from a trial for a variety of reasons: cost, confusion, or just lack of urgency. Looking at analytics on viewed web pages could indicate which bucket a prospect falls into, informing the best sales strategy to use with them.
Anyone on the data team is likely to express excitement when the data they produce—like informative data on sales prospects—is being used more. If the data coming from Snowflake is randomly zero due to a bug, when it goes into a lead scoring model in Salesforce potentially-hot leads won’t be flagged. Data use is only good when the data is high quality.
Low-quality data that’s heavily used will only result in one thing: loss of trust in the analytics team. Testing data can head off that path before it becomes inevitable and keep cross-team trust strong.
That’s where unit testing with Great Expectations comes in.
Granular unit testing on data at rest
Great Expectations is a Python-based framework for running unit tests on data. Analytics teams integrate GX tests in a pipeline directly, stopping bad data from propagating downstream.
The community-driven development of test types has allowed the framework to support everything from null percentages to valid-value checks to statistical tests. The out-of-the-box test types allow users to declare easy-to-read tests that are translated into SQL statements run directly on Snowflake. (Learn more about Expectations here.)
In the Salesforce use case, GX, when integrated with a data orchestrator running transformations on Snowflake directly, could prevent missing data getting into the hands of business teams. Moreover, failing tests can alert the analytics team immediately where the issue stems from, so they don’t need to back-track issues from other tools far downstream.
With Snowflake’s solidified place in many analytics stacks, there’s only one way to increase its usage: by building data trust with unit tests.
Get started with Great Expectations on Snowflake by visiting our documentation. Already a GX user? Consider exposing your clean data through Snowflake and all the integrations they offer here.
