Data teams are increasingly taking on responsibility for the quality of their data in addition to data workflows. A new breed of integrated open source tools is helping them get the right data to the people and systems who need it at the right time.
Originally done manually, most data teams now use workflow orchestrators like Prefect, which speed development and reduce overhead through automation, detect and manage failures, and configure alerts.
Together, Prefect and Great Expectations can be used to ensure data quality as the data is piped in.
Perfect Pairing for Continuous Data Validation
Data engineers are accustomed to managing data workflows but pipeline reliability is only part of the equation. It’s also crucial that the data these pipelines carry is accurate and timely.
Most of us have experienced getting information from a dashboard or report only later to be told it is wrong or out of date. When this happens, it can lead to incorrect insights and poor decisions, not to mention frustration and distrust.
Great Expectations empowers data teams and end users to check for anomalies, expired or missing data, and other errors before loading into systems of record. By adding Great Expectations quality checks into a Prefect workflow, data teams can increase overall data reliability and confidence.
For example,
Error handling and notifications can be combined into a single Prefect workflow so management and monitoring are easier.
You can get a single view of your pipeline.
Common logging makes it easier to diagnose and recover from errors as well as prevent them from reoccurring.
In this way, workflow orchestrators can dramatically reduce failures so downstream users have the data they need when they need it.
Prefect users have reported pipeline error reductions of up to 75 percent..
Even Better Together
Since Great Expectations and Prefect are open source Python frameworks, they offer the flexibility and optionality many data teams want.
For those data engineers using Great Expectations without orchestration, Prefect enables continuous validation as well as including Great Expectations tasks as part of broader workflows. This means less manual work and a centralized operation and observation of errors.
Prefect has had an integration with Great Expectations for nearly a year, with hundreds of joint users. Last December we updated our integration for Prefect Core (now Prefect 1.0).
Today we’re excited to announce the next stage of our partnership to create even better experiences for our joint users across both open source and cloud.
There are two areas we’re focused on:
Product integrations: We’ve updated and verified the Great Expectations task to leverage the latest version (v3), while maintaining backwards compatibility. And we just released a task for Prefect 2.0, our next generation platform.
Go-forward partnership: Besides making our current products work well together, we’re already working on what’s next as our platforms evolve. That means Prefect will have a task ready for Great Expectations’ Cloud offering once it’s released, and Great Expectations will be one of the first tasks available in our new Prefect 2.0 platform. You’ll also see more joint thought leadership as we learn from each other and our communities.
Of course, none of this would have been possible without both the Prefect and Great Expectations community contributors. Both the integrations they created and the ideas put forward helped us shape our joint vision, and will continue into the future.
Ben Castleton
Co-Founder and Head of Business Development at Superconductive (makers of Great Expectations)
Want to Learn More?
Learn how to get started with Prefect and Great Expectations.
Join the Prefect and Great Expectations communities.
Read the technical blog from Alex Streed, Integrations Team Lead at Prefect
