We get the community together on the third Tuesday of every month. At each meetup, we discuss the Great Expectations Roadmap, watch ecosystem integration demos, and explore different ways data leaders have implemented Great Expectations. Sign up here to join the next one!
Now, let's dive into the roundup.
Feature Demo
For September’s feature demo, James Campbell, Great Expectations co-founder and CTO, shared our new GX feature, Data Assistants! James answered questions including:
Why did we make Data Assistants?
Data Assistants help you solve the “pale blue dot problem”—when you’re staring down all the ways you can approach data quality, where do you even start? Data Assistants get you moving with curated, relevant Expectations.
What is a Data Assistant?
You can think of a Data Assistant as a partner that can help you understand the shape of your data within minutes, transforming tacit knowledge into something that can be explicitly shared across the business—not a black box that tries to guess your intent.
Data Assistants make observations about your initial data and propose Expectations based on the data’s characteristics. This creates a comprehensive and semantically-grounded picture of your data that you can use to guide your next steps.
How do Data Assistants work?
Data Assistants wrap functionality from rule-based profilers into rapid, one-shot profiling. Their pre-packaged rules have simple parameters but have been carefully selected to guide users about what to explore in their data, what is (or isn’t) ready for use, and other insights.
How do I use Data Assistants today?
A Data Assistant can be run in two different modes, to support different parts of your process.
For your processed/normalized data, Data Assistants can help you launch your data quality project quickly. The Data Assistant will build Expectations based on the data you know and are confident about, which you can immediately start using on data that you expect to be similar.
(But don’t forget to bring in your SMEs—they’re key to identifying which Expectations are most useful and whether the defaults need adjustment!)
For your new/unprocessed data, Data Assistants are great for bootstrapping your initial data exploration. The Data Assistant builds Expectations that the initial data already fails, identifying areas with significant variances or other reasons for further investigation.
(Why is this better than anomaly detection? Because it’s more transparent and you understand how it works, so you have more control.)
Download the slides here: Announcing Data Assistants
And access the documentation here: https://docs.greatexpectations.io/docs/guides/expectations/data_assistants/how_to_create_an_expectation_suite_with_the_onboarding_data_assistant
Questions? Feedback? Jump into our Slack channel or this GitHub Discussion
Roadmap Update
We covered:
Data Assistants launch today!
What’s in a launch? Gathering user feedback, documentation updates, and updates in Slack.
Next up: Improvements to DataContext, the heart of Great Expectations, plus updates to Datasources that will increase user-friendliness and move toward better batch metrics.
@tal in Slack if you’d like to be part of the developer/user experience journey.
And more!
Join the Conversation
Whether you’re a fledgling data practitioner or a seasoned data expert, our community welcomes you! Here are some conversations happening now:
Adarsha sought advice on testing data from multiple files or multiple tables in a database
Zeta Gundam asked for tips on creating a Docker Image with GX
Check out Sonal’s talk about Entity Resolution with Open Source Zingg at Databricks Data and AI Summit.
Check out Goku’s use of dbt and Great Expectations together in his MadeWithML Course in his tweet here.
Additional Updates
Interested in learning more about Great Expectations Cloud? Please reach out to Matthew on Slack or via email!
We’re hiring in engineering, product, and analytics! Check out our open roles.
Data Assistants close the data trust gap: read about why we built them on the blog.
Great Expectations and DataHub integrate seamlessly to give end users a 360-degree view of their data.
Have you done something cool with Great Expectations that you'd like to share? If you want to demo or have some data quality content you'd like us to feature, DM @kyle on Slack.
