At this month’s meetup, we:
Learned about exciting work by this month’s contributors
Saw a demo of Great Expectations Cloud’s new one-click-Expectation-Suite-and-Checkpoint feature
Heard from Fennel.ai founding engineer Aditya Nambiar about data quality in feature engineering
We’re hiring in developer relations! And we especially encourage community members to apply, so don’t hesitate if you’re interested.
Watch the recording of the meetup here:
Note that Cloud feature demos are currently not included in the recordings: be sure to sign up for future meetups to see them live!
Thanks and kudos
Our Slack supporters are an indispensable part of the GX community. Thank you to all the new and returning top supporters for June:
This month we especially want to highlight two new contributions:
GX now has Clickhouse integration, courtesy of Plozano94! Lots of people have asked about this one, and it took a ton of work from Plozano94 to make it happen. This integration includes some specific Expectation implementations to work with Clickhouse and some great tests. Definitely check it out.
GX now supports AWS System Manager Parameter Stores, including AWS Secrets Manager, which was implemented by Isaac Yuen. This means users who have multiple environments deployed can use AWS Parameter Stores to declare their values.
Thank you to everyone who contributed this month! You all make GX better every day:
Product update: the GX Agent
GX product manager Erik Hencier presented the newest in GX Cloud: the GX Agent, which is a one-click way to set up an Expectation Suite and Checkpoint for any Data Asset.
Erik demonstrated how simple it is to activate the agent in your environment, then click the link that automatically appears for your Data Assets.
The agent takes care of running the Onboarding Data Assistant, and the Expectations and Checkpoints that it identifies automatically appear in the Cloud UI, ready to edit and run!
If you’d like to try out Cloud through our Beta program, DM @Matthew Lundgren on our Slack. We’d especially love to hear more about what kind of workflows you would find useful for the agent.
Beyond Expectations: unpacking data quality in feature engineering
We heard a presentation on Fennel.ai’s use of GX for data quality in feature engineering from Aditya Nambiar. Aditya is an engineer and founding member at Fennel, and an alumnus of Instagram, Facebook, and Google.
In his information-packed talk, we learned:
What is feature engineering?
Where does data quality come into feature engineering?
What are the main causes of data corruption in feature engineering?
What steps can defend against data corruption?
What does GX do to help you implement these defenses?
How can you use GX with streaming data?
He also shared some code examples of how Fennel implements GX in its processes, then answered follow-up questions.
Thank you for sharing your insight, Aditya!
Watch his talk here:
You can also view his slides at go.fennel.ai/data-quality and learn more about Fennel in their documentation.
Join the conversation
Frederick got some tips about integrating pandas-profiling with GX 0.16.15.
Cesar Garcia Saez asked about experiences creating data quality scorecards with GX.
Oliver Angelil sought advice on where to use GX versus the native expectations when working with Delta Live Tables.
Thomas Chung invited Bay Area data professionals to a social & networking event on June 27, hosted by Mage and Data Engineer Things. The meetup’s featured talks will also be livestreamed: one is by Julian Jaffe and Ramayan Tiwari of Netflix and the other is from Joe Reis, co-author of Fundamentals of Data Engineering.
Additional updates
Next month, we’re meeting on Tuesday, July 18! Get the invite here.
Check out this code snippet for validating a Google Sheet using PandasDatasource and CSVAsset.
If you’re even thinking about contributing to GX, this guide to our community contribution process will point you where you need to go.
We took a look into the chaotic world of real estate data with data scientist Lucas Roy.
Get to know GX contributor Aleksei Chumagin!
