GX 1.0 is finally here! And in addition to all the upgrades I’ll cover in a minute, GX OSS is also getting a new name: GX Core.
GX Core better describes the central role our Python framework plays in the GX data quality platform as a whole.
We’re starting GX Core off at version 1.0, and will be following Semantic Versioning 2.0.0 from now on.
What’s new
The changes we’ve made in GX Core are all focused around radically simplifying the API while maintaining our feature set.
Our strategy for this was straightforward: we picked the approach for each task that will best serve the vast majority of people, and pruned the rest.
When we started GX back in 2018, we didn’t have the experience to make these calls—which is why GX pre-1.0 evolved the way it did, with multiple ways to accomplish any given objective.
Six years on, it’s a different story. We have much more expertise in building data quality processes, and the confidence that we can make meaningful decisions about the best way to do things. And as a full-fledged company with nearly 60 employees instead of a founding team of two, we now have the people power that’s needed to make an effort of this magnitude happen.
We are delighted to finally be able to share GX Core with you.
Documentation
GX Core 1.0’s documentation is your guide for everything we’ll talk about here.
With the simplified API and workflow, GX Core’s documentation is intuitive to navigate and easy to follow. Check it out here!
Setup and configuration
You start out your GX Core experience with a quick and straightforward installation process.
You can start connecting to data immediately, providing credentials for your data source through a secrets manager or via environment variables.
Expectations
If you’re new to GX: Expectations are how GX implements data quality tests. They’re simple declarative statements that make a verifiable assertion about an aspect of the data. You use GX to test whether your data meets all of the assertions you’ve made about it. Expectations have descriptive, plain-language names.
In GX Core, you create Expectations with a simple workflow.
There are 47 Expectations, suitable for addressing a variety of data quality issues, fully configured and ready to use in GX Core! You can explore these Expectations in our Expectations Gallery—and, if none of them quite meet your needs, create your own custom Expectations.
Expectations are now fully-typed objects with IDE integration that includes name auto-fill and parameter hints as you work.
Data validation
It’s super simple to validate data in GX Core:
Specify the data you want to validate as a Batch, using a BatchDefinition.
Specify which Expectations you want to run on your Batch using a ValidationDefinition.
Execute the ValidationDefinition.
You’re done!
Optionally, you can also pair a ValidationDefinition with one or more Actions.
An Action is an automated task that will be executed (or not) based on the results of your validation. The simplest example: you can create an Action to send a Slack alert if any of your validations are failed.
A ValidationDefinition and its Actions are stored as a Checkpoint. If you have multiple ValidationDefinitions that are operating on the same Batch, you can group them into a single Checkpoint—making your data testing even faster to start up!
Support
Here’s a rundown of what you can expect in terms of support for different kinds of data sources: GX’s integration support policy.
And if you have questions about GX Core, here are our support options.
Thank you
Thank you for joining me on this introduction to GX Core 1.0.
We—me as an individual as well as GX as a company—want to most sincerely thank everyone who’s contributed to GX over the years. Even though creating GX Core meant removing a lot of code, we wouldn’t have been in a position to do this work without all that code existing in the first place. Every PR, comment, and piece of feedback from the GX community has been a crucial stepping stone in getting here.
I’m immensely proud of what we’ve accomplished with GX Core 1.0 and I can’t wait to hear what you think.
We want to hear your feedback! Let us know what you think about GX Core 1.0 in the #gx-feedback channel of our community Slack or the Feedback board of our Discourse forum.
Interested in contributing? We’re now accepting PRs for bug fixes and parts of the GX Core 1.0 codebase! See our GitHub README for more information.
