Changes to know for GX Core 1.0

Yesterday, we officially introduced GX Core!

If you’re using GX v0.18 or prior, you’ll see a lot of changes for the better! Here are some of the highlights.

Setup and configuration

For ongoing work, there’s no more config file editing. All configurations except credentials are now in Python. 

And, of course, you too can benefit from the improved setup process whenever you need to deploy GX Core.

We’ve also removed block-style Data Sources, which were deprecated in 2023.

Expectations

The IDE upgrades mean it’s now way easier to work with Expectations without breaking your flow.

We’ve also overhauled how you create custom Expectations: you now subclass a Core Expectation!

Custom Expectations as subclasses allow you to do all the key things—customize business logic, change Expectation names, and add specialized rendering for your Data Docs—without having to write an Expectation class from scratch. Here’s how.

Finally, we’ve added the

class so you can create Expectations that use SQL or Spark-SQL queries as the core logic, instead of Python. Read more here.

Data validation

Catch up on the overhauled data validation process in our GX Core 1.0 demo video:

A couple of additional noteworthy changes are:

• The Validator and its workflow are no more

• batch_request
  and
  batching_regex
  are gone; use BatchDefinition instead

And the Checkpoint in particular has gotten a lot of updates:

• No more template or reference configuration

• batch_request
  has been removed from multiple places

• The Validations list has now been replaced by Expectation Bindings (the ability to connect a BatchDefinition and a ValidationDefinition)

All this means a lot of configuration changes for the new Checkpoint: see the documentation for complete details.

Important links

Here are some key links with resources for learning more about and migrating to GX Core 1.0!

Read the GX Core 1.0 migration guide.

Go here to see the GX Core documentation.

Get help and ask questions in our Discourse forum.

Share feedback in our Discourse forum or community Slack.