March 2024: community roundup

At this month’s meetup, we:

• Learned what’s coming next for GX in a product roadmap update

• Heard details about the latest usability updates in GX 1.0

• Saw a demo of GX Cloud with a Postgres data source

• And more

If you missed it, you can catch up on YouTube:

 

Follow our calendar to join future meetups!

Community celebration

GX’s vibrant community is one of the things that sets it apart. Thank you to all the community members who responded to questions or otherwise gave a helping hand to other users last month:

We especially want to recognize GX Champion Toivo Mattila for going above and beyond in the past couple of months! 

Community talk

If you haven’t seen it yet, be sure to take a look at this talk that César García, another of our GX Champions, delivered at PyData Global! It’s about using GX for data quality on open data:

In this session we will demonstrate how to measure and improve the quality of open data using the open source Python library Great Expectations. Attendees will learn quality testing techniques and methodologies to prepare high-quality longitudinal datasets using Open Data from cities and regional portals.

 

Thanks for sharing this, César!

Roadmap

Anna Barr, one of GX’s senior product managers, shared what’s been new in GX lately and what’s coming up next.

Two items of special note:

• We’re working on improving the getting-started experience for GX Cloud! If you’ve tried it out and have any feedback about your initial experiences, we’d love to hear them. You can reach out to @Anna Barr on Slack or Discourse, or email our product team at product_feedback@greatexpectations.io.

• We introduced Fluent Data Sources just under a year ago and deprecated the old block style of Data Source configuration. In the very near future, we’ll be taking the final step of removing the code that supports block style from GX OSS. If you have Data Sources that aren’t yet using fluent configuration, be sure to migrate in the next couple of weeks!

GX 1.0 ease of use improvements

James Campbell, GX's CTO and co-founder, reviewed some of the major changes in GX 1.0. 

⚠️ Important note: These improvements are not live and GX 1.0 is still in active development! These names and implementations could still change.

In 1.0, Expectations use class names instead of method names. For example,

will become

ExpectColumnValuesToBeInSet

.

The Validator has also been removed: GX 1.0 will have specific, separate Batch Definitions and Validation Definitions. Batch Definitions are attached to Data Assets, while Validation Definitions associate Batch Definitions with Expectation Suites. 

Validation Definitions can be created independently of a Checkpoint, which is also a major departure from pre-1.0 GX.

Speaking of Checkpoints, the Checkpoint API has been radically simplified:

• It’s declared using only two parameters: the Validation Definition(s) you want to run, and any Actions that you want to take based on the results.

• When you run a Checkpoint, you define any dynamic Batch or Expectation Suite parameters that are needed.

As part of the streamlining process, some things have been removed from the GX 1.0 API entirely:

• The

  add_or_update
  methods have been removed in favor of explicit add, save, and delete options.

• Convenience methods on Data Contexts (such as

  context.suites.get
  ) have also been removed because they were confusing.

• The Dataset and ValidationOperator APIs are being removed: they’ve been effectively deprecated for years, and their functionality is replicated elsewhere.

• The Profiler and Data Assistant APIs are being removed so we can reintroduce profiling from scratch. The new profiling functionality will reflect the lessons we learned from the Profiler and Data Assistant implementations and in creating the new API.

GX 1.0 will still have some experimental features (like

), but GX 1.0 will be clearer about their experimental status.

GX Cloud + PostgreSQL

Austin Robinson, one of GX's developer advocates, showed off a demo of connecting to Postgres in the GX Cloud UI:

You can try it yourself by signing up for GX Cloud, or sign up for our upcoming GX Cloud + Postgres workshop.

Public preview recap

Mollie Pettit, GX’s senior community product manager, recapped our February 21 public preview event. You can watch it anytime on YouTube:

 

Click here to learn more about public preview.

Additional updates

• Our next meetup is Tuesday, April 16 at 12pm ET (9am PT)! If you haven’t already subscribed to our calendar, click here.

• Karin Wolok shared a discount code for Data Council Austin, happening March 26-28, 2024. Use GREAT20 to get 20% off a regular ticket. Learn more at https://www.datacouncil.ai/. 

• We’re hiring! You can find our postings for a senior developer advocate and senior product manager at https://jobs.greatexpectations.io/.