Community roundup: September 2023

At this month’s meetup, we:

• Shared thanks and kudos to the community

• Heard the latest product updates and looking ahead

• Saw new Databricks and SQL getting started guides

• Answered questions about ID/PK, hiding the Data Doc walkthrough, and machine learning

Catch up with the recording:

And sign up here to join next month!

Thanks and kudos

As always, we give a huge thank you to our contributors, who give their time and effort to make GX better!

We especially want to highlight these contributors:

• You can now specify the severity of a PagerDuty Validation Action, thanks to Son Chaegeon!

• There’s a brand new Expectation, expect_multicolumn_values_to_be_equal, which Karthigai Selvan Murugesan implemented in three backends!

Thanks to all of this month’s contributors:

We have nearly 11,000 community members in Slack—a number that grows every month! Our Slack supporters, who jump in to help and support others, are an essential part of keeping the GX community strong. Kudos to this month’s top Slack supporters:

Roadmap update

Here’s an overview of the most recent work on the GX platform.

• We had some important bugfixes around Snowflake column name case sensitivity and creating temporary views for Databricks SQL.

• GX Cloud got aggregate overview charts for validation runs and a rich, form-based Expectations editor!

Plus, we’d still love feedback on the experimental Missingness Assistant—please try it out and let us know what you think!

Coming up soon is a ton of GX Cloud work, including column descriptive metrics for Data Asset overviews, enabling Data Asset creation in the Cloud UI, and bringing the Missingness Assistant to Cloud.

Getting started guides

Developer advocate Austin Robinson walked through the new getting started guides for Databricks and SQL:

• Get started with GX and Databricks

• Get started with GX and SQL

These guides are meant to be the next step after the GX overview. They’re focused on getting you set up for a real production workflow, including real Data Sources and Assets and persisting Expectation Suites.

If you’re a Databricks or SQL user, check them out! We’d love feedback of any kind—whether or not you’re actually getting started or a GX veteran—about these guides, in the Slack #gx-feedback or Discourse feedback.

Community questions

We also reviewed some of the more recent questions from the community, plus had time for Q&A.

Why isn’t complete result_format (ID/PK) working with my `distinct_value` Expectations?

If you’re using Expectations that include

, GX does aggregation by design: it looks at the distinct values first, and then compares them to the value set. This aggregation means that using complete result_format to retrieve your primary keys of failing records (ID/PK) can’t retrieve the key of every single row.

If you want an ID from every failing row, use the equivalent non-

Expectation: for example,

expect_column_values_to_be_in_set

instead of

expect_column_distinct_values_to_be_in_set

.

How can I disable “Show walkthrough” when generating Data Docs?

In your great_expectations.yaml, you can set

to

false

:

1data_docs_sites:
2  local_site:
3    show_how_to_buttons: false
4    ...

And as a side note, this question illustrates the value of Discourse. This is a perennial question, and this answer for it from September 2022 remains findable and reachable in Discourse—it would have been long gone in Slack.

Do you have any pointers for using GX with ML?

One interesting place that we’ve seen machine learning integrated with GX is in Custom Expectations! Custom Expectations can integrate ML workflows and tools to create some pretty powerful enablement. 

Check out the CapitalOne DataProfiler and our Ethical AI package for some examples.

Join the conversation

• Share your advice for implementing GX in GCP cloud storage with nainesh khanjire.

• Ricko is seeking experiences from anyone that uses `content` in their Expectation Suites.

• José Correia is looking to automatically generate a JSON with all supported Spark Expectations per GX release: do you have any thoughts?

Additional updates

• Next month, we’re meeting on Tuesday, October 18 at 12pmET—get the invite!

• Watch out for these 4 kinds of data quality issues.

• Explore the 10 most popular Expectations!

• Get to know GX community member Bogdan Volodarskiy.

• We’re hiring in engineering and product! Check out our open roles.

• Interested in GX Cloud? Sign up for the Beta to try it out.

Have you done something cool with Great Expectations that you'd like to share? If you're interested in demoing or have a GX project you'd be interested in presenting, DM @Mollie Pettit on our Slack or @mollie.pettit on our Discourse.