backgroundImage

What's new in GX: April 2023

New and updated Expectations, a bunch of Batch-related features, continued work on SQLAlchemy 2.0 and Fluent Datasources, and more

Erin Kapp
May 05, 2023
A red-throated loon in winter plumage is being set into the water by a pair of hands
Get the scoop on everything GX that was released into the wild last month. Also released into the wild: this red-throated loon. (📸: Jonathan L Fiely, USGS, 2000)

Great taste: new and updated Expectations

The streak continue! Three community members contributed new or updated Expectations in April (and a little bit of March).

Flavors:

  • [CONTRIB] - Add new column expectation not be null and empty (#7449) (thanks @tmilitino)

  • [CONTRIB] Added Expect Column Datetime Difference in Months (#7576) (thanks @tb102122)

  • [CONTRIB] Expect Column Values to be Valid UUID - Added SqlAlchemyExecutionEngine support (#7592) (thanks @asafla)

  • [DOCS] Improve expect_column_values_to_be_of_type docstring (#7632)

Only the freshest: Batch-related updates

GX’s Batch capabilities just keep rising.

Varieties:

  • [FEATURE]

    BatchMetadata
    for all fluent
    DataAsset
    s (#7392)

  • [FEATURE] Add batch metadata to sql datasources. (#7499)

  • [FEATURE] Fluent

    DataAsset batch_metadata
    config variables (#7513)

  • [FEATURE] Add batch metadata to spark add_*_asset methods (#7534)

  • [MAINTENANCE] Deprecate TableExpectation in favor of BatchExpectation (#7610)

Bubbling up: more SQLAlchemy 2.0 work

We recommend using SQLAlchemy 1.4 until work on full support for 2.0 is complete.

Fizzies:

  • [DOCS] Postgresql drivername fix for SQLAlchemy compatibility. Closes #7464 (#7466) (thanks @Itzblend)

  • [MAINTENANCE] SqlAlchemy2 Compatibility -

    Row.keys()
    (#7520)

  • [MAINTENANCE] SqlAlchemy 2 Compatibility - Autoload Parameter deprecation (#7526)

Get in the flow: Fluent Datasources

The work on Fluent Datasources continues!

Go with it:

  • [FEATURE] Cloud support FDS deletes (#7682)

  • [FEATURE] Implementing Fluent Datasources Support for Checkpoint (#7697)

  • [FEATURE] FDS persist

    DataAsset
    to YAML file immediately on creation (#7705)

  • [DOCS] Updating cross-table comparison guide with Fluent Datasources (#7691)

  • [DOCS] Add fluent docs and test create and edit expectations with profiler (#7696)

  • [DOCS] FDS Deployment Pattern - AWS S3 Pandas (#7718)

  • [BUGFIX] Fluent Datasource load from config fixes for remaining Pandas Datasources (#7442)

  • [BUGFIX] Data Context Datasource CRUD support for Fluent Datasources (#7660)

  • [MAINTENANCE] FluentDatasources - Quickstart Snippets converted to Named Snippets (#7550)

  • [MAINTENANCE] list_datasources should return FDS configs as well (#7667)

  • [MAINTENANCE] add warning messages when using CLI to edit an expectaiton suite if fluent datasources are present (#7714)

  • [MAINTENANCE] FDS Deployment Guide - Pandas S3 fix reference (#7755)

Sprucing things up: other new features

  • [FEATURE] Add package, contributors and metrics filter in Algolia script for expectation (#7000) (thanks @kod-er)

  • [FEATURE] Introducing CapitalOne DataProfilerColumnDomainBuilder as well as multiple improvements to CapitalOne codebase and testability. (#7498)

  • [FEATURE] Added AssumeRole Feature (#7547)

  • [FEATURE] Persist Cloud DataAssets on creation (#7748)

New chapters: documentation updates

  • [DOCS] New ADR proposal for patch version support (#7451)

  • [DOCS] Adds guides on using Ephemeral Data Contexts and updates Quickstart Next Steps (#7500)

  • [DOCS] Add GX Cloud Onboarding Script (#7517)

  • [DOCS] removes outdated v2/v3 references from the docs (#7519)

  • [DOCS] Quickstart code under test (#7542)

  • [DOCS] Fix code snippets for earlier versions (#7554)

  • [DOCS] Added IAM user and IAM assume role doc (#7634) (thanks @Reactor11)

  • [DOCS] Add lakeFS to list of data version control tools (#7642) (thanks @rmoff)

Pest control: other bug fixes

  • [BUGFIX] Skip dialect specific tests if no flag passed or flag not available (#7443)

  • [BUGFIX] Repair SparkSession initialization behavior to disallow restarting, unless explicitly instructed through configuration (#7444)

  • [BUGFIX] Fix issue running quickstart (#7539)

  • [BUGFIX] Render prescriptive

    ExpectationConfiguration
    s with evaluation parameters inline (#7552)

  • [BUGFIX] fix rendering data asset name in microsoft teams notification (#7675)

  • [BUGFIX] Render Correct Fonts in Data Assistant Plot graphs (#7676)

  • [BUGFIX] Add missing pyspark reference (#7684) (thanks @willfeltman)

Upkeep: additional maintenance

  • [MAINTENANCE]

    mypy
    1.1.1
    update (#7452)

  • [MAINTENANCE] Lower pydantic requirement to v1.9.2 or greater (#7482)

  • [MAINTENANCE] Bump wheel from 0.37.1 to 0.38.1 in /contrib/cli (#7493)

  • [MAINTENANCE] Bump notebook from 6.4.1 to 6.4.12 in /docs_rtd (#7511)

  • [MAINTENANCE] adding a footer to data docs with a link to the cloud page (#7532)

  • [MAINTENANCE] Harden tests for

    CloudDataContext
    always
    include_rendered_content
    (#7558)

  • [MAINTENANCE] Increase minimum scipy version package to 1.6.0 to take advantage of available capabilities. (#7591)

  • [MAINTENANCE] Remove s3fs dependency and upper bound for boto3 (#7598)

  • [MAINTENANCE] Update NotImported mechanism to use scoped compatibility modules (#7635)

  • [MAINTENANCE] Static type checking with python 3.8 (#7637)

  • [MAINTENANCE] Upgrade mypy to 1.2.0 (#7645)


Get the full GX changelog here.

Like our blogs?

Sign up for emails and get more blogs and news

Great Expectations email sign-up

Hello friend of Great Expectations!

Our email content features product updates from the open source platform and our upcoming Cloud product, new blogs and community celebrations.

Error message placeholder

Error message placeholder

Error message placeholder

Error message placeholder

Error message placeholder

Error message placeholder

Error message placeholder

Error message placeholder

Error message placeholder

Banner Image

Search our blog for the latest on data management


©2024 Great Expectations. All Rights Reserved.