What's new in GX: April 2023

Great taste: new and updated Expectations

The streak continue! Three community members contributed new or updated Expectations in April (and a little bit of March).

Flavors:

• [CONTRIB] - Add new column expectation not be null and empty (#7449) (thanks @tmilitino)

• [CONTRIB] Added Expect Column Datetime Difference in Months (#7576) (thanks @tb102122)

• [CONTRIB] Expect Column Values to be Valid UUID - Added SqlAlchemyExecutionEngine support (#7592) (thanks @asafla)

• [DOCS] Improve expect_column_values_to_be_of_type docstring (#7632)

Only the freshest: Batch-related updates

GX’s Batch capabilities just keep rising.

Varieties:

• [FEATURE]

  BatchMetadata
  for all fluent
  DataAsset
  s (#7392)

• [FEATURE] Add batch metadata to sql datasources. (#7499)

• [FEATURE] Fluent

  DataAsset batch_metadata
  config variables (#7513)

• [FEATURE] Add batch metadata to spark add_*_asset methods (#7534)

• [MAINTENANCE] Deprecate TableExpectation in favor of BatchExpectation (#7610)

Bubbling up: more SQLAlchemy 2.0 work

We recommend using SQLAlchemy 1.4 until work on full support for 2.0 is complete.

Fizzies:

• [DOCS] Postgresql drivername fix for SQLAlchemy compatibility. Closes #7464 (#7466) (thanks @Itzblend)

• [MAINTENANCE] SqlAlchemy2 Compatibility -

  Row.keys()
  (#7520)

• [MAINTENANCE] SqlAlchemy 2 Compatibility - Autoload Parameter deprecation (#7526)

Get in the flow: Fluent Datasources

The work on Fluent Datasources continues!

Go with it:

• [FEATURE] Cloud support FDS deletes (#7682)

• [FEATURE] Implementing Fluent Datasources Support for Checkpoint (#7697)

• [FEATURE] FDS persist

  DataAsset
  to YAML file immediately on creation (#7705)

• [DOCS] Updating cross-table comparison guide with Fluent Datasources (#7691)

• [DOCS] Add fluent docs and test create and edit expectations with profiler (#7696)

• [DOCS] FDS Deployment Pattern - AWS S3 Pandas (#7718)

• [BUGFIX] Fluent Datasource load from config fixes for remaining Pandas Datasources (#7442)

• [BUGFIX] Data Context Datasource CRUD support for Fluent Datasources (#7660)

• [MAINTENANCE] FluentDatasources - Quickstart Snippets converted to Named Snippets (#7550)

• [MAINTENANCE] list_datasources should return FDS configs as well (#7667)

• [MAINTENANCE] add warning messages when using CLI to edit an expectaiton suite if fluent datasources are present (#7714)

• [MAINTENANCE] FDS Deployment Guide - Pandas S3 fix reference (#7755)

Sprucing things up: other new features

• [FEATURE] Add package, contributors and metrics filter in Algolia script for expectation (#7000) (thanks @kod-er)

• [FEATURE] Introducing CapitalOne DataProfilerColumnDomainBuilder as well as multiple improvements to CapitalOne codebase and testability. (#7498)

• [FEATURE] Added AssumeRole Feature (#7547)

• [FEATURE] Persist Cloud DataAssets on creation (#7748)

New chapters: documentation updates

• [DOCS] New ADR proposal for patch version support (#7451)

• [DOCS] Adds guides on using Ephemeral Data Contexts and updates Quickstart Next Steps (#7500)

• [DOCS] Add GX Cloud Onboarding Script (#7517)

• [DOCS] removes outdated v2/v3 references from the docs (#7519)

• [DOCS] Quickstart code under test (#7542)

• [DOCS] Fix code snippets for earlier versions (#7554)

• [DOCS] Added IAM user and IAM assume role doc (#7634) (thanks @Reactor11)

• [DOCS] Add lakeFS to list of data version control tools (#7642) (thanks @rmoff)

Pest control: other bug fixes

• [BUGFIX] Skip dialect specific tests if no flag passed or flag not available (#7443)

• [BUGFIX] Repair SparkSession initialization behavior to disallow restarting, unless explicitly instructed through configuration (#7444)

• [BUGFIX] Fix issue running quickstart (#7539)

• [BUGFIX] Render prescriptive

  ExpectationConfiguration
  s with evaluation parameters inline (#7552)

• [BUGFIX] fix rendering data asset name in microsoft teams notification (#7675)

• [BUGFIX] Render Correct Fonts in Data Assistant Plot graphs (#7676)

• [BUGFIX] Add missing pyspark reference (#7684) (thanks @willfeltman)

Upkeep: additional maintenance

• [MAINTENANCE]

  mypy
  1.1.1
  update (#7452)

• [MAINTENANCE] Lower pydantic requirement to v1.9.2 or greater (#7482)

• [MAINTENANCE] Bump wheel from 0.37.1 to 0.38.1 in /contrib/cli (#7493)

• [MAINTENANCE] Bump notebook from 6.4.1 to 6.4.12 in /docs_rtd (#7511)

• [MAINTENANCE] adding a footer to data docs with a link to the cloud page (#7532)

• [MAINTENANCE] Harden tests for

  CloudDataContext
  always
  include_rendered_content
  (#7558)

• [MAINTENANCE] Increase minimum scipy version package to 1.6.0 to take advantage of available capabilities. (#7591)

• [MAINTENANCE] Remove s3fs dependency and upper bound for boto3 (#7598)

• [MAINTENANCE] Update NotImported mechanism to use scoped compatibility modules (#7635)

• [MAINTENANCE] Static type checking with python 3.8 (#7637)

• [MAINTENANCE] Upgrade mypy to 1.2.0 (#7645)

Get the full GX changelog here.