Into hibernation: dropped support for Python 3.7
The day is here: as of GX version 0.16.14, we no longer support Python 3.7.
If you’re still using it:
You’ll still be able to use GX, butwon’t install newer versions, and you may need to pin or set upper bounds to make sure you’re using a version of GX that’s compatible with Python 3.7.
We recommend upgrading to Python 3.8 or later if possible—we want you to be able to use the improvements that we’re working on!
The full announcement with additional detail is on our Slack.
[MAINTENANCE] bump python minimum version to 3.8 (#7916)
Sounding great: new and updated Expectations
Notes in harmony:
[BUGFIX] fix class name (#7734) (thanks @tb102122)
Catching fire: Spark functionality
There are a ton of new updates brightening up GX’s Spark functionality in GX, especially as it relates to Fluent Datasources.
[FEATURE] Spark parquet reader support for fluent datasources (#7754)
[FEATURE] Spark read directory of files as a single batch for CSV (#7777)
[FEATURE] Enable passing "spark_config" through to "SparkDFExecutionEngine" constructor as arguments to "add_spark()" Fluent Datasources methods. (#7810)
[FEATURE] Spark file reader support for fluent datasources (#7844)
[FEATURE] Add Spark DeltaAsset type (#7872)
[FEATURE] Spark directory asset types (#7873)
[DOCS] FDS Deployment Pattern - AWS: Spark and S3 (#7775)
[BUGFIX] Fix sparkDF cannot compute mean for DecimalType (#7867)
May flowers: Fluent Datasource improvements
This month saw a bouquet of improvements and fixes for the Datasource of the future.
[FEATURE] Add tests for #7778)utilizing Fluent Datasources with Pandas, Spark, and SQLAlchemy test cases. (
[FEATURE] Add batch.columns() convenience method to Fluent DataAsset implementation. (#7926)
[FEATURE] NotImplementedErrors for all FDS methods when accessed from BDS (#8002)
[DOCS] FDS Deployment Pattern - Google Cloud: BigQuery and GCS (#7741)
[BUGFIX] Fix remaining FDS config substitution issues (#7917)
Sprucing things up: other new features
[FEATURE] usingfor Windows OS. (#7339) (Thanks @Richard O’Hara )
[FEATURE] DataProfilerStructuredDataAssistant Float Rule (#7842) (thanks @Michael Davis)
[FEATURE] Add DirectoryDeltaAsset (#7877)
[FEATURE] add ssm parameter support for config secrets (#7940) (thanks @Isaacwhyuenac)
New chapters: documentation updates
[DOCS] Remove Redundant Introduction Headings (#7747)
[DOCS] Add CLI Admonition (#7765)
[DOCS] Technical tags in Versioned Docs reference correct version (#7935)
[DOCS] add in-memory add expectation suite (#7973) (thanks @Tobias Bruckert)
Pest control: other bug fixes
[BUGFIX] Repair handling of regular expressions partitioning for cloud file storage environments utilizing prefix directive. (#7798)
[BUGFIX] Fix inability to extend SimpleCheckpoint -- and several additional enhancements and clean up (#7879)
[BUGFIX] Delete ExpectationSuite by name in GX Cloud (#7881)
[BUGFIX] Return qualified name when calling #7942) (thanks @calabozo)(
[BUGFIX] Change GXSqlDialect.AWSATHENA to #7950) (thanks @calabozo)(
[BUGFIX] Cloud - Fix #7989)instead of calls (
Upkeep: additional maintenance
[MAINTENANCE] Enable #7776)rules (
[MAINTENANCE] CLI warnings for #7787)command (
[MAINTENANCE] Update #7985)to return nullable attrs instead of empty strings (
Get the full GX changelog here.