Pass custom parameters in Custom Expectations and optimize your GX workflows

When creating Custom Expectations in Great Expectations, you may find yourself creating a Custom Expectation that depends on a piece of data that can’t or shouldn’t be hard-coded.

That’s when you need to pass custom parameters into your Metric. This can unlock greater functionality for your Custom Expectations, but its implementation will be slightly different depending on the class of Custom Expectation you’re building.

(This is an advanced configuration and customization of your Custom Expectations; see our introduction to creating Custom Expectations to get started!)

All Custom Expectations (and all Expectations in general) have a success_keys attribute. This attribute is a tuple of strings.

The items in the tuple are the names of the parameters which are available for you to use during the evaluation of your Custom Expectation: the default parameters, plus any you have defined.

For example, standard success_keys for a ColumnPairMapExpectation, referencing the two columns being evaluated and the mostly parameter, look like:

1success_keys = (
2   "column_A",
3   "column_B",
4   "mostly",
5)

These parameters will all be passed through as a part of your ExpectationConfiguration and made available to the _validate methods of Expectations. 

Any parameters we wish to also make available to our Metric (for example, the two columns to be evaluated) will need to be listed as a part of an attribute under that Metric.

Consider an example case using ColumnPairMapMetrics, MulticolumnMapMetrics, and ColumnMapMetrics. You could use these parameters to:

• Define a condition_value_keys tuple, to supply the arguments needed to compute your Metric.

• Define a condition_domain_keys tuple, to help define the domain on which the Metric is operating.

⚠️ Warning

If you’re passing only a single parameter into any kind of value_keys or domain_keys in your Metric, it must still be a tuple and have a trailing comma:

1✅ value_keys = ("degrees",) 
2❌ value_keys = "degrees"
3❌ value_keys = ("degrees")

For example, the MulticolumnSumEqual Metric contains:

1condition_domain_keys = (
2   "batch_id",       
3   "table",       
4   "column_list",       
5   "row_condition",       
6   "condition_parser",       
7   "ignore_row_if",   
8)   
9condition_value_keys = ("sum_total",)
10
11ColumnAggregateMetrics, TableMetrics, and QueryMetrics can similarly define a value_keys tuple or a domain_keys tuple:
12
13value_keys = ("column",)
14domain_keys = ("query",)

ℹ️ Info

Any type of Expectation can have some version of value_keys & domain_keys.

Some classes of Expectation judge whether a row-wise condition is met, and these require condition_value_keys and condition_domain_keys. 

Classes that do not use row-wise conditions use value_keys and domain_keys.

Once these attributes have been provided in both the Expectation class & Metric class, your custom parameters can be passed into and utilized within the individual Metric functions, as seen in the ColumnValuesBetween Metric:

1classColumnValuesBetween(ColumnMapMetricProvider):    

1condition_metric_name = "column_values.between"   
2condition_value_keys = (       
3   "min_value",       
4   "max_value",       
5   "strict_min",       
6   "strict_max",       
7   "parse_strings_as_datetimes",
8   "allow_cross_type_comparisons",
9)
10@column_condition_partial(engine=PandasExecutionEngine)   
11def_pandas(       
12   cls,       
13   column,       
14   min_value=None,       
15   max_value=None,       
16   strict_min=None,       
17   strict_max=None,       
18   parse_strings_as_datetimes: bool = False,
19   allow_cross_type_comparisons=None,       
20   **kwargs   
21):

Utilizing these patterns to further customize your Custom Expectations can help you to create powerful, fit-for-purpose validations and optimize your Great Expectations workflows.

For an example of passing custom parameters in the context of a full Custom Expectation, check out expect_column_values_to_be_lat_lon_coordinates_in_range_of_given_point.

A final note: The patterns described in this post are our recommended best-practice when using custom parameters, but it's also possible to access them through kwargs. You can see an example of kwarg-based access in the Custom Expectation linked above.

---

Great Expectations is part of an increasingly flexible and powerful modern data ecosystem. This is just one example of the ways in which Great Expectations is able to give you greater control of your data quality processes within that ecosystem.

We’re committed to supporting and growing the community around Great Expectations. It’s not enough to build a great platform; we want to build a great community as well. Join our public Slack channel here, find us on GitHub, sign up for one of our weekly cloud workshops, or head to https://greatexpectations.io/ to learn more.