Tl;dr:
- We are hosting a few Great Expectations hackathons over the coming weeks and would like you to join: Sign up here
- We are hosting a webinar next Thursday, January 21th @ 2:00pm US Eastern Time, about Modular Expectations that will make you a hackathon hero Sign up here
Greater Scope for Great Expectations
The Great Expectations library currently contains just over 60 Expectations, focused on the most common problems that data scientists and engineers face, across all industries and tech stacks:
- Row counts
- Missing values
- Sets
- Ranges and distributions
- Datetime formats
- etc.
The single most common request that we get from users is more expressivity: Expectations that cover a broader range of cases and get more into the details of specific types of data.
Specific identifiers: zip codes, names, social security numbers, etc. Demographic variables and standard survey questions: names, suffixes (Sr., Jr., etc.), age, race, etc. Expectations expressed in terms of time: % increases and decreases, trends, Expectations to check accounting identities and proper functioning of state machines Statistical tests for machine learning and analytics reporting Conditions and faceting Time series and seasonality etc., etc, etc,
At the end of November, we released v0.13 of Great Expectations, which introduced the concept of Modular Expectations. Modular Expectations dramatically simplify the process of adding and extending the library of Expectations, including full integration with other components of Great Expectations: rendering to documentation, code for executing Expectations in multiple engines (e.g. pandas, SQL, spark), etc.
In internal time trials, developers who had never seen a Modular Expectation before were able to develop 3 or more Expectations in their first afternoon. Once you know how, it’s possible to knock out the core of some types of Expectations in less than 10 minutes.
We are hosting a webinar (optional) next Thursday, January 21th @ 2:00pm US Eastern Time, about Modular Expectations! Join if you want to be a hackathon hero: Sign up here.
You don’t even need to build all the moving parts of an Expectation in order to make a meaningful contribution: we’ve bite-sized the process of creating Expectations, so that the different execution engines, renderers, documentation, tests cases, etc. can be developed incrementally, often by different people.
Why Community Hackathons?
There’s a lot to build here—tremendous scope for creativity expressed in code. Rather than keep all the fun for ourselves, we’ve made a conscious decision to empower the data community to build it together.
The community is clearly eager for this capability: even before we’d finished fully documenting how to add Modular Expectations, we started to receive some really cool suggestions (e.g. Expectations about geo data) and contributions (e.g. Expectations to validate language in free text.)
Community hackathons are a great way to jump start this process: a combination of “teach a man to fish” and “let’s catch a whole bunch of fish together, right now.” We’ll contribute to the library together, empower many new people to keep contributing, and learn more about how the core team can best support contributors.
What’s in it for me?
Besides the eternal fame and glory, you mean?
Hackathons are a great way to meet cool people and pick up new skills. By joining a Great Expectations community hackathon, you will also:
- Join one of the fastest-growing data communities on the web
- Get contributor credit in the Great Expectations public repo
- Receive limited edition Great Expectations swag commemorating the occasion. High-value items for hackathon T-shirt collectors, for sure.
- Cash Prizes!
We have scheduled three different hackathons with hopefully more to come. Here's what we have so far:
- Student Hackathon 1/23 5-9pm PST (students only, must be currently attending a university)
- Data Professionals Hackathon 1/28 5-9 PST
- Student Hackathon 2/6 2-6pm PST (students only, must be currently attending a university)
If this sounds like fun, please sign up here!
To help with this initiative, we’ve partnered with other vibrant data communities. We’re proud to partner with Insight Data Fellows and Data Council. If you're interested in partnering with us for a Great Expectations hackathon event please feel free to contact kyle@superconductive.com
