[Feature] Update is_(not)_in_range (#87) to support max/min limits from col #153
+105
−58
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
karthik-ballullaya-db
requested review from
pratikk-databricks
and removed request for
a team
karthik-ballullaya-db
temporarily deployed
to
tool
GitHub Actions
Inactive
karthik-ballullaya-db
temporarily deployed
to
tool
GitHub Actions
Inactive
|
✅ 126/126 passed, 1 skipped, 17m53s total Running from acceptance #498 |
alexott
requested changes
Feb 5, 2025
Comment on lines
318
to
319
| min_limit: int | datetime.date | datetime.datetime | str, | ||
| max_limit: int | datetime.date | datetime.datetime | str, |
There was a problem hiding this comment.
I would make it a bit different:
- add support for a string as a valid range
- add support for a column object as min/max arguments, because people may use not simple column names but more complex expressions (via
F.expr).
There was a problem hiding this comment.
agree, updated code to support string ranges and column expression
karthik-ballullaya-db
force-pushed
the
feat/(not)_in_range_col_func
branch
from
5734785 to
1bd0402
Compare
karthik-ballullaya-db
temporarily deployed
to
tool
GitHub Actions
Inactive
karthik-ballullaya-db
temporarily deployed
to
tool
GitHub Actions
Inactive
karthik-ballullaya-db
had a problem deploying
to
tool
GitHub Actions
Error
karthik-ballullaya-db
had a problem deploying
to
tool
GitHub Actions
Error
karthik-ballullaya-db
temporarily deployed
to
tool
GitHub Actions
Inactive
karthik-ballullaya-db
temporarily deployed
to
tool
GitHub Actions
Inactive
alexott
reviewed
Feb 7, 2025
Comment on lines
285
to
288
| min_limit: int | datetime.date | datetime.datetime | str = '', | ||
| max_limit: int | datetime.date | datetime.datetime | str = '', | ||
| min_limit_col_expr: str | Column = '', | ||
| max_limit_col_expr: str | Column = '', |
There was a problem hiding this comment.
I would use Optional type or None instead of default of empty string
Comment on lines
311
to
314
| min_limit: int | datetime.date | datetime.datetime | str = '', | ||
| max_limit: int | datetime.date | datetime.datetime | str = '', | ||
| min_limit_col_expr: str | Column = '', | ||
| max_limit_col_expr: str | Column = '', |
Comment on lines
348
to
351
| min_limit: int | datetime.date | datetime.datetime | str = '', | ||
| max_limit: int | datetime.date | datetime.datetime | str = '', | ||
| min_limit_col_expr: str | Column = '', | ||
| max_limit_col_expr: str | Column = '', |
karthik-ballullaya-db
temporarily deployed
to
tool
GitHub Actions
Inactive
karthik-ballullaya-db
temporarily deployed
to
tool
GitHub Actions
Inactive
alexott
requested changes
Feb 12, 2025
Comment on lines
+298
to
+305
| if min_limit_col_expr is None: | ||
| min_limit_expr = F.lit(min_limit) | ||
| else: | ||
| min_limit_expr = F.col(min_limit_col_expr) if isinstance(min_limit_col_expr, str) else min_limit_col_expr | ||
| if max_limit_col_expr is None: | ||
| max_limit_expr = F.lit(max_limit) | ||
| else: | ||
| max_limit_expr = F.col(max_limit_col_expr) if isinstance(max_limit_col_expr, str) else max_limit_col_expr |
There was a problem hiding this comment.
we need to throw an error if both min_limit and min_limit_col_expr are None (same for max_limit and max_limit_col_expr).
Otherwise we'll get conditions like (F.col(col_name) < F.lit(None)) | (F.col(col_name) > F.lit(None))
karthik-ballullaya-db
temporarily deployed
to
tool
GitHub Actions
Inactive
karthik-ballullaya-db
temporarily deployed
to
tool
GitHub Actions
Inactive
alexott
approved these changes
Feb 26, 2025
mwojtyczka
added a commit
that referenced
this pull request
Feb 27, 2025
* Fixed cli installation and demo ([#177](#177)). In this release, changes have been made to adjust the dashboard name, ensuring compliance with new API naming rules. The dashboard name now only contains alphanumeric characters, hyphens, or underscores, and the reference section has been split for clarity. In addition, demo for the tool has been updated to work regardless if a path or UC table is provided in the config. Furthermore, documentation has been refactored and udpated to improve clarity. The following issue have been closed: [#171](#171) and [#198](#198). * [Feature] Update is_(not)_in_range ([#87](#87)) to support max/min limits from col ([#153](#153)). In this release, the `is_in_range` and `is_not_in_range` quality rule functions have been updated to support a column as the minimum or maximum limit, in addition to a literal value. This change is accomplished through the introduction of optional `min_limit_col_expr` and `max_limit_col_expr` arguments, allowing users to specify a column expression as the minimum or maximum limit. Extensive testing, including unit tests and integration tests, has been conducted to ensure the correct behavior of the new functionality. These enhancements offer increased flexibility when defining quality rules, catering to a broader range of use cases and scenarios.
Merged
mwojtyczka
added a commit
that referenced
this pull request
Feb 27, 2025
* Fixed cli installation and demo ([#177](#177)). In this release, changes have been made to adjust the dashboard name, ensuring compliance with new API naming rules. The dashboard name now only contains alphanumeric characters, hyphens, or underscores, and the reference section has been split for clarity. In addition, demo for the tool has been updated to work regardless if a path or UC table is provided in the config. Furthermore, documentation has been refactored and udpated to improve clarity. The following issue have been closed: [#171](#171) and [#198](#198). * [Feature] Update is_(not)_in_range ([#87](#87)) to support max/min limits from col ([#153](#153)). In this release, the `is_in_range` and `is_not_in_range` quality rule functions have been updated to support a column as the minimum or maximum limit, in addition to a literal value. This change is accomplished through the introduction of optional `min_limit_col_expr` and `max_limit_col_expr` arguments, allowing users to specify a column expression as the minimum or maximum limit. Extensive testing, including unit tests and integration tests, has been conducted to ensure the correct behavior of the new functionality. These enhancements offer increased flexibility when defining quality rules, catering to a broader range of use cases and scenarios.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Changes in the is_in_range and is_not_in_range column function to handle a column as the min/max limit along with the literal value.
Linked issues
Resolves #87
Tests