Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle regexp_replace inconsistency with empty strings and zero-repetition patterns [databricks] #5740

Merged
merged 18 commits into from
Jun 8, 2022
Merged

Handle regexp_replace inconsistency with empty strings and zero-repetition patterns [databricks] #5740

merged 18 commits into from
Jun 8, 2022

Conversation

anthony-chang
Copy link
Contributor

@anthony-chang anthony-chang commented Jun 3, 2022 •
edited
Loading

Fixes #5456

When using regexp_replace on an empty string using a regex pattern containing only zero-repetitions (eg. *, ?, {0,}, etc), the behaviour is different depending on which version of Spark.

Given the input

df = spark.sparkContext.parallelize([[""],[""],["AAA"]]).toDF(["a"])
df.selectExpr("regexp_replace(a,'A*','_REPLACED_')").show()

Spark versions 3.1.3, 3.1.4, 3.2.2, 3.3.0+ gives

+-------------------------------------+
|regexp_replace(a, A*, _REPLACED_, 1) |
+-------------------------------------+
|                          _REPLACED_ |
|                          _REPLACED_ |
|                _REPLACED__REPLACED_ |
+-------------------------------------+

but all other versions gives

+-------------------------------------+
|regexp_replace(a, A*, _REPLACED_, 1) |
+-------------------------------------+
|                                     |
|                                     |
|                _REPLACED__REPLACED_ |
+-------------------------------------+

This PR adds shims for the 2nd case to short circuit the plugin regex handling.

Signed-off-by: Anthony Chang antchang@nvidia.com

andygrove and others added 4 commits June 3, 2022 11:35
@andygrove
shim layer for regexp
e3b81d6
@andygrove
rough out handing of empty string bug
cd7dbf3
@anthony-chang
Add licences to shim files, check for empty repetition
40b21c0 
Signed-off-by: Anthony Chang <antchang@nvidia.com>
@anthony-chang
Update integration test
951d455 
Signed-off-by: Anthony Chang <antchang@nvidia.com>
@anthony-chang anthony-chang self-assigned this Jun 3, 2022
@sameerz sameerz added the audit_3.3.0 Audit related tasks for 3.3.0 label Jun 4, 2022
@anthony-chang
Recursively check for only empty repetition in regex
65b9f8e 
Signed-off-by: Anthony Chang <antchang@nvidia.com>
@anthony-chang
Fix license date
6256bd8 
Signed-off-by: Anthony Chang <antchang@nvidia.com>
@andygrove
Copy link
Contributor

@anthony-chang I know this is still WIP but could you update the PR description to explain what is changed in this PR

@anthony-chang
Merge branch 'branch-22.08' of github.com:nvidia/spark-rapids into re…
6a8db81 
…gexp-shim-5456
@anthony-chang
Add 321cdh shim
3345728 
Signed-off-by: Anthony Chang <antchang@nvidia.com>
@anthony-chang
Copy link
Contributor Author

build

@anthony-chang
Move shim file
2dbca58 
Signed-off-by: Anthony Chang <antchang@nvidia.com>
@anthony-chang
Copy link
Contributor Author

build

1 similar comment
@anthony-chang
Copy link
Contributor Author

build

@anthony-chang
Remove a shim file
be93d92 
Signed-off-by: Anthony Chang <antchang@nvidia.com>
@anthony-chang
Copy link
Contributor Author

build

@anthony-chang
Add comma
6e0f2ba 
Signed-off-by: Anthony Chang <antchang@nvidia.com>
@anthony-chang
Handle case where AST root is a RegexSequence
bce08b9 
Signed-off-by: Anthony Chang <antchang@nvidia.com>
@anthony-chang
Add shim to 321db
ce644c7 
Signed-off-by: Anthony Chang <antchang@nvidia.com>
@anthony-chang
Copy link
Contributor Author

build

@anthony-chang anthony-chang changed the title [WIP] Handle regexp_replace inconsistency with empty strings and zero-repetition patterns [WIP] Handle regexp_replace inconsistency with empty strings and zero-repetition patterns [databricks] Jun 7, 2022
@anthony-chang
Handle case where pattern contains more than 1 empty repetition
364764e 
Signed-off-by: Anthony Chang <antchang@nvidia.com>
@anthony-chang
Update comment
e7a6e2c 
Signed-off-by: Anthony Chang <antchang@nvidia.com>
@anthony-chang
Copy link
Contributor Author

build

@anthony-chang anthony-chang marked this pull request as ready for review June 8, 2022 16:30
@anthony-chang anthony-chang changed the title [WIP] Handle regexp_replace inconsistency with empty strings and zero-repetition patterns [databricks] Handle regexp_replace inconsistency with empty strings and zero-repetition patterns [databricks] Jun 8, 2022
@anthony-chang @andygrove
Apply suggestions from code review
a4584d5 
Co-authored-by: Andy Grove <andygrove73@gmail.com>
@anthony-chang
Copy link
Contributor Author

build

@anthony-chang
Refactor isEmptyRepetition logic to RegExpUtils
916ab64 
Signed-off-by: Anthony Chang <antchang@nvidia.com>
@anthony-chang
Fix scalastyle
194a42f 
Signed-off-by: Anthony Chang <antchang@nvidia.com>
@anthony-chang
Copy link
Contributor Author

build

@anthony-chang anthony-chang merged commit 8487b4b into NVIDIA:branch-22.08 Jun 8, 2022
@anthony-chang anthony-chang mentioned this pull request Jun 15, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andygrove andygrove andygrove approved these changes

@NVnavkumar NVnavkumar NVnavkumar approved these changes

Assignees

@anthony-chang anthony-chang

Labels
audit_3.3.0 Audit related tasks for 3.3.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG] Handle regexp_replace inconsistency from https://issues.apache.org/jira/browse/SPARK-39107
4 participants
@anthony-chang @andygrove @NVnavkumar @sameerz