-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
lighting/parser: align NULL and ESCAPED BY with LOAD DATA #40909
Conversation
Signed-off-by: lance6716 <lance6716@gmail.com>
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
/cc @gozssky @dsdashun |
Signed-off-by: lance6716 <lance6716@gmail.com>
/retest |
Signed-off-by: lance6716 <lance6716@gmail.com>
}, | ||
} | ||
|
||
testCases := []testCase{ | ||
{ | ||
input: `\\`, | ||
expected: [][]types.Datum{{nullDatum}}, | ||
expected: [][]types.Datum{{types.NewStringDatum("")}}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
note that the behaviour is changed. empty field inside delimiter is no longer NULL
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh I just noticed we have explicitly stated the behaviour in doc
Quoting does not affect whether a field is null.
https://docs.pingcap.com/tidb/stable/tidb-lightning-data-source#not-null-and-null
I should add a hide configuration for LOAD DATA 😂
Signed-off-by: lance6716 <lance6716@gmail.com>
Signed-off-by: lance6716 <lance6716@gmail.com>
Signed-off-by: lance6716 <lance6716@gmail.com>
/run-integration-br-tests |
@@ -303,20 +303,22 @@ func (parser *blockParser) readBlock() error { | |||
} | |||
} | |||
|
|||
var unescapeRegexp = regexp.MustCompile(`(?s)\\.`) | |||
var chunkParserUnescapeRegexp = regexp.MustCompile(`(?s)\\.`) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Almost LGTM. I don't quite understand why this regexp has been preserved and will review it later.
@lance6716: PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
ping |
csv.EscapedBy = `\` | ||
} | ||
if !csv.BackslashEscape && csv.EscapedBy == `\` { | ||
csv.EscapedBy = "" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is csv.EscapedBy
set to empty if it is set in the config but backslashEscape
is false?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently the default value of EscapedBy is \
, and BackslashEscape is hidden and has default value true. So if BackslashEscape is changed to false, it means this is an old format configuration file with false BackslashEscape, we should disable escaping
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
Signed-off-by: lance6716 <lance6716@gmail.com>
Signed-off-by: lance6716 <lance6716@gmail.com>
Signed-off-by: lance6716 <lance6716@gmail.com>
/merge |
This pull request has been accepted and is ready to merge. Commit hash: 79f2d6b
|
/merge |
/retest |
1 similar comment
/retest |
/test all |
/retest |
Signed-off-by: lance6716 lance6716@gmail.com
What problem does this PR solve?
Issue Number: ref #40499
Problem Summary:
What is changed and how it works?
\
\
),\N
should be treated as NULL. If ENCLOSED BY is set (for example"
), unenclosedNULL
in data file should be treated as NULL. Modify the code so caller should set\N
,\NULL
in the configurationAlso fix a bug in ReadUntilTerminator.
Check List
Tests
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.