Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix line terminator in string literal #1084

Merged
merged 1 commit into from
Feb 1, 2021

Conversation

jevancc
Copy link
Contributor

@jevancc jevancc commented Jan 19, 2021

This Pull Request fixes/closes #1083.

It changes the following:

  • Handle line terminator in string literal
  • Fix broken tests
  • Refactor and fix line terminator in template literal
  • Support invalid character in string/template literal (like "\����")
  • Modify existing tests

Since template literal now has its own method to process characters, I unexported the method StringLiteral::take_string_characters and remove StringTerminator::End. Some tests on take_string_characters are modified to test on lexer directly.

@jevancc jevancc changed the title Fix line terminator in string literal (WIP) Fix line terminator in string literal Jan 19, 2021
@codecov
Copy link

codecov bot commented Jan 19, 2021

Codecov Report

Merging #1084 (bd0742c) into master (2f3b926) will increase coverage by 0.06%.
The diff coverage is 61.11%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1084      +/-   ##
==========================================
+ Coverage   58.74%   58.81%   +0.06%     
==========================================
  Files         176      176              
  Lines       12477    12467      -10     
==========================================
+ Hits         7330     7332       +2     
+ Misses       5147     5135      -12     
Impacted Files Coverage Δ
boa/src/syntax/lexer/string.rs 64.40% <58.33%> (+3.66%) ⬆️
boa/src/syntax/lexer/template.rs 58.33% <66.66%> (+9.55%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2f3b926...8937bb2. Read the comment docs.

@Razican
Copy link
Member

Razican commented Jan 20, 2021

Numbers are looking good :)

Test262 conformance changes:

Test result master count PR count difference
Total 78,497 78,497 0
Passed 24,813 24,848 +35
Ignored 15,587 15,587 0
Failed 38,097 38,062 -35
Panics 20 20 0
Conformance 31.61 31.65 +0.04%

@jevancc jevancc changed the title (WIP) Fix line terminator in string literal Fix line terminator in string literal Jan 20, 2021
@jevancc jevancc marked this pull request as ready for review January 20, 2021 23:08
@RageKnify RageKnify added the lexer Issues surrounding the lexer label Jan 21, 2021
Copy link
Member

@RageKnify RageKnify left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused by some of the changes in the tests. The implementation seems good.

The test262 numbers have changed, latest commit reports:

Test262 conformance changes:

Test result master count PR count difference
Total 78,497 78,497 0
Passed 24,850 24,862 +12
Ignored 15,587 15,587 0
Failed 38,060 38,048 -12
Panics 18 18 0
Conformance 31.66 31.67 +0.02%

boa/src/builtins/string/tests.rs Show resolved Hide resolved
boa/src/builtins/string/tests.rs Show resolved Hide resolved
Copy link
Contributor

@tofpie tofpie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Great job!

Fix octal escape in string literal


Add tests


Fix zero escape


Fix zero escape lookahead


Rename variables


Rename helper functions


Refactor match arms


Fix escape line terminator sequence


Fix single character escape


Fix line terminator and escape followed by unicode char


Fix broken tests


Add NonOctalDecimalEscapeSequence


Fix comment


Refactor


Modify error message


Add tests


Rename tests


Add test for error


Add comments for unsafe bytes to str


Update boa/src/syntax/lexer/string.rs

Co-authored-by: tofpie <75836434+tofpie@users.noreply.github.com>
Minor refactor


Remove unsafe bytes to str


Fix panic when reading invalid utf-8 chars


Refactor string literal


Support invalid utf-8 chars in string literal input


Add cook function for template literal


Fix line continuation bug


Add methods for utf16 buffer trait


Add trait comments


Add error message for template literal


Add and fix comments


Hide unused exported function and modify tests


Fix bug


Fix merge bug
@RageKnify RageKnify merged commit 038acb4 into boa-dev:master Feb 1, 2021
Razican pushed a commit that referenced this pull request May 22, 2021
Fix octal escape in string literal


Add tests


Fix zero escape


Fix zero escape lookahead


Rename variables


Rename helper functions


Refactor match arms


Fix escape line terminator sequence


Fix single character escape


Fix line terminator and escape followed by unicode char


Fix broken tests


Add NonOctalDecimalEscapeSequence


Fix comment


Refactor


Modify error message


Add tests


Rename tests


Add test for error


Add comments for unsafe bytes to str


Update boa/src/syntax/lexer/string.rs

Co-authored-by: tofpie <75836434+tofpie@users.noreply.github.com>
Minor refactor


Remove unsafe bytes to str


Fix panic when reading invalid utf-8 chars


Refactor string literal


Support invalid utf-8 chars in string literal input


Add cook function for template literal


Fix line continuation bug


Add methods for utf16 buffer trait


Add trait comments


Add error message for template literal


Add and fix comments


Hide unused exported function and modify tests


Fix bug


Fix merge bug
@RageKnify RageKnify added this to the v0.12.0 milestone May 31, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lexer Issues surrounding the lexer
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Line terminator in string literal should raise a syntax error
5 participants