Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New lexer #559

Merged
merged 321 commits into from
Aug 27, 2020
Merged

New lexer #559

merged 321 commits into from
Aug 27, 2020

Conversation

Lan2u
Copy link

@Lan2u Lan2u commented Jul 9, 2020 •
edited
Loading

This is the same as the previous branch but based on a branch on the boa repo ( we are troubleshooting why there is no benchmark comment).

Closes:
#294
#456
#288 (at least with reference to using streams but not multithreaded)
#588

Paul Lancaster added 30 commits June 22, 2020 16:35
Continuing to adapt parser to new cursor returns
910ba1a
Fix multiple mutable cursor reference
61b1c68
Updated benchmarks lexer/parser usage
354cf1a
Updated cli/wasm usage of parser
3dda033
Back to a compiling state
9715bae
Seperated cursor peek and peek(skip > 0)
68f41fa
VecDeque used to allow peeking 1 ahead
067e310
Removing back/prev from parser cursor
efb0ab3
Fixed peek_more parser cursor which caused infinite loop
efc6bc1
Modified parser cursor.expect behaviour to fit usage
8dc3159
Traced down / fixed bug related to semicolon insertion
0f9b23b
Added push-back() which seems to be required for parsing the differen…
da68b13 
…t between (expr) and an arrow function
Fixed bug where line terminators were not ignored during object liter…
554ae90 
…al parsing
Fixed bug where line terminators were rejected within object literal …
e48de3d 
…definition
fixed bug where anonymous functions as function arguments could not b…
16480d9 
…e parsed
Fixed bug with line terminators being supported in WhileLoop
bd08224
Removed need for special peek_semicolon with do_while flag
8b57683
Added dowhile semicolon insertion
4c0b798
Fixed bug where line terminators were not allowed between parts of do…
273b1e4 
…while
Disabled 2 new lexer tests which didn't pass anyway
a5cd476
Cargo fmt
9370e6f
Cleanup
548ae5e
Fmt
f22f171
Tidying
ac1d238
Merge pull request #19 from Lan2u/bug_fix
32fb242 
Bug fixes
Starting to add goal symbol set statements
dde8893
Added goal symbol set to expression parser macro
7af447a
Trying to workout where to put goal symbols
03690ff
Merge pull request #20 from Lan2u/new_parser_lexer
7932837 
New parser lexer
Added a 'fallback' mechanism to try and regex if a div is found unexp…
40db5f7 
…ectadly - all regex tests currently pass
@github-actions
Copy link

Benchmark for 51f4e7a

Click to view benchmark
Test PR Benchmark Master Benchmark %
Arithmetic operations (Execution) 358.3±25.79ns 343.1±22.62ns +4.43%
Arithmetic operations (Full) 240.1±49.15µs 232.9±12.65µs +3.09%
Array access (Execution) 7.6±0.37µs 7.7±0.42µs -1.30%
Array access (Full) 251.5±13.76µs 251.1±20.18µs +0.16%
Array creation (Execution) 3.2±0.15ms 3.2±0.12ms 0.00%
Array creation (Full) 3.4±0.13ms 3.5±0.13ms -2.86%
Array pop (Execution) 1122.2±76.83µs 1107.8±67.58µs +1.30%
Array pop (Full) 1501.2±109.68µs 1485.3±103.39µs +1.07%
Boolean Object Access (Execution) 4.3±0.23µs 4.4±0.24µs -2.27%
Boolean Object Access (Full) 254.3±13.46µs 240.4±16.11µs +5.78%
Clean js (Execution) 679.2±32.84µs 690.8±43.32µs -1.68%
Clean js (Full) 949.8±40.84µs 945.8±54.29µs +0.42%
Clean js (Parser) 35.5±2.05µs 22.7±1.57µs +56.39%
Create Realm 457.8±29.91ns 446.9±37.14ns +2.44%
Dynamic Object Property Access (Execution) 5.1±0.29µs 5.1±0.25µs 0.00%
Dynamic Object Property Access (Full) 247.5±14.28µs 242.5±11.25µs +2.06%
Expression (Lexer) 2.2±0.15µs N/A N/A
Expression (Parser) 7.1±0.45µs 5.1±0.28µs +39.22%
Fibonacci (Execution) 758.1±54.01µs 748.3±45.11µs +1.31%
Fibonacci (Full) 1033.2±68.19µs 1021.5±46.49µs +1.15%
For loop (Execution) 20.9±1.08µs 20.8±0.95µs +0.48%
For loop (Full) 270.2±16.21µs 263.1±15.76µs +2.70%
For loop (Lexer) 4.1±0.18µs N/A N/A
For loop (Parser) 17.2±0.89µs 11.6±0.73µs +48.28%
Goal Symbols (Parser) 11.8±0.59µs 7.6±0.43µs +55.26%
Hello World (Lexer) 799.0±45.92ns N/A N/A
Hello World (Parser) 3.0±0.16µs 2.0±0.14µs +50.00%
Long file (Parser) 719.4±57.80ns 6.0±0.19ms -99.99%
Mini js (Execution) 630.9±43.57µs 632.1±39.28µs -0.19%
Mini js (Full) 865.5±36.03µs 871.2±38.59µs -0.65%
Mini js (Parser) 31.3±1.81µs 20.4±1.46µs +53.43%
Number Object Access (Execution) 3.7±0.18µs 3.5±0.27µs +5.71%
Number Object Access (Full) 244.8±15.40µs 242.9±26.86µs +0.78%
Object Creation (Execution) 4.5±0.22µs 4.2±0.31µs +7.14%
Object Creation (Full) 298.5±14.45µs 291.2±18.10µs +2.51%
RegExp (Execution) 66.2±5.58µs 63.2±3.88µs +4.75%
RegExp (Full) 387.9±26.03µs 381.2±18.77µs +1.76%
RegExp Literal (Execution) 68.7±3.28µs 65.8±3.50µs +4.41%
RegExp Literal (Full) 321.8±20.42µs 309.4±21.83µs +4.01%
RegExp Literal Creation (Execution) 65.7±4.73µs 63.4±3.79µs +3.63%
RegExp Literal Creation (Full) 393.3±24.15µs 374.4±24.53µs +5.05%
Static Object Property Access (Execution) 4.4±0.22µs 4.6±0.28µs -4.35%
Static Object Property Access (Full) 247.1±13.99µs 240.0±11.12µs +2.96%
String Object Access (Execution) 6.5±0.42µs 6.5±0.35µs 0.00%
String Object Access (Full) 255.4±12.79µs 239.8±17.75µs +6.51%
String comparison (Execution) 6.2±0.50µs 6.1±0.42µs +1.64%
String comparison (Full) 249.3±17.94µs 238.8±21.40µs +4.40%
String concatenation (Execution) 4.9±0.24µs 4.9±0.30µs 0.00%
String concatenation (Full) 256.0±14.88µs 230.1±11.84µs +11.26%
String copy (Execution) 3.8±0.26µs 3.6±0.21µs +5.56%
String copy (Full) 237.1±12.70µs 231.2±12.52µs +2.55%
Symbols (Execution) 3.0±0.23µs 3.0±0.17µs 0.00%
Symbols (Full) 222.8±14.30µs 220.2±11.78µs +1.18%

@github-actions
Copy link

Benchmark for c16865e

Click to view benchmark
Test PR Benchmark Master Benchmark %
Arithmetic operations (Execution) 381.7±5.75ns 364.0±11.03ns +4.86%
Arithmetic operations (Full) 259.6±5.04µs 251.8±6.98µs +3.10%
Array access (Execution) 8.4±0.51µs 8.4±0.28µs 0.00%
Array access (Full) 285.8±6.54µs 280.2±10.19µs +2.00%
Array creation (Execution) 3.1±0.07ms 3.3±0.06ms -6.06%
Array creation (Full) 3.6±0.05ms 3.6±0.17ms 0.00%
Array pop (Execution) 1091.6±14.47µs 1182.7±183.16µs -7.70%
Array pop (Full) 1622.2±23.94µs 1536.0±42.26µs +5.61%
Boolean Object Access (Execution) 4.8±0.07µs 4.8±0.06µs 0.00%
Boolean Object Access (Full) 276.3±5.33µs 266.2±7.63µs +3.79%
Clean js (Execution) 734.0±15.38µs 742.5±21.89µs -1.14%
Clean js (Full) 1062.7±41.58µs 1037.9±28.81µs +2.39%
Clean js (Parser) 40.4±0.94µs 25.9±0.52µs +55.98%
Create Realm 486.4±17.45ns 475.8±9.42ns +2.23%
Dynamic Object Property Access (Execution) 5.6±0.15µs 5.6±0.22µs 0.00%
Dynamic Object Property Access (Full) 278.0±9.51µs 266.3±6.46µs +4.39%
Expression (Lexer) 2.4±0.03µs N/A N/A
Expression (Parser) 7.8±0.17µs 5.3±0.12µs +47.17%
Fibonacci (Execution) 889.0±23.87µs 889.4±22.44µs -0.04%
Fibonacci (Full) 1172.5±16.36µs 1181.6±40.15µs -0.77%
For loop (Execution) 23.1±0.94µs 23.7±2.77µs -2.53%
For loop (Full) 300.6±6.12µs 285.9±8.30µs +5.14%
For loop (Lexer) 4.2±0.10µs N/A N/A
For loop (Parser) 19.9±0.66µs 12.4±0.36µs +60.48%
Goal Symbols (Parser) 13.2±0.36µs 8.4±0.21µs +57.14%
Hello World (Lexer) 882.0±26.66ns N/A N/A
Hello World (Parser) 3.3±0.10µs 2.2±0.07µs +50.00%
Long file (Parser) 802.6±12.43ns 6.6±0.12ms -99.99%
Mini js (Execution) 659.5±12.88µs 670.4±16.12µs -1.63%
Mini js (Full) 959.3±32.85µs 947.3±20.24µs +1.27%
Mini js (Parser) 35.5±0.68µs 23.2±0.59µs +53.02%
Number Object Access (Execution) 3.7±0.06µs 3.8±0.11µs -2.63%
Number Object Access (Full) 271.9±5.76µs 259.9±11.30µs +4.62%
Object Creation (Execution) 4.7±0.11µs 4.8±0.13µs -2.08%
Object Creation (Full) 329.5±10.34µs 317.2±9.21µs +3.88%
RegExp (Execution) 71.7±1.75µs 69.0±1.97µs +3.91%
RegExp (Full) 428.0±10.41µs 420.3±10.69µs +1.83%
RegExp Literal (Execution) 75.6±2.73µs 73.4±1.85µs +3.00%
RegExp Literal (Full) 354.2±7.11µs 346.3±10.20µs +2.28%
RegExp Literal Creation (Execution) 70.1±1.76µs 73.4±22.23µs -4.50%
RegExp Literal Creation (Full) 424.0±10.66µs 419.3±9.29µs +1.12%
Static Object Property Access (Execution) 5.0±0.08µs 5.0±0.14µs 0.00%
Static Object Property Access (Full) 275.0±6.43µs 267.2±7.85µs +2.92%
String Object Access (Execution) 7.0±0.09µs 7.0±0.33µs 0.00%
String Object Access (Full) 278.0±6.96µs 265.5±6.09µs +4.71%
String comparison (Execution) 6.5±0.13µs 6.5±0.17µs 0.00%
String comparison (Full) 277.4±5.08µs 267.3±7.19µs +3.78%
String concatenation (Execution) 5.3±0.16µs 5.3±0.11µs 0.00%
String concatenation (Full) 273.0±9.16µs 258.9±4.62µs +5.45%
String copy (Execution) 4.0±0.09µs 4.0±0.16µs 0.00%
String copy (Full) 268.7±5.30µs 259.8±11.07µs +3.43%
Symbols (Execution) 3.3±0.05µs 3.3±0.13µs 0.00%
Symbols (Full) 247.5±7.99µs 244.2±20.54µs +1.35%

@Lan2u Lan2u requested a review from Razican August 25, 2020 07:55
Razican
Razican reviewed Aug 25, 2020
View reviewed changes
Copy link
Member

@Razican Razican left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should add a test for the lexing error positions.

@Lan2u
Copy link
Author

Lan2u commented Aug 25, 2020

I think we should add a test for the lexing error positions.

Added some :)

@github-actions
Copy link

Benchmark for 541b5fa

Click to view benchmark
Test PR Benchmark Master Benchmark %
Arithmetic operations (Execution) 350.7±15.57ns 383.7±21.11ns -8.60%
Arithmetic operations (Full) 246.5±12.95µs 255.9±12.72µs -3.67%
Array access (Execution) 7.7±0.48µs 8.0±0.43µs -3.75%
Array access (Full) 285.3±20.30µs 273.8±9.66µs +4.20%
Array creation (Execution) 3.4±0.15ms 3.4±0.17ms 0.00%
Array creation (Full) 3.7±0.15ms 3.8±0.13ms -2.63%
Array pop (Execution) 1246.1±35.73µs 1160.8±58.88µs +7.35%
Array pop (Full) 1669.5±80.01µs 1653.2±65.49µs +0.99%
Boolean Object Access (Execution) 4.8±0.08µs 4.8±0.22µs 0.00%
Boolean Object Access (Full) 254.4±14.85µs 265.8±11.32µs -4.29%
Clean js (Execution) 707.4±38.87µs 771.6±39.60µs -8.32%
Clean js (Full) 1054.4±78.02µs 1055.4±40.81µs -0.09%
Clean js (Parser) 37.1±1.95µs 24.4±0.53µs +52.05%
Create Realm 462.4±31.62ns 458.5±40.43ns +0.85%
Dynamic Object Property Access (Execution) 5.2±0.30µs 5.2±0.57µs 0.00%
Dynamic Object Property Access (Full) 259.7±15.18µs 265.3±9.51µs -2.11%
Expression (Lexer) 2.4±0.12µs N/A N/A
Expression (Parser) 7.6±0.26µs 5.4±0.16µs +40.74%
Fibonacci (Execution) 782.9±40.05µs 777.4±43.11µs +0.71%
Fibonacci (Full) 1102.0±23.87µs 1131.7±43.18µs -2.62%
For loop (Execution) 22.0±1.39µs 21.6±1.01µs +1.85%
For loop (Full) 282.9±15.98µs 291.7±14.21µs -3.02%
For loop (Lexer) 4.4±0.22µs N/A N/A
For loop (Parser) 19.0±0.72µs 12.6±0.48µs +50.79%
Goal Symbols (Parser) 12.7±0.81µs 8.6±0.25µs +47.67%
Hello World (Lexer) 866.6±29.01ns N/A N/A
Hello World (Parser) 3.2±0.12µs 2.3±0.11µs +39.13%
Long file (Parser) 786.6±38.62ns 6.3±0.11ms -99.99%
Mini js (Execution) 638.4±33.11µs 700.8±44.31µs -8.90%
Mini js (Full) 981.9±35.26µs 957.1±25.87µs +2.59%
Mini js (Parser) 33.1±1.60µs 22.3±0.81µs +48.43%
Number Object Access (Execution) 3.9±0.19µs 3.7±0.13µs +5.41%
Number Object Access (Full) 255.9±14.97µs 258.4±5.92µs -0.97%
Object Creation (Execution) 4.6±0.15µs 4.3±0.23µs +6.98%
Object Creation (Full) 311.1±15.53µs 318.5±11.61µs -2.32%
RegExp (Execution) 66.0±4.14µs 67.6±4.59µs -2.37%
RegExp (Full) 402.2±30.70µs 418.2±12.54µs -3.83%
RegExp Literal (Execution) 71.3±2.24µs 69.1±3.69µs +3.18%
RegExp Literal (Full) 334.5±15.43µs 345.9±12.99µs -3.30%
RegExp Literal Creation (Execution) 65.0±3.92µs 65.6±7.75µs -0.91%
RegExp Literal Creation (Full) 396.8±18.94µs 421.4±21.84µs -5.84%
Static Object Property Access (Execution) 4.8±0.35µs 4.6±0.25µs +4.35%
Static Object Property Access (Full) 254.5±10.97µs 267.5±6.64µs -4.86%
String Object Access (Execution) 6.6±0.29µs 7.1±0.55µs -7.04%
String Object Access (Full) 260.5±12.87µs 276.7±42.88µs -5.85%
String comparison (Execution) 5.9±0.28µs 6.3±0.38µs -6.35%
String comparison (Full) 252.1±14.14µs 265.6±13.80µs -5.08%
String concatenation (Execution) 5.0±0.21µs 4.9±0.26µs +2.04%
String concatenation (Full) 251.3±12.38µs 261.1±11.27µs -3.75%
String copy (Execution) 3.8±0.22µs 3.9±0.17µs -2.56%
String copy (Full) 252.2±14.32µs 259.8±12.66µs -2.93%
Symbols (Execution) 3.1±0.19µs 3.0±0.18µs +3.33%
Symbols (Full) 255.6±26.08µs 243.2±8.88µs +5.10%

@github-actions
Copy link

Benchmark for 70a9156

Click to view benchmark
Test PR Benchmark Master Benchmark %
Arithmetic operations (Execution) 377.0±6.33ns 379.3±11.80ns -0.61%
Arithmetic operations (Full) 259.8±9.39µs 258.9±8.38µs +0.35%
Array access (Execution) 8.3±0.17µs 8.3±0.17µs 0.00%
Array access (Full) 285.0±10.36µs 279.2±4.13µs +2.08%
Array creation (Execution) 3.6±0.05ms 3.5±0.04ms +2.86%
Array creation (Full) 4.0±0.05ms 3.8±0.05ms +5.26%
Array pop (Execution) 1275.2±30.07µs 1258.8±36.61µs +1.30%
Array pop (Full) 1766.7±29.83µs 1646.2±28.46µs +7.32%
Boolean Object Access (Execution) 4.8±0.11µs 4.9±0.17µs -2.04%
Boolean Object Access (Full) 273.8±9.10µs 266.0±3.48µs +2.93%
Clean js (Execution) 770.1±12.87µs 767.4±19.72µs +0.35%
Clean js (Full) 1093.8±41.68µs 1065.7±21.43µs +2.64%
Clean js (Parser) 39.9±1.45µs 24.9±0.30µs +60.24%
Create Realm 470.8±10.60ns 476.1±7.26ns -1.11%
Dynamic Object Property Access (Execution) 5.4±0.08µs 5.5±0.08µs -1.82%
Dynamic Object Property Access (Full) 272.6±4.60µs 271.6±9.15µs +0.37%
Expression (Lexer) 2.4±0.04µs N/A N/A
Expression (Parser) 7.9±0.33µs 5.5±0.15µs +43.64%
Fibonacci (Execution) 827.9±17.18µs 819.9±24.67µs +0.98%
Fibonacci (Full) 1122.8±25.84µs 1124.0±29.52µs -0.11%
For loop (Execution) 23.3±1.67µs 22.9±0.48µs +1.75%
For loop (Full) 297.6±6.17µs 289.9±4.50µs +2.66%
For loop (Lexer) 4.4±0.18µs N/A N/A
For loop (Parser) 19.5±0.40µs 12.8±0.51µs +52.34%
Goal Symbols (Parser) 13.2±0.24µs 8.6±0.11µs +53.49%
Hello World (Lexer) 865.8±14.75ns N/A N/A
Hello World (Parser) 3.3±0.04µs 2.2±0.03µs +50.00%
Long file (Parser) 788.8±15.61ns 6.4±0.10ms -99.99%
Mini js (Execution) 700.0±15.52µs 690.4±11.53µs +1.39%
Mini js (Full) 999.2±37.25µs 967.6±12.40µs +3.27%
Mini js (Parser) 35.1±0.84µs 22.2±0.49µs +58.11%
Number Object Access (Execution) 3.9±0.05µs 3.9±0.05µs 0.00%
Number Object Access (Full) 273.0±11.01µs 272.4±25.79µs +0.22%
Object Creation (Execution) 4.5±0.06µs 4.6±0.05µs -2.17%
Object Creation (Full) 328.0±6.62µs 320.6±6.24µs +2.31%
RegExp (Execution) 69.1±1.63µs 68.1±1.53µs +1.47%
RegExp (Full) 431.3±5.46µs 427.0±8.73µs +1.01%
RegExp Literal (Execution) 71.7±1.52µs 71.1±1.91µs +0.84%
RegExp Literal (Full) 348.7±6.68µs 348.3±7.21µs +0.11%
RegExp Literal Creation (Execution) 69.1±1.96µs 68.4±2.85µs +1.02%
RegExp Literal Creation (Full) 425.1±7.26µs 417.9±8.77µs +1.72%
Static Object Property Access (Execution) 4.9±0.08µs 4.9±0.10µs 0.00%
Static Object Property Access (Full) 271.6±4.01µs 268.0±3.17µs +1.34%
String Object Access (Execution) 7.0±0.10µs 7.1±0.12µs -1.41%
String Object Access (Full) 276.0±10.55µs 269.9±8.69µs +2.26%
String comparison (Execution) 6.4±0.09µs 6.6±0.11µs -3.03%
String comparison (Full) 272.8±6.63µs 271.0±9.44µs +0.66%
String concatenation (Execution) 5.2±0.15µs 5.3±0.12µs -1.89%
String concatenation (Full) 269.3±8.65µs 263.7±7.02µs +2.12%
String copy (Execution) 3.9±0.15µs 3.9±0.07µs 0.00%
String copy (Full) 262.9±9.44µs 260.7±4.52µs +0.84%
Symbols (Execution) 3.2±0.08µs 3.3±0.06µs -3.03%
Symbols (Full) 247.3±7.19µs 245.0±4.14µs +0.94%

Razican
Razican approved these changes Aug 27, 2020
View reviewed changes
Copy link
Member

@Razican Razican left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. I would just change a couple of things as per my comments. If you think they are reasonable, I can do the change myself today. I think we can merge this today, if the team is fine with it :)

boa/src/syntax/lexer/cursor.rs Outdated Show resolved Hide resolved
boa/src/syntax/lexer/tests.rs Outdated Show resolved Hide resolved
@Razican Razican requested a review from HalidOdat August 27, 2020 08:46
@jasonwilliams
Copy link
Member

I will try to take a look today

Paul Lancaster and others added 2 commits August 27, 2020 18:34
@Razican
Update boa/src/syntax/lexer/cursor.rs
f6fd3bb 
Co-authored-by: Iban Eguia <razican@protonmail.ch>
@Razican
Update boa/src/syntax/lexer/tests.rs
0d7ad6c 
Co-authored-by: Iban Eguia <razican@protonmail.ch>
@github-actions
Copy link

Benchmark for 52dedb1

Click to view benchmark
Test PR Benchmark Master Benchmark %
Arithmetic operations (Execution) 350.2±11.06ns 371.0±8.79ns -5.61%
Arithmetic operations (Full) 241.2±5.32µs 243.2±10.54µs -0.82%
Array access (Execution) 7.6±0.27µs 7.9±0.16µs -3.80%
Array access (Full) 249.4±9.31µs 257.5±6.18µs -3.15%
Array creation (Execution) 3.3±0.09ms 3.2±0.06ms +3.13%
Array creation (Full) 3.6±0.09ms 3.6±0.05ms 0.00%
Array pop (Execution) 1155.7±39.55µs 1137.4±24.11µs +1.61%
Array pop (Full) 1559.9±40.68µs 1534.9±20.67µs +1.63%
Boolean Object Access (Execution) 4.5±0.15µs 4.6±0.09µs -2.17%
Boolean Object Access (Full) 253.5±5.49µs 250.1±4.61µs +1.36%
Clean js (Execution) 708.6±18.57µs 709.2±13.55µs -0.08%
Clean js (Full) 1004.8±23.36µs 972.7±20.73µs +3.30%
Clean js (Parser) 35.1±0.46µs 22.2±0.33µs +58.11%
Create Realm 447.6±12.44ns 450.2±11.84ns -0.58%
Dynamic Object Property Access (Execution) 5.0±0.16µs 5.3±0.07µs -5.66%
Dynamic Object Property Access (Full) 251.9±8.47µs 252.8±4.85µs -0.36%
Expression (Lexer) 1612.0±31.17ns N/A N/A
Expression (Parser) 6.7±0.12µs 4.6±0.11µs +45.65%
Fibonacci (Execution) 743.9±19.93µs 782.6±11.62µs -4.95%
Fibonacci (Full) 1049.2±31.54µs 1056.2±19.67µs -0.66%
For loop (Execution) 20.5±0.60µs 21.7±0.41µs -5.53%
For loop (Full) 263.3±8.09µs 269.4±6.62µs -2.26%
For loop (Lexer) 3.2±0.11µs N/A N/A
For loop (Parser) 17.2±0.35µs 11.1±0.16µs +54.95%
Goal Symbols (Parser) 11.7±0.16µs 7.5±0.14µs +56.00%
Hello World (Lexer) 674.6±14.05ns N/A N/A
Hello World (Parser) 2.9±0.06µs 1951.0±36.65ns +48.64%
Long file (Parser) 731.9±12.59ns 5.7±0.06ms -99.99%
Mini js (Execution) 632.6±19.42µs 637.5±16.55µs -0.77%
Mini js (Full) 918.4±15.02µs 910.8±17.31µs +0.83%
Mini js (Parser) 30.0±0.62µs 19.5±0.26µs +53.85%
Number Object Access (Execution) 3.5±0.14µs 3.7±0.06µs -5.41%
Number Object Access (Full) 251.2±5.26µs 251.6±4.30µs -0.16%
Object Creation (Execution) 4.2±0.12µs 4.5±0.06µs -6.67%
Object Creation (Full) 295.4±9.81µs 308.6±3.71µs -4.28%
RegExp (Execution) 61.7±2.37µs 63.6±1.33µs -2.99%
RegExp (Full) 401.4±6.40µs 405.3±7.38µs -0.96%
RegExp Literal (Execution) 63.6±2.40µs 67.2±1.05µs -5.36%
RegExp Literal (Full) 322.9±9.85µs 323.9±6.84µs -0.31%
RegExp Literal Creation (Execution) 60.8±2.27µs 63.9±1.39µs -4.85%
RegExp Literal Creation (Full) 402.1±6.16µs 405.0±5.71µs -0.72%
Static Object Property Access (Execution) 4.5±0.12µs 4.7±0.07µs -4.26%
Static Object Property Access (Full) 245.5±7.10µs 250.2±4.57µs -1.88%
String Object Access (Execution) 6.5±0.20µs 6.7±0.14µs -2.99%
String Object Access (Full) 256.5±5.46µs 251.4±6.60µs +2.03%
String comparison (Execution) 5.8±0.18µs 6.0±0.13µs -3.33%
String comparison (Full) 259.5±2.12µs 253.1±5.22µs +2.53%
String concatenation (Execution) 4.8±0.15µs 4.8±0.11µs 0.00%
String concatenation (Full) 252.6±4.85µs 246.7±5.79µs +2.39%
String copy (Execution) 3.6±0.12µs 3.7±0.07µs -2.70%
String copy (Full) 249.9±11.91µs 249.4±4.31µs +0.20%
Symbols (Execution) 2.9±0.09µs 3.1±0.06µs -6.45%
Symbols (Full) 221.5±6.87µs 231.7±4.18µs -4.40%

@github-actions
Copy link

Benchmark for 52dedb1

Click to view benchmark
Test PR Benchmark Master Benchmark %
Arithmetic operations (Execution) 352.3±18.02ns 379.9±23.65ns -7.27%
Arithmetic operations (Full) 245.3±10.55µs 256.3±11.59µs -4.29%
Array access (Execution) 7.5±0.23µs 8.4±0.38µs -10.71%
Array access (Full) 269.8±15.01µs 280.7±8.39µs -3.88%
Array creation (Execution) 3.0±0.09ms 3.2±0.07ms -6.25%
Array creation (Full) 3.2±0.10ms 3.6±0.23ms -11.11%
Array pop (Execution) 1053.5±48.50µs 1138.8±44.12µs -7.49%
Array pop (Full) 1465.8±101.23µs 1507.3±67.91µs -2.75%
Boolean Object Access (Execution) 4.9±0.23µs 4.9±0.15µs 0.00%
Boolean Object Access (Full) 276.1±13.25µs 269.4±8.50µs +2.49%
Clean js (Execution) 716.3±38.85µs 747.5±17.67µs -4.17%
Clean js (Full) 975.0±49.02µs 1028.8±46.99µs -5.23%
Clean js (Parser) 37.6±1.44µs 23.0±0.54µs +63.48%
Create Realm 410.1±10.22ns 452.7±26.05ns -9.41%
Dynamic Object Property Access (Execution) 5.1±0.19µs 5.5±0.20µs -7.27%
Dynamic Object Property Access (Full) 264.6±18.22µs 274.2±9.17µs -3.50%
Expression (Lexer) 1625.8±47.81ns N/A N/A
Expression (Parser) 6.7±0.37µs 4.4±0.21µs +52.27%
Fibonacci (Execution) 804.3±33.78µs 880.6±45.06µs -8.66%
Fibonacci (Full) 1127.7±83.07µs 1209.1±43.47µs -6.73%
For loop (Execution) 20.5±0.62µs 21.9±0.87µs -6.39%
For loop (Full) 290.9±23.70µs 292.9±9.39µs -0.68%
For loop (Lexer) 3.2±0.05µs N/A N/A
For loop (Parser) 19.3±4.57µs 11.6±0.25µs +66.38%
Goal Symbols (Parser) 12.4±0.61µs 7.9±0.20µs +56.96%
Hello World (Lexer) 721.0±42.76ns N/A N/A
Hello World (Parser) 2.9±0.18µs 2.1±0.06µs +38.10%
Long file (Parser) 811.0±55.97ns 6.1±0.09ms -99.99%
Mini js (Execution) 638.4±36.28µs 669.1±14.00µs -4.59%
Mini js (Full) 919.1±31.74µs 942.3±19.69µs -2.46%
Mini js (Parser) 33.1±1.85µs 20.4±0.37µs +62.25%
Number Object Access (Execution) 3.7±0.23µs 3.9±0.18µs -5.13%
Number Object Access (Full) 279.3±7.54µs 261.7±7.45µs +6.73%
Object Creation (Execution) 4.3±0.17µs 4.7±0.23µs -8.51%
Object Creation (Full) 311.5±21.47µs 318.9±11.49µs -2.32%
RegExp (Execution) 62.5±3.69µs 69.2±7.89µs -9.68%
RegExp (Full) 412.4±24.44µs 429.6±11.50µs -4.00%
RegExp Literal (Execution) 64.3±2.70µs 72.4±4.23µs -11.19%
RegExp Literal (Full) 345.6±17.46µs 352.4±6.70µs -1.93%
RegExp Literal Creation (Execution) 61.1±1.89µs 67.5±2.72µs -9.48%
RegExp Literal Creation (Full) 430.4±18.63µs 429.1±12.96µs +0.30%
Static Object Property Access (Execution) 4.6±0.24µs 5.1±0.16µs -9.80%
Static Object Property Access (Full) 243.9±8.78µs 268.6±12.90µs -9.20%
String Object Access (Execution) 6.7±0.30µs 7.1±0.21µs -5.63%
String Object Access (Full) 270.3±27.05µs 269.3±10.88µs +0.37%
String comparison (Execution) 6.2±0.41µs 6.7±0.37µs -7.46%
String comparison (Full) 275.9±16.23µs 269.8±6.32µs +2.26%
String concatenation (Execution) 5.0±0.43µs 5.4±0.14µs -7.41%
String concatenation (Full) 272.0±11.86µs 271.6±9.92µs +0.15%
String copy (Execution) 3.8±0.20µs 4.0±0.12µs -5.00%
String copy (Full) 265.6±7.95µs 259.3±6.60µs +2.43%
Symbols (Execution) 2.9±0.09µs 3.1±0.15µs -6.45%
Symbols (Full) 228.3±10.09µs 246.8±11.39µs -7.50%

@github-actions
Copy link

Benchmark for ca94d59

Click to view benchmark
Test PR Benchmark Master Benchmark %
Arithmetic operations (Execution) 361.0±35.58ns 418.0±32.66ns -13.64%
Arithmetic operations (Full) 243.4±13.18µs 252.1±19.71µs -3.45%
Array access (Execution) 8.2±0.50µs 8.3±0.62µs -1.20%
Array access (Full) 267.6±20.41µs 269.3±15.67µs -0.63%
Array creation (Execution) 3.1±0.21ms 3.0±0.18ms +3.33%
Array creation (Full) 3.3±0.35ms 3.3±0.19ms 0.00%
Array pop (Execution) 1117.5±120.67µs 1119.5±71.15µs -0.18%
Array pop (Full) 1477.0±134.34µs 1392.4±86.29µs +6.08%
Boolean Object Access (Execution) 4.7±0.36µs 4.7±0.64µs 0.00%
Boolean Object Access (Full) 267.1±13.96µs 248.2±18.41µs +7.61%
Clean js (Execution) 711.8±48.10µs 695.0±48.82µs +2.42%
Clean js (Full) 997.9±56.48µs 1009.6±76.07µs -1.16%
Clean js (Parser) 34.4±3.29µs 22.3±1.58µs +54.26%
Create Realm 440.1±30.42ns 463.1±35.13ns -4.97%
Dynamic Object Property Access (Execution) 5.6±0.37µs 5.5±0.34µs +1.82%
Dynamic Object Property Access (Full) 262.3±22.08µs 253.4±20.72µs +3.51%
Expression (Lexer) 1497.1±119.79ns N/A N/A
Expression (Parser) 6.1±0.61µs 4.0±0.35µs +52.50%
Fibonacci (Execution) 863.9±44.50µs 855.3±68.25µs +1.01%
Fibonacci (Full) 1130.8±86.01µs 1136.8±72.04µs -0.53%
For loop (Execution) 22.3±1.46µs 21.2±1.82µs +5.19%
For loop (Full) 288.0±23.36µs 283.5±20.88µs +1.59%
For loop (Lexer) 3.0±0.19µs N/A N/A
For loop (Parser) 18.4±1.25µs 11.1±0.76µs +65.77%
Goal Symbols (Parser) 11.7±0.70µs 7.3±0.55µs +60.27%
Hello World (Lexer) 626.2±49.47ns N/A N/A
Hello World (Parser) 2.7±0.22µs 1844.7±120.53ns +46.37%
Long file (Parser) 778.5±64.65ns 5.9±0.33ms -99.99%
Mini js (Execution) 663.1±47.85µs 665.3±44.82µs -0.33%
Mini js (Full) 887.3±57.32µs 922.3±87.46µs -3.79%
Mini js (Parser) 30.2±2.45µs 19.4±1.44µs +55.67%
Number Object Access (Execution) 3.8±0.30µs 3.8±0.33µs 0.00%
Number Object Access (Full) 271.2±16.65µs 246.4±16.89µs +10.06%
Object Creation (Execution) 4.7±0.59µs 4.6±0.41µs +2.17%
Object Creation (Full) 303.8±23.01µs 306.0±18.02µs -0.72%
RegExp (Execution) 67.0±4.63µs 66.4±5.02µs +0.90%
RegExp (Full) 379.7±32.83µs 404.8±37.24µs -6.20%
RegExp Literal (Execution) 68.6±4.70µs 70.6±4.43µs -2.83%
RegExp Literal (Full) 338.4±30.43µs 324.3±22.39µs +4.35%
RegExp Literal Creation (Execution) 63.5±4.22µs 65.8±4.08µs -3.50%
RegExp Literal Creation (Full) 408.6±33.06µs 401.3±30.49µs +1.82%
Static Object Property Access (Execution) 5.7±0.54µs 5.0±0.26µs +14.00%
Static Object Property Access (Full) 268.6±18.84µs 253.6±14.63µs +5.91%
String Object Access (Execution) 6.6±0.62µs 7.1±0.54µs -7.04%
String Object Access (Full) 267.3±22.06µs 243.4±18.62µs +9.82%
String comparison (Execution) 6.3±0.49µs 6.4±0.45µs -1.56%
String comparison (Full) 263.1±16.00µs 254.5±18.33µs +3.38%
String concatenation (Execution) 5.4±0.35µs 5.2±0.31µs +3.85%
String concatenation (Full) 261.4±18.43µs 248.1±17.08µs +5.36%
String copy (Execution) 3.9±0.31µs 3.8±0.29µs +2.63%
String copy (Full) 250.7±16.96µs 250.6±19.45µs +0.04%
Symbols (Execution) 3.4±0.29µs 3.3±0.26µs +3.03%
Symbols (Full) 231.0±17.37µs 227.8±18.63µs +1.40%

@github-actions
Copy link

Benchmark for ca94d59

Click to view benchmark
Test PR Benchmark Master Benchmark %
Arithmetic operations (Execution) 360.2±13.54ns 344.8±13.58ns +4.47%
Arithmetic operations (Full) 259.0±17.45µs 248.3±11.23µs +4.31%
Array access (Execution) 8.5±0.31µs 8.7±0.38µs -2.30%
Array access (Full) 286.7±6.79µs 275.0±23.85µs +4.25%
Array creation (Execution) 3.2±0.11ms 3.2±0.13ms 0.00%
Array creation (Full) 3.5±0.12ms 3.4±0.13ms +2.94%
Array pop (Execution) 1123.8±40.95µs 1137.8±43.63µs -1.23%
Array pop (Full) 1501.9±60.61µs 1411.0±58.07µs +6.44%
Boolean Object Access (Execution) 5.0±0.40µs 4.7±0.20µs +6.38%
Boolean Object Access (Full) 278.7±12.56µs 257.3±10.39µs +8.32%
Clean js (Execution) 739.1±41.94µs 774.3±29.93µs -4.55%
Clean js (Full) 1008.8±41.78µs 978.6±35.99µs +3.09%
Clean js (Parser) 37.0±1.40µs 23.3±0.90µs +58.80%
Create Realm 464.7±17.31ns 449.5±25.93ns +3.38%
Dynamic Object Property Access (Execution) 5.7±0.22µs 5.5±0.18µs +3.64%
Dynamic Object Property Access (Full) 272.7±16.36µs 268.1±17.52µs +1.72%
Expression (Lexer) 1516.8±74.72ns N/A N/A
Expression (Parser) 7.0±0.34µs 4.7±0.14µs +48.94%
Fibonacci (Execution) 892.2±37.48µs 903.4±31.52µs -1.24%
Fibonacci (Full) 1190.4±45.36µs 1154.5±47.61µs +3.11%
For loop (Execution) 22.7±0.75µs 23.3±1.24µs -2.58%
For loop (Full) 283.0±12.92µs 287.6±11.17µs -1.60%
For loop (Lexer) 3.0±0.14µs N/A N/A
For loop (Parser) 18.1±0.91µs 12.0±0.36µs +50.83%
Goal Symbols (Parser) 12.0±0.43µs 7.7±0.43µs +55.84%
Hello World (Lexer) 682.6±23.70ns N/A N/A
Hello World (Parser) 3.0±0.12µs 2.1±0.08µs +42.86%
Long file (Parser) 763.6±39.22ns 5.9±0.17ms -99.99%
Mini js (Execution) 652.7±26.79µs 689.4±34.08µs -5.32%
Mini js (Full) 915.8±34.92µs 916.5±33.67µs -0.08%
Mini js (Parser) 32.6±1.04µs 20.7±0.93µs +57.49%
Number Object Access (Execution) 3.9±0.33µs 3.9±0.23µs 0.00%
Number Object Access (Full) 274.9±10.72µs 258.1±9.44µs +6.51%
Object Creation (Execution) 4.8±0.36µs 4.9±0.17µs -2.04%
Object Creation (Full) 319.8±9.78µs 312.4±14.87µs +2.37%
RegExp (Execution) 68.1±3.76µs 67.0±3.16µs +1.64%
RegExp (Full) 415.9±14.43µs 434.6±18.15µs -4.30%
RegExp Literal (Execution) 70.7±3.50µs 70.0±5.07µs +1.00%
RegExp Literal (Full) 336.7±12.98µs 358.9±16.93µs -6.19%
RegExp Literal Creation (Execution) 67.5±2.78µs 67.4±4.45µs +0.15%
RegExp Literal Creation (Full) 409.8±25.01µs 407.1±16.08µs +0.66%
Static Object Property Access (Execution) 5.2±0.21µs 5.3±0.25µs -1.89%
Static Object Property Access (Full) 271.9±11.36µs 257.7±10.05µs +5.51%
String Object Access (Execution) 7.2±0.64µs 7.0±0.18µs +2.86%
String Object Access (Full) 274.6±10.97µs 261.4±14.36µs +5.05%
String comparison (Execution) 6.9±0.28µs 6.5±0.70µs +6.15%
String comparison (Full) 268.9±13.92µs 263.0±10.16µs +2.24%
String concatenation (Execution) 5.7±0.19µs 5.2±0.22µs +9.62%
String concatenation (Full) 263.2±7.67µs 263.8±12.61µs -0.23%
String copy (Execution) 4.2±0.25µs 4.0±0.13µs +5.00%
String copy (Full) 273.5±9.46µs 257.6±10.76µs +6.17%
Symbols (Execution) 3.3±0.15µs 3.2±0.20µs +3.12%
Symbols (Full) 242.5±10.19µs 235.7±9.91µs +2.89%

@HalidOdat
Copy link
Member

I think we can merge this today, if the team is fine with it :)

Lets merge this :)

@HalidOdat HalidOdat merged commit cb93472 into master Aug 27, 2020
@HalidOdat HalidOdat deleted the new_lexer branch August 27, 2020 18:30
@Lan2u Lan2u mentioned this pull request Aug 27, 2020
Closed
@HalidOdat HalidOdat linked an issue Aug 27, 2020 that may be closed by this pull request
Assignment with invalid LHS does not throw SyntaxError #508
Closed
@github-actions
Copy link

Benchmark for 66ba9da

Click to view benchmark
Test PR Benchmark Master Benchmark %
Arithmetic operations (Execution) 347.9±7.53ns 362.2±6.31ns -3.95%
Arithmetic operations (Full) 240.7±3.73µs 239.7±4.96µs +0.42%
Array access (Execution) 7.6±0.12µs 7.8±0.16µs -2.56%
Array access (Full) 262.7±6.45µs 260.9±5.46µs +0.69%
Array creation (Execution) 3.2±0.05ms 3.1±0.04ms +3.23%
Array creation (Full) 3.6±0.07ms 3.6±0.07ms 0.00%
Array pop (Execution) 1148.7±18.75µs 1119.0±18.22µs +2.65%
Array pop (Full) 1575.1±45.00µs 1532.6±24.67µs +2.77%
Boolean Object Access (Execution) 4.5±0.11µs 4.5±0.06µs 0.00%
Boolean Object Access (Full) 250.3±4.92µs 246.5±5.43µs +1.54%
Clean js (Execution) 711.5±18.33µs 696.3±12.38µs +2.18%
Clean js (Full) 978.3±13.88µs 996.1±16.93µs -1.79%
Clean js (Parser) 34.3±0.94µs 22.1±0.48µs +55.20%
Create Realm 444.9±14.88ns 438.3±6.36ns +1.51%
Dynamic Object Property Access (Execution) 4.9±0.11µs 5.1±0.09µs -3.92%
Dynamic Object Property Access (Full) 257.6±4.82µs 246.0±5.59µs +4.72%
Expression (Lexer) 1609.0±30.87ns N/A N/A
Expression (Parser) 6.5±0.10µs 4.5±0.09µs +44.44%
Fibonacci (Execution) 748.6±14.77µs 761.2±11.21µs -1.66%
Fibonacci (Full) 1047.5±21.06µs 1049.7±21.76µs -0.21%
For loop (Execution) 20.7±0.39µs 21.3±0.33µs -2.82%
For loop (Full) 276.3±5.73µs 273.1±4.08µs +1.17%
For loop (Lexer) 3.3±0.05µs N/A N/A
For loop (Parser) 16.9±0.29µs 11.1±0.28µs +52.25%
Goal Symbols (Parser) 11.3±0.21µs 7.3±0.11µs +54.79%
Hello World (Lexer) 670.3±14.38ns N/A N/A
Hello World (Parser) 2.9±0.05µs 1919.4±42.81ns +51.09%
Long file (Parser) 740.1±14.10ns 5.7±0.08ms -99.99%
Mini js (Execution) 628.9±11.42µs 626.9±11.73µs +0.32%
Mini js (Full) 923.5±21.16µs 890.4±15.64µs +3.72%
Mini js (Parser) 30.6±0.58µs 19.1±0.29µs +60.21%
Number Object Access (Execution) 3.6±0.08µs 3.7±0.05µs -2.70%
Number Object Access (Full) 248.2±3.89µs 244.1±4.49µs +1.68%
Object Creation (Execution) 4.2±0.09µs 4.3±0.07µs -2.33%
Object Creation (Full) 302.5±5.75µs 298.9±6.62µs +1.20%
RegExp (Execution) 60.9±2.26µs 61.9±1.65µs -1.62%
RegExp (Full) 404.7±6.32µs 385.2±6.56µs +5.06%
RegExp Literal (Execution) 64.5±1.53µs 64.9±1.58µs -0.62%
RegExp Literal (Full) 323.7±6.80µs 320.9±6.66µs +0.87%
RegExp Literal Creation (Execution) 61.2±1.49µs 62.0±1.45µs -1.29%
RegExp Literal Creation (Full) 390.0±11.94µs 389.4±6.45µs +0.15%
Static Object Property Access (Execution) 4.4±0.09µs 4.5±0.07µs -2.22%
Static Object Property Access (Full) 252.3±5.50µs 241.3±3.32µs +4.56%
String Object Access (Execution) 6.5±0.13µs 6.6±0.10µs -1.52%
String Object Access (Full) 252.2±4.80µs 255.6±4.85µs -1.33%
String comparison (Execution) 5.8±0.12µs 6.0±0.09µs -3.33%
String comparison (Full) 251.4±5.18µs 248.4±5.95µs +1.21%
String concatenation (Execution) 4.6±0.13µs 4.7±0.09µs -2.13%
String concatenation (Full) 246.1±4.28µs 248.7±4.82µs -1.05%
String copy (Execution) 3.6±0.07µs 3.6±0.08µs 0.00%
String copy (Full) 240.4±3.66µs 244.5±5.21µs -1.68%
Symbols (Execution) 3.0±0.06µs 3.0±0.06µs 0.00%
Symbols (Full) 230.2±6.46µs 231.6±11.52µs -0.60%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Razican Razican Razican approved these changes

@jasonwilliams jasonwilliams jasonwilliams approved these changes

@HalidOdat HalidOdat HalidOdat approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request lexer Issues surrounding the lexer
Projects
None yet
Milestone
v0.10.0
4 participants
@Lan2u @Razican @HalidOdat @jasonwilliams