The lexer might create invalid spans

[alexanderkjall]

Describe the bug

Expected an Error, not a panic.

To Reproduce

Can be reproduced with this program

fn main() {
    let data: Vec<u8> = vec![0x2d, 0x2d, 0x0d, 0x00, 0x33];
    if let Ok(s) = std::str::from_utf8(&data) {
        let _ = boa::parse(s);
    }
}

Expected behavior

An Error, not a panic

Build environment (please complete the following information):

• OS: Ubuntu 20.04
• Version: 0.10.0
• Target triple: [e.g. x86_64-unknown-linux-gnu]
• Rustc version: 1.48.0-nightly (d006f5734 2020-08-28)

Additional context

Full stacktrace:

thread 'main' panicked at 'a span cannot start after its end', /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/lexer/mod.rs:189:17
stack backtrace:
   0: std::panicking::begin_panic
             at /home/capitol/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:505
   1: boa::syntax::ast::position::Span::new
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/ast/position.rs:71
   2: boa::syntax::lexer::Lexer<R>::next
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/lexer/mod.rs:189
   3: boa::syntax::parser::cursor::buffered_lexer::BufferedLexer<R>::fill
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/cursor/buffered_lexer/mod.rs:116
   4: boa::syntax::parser::cursor::buffered_lexer::BufferedLexer<R>::peek
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/cursor/buffered_lexer/mod.rs:201
   5: boa::syntax::parser::cursor::Cursor<R>::peek
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/cursor/mod.rs:56
   6: <boa::syntax::parser::expression::left_hand_side::member::MemberExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/left_hand_side/member.rs:64
   7: <boa::syntax::parser::expression::left_hand_side::LeftHandSideExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/left_hand_side/mod.rs:66
   8: <boa::syntax::parser::expression::update::UpdateExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/update.rs:70
   9: <boa::syntax::parser::expression::assignment::exponentiation::ExponentiationExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/assignment/exponentiation.rs:91
  10: <boa::syntax::parser::expression::MultiplicativeExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
  11: <boa::syntax::parser::expression::AdditiveExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
  12: <boa::syntax::parser::expression::ShiftExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
  13: <boa::syntax::parser::expression::RelationalExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
  14: <boa::syntax::parser::expression::EqualityExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
  15: <boa::syntax::parser::expression::BitwiseANDExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
  16: <boa::syntax::parser::expression::BitwiseXORExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
  17: <boa::syntax::parser::expression::BitwiseORExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
  18: <boa::syntax::parser::expression::LogicalANDExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
  19: <boa::syntax::parser::expression::LogicalORExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
  20: <boa::syntax::parser::expression::assignment::conditional::ConditionalExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/assignment/conditional.rs:69
  21: <boa::syntax::parser::expression::assignment::AssignmentExpression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/assignment/mod.rs:175
  22: <boa::syntax::parser::expression::Expression as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/expression/mod.rs:71
  23: <boa::syntax::parser::statement::expression::ExpressionStatement as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/statement/expression/mod.rs:46
  24: <boa::syntax::parser::statement::Statement as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/statement/mod.rs:197
  25: <boa::syntax::parser::statement::StatementListItem as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/statement/mod.rs:379
  26: <boa::syntax::parser::statement::StatementList as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/statement/mod.rs:315
  27: <boa::syntax::parser::ScriptBody as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/mod.rs:148
  28: <boa::syntax::parser::Script as boa::syntax::parser::TokenParser<R>>::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/mod.rs:125
  29: boa::syntax::parser::Parser<R>::parse_all
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/syntax/parser/mod.rs:104
  30: boa::parse
             at /home/capitol/.cargo/registry/src/github.com-1ecc6299db9ec823/Boa-0.10.0/src/lib.rs:73
  31: boa_reproduce::main
             at ./src/main.rs:4
  32: core::ops::function::FnOnce::call_once
             at /home/capitol/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227

[vgel]

I can take a look at this.

[vgel]

Simpler repro:

let str = "-\r3";
crate::parse(str).unwrap(); // panics

[vgel]

Problematic code is syntax::lexer::cursor::Cursor::carriage_return, which is simulating a real carriage return (going back to column 1, but staying on the same line). I don't think this behavior makes sense in a syntax environment? I wouldn't expect the column numbering to reset for a bare \r, and for a \r\n Cursor already resets the column on \n regardless.

Interestingly, node 14.5.0 appears to treat \r as a whole newline, but only when not followed by a \n:

$ echo "1invalid" | node
[stdin]:1
1invalid
^

SyntaxError: Invalid or unexpected token

$ echo "1\rinvalid" | node
[stdin]:2
invalid
^

ReferenceError: invalid is not defined
    at [stdin]:2:1

$ echo "1\r\ninvalid" | node
[stdin]:2
invalid
^

ReferenceError: invalid is not defined
    at [stdin]:2:1

$ echo "1\n\rinvalid" | node
[stdin]:3
invalid
^

ReferenceError: invalid is not defined
    at [stdin]:3:1

Might be trying to recognize (very) old Mac OS 9 line endings, which only used CR (https://en.wikipedia.org/wiki/Newline#Unicode). Seems reasonable to follow that -- we can peek ahead in Cursor to see if the next char is a \n when we get a \r. If that solution sounds good (treat lone \r as newline, but combined \r\n as only single newline), I can make a PR.