…+26.

🛠 [Frontend] Ensure commas inserted by #embed are properly serialized to output

⚡ [Lex] Better reservations for improved performance/memory usage.

🛠 [Lex, Frontend] Remove comma hardcoding since we are servicing a full file

apply suggestions from git-clang-format

This way we're not tied to a SmallVector<Token, 2> specifically in callers.

If the user passes -fno-builtin, then the call to getValue() will
assert due to an out of bounds access. So we check to see which form
the user passes (-fno-builtin or -fno-builtin-pp_embed).

Additionally, we need to round trip the argument properly depending on
which form the user passed.

This fixes a few hundred failing test cases for me; still several left
failing though.

The test needs to care about the three new predefined macros.

This is a weird builtin function that's more like __builtin_offsetof
in that it takes a type argument. Therefore, it's not really a function
call like other builtins (we wouldn't check its validity in
SemaChecking.cpp).

There is likely more work to be done here to split parsing and semantic
concerns. This also pointed out an issue where __builtin_pp_embed seems
to have a non-void return type, but who knows what it actually returns
as a value.

The options don't make sense outside of Clang currently.

The issue would previously manifest in -E output where we would print:
  1>
instead of:
  <built-in:embed:1>

We do not want to have a builtin for embed because it poses too many
problems. For example, it allows for recursive embeds through:

  __builtin_pp_embed(
    #embed "file containing a file name.txt"
    ,
    #embed "file containing base64 data.txt"
  )

Instead, we'll use annotation tokens to pass information from the
preprocessor into the parser.

Previously, we would scan the prefix and suffix tokens to see if they
were a simple sequence of integer literals and commas, and if so, we
would encode their data as part of the binary contents for the embed
expression. If they were not a simple sequence, we would fall back to
the "naive" implementation.

This removes the naive implementation entirely; that would produce six
tokens for every byte in the embedded file:
  ( unsigned char ) <value> ,
which is not going to have acceptable memory overhead for files over a
particular size.

Now, we stream the prefix tokens first, then the embed expression
tokens, then the suffix tokens. This way, the parser always sees the
correct prefix and suffix and non-idiomatic uses won't suffer an
extreme compile-time performance penalty.

This was added in an earlier refactoring to support diagnosing issues
with calls to __builtin__pp_embed, no longer needed now that we've
removed the builtin.

The previous diagnostic wording was incorrect and untested; this
updates the wording, changes the groups the diagnostics are in, adds an
explicit test for the diagnostics, and corrects test RUN lines to
specify the standards mode (to disable diagnostics we don't intend to
test).

We now correctly emit the #embed directive in -dE mode, including all
parameters. We now also stop emitting internal tokens like the base64
encoded data when in regular -E mode.

This also updates the EmbedDirective() callback parameter list so that
information about the parameters is passed to callbacks.

This identified a potential issue with the clang::offset parameter,
which behaves in a somewhat surprising manner when combined with a
limit parameter. The test case now has a comment explaining the issue.

Add an iterator interface to PPEmbedExpr to iterate through the
contents of the resource as though they were IntegerLiteral AST nodes,
but without creating one AST node per byte in the resource.

We now iterate over those fake nodes from some of the recrusive AST
visitors, like constant expression evaluation, so that we can remove
some kludges.

We were calling LookupEmbedFile() differently between the two uses and
that difference resulted in situations where __has_embed would return
__STDC_EMBED_FOUND__ and #embed would then say the file cannot be
found.

This also corrects the path handling for quoted lookup -- we would
previously find files from `--embed-dir=` before we would find files
next to the source file. Include path search heuristics are that quoted
header names are looked up starting from the directory containing the
source file.

This adds back some changes from d6c3c90
that were necessary after all, but were untested.

Both __has_embed and #embed will contribute to the dependency file.

The changes weren't correct and the clang-format folks are in a better
position to lead this effort anyway. It will be handled in a follow-up.

Instead, pass the binary data. If we want to add support for something
like #embed_base64, we can do so by taking the base64 encoded data from
the directive and decoding it into a binary blob and pass it off like
we would for #embed.

These changes also remove diagnostics related to CHAR_BIT != 8; we do
not support any such targets currently and so that code is untestable.

We don't support any CHAR_BIT other than 8, so none of this code could
be tested. Pulling it out for now; we can bring it back if we ever
start to support a host or target where this is possible.

Note, the original code was also quite wrong, it was using the CHAR_BIT
for the machine compiling Clang, not the CHAR_BIT for the machine
running Clang or the target architecture.

There was an issue with parsing the binary data from ParseExpr.cpp; we
want to parse it as an unevaluated string, otherwise the string literal
will contain space for a null character. But we can't parse it as an
unevaluated string literal because there are assertions ensuring we
don't get one of those in a place where a string should be evaluated,
like an InitListExpr.

This tosses the custom parsing logic and instead uses an annotation
token value to squirrel the data from the preprocessor to the parser.
This does involve copying data -- we cannot pass a StringRef to the
memory buffer containing the file because that reference does not live
long enough.

1) The code was originally passing OpenFile as the wrong argument to
getFileRef. This was corrected.
2) The code for processing #embed was asking for the file to not be
opened, which was corrected.
3) The logic for looking up the files was simplified a bit.

There may be further work necessary here as I'd like to retain the
opened file as part of the annotation token data still.

Instead, we open the file through the SourceManager so we can pass a
StringRef for the buffer contents through the annotation token instead
of copying data into a buffer.

This adds test to ensure we properly handle eliding prefix and suffix
tokens for an empty file, and that we do not allow the string literal
implementation detail to leak into letting the user initialize a
pointer from an embedded resource unless the "pointer" is a single null
byte.

I started adding comprehensive tests and ran into a significant number
of crashes with the original approach, so this rewrites the parsing
code to be a bit more generic and hopefully easier to reason about. It
also fixes numerous parsing crashes.

Note: __has_embed should get the same set of comprehensive tests; that
will be done in a follow-up as the parsing code is shared between

This coverage is duplicated within this test file.

We can't test this in the same file as with chevrons because the file
not being found is a fatal error (so the first diagnostic will swallow
all the rest).

This also starts grabbing test cases from the standard.

Note, the embedded resources are all made up data (complying with what
the standard implied the contents would be).

This specifies that we're backporting support for #embed to earlier C
modes and C++. It also documents the `clang::offset` parameter
extension.

This looks to have been a work in progress from the initial commit but
is not needed.

This addresses the issues found by Misc/warning-flags.c

WG21 has not adopted this feature yet so we should not provide a
feature test macro suggesting otherwise. We'll wait for WG21 to
finalize their plans, then add it back.

This allows us to iterate over even larger files without adding extra
memory overhead (thanks to alignment of other fields).