JIT: add more inlining bail out checks

[AndyAyersMS]

When inlining, new temp allocation may fail, and callers of methods
that allocate new temps must do suitable checks.

Add a few that were missing.

Fixes #64787.

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

When inlining, new temp allocation may fail, and callers of methods
that allocate new temps must do suitable checks.

Add a few that were missing.

Fixes #64787.

Author:	AndyAyersMS
Assignees:	-
Labels:	area-CodeGen-coreclr
Milestone:	-

[AndyAyersMS]

@EgorBo PTAL
cc @dotnet/jit-contrib

We've known for a while this whole design needs rethinking (#8713). Seems like with temps we can just over-allocate and do this sort of check periodically at a higher level; failed inlines should scrub out their side effects to the locals table.

[jakobbotsch]

I wonder if we could use longjmp or an exception to bail out of inlining in a non-local manner. Is there anything that stops us from doing this given that the design of the JIT is already to leave it to the runtime to clean up after us (so not executing destructors and the likes is not super problematic)?

[AndyAyersMS]

I wonder if we could use longjmp or an exception to bail out of inlining in a non-local manner. Is there anything that stops us from doing this given that the design of the JIT is already to leave it to the runtime to clean up after us (so not executing destructors and the likes is not super problematic)?

Probably doable. I think the only clunky part is we'd need to generalize runWithErrorTrap to recognize yet another type of handled failure.

Not sure if the runtime needs to get involved in cleanup as there is only one jit request and I think the jit interface state/caching is tied to that, not to the particular compiler object the jit is using.