Use a single OffscreenCanvas for rendering in CanvasKit

[harryterkelsen]

This changes CanvasKit's rendering model. Previously, each overlay canvas induced a new WebGL context and SkSurface. Now, there is only ever one SkSurface (backed by an OffscreenCanvas), and the overlays are HTML canvases with a bitmaprenderer context. The content is rendered using the SkSurface and passed to the overlays using transferToImageBitmap. This way, there is only one WebGL context in the lifetime of the app.

Pre-launch Checklist

• I read the Contributor Guide and followed the process outlined there for submitting PRs.
• I read the Tree Hygiene wiki page, which explains my responsibilities.
• I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
• I listed at least one issue that this PR fixes in the description above.
• I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
• I updated/added relevant documentation (doc comments with ///).
• I signed the CLA.
• All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

[flutter-dashboard]

Golden file changes have been found for this pull request. Click here to view and triage (e.g. because this is an intentional change).

If you are still iterating on this change and are not ready to resolve the images on the Flutter Gold dashboard, consider marking this PR as a draft pull request above. You will still be able to view image results on the dashboard, commenting will be silenced, and the check will not try to resolve itself until marked ready for review.

Changes reported for pull request #42672 at sha 2d76c46

[flutter-dashboard]

Golden file changes are available for triage from new commit, Click here to view.

Changes reported for pull request #42672 at sha bb5b746

[flutter-dashboard]

Golden file changes are available for triage from new commit, Click here to view.

Changes reported for pull request #42672 at sha c29808b

[jezell]

So this is an interesting approach, it seems to use a single canvas to render each layer, and then use canvas to canvas drawing operations to output it to a non webgl canvas. The goals of this change, I believe, are to avoid the multiple webGL context limit enforced by the browser. Some observations:

This still doesn't actually solve the root problems related to too many layers when HTML elements get introduced into the DOM and we need overlay canvases. Instead of the browsers built-in error limit, now we have a new software defined overlay canvas limit (maximumOverlays), which is 7 vs the browsers 16 (and flutter's current 8?).

Am I missing something, or will every case that would fail before still fail under this proposed approach, with the caveat that in theory you can make that number whatever you want (if it's exposed) and avoid the hard cap of the browser, but in return we need to do tons of bitmap copy operations which might blow up perf (Firefox and Safari I'm looking at you).

I guess the proof is in the perf as to whether this is an improvement or not (I understand there might be other values to an offscreen canvas being used like rendering off the main thread which could be worth it), but how does it solve the problems behind #37890 and the other related issues if it doesn't reduce the number of overlays required and has an even lower limit of overlays due to the perf required for the copies?

The proposal in #41562 solves the problem that leads to the additional overlays being needed for things like video elements and image elements. Do we need to resurrect that even if this change gets merged?

Instead of going through all this work, or maybe as part of this work, why not just allow the overlays to render themselves into the canvas? The open issues related to Video and Image elements causing canvases to be created would require a lot less copying if they could just paint their media element directly into the canvas instead of copying around everything besides them. Skia supports it already so no reason it can't be supported with a special image constructor in web_ui that accepts an HTMLMediaElement.

[harryterkelsen]

@jezell I have a follow-up change which reduces the overlays induced by HTML elements which uses the algorithm here: https://docs.google.com/document/d/1aCIOEEYe31HqCaCGUKsAmUu9T0U6o2zUMGpqdPEdJoQ/edit#heading=h.kj5jgtvn1u89

[eyebrowsoffire]

@jezell Also, if all you're looking for is a way to create a ui.Image from a browser resource, I recently added a web-specific API for making a ui.Image from an ImageBitmap: #45256

You can use createImageBitmap to make an image bitmap from most browser sources you might care about, such as a video or image element.

[jezell]

@harryterkelsen @eyebrowsoffire awesome news!

[jezell]

@harryterkelsen you can in many cases make a further optimization and reduce the required platform views to one as I did in #41562. As long as the platform views are rectangular, opaque, and ignore pointer events, you can place them behind the canvas and render their region with a clear operation. While some platform views might not meet these conditions, many commonly problematic ones such as camera / video feeds or jpegs do. To the viewer, it appears that they are sitting in the correct location because the things that would have been rendered on top (because they are moved behind the canvas z-index) get erased during the drawing operation instead of creating a new canvas. I believe his approach is superior in these cases as it completely eliminates the need for the overlays when these constraints hold true.

[eyebrowsoffire]

@jezell We have actually discussed the "cutout" approach in the past, but as you've already described it only works on completely opaque rectangular views, so we cannot use it as a general purpose solution for platform views.

To help reassure a bit, we have done benchmarking of this change and haven't found any major issues with the performance of this strategy. This is also roughly equivalent to the strategy we are using in the experimental Skwasm renderer, which has proven to be quite performant so far. The amount of copying happening under the hood is actually quite a bit less than you might imagine, as transferFromImageBitmap does not actually do any copying, but just replaces the backing store of the canvas itself with the contents of the ImageBitmap.

The main thing here is that practically speaking, there are hard limits and very real perf ramifications to creating multiple GL contexts, but those same limitations do not apply to making ImageBitmap objects. I don't know of any upper limit to the number of ImageBitmap objects one can create, but if there is one it's nowhere near the scale of the limit of the number of WebGL contexts that can be created. We have spent some time discussing this with browser folks too. There is a lot of intelligent resource pooling and reuse that happens under the hood in the browser when creating ImageBitmap objects, and they have said this is a reasonable use pattern.

The main thing we achieve from having a single GL context is that we can safely use the same GPU-side resources across multiple overlays. Right now, having multiple GL contexts is unfortunately pretty error prone and has performance cliffs when we have to move resources from one context to another.

Anyway, we really appreciate your engagement (and your patience) on these issues. I'm hoping the new ui_web API might be the best way forward for your use-case to avoid making platform views at all, but I'd be curious to hear about the outcomes of any experimentations with it, so please keep us informed. Thank you!

[jezell]

@eyebrowsoffire I agree that the ImageBitmap approach is the best general purpose solution, looking forward to giving it a shot. Should solve a lot of the cases for us that get us into trouble. Is the video_player web package being updated to use it? I'll work with the flutter_webrtc guys and see if we can get them to move over to it as well. I believe it should be good enough to solve our current blockers.

[eyebrowsoffire]

LGTM! One small comment.

[flutter-dashboard]

Golden file changes are available for triage from new commit, Click here to view.

Changes reported for pull request #42672 at sha 32f1c24