[Impeller] Scale down the Gaussian blur in both directions prior to blurring. #131580

bdero · 2023-07-30T18:42:16Z

... as opposed to having only two passes and downscaling in each blur direction per-pass.

This is what Skia does, and would allow us to reduce fragment load in two significant ways:

The first pass will benefit from having fewer fragments to compute colors.
On both passes, each fragment will require proportionally fewer samples.

Doing #131579 would help to make this less confusing.

…n. (#45520) Resolves flutter/flutter#122658. Resolves remaining blur issues visible in: flutter/flutter#132839 The current blur style implementation hasn't been working since we added the downscaling optimization, and it will continue not working after we switch to flutter/flutter#131580. In it's current state, it at best causes awful looking undesirable results that users are forced to work around anyhow. We should just remove it in the meantime. Before: ![image](https://github.com/flutter/engine/assets/919017/fee57875-bd56-4a97-b8c7-353328935298) After: ![image](https://github.com/flutter/engine/assets/919017/03d6a6dd-3789-4629-bae2-61cf6f5bb054)

bdero · 2023-10-23T19:07:09Z

To be more specific, the big difference is that Skia downscales the source image before sampling from it and reduces the kernel size to match, whereas our approach scales down while rendering the blur pass into the texture, and so we must keep the original kernel size to get the right result.

So our current downscale optimization reduces the number of blurred pixels we need to draw, but the pixels we do draw still need to get fully integrated. And so the downscale is taking an already correct blur and causing it to be more blurry.

gaaclarke · 2023-10-24T22:40:08Z

@bdero to make sure I understand, currently it works like this:

input[586x678] -> pass1[97x678] ->pass2[87x102]

Is the proposal that we shrink the dimensions in the first pass such that it looks more like this, using nearest-neighbor to downsample between input and pass1 while calculating pass1?

input[586x678] -> pass1[Nx87] ->pass2[87x102]

Or are you suggesting something like this where there is a separate pass to downsample the input?

input[586x678] -> input'[87x102] -> pass1[Nx102] ->pass2[87x102]

How do you expect to do this downsampling, nearest, linear, cubic?

Do you have a link to the skia implementation?

gaaclarke · 2023-10-26T00:05:49Z

We had a sync, here are some quick notes:

The idea is to downscale first
The scalar of the kernel should be proportional to the scale of the image
We can ping pong textures, or use read-from-resolve as an optimization where available
The scaling of the image should use the linear algorithm
watch out for:
- coverage_hint, which limits the blur so it doesn't go beyond edges sometimes
- effect transforms that are not axis aligned since the blur is hardwired to go in the +x +y directions

in pursuit of: flutter/flutter#131580 I'm starting to write tests for the new blur, but I wanted to make tests for the old blur to make sure I understood the coverage rules. I decided to check this in separately while I work on the other blur. Even though the goal is to delete this class eventually. Most of this work transfers over to the new blur. [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style

bdero · 2023-11-10T18:43:27Z

re: our conversation earlier:

The playground with all the interactive doodads is Play/EntityTest.GaussianBlurFilter/Metal: https://github.com/flutter/engine/blob/77349dc8e27b7799345d57c2f0eb7668a5521b10/impeller/entity/entity_unittests.cc#L983-L1135

The effect transform should be applied to the effect (i.e. the blur sigma vector) in addition to the entity transform.
There are exceptions for SaveLayer filters and backdrop filters because the geometry we generate for those draws don't inherit the current transform of the canvas.

So for most things, the effect transform is just identity. A bunch of canvas operations call Paint::WithFilters to wrap the filter: https://github.com/flutter/engine/blob/42952d9a50f4d4bcb763c61193aac8f7784f9493/impeller/aiks/paint.cc#L75-L76
For both SaveLayer filters and backdrop filters, the Entities we generate for those while rendering are not given the Canvas CTM as their transform (their source geometry is generated in a different space on the fly), and so their effect transforms are set to the Canvas CTM:
- For backdrop filters, that originates here: https://github.com/flutter/engine/blob/42952d9a50f4d4bcb763c61193aac8f7784f9493/impeller/entity/entity_pass.cc#L126-L129
- For SaveLayer filters, that originates here: https://github.com/flutter/engine/blob/42952d9a50f4d4bcb763c61193aac8f7784f9493/impeller/entity/entity_pass.cc#L651-L655

…the blur (#47576) A partial implementation of the new gaussian blur effect. This should perform enough of the code to start getting some performance numbers. Known outstanding problems: 1) The edges of the blur are clipped. I have notes on how I plan on expanding render space in the PR. 1) Animating the sigma causes some "jumping around artifacts" resulting from the downsampling (maybe the discrete nature of texture pixel size?) 1) Coverage hints are ignored. I think depth tests will make that not much of an issue. 1) We aren't ping ponging textures yet 1) The snapshot's transform is ignored. issue: flutter/flutter#131580 [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style

gaaclarke · 2023-11-16T22:46:22Z

Known outstanding problems after flutter/engine#47576:

The edges of the blur are clipped. I have notes on how I plan on expanding render space in the PR.
Animating the sigma causes some "jumping around artifacts" resulting from the downsampling (maybe the discrete nature of texture pixel size?)
Coverage hints are ignored. I think depth tests will make that not much of an issue.
We aren't ping ponging textures yet
The snapshot's transform is ignored.

This makes the blurred item have a halo effect that is rendered outside of the bounds of the original thing that was rendered. issue: flutter/flutter#131580 [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style

issue: flutter/flutter#131580 [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style

This will run the blur passes on a subset of the texture passed into it that will actually be blurred. This is an optimization for backdrop filters. issue: flutter/flutter#131580 test coverage: devicelab `backdrop_filter_perf_ios__timeline_summary` ## performance results This results in a 50% reduction in average GPU time in our benchmark, 79% reduction in the "99% percentile". ``` BEFORE "average_vsync_transitions_missed": 3.889423076923077, "90th_percentile_vsync_transitions_missed": 4.0, "99th_percentile_vsync_transitions_missed": 4.0, "average_vsync_frame_lag": 0.0, "90th_percentile_vsync_frame_lag": 0.0, "99th_percentile_vsync_frame_lag": 0.0, "average_layer_cache_count": 0.0, "90th_percentile_layer_cache_count": 0.0, "99th_percentile_layer_cache_count": 0.0, "average_frame_request_pending_latency": 16654.368333333332, "90th_percentile_frame_request_pending_latency": 16692.0, "99th_percentile_frame_request_pending_latency": 16749.0, "worst_layer_cache_count": 0.0, "average_layer_cache_memory": 0.0, "90th_percentile_layer_cache_memory": 0.0, "99th_percentile_layer_cache_memory": 0.0, "worst_layer_cache_memory": 0.0, "average_picture_cache_count": 0.0, "90th_percentile_picture_cache_count": 0.0, "99th_percentile_picture_cache_count": 0.0, "worst_picture_cache_count": 0.0, "average_picture_cache_memory": 0.0, "90th_percentile_picture_cache_memory": 0.0, "99th_percentile_picture_cache_memory": 0.0, "worst_picture_cache_memory": 0.0, "total_ui_gc_time": 0.659, "30hz_frame_percentage": 0.0, "60hz_frame_percentage": 100.0, "80hz_frame_percentage": 0.0, "90hz_frame_percentage": 0.0, "120hz_frame_percentage": 0.0, "illegal_refresh_rate_frame_count": 0, "average_gpu_frame_time": 52.13341346153846, "90th_percentile_gpu_frame_time": 62.5, "99th_percentile_gpu_frame_time": 62.5, "worst_gpu_frame_time": 62.5, "average_cpu_usage": 69.08979595918369, "90th_percentile_cpu_usage": 70.4, "99th_percentile_cpu_usage": 71.699999, "average_gpu_usage": 100.0, "90th_percentile_gpu_usage": 100.0, "99th_percentile_gpu_usage": 100.0, "average_memory_usage": 118.79942602040816, "90th_percentile_memory_usage": 138.125, "99th_percentile_memory_usage": 143.65625 AFTER "average_vsync_transitions_missed": 2.0, "90th_percentile_vsync_transitions_missed": 2.0, "99th_percentile_vsync_transitions_missed": 2.0, "average_vsync_frame_lag": 0.0, "90th_percentile_vsync_frame_lag": 0.0, "99th_percentile_vsync_frame_lag": 0.0, "average_layer_cache_count": 0.0, "90th_percentile_layer_cache_count": 0.0, "99th_percentile_layer_cache_count": 0.0, "average_frame_request_pending_latency": 16635.025, "90th_percentile_frame_request_pending_latency": 16715.0, "99th_percentile_frame_request_pending_latency": 16802.0, "worst_layer_cache_count": 0.0, "average_layer_cache_memory": 0.0, "90th_percentile_layer_cache_memory": 0.0, "99th_percentile_layer_cache_memory": 0.0, "worst_layer_cache_memory": 0.0, "average_picture_cache_count": 0.0, "90th_percentile_picture_cache_count": 0.0, "99th_percentile_picture_cache_count": 0.0, "worst_picture_cache_count": 0.0, "average_picture_cache_memory": 0.0, "90th_percentile_picture_cache_memory": 0.0, "99th_percentile_picture_cache_memory": 0.0, "worst_picture_cache_memory": 0.0, "total_ui_gc_time": 1.732, "30hz_frame_percentage": 0.0, "60hz_frame_percentage": 100.0, "80hz_frame_percentage": 0.0, "90hz_frame_percentage": 0.0, "120hz_frame_percentage": 0.0, "illegal_refresh_rate_frame_count": 0, "average_gpu_frame_time": 25.01558603491272, "90th_percentile_gpu_frame_time": 31.25, "99th_percentile_gpu_frame_time": 31.25, ``` [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style

issue: flutter/flutter#131580 This factors out the gaussian blur fragment shader to be a more generic kernel fragment shader. We can't use this for the old gaussian blur because it's kernel is unbounded in one dimension. I implemented this by passing in the coefficients into the UBO so it can't support that. Profile after change: ``` "average_vsync_transitions_missed": 2.0, "90th_percentile_vsync_transitions_missed": 2.0, "99th_percentile_vsync_transitions_missed": 2.0, "average_vsync_frame_lag": 0.0, "90th_percentile_vsync_frame_lag": 0.0, "99th_percentile_vsync_frame_lag": 0.0, "average_layer_cache_count": 0.0, "90th_percentile_layer_cache_count": 0.0, "99th_percentile_layer_cache_count": 0.0, "average_frame_request_pending_latency": 16636.624373956594, "90th_percentile_frame_request_pending_latency": 16718.0, "99th_percentile_frame_request_pending_latency": 16761.0, "worst_layer_cache_count": 0.0, "average_layer_cache_memory": 0.0, "90th_percentile_layer_cache_memory": 0.0, "99th_percentile_layer_cache_memory": 0.0, "worst_layer_cache_memory": 0.0, "average_picture_cache_count": 0.0, "90th_percentile_picture_cache_count": 0.0, "99th_percentile_picture_cache_count": 0.0, "worst_picture_cache_count": 0.0, "average_picture_cache_memory": 0.0, "90th_percentile_picture_cache_memory": 0.0, "99th_percentile_picture_cache_memory": 0.0, "worst_picture_cache_memory": 0.0, "total_ui_gc_time": 1.65, "30hz_frame_percentage": 0.0, "60hz_frame_percentage": 100.0, "80hz_frame_percentage": 0.0, "90hz_frame_percentage": 0.0, "120hz_frame_percentage": 0.0, "illegal_refresh_rate_frame_count": 0, "average_gpu_frame_time": 21.875, "90th_percentile_gpu_frame_time": 31.25, "99th_percentile_gpu_frame_time": 31.25, "worst_gpu_frame_time": 31.25, "average_cpu_usage": 162.86326532653064, "90th_percentile_cpu_usage": 167.3, "99th_percentile_cpu_usage": 168.500003, "average_gpu_usage": 93.26530612244898, "90th_percentile_gpu_usage": 94.0, "99th_percentile_gpu_usage": 95.0, "average_memory_usage": 113.56696428571429, "90th_percentile_memory_usage": 145.484375, "99th_percentile_memory_usage": 152.75 ``` This is a 13% (21.25 / 25 ) drop in `average_gpu_frame_time` from the last profile in #49299. [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style

This new blur should perform faster since it scales down the image before blurring it. Jonah did early testing of it and found it to be faster. Scrolling around with the blur perf bug it seems faster. It also has a wider test bed and is hopefully easier to maintain since it contains all of its logic for both directions. testing: There are existing blur tests and we've backfilled more as we've added features to this blur. fixes flutter/flutter#131580 fixes flutter/flutter#138259 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide] and the [C++, Objective-C, Java style guides]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I added new tests to check the change I am making or feature I am adding, or the PR is [test-exempt]. See [testing the engine] for instructions on writing and running engine tests. - [x] I updated/added relevant documentation (doc comments with `///`). - [x] I signed the [CLA]. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord].  [Contributor Guide]: https://github.com/flutter/flutter/wiki/Tree-hygiene#overview [Tree Hygiene]: https://github.com/flutter/flutter/wiki/Tree-hygiene [test-exempt]: https://github.com/flutter/flutter/wiki/Tree-hygiene#tests [Flutter Style Guide]: https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style [testing the engine]: https://github.com/flutter/flutter/wiki/Testing-the-engine [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/wiki/Chat

This new blur should perform faster since it scales down the image before blurring it. Jonah did early testing of it and found it to be faster. Scrolling around with the blur perf bug it seems faster. It also has a wider test bed and is hopefully easier to maintain since it contains all of its logic for both directions. testing: There are existing blur tests and we've backfilled more as we've added features to this blur. fixes flutter/flutter#131580 fixes flutter/flutter#138259 - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide] and the [C++, Objective-C, Java style guides]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I added new tests to check the change I am making or feature I am adding, or the PR is [test-exempt]. See [testing the engine] for instructions on writing and running engine tests. - [x] I updated/added relevant documentation (doc comments with `///`). - [x] I signed the [CLA]. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord].  [Contributor Guide]: https://github.com/flutter/flutter/wiki/Tree-hygiene#overview [Tree Hygiene]: https://github.com/flutter/flutter/wiki/Tree-hygiene [test-exempt]: https://github.com/flutter/flutter/wiki/Tree-hygiene#tests [Flutter Style Guide]: https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style [testing the engine]: https://github.com/flutter/flutter/wiki/Testing-the-engine [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/wiki/Chat

This is a reland of the new gaussian blur. Changes since revert: 1) Textures are now recycled with ping ponging to reduce memory usage 1) Mipmaps are generated to diminish the shimmering that happens with animated blurs (in metal) --- This new blur should perform faster since it scales down the image before blurring it. Jonah did early testing of it and found it to be faster. Scrolling around with the blur perf bug it seems faster. It also has a wider test bed and is hopefully easier to maintain since it contains all of its logic for both directions. testing: There are existing blur tests and we've backfilled more as we've added features to this blur. fixes flutter/flutter#131580 fixes flutter/flutter#138259 https://github.com/flutter/flutter/wiki/Tree-hygiene#overview [Tree Hygiene]: https://github.com/flutter/flutter/wiki/Tree-hygiene [test-exempt]: https://github.com/flutter/flutter/wiki/Tree-hygiene#tests [Flutter Style Guide]: https://github.com/flutter/flutter/wiki/Style-guide-for-Flutter-repo [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style [testing the engine]: https://github.com/flutter/flutter/wiki/Testing-the-engine [CLA]: https://cla.developers.google.com/ https://github.com/flutter/flutter/wiki/Tree-hygiene#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/wiki/Chat

github-actions · 2024-02-01T01:23:53Z

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

bdero added the e: impeller Impeller rendering backend issues and features requests label Jul 30, 2023

bdero mentioned this issue Jul 30, 2023

[Impeller] Very slow performance of blur with wide gamut enabled #131573

Closed

bdero added P1 High-priority issues at the top of the work list P2 Important issues not at the top of the work list engine flutter/engine repository. See also e: labels. and removed P1 High-priority issues at the top of the work list labels Jul 30, 2023

chinmaygarde added team-engine Owned by Engine team triaged-engine Triaged by Engine team labels Jul 31, 2023

chinmaygarde mentioned this issue Aug 7, 2023

[Impeller] Reduce render-pass overhead when using wide gamut (iOS) #131567

Closed

chinmaygarde added P1 High-priority issues at the top of the work list and removed P2 Important issues not at the top of the work list labels Aug 7, 2023

bdero mentioned this issue Sep 6, 2023

[Impeller] Gaussian blur: Remove the current blur style implementation. flutter/engine#45520

Merged

jonahwilliams self-assigned this Sep 15, 2023

jonahwilliams mentioned this issue Sep 15, 2023

[Impeller] Blur performance issue. #132735

Closed

gaaclarke mentioned this issue Oct 23, 2023

[Impeller] Imparity of blur result with MaskFilter.blur in custom painter with skia on master #127770

Closed

2 tasks

jonahwilliams removed their assignment Oct 26, 2023

gaaclarke mentioned this issue Nov 2, 2023

[Impeller] scales blur coverage to match rendered output flutter/engine#47621

Merged

8 tasks

gaaclarke mentioned this issue Nov 15, 2023

[Impeller] Implements gaussian blur that scales down before applying the blur flutter/engine#47576

Merged

8 tasks

gaaclarke mentioned this issue Nov 16, 2023

[Impeller] implements gaussian "blur halo" flutter/engine#48149

Merged

8 tasks

This was referenced Dec 11, 2023

[Impeller] Turned on new blur. flutter/engine#48472

Merged

[Impeller] Made the new blur support 1D blurs flutter/engine#49001

Merged

auto-submit bot pushed a commit to flutter/engine that referenced this issue Dec 14, 2023

[Impeller] Made the new blur support 1D blurs (#49001)

b19c2f1

issue: flutter/flutter#131580 [C++, Objective-C, Java style guides]: https://github.com/flutter/engine/blob/main/CONTRIBUTING.md#style

gaaclarke closed this as completed in flutter/engine#48472 Dec 14, 2023

gaaclarke reopened this Dec 18, 2023

tugorez closed this as completed in tugorez/engine@d0464cf Dec 19, 2023

bdero reopened this Dec 19, 2023

gaaclarke mentioned this issue Jan 2, 2024

[Impeller] new blur: limit uvs to blur region flutter/engine#49299

Merged

8 tasks

gaaclarke mentioned this issue Jan 4, 2024

[Impeller] migrate blur to calculating coefficients on the cpu flutter/engine#49512

Merged

8 tasks

gaaclarke mentioned this issue Jan 9, 2024

Reland: [Impeller] Turned on new blur. (#48472) flutter/engine#49642

Merged

8 tasks

auto-submit bot closed this as completed in flutter/engine#49642 Jan 18, 2024

github-actions bot locked as resolved and limited conversation to collaborators Feb 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Impeller] Scale down the Gaussian blur in both directions prior to blurring. #131580

[Impeller] Scale down the Gaussian blur in both directions prior to blurring. #131580

bdero commented Jul 30, 2023

bdero commented Oct 23, 2023

gaaclarke commented Oct 24, 2023

gaaclarke commented Oct 26, 2023

bdero commented Nov 10, 2023

gaaclarke commented Nov 16, 2023

github-actions bot commented Feb 1, 2024

[Impeller] Scale down the Gaussian blur in both directions prior to blurring. #131580

[Impeller] Scale down the Gaussian blur in both directions prior to blurring. #131580

Comments

bdero commented Jul 30, 2023

bdero commented Oct 23, 2023

gaaclarke commented Oct 24, 2023

gaaclarke commented Oct 26, 2023

bdero commented Nov 10, 2023

gaaclarke commented Nov 16, 2023

github-actions bot commented Feb 1, 2024