Practical Fragment Shaders in Flutter | Guide – Shading Widgets

The first 3 parts of this guide focused mostly on getting a grip on the basics of fragment shaders in Flutter. Now that we know our way around them, and how to interact between GLSL and Flutter code, let’s learn how to merge the two together. 

Find the previous lessons here:

• Practical Fragment Shaders in Flutter | Guide – Introduction
• Practical Fragment Shaders in Flutter | Guide – Generative Art Part 1
• Practical Fragment Shaders in Flutter | Guide – Generative Art Part 2

The simple way

For starters, let’s think about how we could use the WARP shader we were benchmarking in the previous part of the guide. It’s obviously a great art piece for a background somewhere, but it can be a little overwhelming if we try to put some actual readable stuff on top of it. So, maybe we could reverse the roles? Make the background a plain, simple color, and apply the WARP effect to text on the foreground? That sounds great and we already have a perfect candidate app to try it on – the counter. Flutter’s default app that every Flutter developer is more than familiar with!

This time, I’ll skip some of the details covered in the previous articles, but if you ever get lost, feel free to refer to the code on my Github repo.

So, let’s start coding!

As you can see, we’ve wrapped the Scaffold with ShaderBuilder to load the shader. 

From there it’s just the standard counter app widget until we get to a new widget called ShaderMask. Its purpose is to apply shader effects to the child widget. Here, we set up our shader the same way we would in the CustomPainter from previous articles, just in a shaderCallback instead of the paint method. Remember to give your StatefulWidget a SingleTickerProviderStateMixin and create a Ticker – without that, the WARP shader will not animate. 

Next, we need to set a blendMode which will tell ShaderMask how we want to blend output colors of the shader with colors of the child widget. There is a lot of available BlendModes to choose from, each serving a different purpose, but for now we’ll only be using BlendMode.srcIn, which means that the ShaderMask will draw our shader without changing its colors, but only wherever the child widget is supposed to be drawn. It’ll basically create a cutout with the shape of the child widget and the colors of the shader. 

Lastly, as our child widget, we’ll use just a standard Text showing the value of the counter. 

Here’s how it should look when run:

Awesome!

The still simple but slightly less simple way

While the ShaderMask method is super easy to apply anywhere in your app, it has one major limitation. Can you spot it? What would happen if some of the calculations inside the shader needed to access the information about the widget it’s styling? Yeah, we could try using different BlendModes to try to achieve the effect we want, but it’s still very limiting and would only work for specific cases. We’d have no way to do that efficiently with ShaderMask, so we need another solution.

Enter AnimatedSampler. It’s another handy widget from the flutter_shaders package and, contrary to its name, has little to do with animating and more to do with sampling stuff. What it actually does is take its child widget and, instead of drawing it on the screen straight away, it renders the widget into a ui.Image object. And you might see where we’re going with this – ui.Image is the exact type we need to insert an image into our shader as a sampler. To use this image, we provide a second parameter to the AnimatedSampler – an AnimatedSamplerBuilder that will give us access to the image, its size and a Canvas object – exactly the same one we’d get with a CustomPainter. So now it should be clear what to do – add the parameters to the shader and draw it on the screen using canvas.

Here’s the code:

Now let’s make a simple shader that will turn any widget into a grayscale version of itself. We’re going to use an algorithm called luminosity conversion, which just means we’ll multiply the red, green and blue channels by some magic numbers which should give us a nice grayscale color.

And here’s the final effect:

As you can see, it’s not that complicated – once you wrap your head around the concept of rendering a widget into a texture instead of directly on screen, and then sampling that texture inside the shader, it’s downhill from there.

But let’s do something more with that new skill!

Shading the world

Shading widgets is nice and all, but how about we shade something from the real world for a change? That’s right, let’s take a view from the smartphone’s camera and apply a filter in real time!

Begin by adding Flutter’s camera package to the project. Add the required configuration on the native side of the project, depending on your platform, and then proceed to import the package into your widget and start coding!

In a StatefulWidget, load the list of available cameras and create and initialize a CameraController. When the controller is initialized, start an image stream and rebuild the widget every time you get a new image from the camera. Remember to dispose of the controller via the widget’s dispose() method! Next, in the build() method, just add a CameraPreview widget with the controller you just created and you’re set!

Here’s how it should look:

Okay, we’ve got the camera view, so now for the filter – let’s do something that would probably be the first real algorithm you’d learn if you took a computer vision basics course – the Sobel filter!

“What on earth is the Sobel filter?” you might ask. It’s a very simple convolutional edge detection algorithm that uses an isotropic 3×3 image gradient operator. Which is just a bunch of big words for doing some addition and multiplication of neighboring pixels to check if there’s a significant color difference between them, which will allow us to guess that there is or isn’t an edge in that spot. It’ll all be clear when you see the final effect, I promise.

First, we need a convolution matrix, which is just an array pointing us to the pixels we’ll be taking into consideration for the edge detection. We’ll then fetch each pixel’s color and convert it to grayscale.

Note that, when we want to get the neighboring pixel’s color, we can’t just use texture(image, currentPixelCoords + vec2(1, 0)), because the texture() function operates on UV coordinates – meaning all the coords we pass need to be between 0.0 and 1.0. That’s why we need to divide the value from the convolutionMatrix array by the resolution.

Next, we do the math required to calculate the edges:

And we’re ready to go!

Here’s the shader in full:

And now we wrap our CameraPreview widget with AnimatedSampler just like before.

Let’s launch the app!

Wrapping Up

In this guide, we’ve explored two powerful methods to apply fragment shader effects in Flutter, enhancing widgets with visually dynamic backgrounds and real-time camera filters. These techniques not only broaden the horizon for app aesthetics but also empower developers to innovate and customize their UI/UX with unprecedented control and artistic flair. Remember that these approaches are just the beginning – your journey into the vast world of shader effects in Flutter is bound to be as unique and inventive as the projects you choose to embark on. Happy coding!

Related posts