# Real-time dreamy Cloudscapes with Volumetric Raymarching

the past few months

DATE

diving into the realm of Raymarching and studying some of its applications that may come in handy for future 3D projects, and while I managed to build a pretty diverse set of scenes, all of them consisted of rendering surfaces or solid objects. My blog post on Raymarching covered some of the many impressive capabilities of this rendering technique, and as I mentioned at the end of that post, that was only the tip of the iceberg; there is a lot more we can do with it.

One

CARDINAL

fascinating aspect of Raymarching I quickly encountered in my study was its capacity to be tweaked to render volumes. Instead of stopping the raymarched loop once the ray hits a surface, we push through and continue the process to sample the inside of an object. That is where my obsession with volumetric clouds started, and I think

the countless hours

TIME

I spent exploring the many

Sky Islands

LOC

in

Zelda Tears of the Kingdom

WORK_OF_ART

contributed a lot to my curiosity to learn more about how they work. I thus studied a lot of

Shadertoy

ORG

scenes such as "Clouds" by

Inigo Quilez

PERSON

, "

Starry Night

WORK_OF_ART

" by

al-ro

PERSON

, and "

Volumetric Raymarching

WORK_OF_ART

sample" by

Suyoku

ORG

leveraging many

Volumetric Raymarching

PRODUCT

techniques to render smoke, clouds, and

cloudscapes

PERSON

, which I obviously couldn’t resist giving a try rebuilding myself:

I spent a great deal of time exploring the different ways I could use Raymarching to render clouds, from fully wrapping my head around the basics of

Volumetric Raymarching

ORG

to leveraging physically based properties of clouds to try getting a more realistic output while also trying to squeeze as much performance out of my scenes with neat performance improvement tips I learned along the way. I cover all of that in this article, which I hope can serve you as a field guide for your own volumetric rendering experiments and learnings.

Before you start 👉 This article assumes you have basic knowledge about shaders, noise, and GLSL, or read

The Study of Shaders

WORK_OF_ART

with

React Three Fiber

LAW

as well as some notions about Raymarching, which you can learn more about in Painting with Math: A Gentle Study of Raymarching.

Volumetric rendering: Raymarching with a twist In my previous blog post on Raymarching, we saw that the technique relied on:

Arrow An

ORG

icon representing an arrow Signed Distance Fields : functions that return the distance of a given point in space to the surface of an object

Arrow

ORG

An icon representing an arrow A Raymarching loop where we march step-by-step alongside rays cast from an origin point (a camera, the observer’s eye) through each pixel of an output image, and we calculate the distance to the object’s surface using our

SDF

ORG

. Once that distance is small enough, we can draw a pixel. If you’ve practiced this technique on some of your own scenes, you’re in luck: Volumetric Raymarching relies on the same principles: there’s a loop, rays cast from an origin, and SDFs. However, since we’re rendering volumes instead of surfaces, there’s a tiny twist to the technique 👀. How to sample a volume The

first

ORDINAL

time we got introduced to the concept of

SDF

ORG

, we learned that it was important not to step inside the object during our Raymarching loop to have a beautiful render. I even emphasized that fact in one of my diagrams showcasing

3

CARDINAL

points relative to an object:

Arrow An

ORG

icon representing an arrow

P1

ORG

is located far from the surface, in green, representing a positive distance to the surface.

Arrow

ORG

An icon representing an arrow P2 is located at a close distance ε to the surface, in orange,

Arrow

ORG

An icon representing an arrow

P3

PRODUCT

positioned inside the object, in red, representing a negative distance to the surface.

Diagram

ORG

showcasing

3

CARDINAL

points,

P1

ORG

,

P2

CARDINAL

, and

P3

PRODUCT

, being respectively, at a positive distance, small distance, and inside a sphere. When sampling a volume, we’ll need to actually raymarch inside our object and reframe how we think of

SDF

ORG

: instead of representing the distance to the surface, we will now use it as the density of our volume.

Arrow

ORG

An icon representing an arrow When raymarching outside, the density is null, or

0

CARDINAL

.

Arrow

ORG

An icon representing an arrow Once we raymarch inside, it is positive. To illustrate this new way of thinking about Raymarching in the context of volume, here’s a modified version of the widget I introduced in my blog post on the topic

earlier this year

DATE

.

-0.5 ,

CARDINAL

0.5 0.5

CARDINAL

,

0.5

CARDINAL

-0.5 , -0.5 0.5 , -0.5 Repeat An icon representing an arrow twisted so it makes a loop

Arrow

ORG

An icon representing an arrow Step: 0 Info An icon representing the letter ‘i‘ in a circle Notice how now we keep sampling once inside the object, and each step returns a layer of the volume until the ray has entirely cast through it and the density is once again 0 . That reframing of what an

SDF

ORG

represents ends up changing

two

CARDINAL

core principles in our Raymarching technique that will have to be reflected in our code:

Arrow An

ORG

icon representing an arrow We have to march step-by-step with a constant step size along our rays. We no longer use the distance returned by the

SDF

ORG

.

Arrow

ORG

An icon representing an arrow Our

SDF

ORG

now returns the opposite of the distance to the surface to properly represent the density of our object (positive on the inside,

0

CARDINAL

on the outside)

Diagram

ORG

showcasing

3

CARDINAL

points,

P1

ORG

,

P2

CARDINAL

, and

P3

PRODUCT

, being respectively, at a positive distance, small distance, and inside a sphere. Only P3 is considered ‘valid’ in the context of

Volumetric Raymarching Our first

ORG

Volumetric Raymarching scene Now that we have a grasp of sampling volumes using what we know about Raymarching, we can try implementing it by modifying an existing scene. For brevity, I’m not detailing the setup of a basic of Raymarching scenes. If you want a good starting point you can head to my Raymarching setup I already introduced in a previous article. The setup of the scene is quite similar to what we’re familiar with in classic Raymarching; the modifications we’ll need to do are located in:

Arrow An

ORG

icon representing an arrow Our

SDF

ORG

functions: we’ll need to return the opposite of the distance: -d instead of d . Example of

SDF

ORG

used in

Volumetric Raymarching 1

PRODUCT

float sdSphere ( vec3 p , float radius ) {

2

CARDINAL

return length ( p ) – radius ;

3

CARDINAL

}

4 5

CARDINAL

float scene ( vec3 p ) {

6

CARDINAL

float distance = sdSphere ( p ,

1.0

CARDINAL

) ;

7

CARDINAL

return – distance ;

8

CARDINAL

}

Arrow

ORG

An icon representing an arrow Our raymarch function: we’ll need to march at a constant step size and start drawing only once the density is over 0. Volumetric Raymarching loop with constant step size

1 #

CARDINAL

define MAX_STEPS 100 2 3 const float MARCH_SIZE =

0.08

CARDINAL

;

4

CARDINAL

5 6 float depth =

0.0

CARDINAL

;

7

CARDINAL

vec3 p = rayOrigin + depth * rayDirection ;

8

CARDINAL

9 vec4 res = vec4 (

0.0

CARDINAL

) ;

10 11

CARDINAL

for ( int i = 0 ; i < MAX_STEPS ; i ++ ) {

12

CARDINAL

float density = scene ( p ) ;

13

CARDINAL

if ( density >

0.0

CARDINAL

) {

14 15

CARDINAL

}

16 17

QUANTITY

depth += MARCH_SIZE ;

18

CARDINAL

p = rayOrigin + depth * rayDirection ;

19

CARDINAL

} Now comes another question: what shall we draw once our density is positive to represent a volume? For this

first

ORDINAL

example, we can keep things simple and play with the alpha channel of our colors to make it proportional to the density of our volume: the denser our object gets as we march into it, the more opaque/darker it will be. Simple Volumetric Raymarching loop

1

CARDINAL

const float MARCH_SIZE =

0.08

CARDINAL

;

2

CARDINAL

3 vec4 raymarch ( vec3 rayOrigin , vec3 rayDirection ) {

4

CARDINAL

float depth =

0.0

CARDINAL

;

5

CARDINAL

vec3 p = rayOrigin + depth * rayDirection ;

6

CARDINAL

7 vec4 res = vec4 (

0.0

CARDINAL

) ;

8 9

CARDINAL

for ( int i = 0 ; i < MAX_STEPS ; i ++ ) {

10

CARDINAL

float density = scene ( p ) ;

11 12 13

DATE

if ( density >

0.0

CARDINAL

) {

14

CARDINAL

vec4 color = vec4 ( mix ( vec3 (

1.0

CARDINAL

,

1.0

CARDINAL

,

1.0

CARDINAL

) ,

vec3

PERSON

(

0.0

CARDINAL

,

0.0

CARDINAL

,

0.0

CARDINAL

) , density ) , density ) ;

15

CARDINAL

color . rgb *= color . a ;

16

CARDINAL

res += color * (

1.0

CARDINAL

– res . a ) ;

17

CARDINAL

}

18 19

QUANTITY

depth += MARCH_SIZE ;

20

CARDINAL

p = rayOrigin + depth * rayDirection ;

21

CARDINAL

}

22 23

CARDINAL

return res ;

24

CARDINAL

} If we try to render this code in our React

Three

CARDINAL

Fiber canvas, we should get the following result

👀

ORG

Drawing Fluffy Raymarched Clouds We now know and applied the basics of

Volumetric Raymarching

ORG

. So far, we only rendered a simple volumetric sphere with constant density as we march through the volume, which is a good start. We can now try using that simple scene as a foundation to render something more interesting: clouds! Noisy Volume Going from our simple

SDF

ORG

of a sphere to a cloud consists of drawing it with a bit more noise. Clouds don’t have a uniform shape nor do they have a uniform density, thus we need to introduce some organic randomness through noise in our Raymarching loop. If you read some of my previous articles, you should already be familiar with the concept of:

Arrow An

ORG

icon representing an arrow

Noise

PRODUCT

,

Perlin

ORG

noise, and value noise derivative

Arrow

ORG

An icon representing an arrow

Fractal Brownian Motion

ORG

, or FBM.

Arrow

ORG

An icon representing an arrow Texture based noise. To generate raymarched landscapes, we used a noise texture, noise derivatives, and

FBM

ORG

to get a detailed organic result. We’ll rely on some of those concepts to create organic randomness and obtain a cloud from our

SDF

ORG

☁️. Noise function for Raymarched landscape

1 vec3 noise

QUANTITY

( vec2 x ) {

2

CARDINAL

vec2 p = floor ( x ) ;

3

CARDINAL

vec2 f = fract ( x ) ;

4

CARDINAL

vec2 u = f * f * (

3

CARDINAL

. – 2. * f ) ;

5 6

CARDINAL

float a = textureLod (

uTexture

ORG

, ( p + vec2 ( .0 , .0 ) ) /

256

CARDINAL

. ,

0.

CARDINAL

) . x ;

7

CARDINAL

float b = textureLod (

uTexture

ORG

, ( p + vec2 (

1.0

CARDINAL

, .0 ) ) /

256

CARDINAL

. ,

0.

CARDINAL

) . x ;

8

CARDINAL

float c = textureLod (

uTexture

ORG

, ( p + vec2 ( .0 ,

1.0

CARDINAL

) ) /

256

CARDINAL

. ,

0.

CARDINAL

) . x ;

9

CARDINAL

float d = textureLod (

uTexture

ORG

, ( p + vec2 (

1.0

CARDINAL

,

1.0

CARDINAL

) ) /

256

CARDINAL

. ,

0.

CARDINAL

) . x ;

10 11

CARDINAL

float noiseValue = a + ( b – a ) * u . x + ( c – a ) * u . y + ( a – b – c + d ) * u . x * u . y ;

12

CARDINAL

vec2 noiseDerivative =

6

CARDINAL

. * f * (

1

CARDINAL

. – f ) * ( vec2 ( b – a , c – a ) + ( a – b – c + d ) * u . yx ) ;

13 14

DATE

return vec3 ( noiseValue , noiseDerivative ) ;

15

CARDINAL

} For clouds, our noise function looks a bit different: Noise function for

Volumetric

PRODUCT

clouds

1

CARDINAL

float noise ( vec3 x ) {

2

CARDINAL

vec3 p = floor ( x ) ;

3

CARDINAL

vec3 f = fract ( x ) ;

4

CARDINAL

vec2 u = f * f * (

3

CARDINAL

. – 2. * f ) ;

5 6

CARDINAL

vec2 uv = ( p . xy + vec2 (

37.0

CARDINAL

,

239.0

CARDINAL

) * p . z ) + u .

xy

ORG

;

7

CARDINAL

vec2 tex = textureLod ( uNoise , ( uv + 0.5 ) / 256.0 ,

0.0

CARDINAL

) . yx ;

8 9

CARDINAL

return mix ( tex . x , tex . y ,

u .

GPE

z ) * 2.0 – 1.0 ;

10

CARDINAL

} To tell you the truth, I saw this function in many

Shadertoy

ORG

demos without necessarily seeing a credited author or even a link to an explanation; I kept using it throughout my work as it still yielded a convincing cloud noise pattern. Here’s an attempt at gathering together some of its specificities from my own understanding:

Arrow An

ORG

icon representing an arrow

Clouds

PRODUCT

are 3D structures, so our function takes in a vec3 as input: a point in space within our cloud.

Arrow

ORG

An icon representing an arrow The texture lookup differs from its landscape counterpart: we’re sampling it as a

2D

CARDINAL

slice from a 3D position. The

vec2(37.0

PERSON

,

239.0

CARDINAL

) *

p.z

PERSON

seems a bit arbitrary to me, but from what I gathered, it allows for more variation in the resulting noise.

Arrow

ORG

An icon representing an arrow We then mix

two

CARDINAL

noise values from our texture lookup based on the z value to generate a smooth noise pattern and rescale it within the [-1,

1

CARDINAL

] range. Info An icon representing the letter ‘i‘ in a circle During my research, I found that game devs and other 3D creators use a large variety of noises to get more realistic-looking clouds. If you’re interested in reading more on that topic, you can check the following write-ups (also quoted as source):

Arrow

ORG

An icon representing an arrow How Big Budget AAA Games Render Clouds

Arrow

ORG

An icon representing an arrow This presentation on the

EA

ORG

frostbite engine starting at page

30

CARDINAL

Applying this noise along with a

Fractal Brownian

PRODUCT

motion is pretty similar to what we’re used to with

Raymarched

PERSON

landscapes:

Fractal Brownian Motion

ORG

applied to our

Volumetric Raymarching

PRODUCT

scene

1

CARDINAL

float fbm ( vec3 p ) {

2

CARDINAL

vec3 q = p + uTime *

0.5

CARDINAL

* vec3 (

1.0

CARDINAL

, –

0.2

CARDINAL

, –

1.0

CARDINAL

) ;

3 float g

QUANTITY

= noise ( q ) ;

4 5

CARDINAL

float f =

0.0

CARDINAL

;

6

CARDINAL

float scale =

0.5

CARDINAL

;

7

CARDINAL

float factor =

2.02

CARDINAL

;

8 9

CARDINAL

for ( int i = 0 ; i < 6 ; i ++ ) {

10

CARDINAL

f += scale * noise ( q ) ;

11

CARDINAL

q *= factor ;

12

CARDINAL

factor +=

0.21

CARDINAL

;

13

CARDINAL

scale *=

0.5

CARDINAL

;

14

CARDINAL

}

15 16

CARDINAL

return f ;

17

CARDINAL

}

18 19

CARDINAL

float scene ( vec3 p ) {

20

CARDINAL

float distance = sdSphere ( p ,

1.0

CARDINAL

) ;

21

CARDINAL

float f = fbm ( p ) ;

22 23

CARDINAL

return – distance + f ;

24

CARDINAL

} If we apply the code above to our previous demo, we do get something that starts to look like a cloud 👀: Adding light Once again, we’re just "starting" to see something approaching our goal, but a crucial element is missing to make our cloud feel more cloudy: light. The demo we just saw in the previous part lacks depth and shadows and thus doesn’t feel very realistic overall, and that’s due to the lack of diffuse light. Light goes a long way We had similar issues in:

Arrow An

ORG

icon representing an arrow Our

first

ORDINAL

demo from the Raymarching blog post

Arrow

ORG

An icon representing an arrow Our Dispersion shader Adding diffuse and optionally specular lighting can go a long way to give your shader material or scene a sense of depth. To add light to our cloud and consequentially obtain better shadows, one may want to apply the same lighting we used in standard Raymarching scenes:

Arrow An

ORG

icon representing an arrow Calculate the normal of each sample point using our scene function

Arrow

ORG

An icon representing an arrow Use the dot product of the normal and the light direction

Diffuse

PRODUCT

lighting in Raymarched scene using normals

1 vec3 getNormal

QUANTITY

( vec3 p ) {

2

CARDINAL

vec2 e = vec2 (

.01

CARDINAL

, 0 ) ;

3

CARDINAL

4 vec3 n = scene ( p ) – vec3 (

5

CARDINAL

scene ( p – e .

xyy

ORG

) ,

6

CARDINAL

scene ( p – e .

yxy

PERSON

) ,

7

CARDINAL

scene ( p – e .

yyx

ORG

) ) ;

8 9

CARDINAL

return normalize ( n ) ;

10

CARDINAL

}

11 12

CARDINAL

void main ( ) {

13

CARDINAL

14 15

vec3 ro = vec3

PERSON

(

0.0

CARDINAL

,

0.0

CARDINAL

,

5.0

CARDINAL

) ;

16 vec3 rd

TIME

= normalize ( vec3 ( uv , –

1.0

CARDINAL

) ) ;

17

CARDINAL

vec3 lightPosition = vec3

PERSON

(

1.0

CARDINAL

) ;

18 19

CARDINAL

float d = raymarch ( ro ,

rd

GPE

) ;

20

CARDINAL

vec3 p = ro + rd * d ;

21

CARDINAL

22 23 vec3 color = vec3 (

0.0

CARDINAL

) ;

24 25

CARDINAL

if ( d <

MAX_DIST

ORG

) {

26

CARDINAL

vec3 normal = getNormal ( p ) ;

27

CARDINAL

vec3 lightDirection

PERSON

= normalize ( lightPosition – p ) ;

28 29

CARDINAL

float diffuse = max ( dot ( normal , lightDirection ) ,

0.0

CARDINAL

) ;

30

CARDINAL

color = vec3 (

1.0

CARDINAL

,

1.0

CARDINAL

,

1.0

CARDINAL

) * diffuse ;

31

CARDINAL

}

32 33

CARDINAL

gl_FragColor = vec4 ( color ,

1.0

CARDINAL

) ;

34

CARDINAL

} That would work in theory, but it’s not the optimal choice for

Volumetric Raymarching

ORG

: Arrow An icon representing an arrow The

getNormal

ORG

function requires a lot of sample points to estimate the "gradient" in every direction. In the code above, we need

4

CARDINAL

, but there are code snippets that require

6

CARDINAL

for a more accurate result.

Arrow

ORG

An icon representing an arrow Our volumetric raymarching loop is more resource-intensive: we’re walking at a constant step size along our ray to sample the density of our volume. Thus, we need another method or approximation for our diffuse light. Luckily,

Inigo Quilez

PERSON

presents a technique to solve this problem in his article on directional derivatives. Instead of having to sample our density in every direction like

getNormal

ORG

, this method simplifies the problem by sampling the density at our sampling point p and at an offset in the direction of the light and getting the difference between those values to approximate how the light scatters roughly inside our volume.

Diagram

ORG

showcasing

2

CARDINAL

sampled points

P1

PRODUCT

and

P2

CARDINAL

with both their diffuse lighting calculated by sampling extra points

P1

ORG

‘ and P2′ in the direction of the light In the diagram above, you can see that we’re sampling our density at p1 and at another point p1’ that’s a bit further along the light ray:

Arrow An

ORG

icon representing an arrow If the density increases along that path, that means the volume gets denser, and light will scatter more

Arrow

ORG

An icon representing an arrow If the density gets smaller, our cloud is less thick, and thus, the light will scatter less. This method only requires

2

CARDINAL

sampling points and consequentially requires fewer resources to give us a good approximation of how the light behaves with the volume around p1 . Alert An icon representing an exclamation mark in an octogone The directional derivative method to calculate diffuse lighting works only with a few light sources. That is an acceptable limitation, as our scenes only feature

one

CARDINAL

light source: the sun. We can apply this diffuse formula to our demo as follows: Diffuse lighting using directional derivatives

1 2

CARDINAL

if ( density >

0.0

CARDINAL

) {

3

CARDINAL

4 5 float diffuse = clamp ( ( scene ( p ) – scene ( p + 0.3 * sunDirection ) ) /

0.3

CARDINAL

,

0.0

CARDINAL

,

1.0

CARDINAL

) ;

6 7 vec3 lin

TIME

=

vec3

PERSON

(

0.60

CARDINAL

,

0.60

CARDINAL

,

0.75

CARDINAL

) *

1.1

CARDINAL

+ 0.8 * vec3 (

1.0

CARDINAL

,

0.6

DATE

,

0.3

DATE

) * diffuse ;

8

CARDINAL

vec4 color = vec4 ( mix ( vec3 (

1.0

CARDINAL

,

1.0

CARDINAL

,

1.0

CARDINAL

) ,

vec3

PERSON

(

0.0

CARDINAL

,

0.0

CARDINAL

,

0.0

CARDINAL

) , density ) , density ) ;

9

CARDINAL

color . rgb *= lin ;

10 11

CARDINAL

color . rgb *= color . a ;

12

CARDINAL

res += color * (

1.0

CARDINAL

– res . a ) ;

13

CARDINAL

}

14

CARDINAL

That is, once again, very similar to what we were doing in standard Raymarching, except that now, we have to include it inside

the Raymarching loop

LOC

as we’re sampling a volume and thus have to run the calculation multiple times throughout the volume as the density may vary whereas a surface required

only one

CARDINAL

diffuse lighting computation (at the surface). You can observe the difference between our cloud without lighting and with diffuse lighting below 👇

Arrow

ORG

An icon representing an arrow

Arrow

ORG

An icon representing an arrow Before/After comparison of our

Volumetric

PRODUCT

cloud without and with diffuse lighting. And here’s the demo featuring the concept and code we just introduced 👀: Info An icon representing the letter ‘i‘ in a circle In this demo above, try to move the "sun" by modifying the SUN_POSITION vector and see the changes in light scattering. Morphing clouds Let’s take a little break to tweak our scene and have some fun with what we built so far! Despite the differences between the standard Raymarching and its volumetric counterpart, there are still a lot of

SDF

ORG

-related concepts you can apply when building cloudscapes. You can try to make a cloud in fun shapes like a cross or a torus, or even better, try to make it morph from

one

CARDINAL

form to another over time: Mixing SDF to morph volumetric clouds into different shapes

1 mat2

QUANTITY

rotate2D ( float a ) {

2

CARDINAL

float s = sin ( a ) ;

3

CARDINAL

float c = cos ( a ) ;

4

CARDINAL

return mat2 ( c , – s , s , c ) ;

5

CARDINAL

}

6 7

CARDINAL

float nextStep ( float t , float len , float smo ) {

8

CARDINAL

float tt = mod ( t += smo ,

len

PERSON

) ;

9

CARDINAL

float stp = floor ( t / len ) –

1.0

CARDINAL

;

10

CARDINAL

return smoothstep (

0.0

CARDINAL

,

smo

ORG

, tt ) + stp ;

11

CARDINAL

}

12 13

CARDINAL

float scene ( vec3 p ) {

14

CARDINAL

vec3 p1 = p ;

15

CARDINAL

p1 . xz *= rotate2D ( – PI *

0.1

CARDINAL

) ;

16

CARDINAL

p1 . yz *= rotate2D ( PI *

0.3

CARDINAL

) ;

17 18

CARDINAL

float s1 = sdTorus ( p1 , vec2 (

1.3

CARDINAL

,

0.9

DATE

) ) ;

19

CARDINAL

float s2 = sdCross ( p1 * 2.0 ,

0.6

CARDINAL

) ;

20

CARDINAL

float s3 = sdSphere ( p ,

1.5

CARDINAL

) ;

21

CARDINAL

float

s4

PRODUCT

= sdCapsule ( p ,

vec3

PERSON

( – 2.0 , –

1.5

CARDINAL

,

0.0

CARDINAL

) ,

vec3

PERSON

( 2.0 ,

1.5

CARDINAL

,

0.0

CARDINAL

) ,

1.0

CARDINAL

) ;

22 23

CARDINAL

float t = mod ( nextStep (

uTime

ORG

,

3.0

CARDINAL

,

1.2

CARDINAL

) ,

4.0

CARDINAL

) ;

24 25

CARDINAL

float distance = mix ( s1 , s2 , clamp ( t ,

0.0

CARDINAL

,

1.0

CARDINAL

) ) ;

26

CARDINAL

distance = mix ( distance ,

s3

ORG

, clamp ( t –

1.0

CARDINAL

,

0.0

CARDINAL

,

1.0

CARDINAL

) ) ;

27

CARDINAL

distance = mix ( distance ,

s4

PRODUCT

, clamp ( t – 2.0 ,

0.0

CARDINAL

,

1.0

CARDINAL

) ) ;

28

CARDINAL

distance = mix ( distance , s1 , clamp ( t –

3.0

CARDINAL

,

0.0

CARDINAL

,

1.0

CARDINAL

) ) ;

29 30

CARDINAL

float f = fbm ( p ) ;

31 32

CARDINAL

return – distance + f ;

33

CARDINAL

} This demo is a reproduction of this volumetric rendering related

Shadertoy

ORG

scene. I really like this creation because the result is very organic, and it gives the impression that the cloud is rolling into its next shape naturally. You can also try to render:

Arrow An

ORG

icon representing an arrow Clouds merging together using the min and smoothmin of

two

CARDINAL

SDFs

Arrow

ORG

An icon representing an arrow Repeating clouds through space using the mod function There are a lot of creative compositions to try!

Performance optimization You may notice that running the scenes we built so far may make your computer sound like a jet engine at high resolution or at least not look as smooth as they could. Luckily, we can do something about it and use some performance optimization techniques to strike the right balance between

FPS

ORG

count and output quality. Acknowledgments Thank you @N8Programs and

@Cody_J_Bennett

PERSON

for taking the time to introduce me to the techniques showcased in this section and giving me some helpful links and examples. Your help has been essential for the writing of this article!

N8 Programs

ORG

@ N8Programs @0xca0a @MaximeHeckel I’d be happy too: some recommendations – calculate at

half-res &

MONEY

dither w/ blue noise, upscale and then apply light blur. Lots of things to try.

4:09 PM – Aug 8,

TIME

2023 2 3

Cody Bennett

PERSON

@

Cody_J_Bennett @N8Programs @0xca0a @MaximeHeckel

PERSON

I’d add that dithering your sample point lets you cut down on samples/sample distance since it hides the usual stepping from under sampling. This is a good reference which covers a lot of now usual optimizations and experiments: https://t.co/RTXKVYTYGG. https://t.co/LHAoE6W5Ji

4:35 PM – Aug 8

TIME

, 2023 2 2 Blue noise dithering

One

CARDINAL

of the main performance pitfalls of our current

raymarched cloudscape

PERSON

scene is due to:

Arrow An

ORG

icon representing an arrow the number of steps we have to perform to sample our volume and the small marchSize

Arrow

ORG

An icon representing an arrow some heavy computation we have to do within our loop, like our directional derivative or FBM. This issue will only worsen as we attempt to make more computations to achieve a more physically accurate output in the next part of this article.

One

CARDINAL

of the

first

ORDINAL

things we could do to make this scene more efficient would be to reduce the amount of steps we perform when sampling our cloud and increase the step size. However, if we attempt this on some of our previous examples (I invite you to try), some layering will be visible, and our volume will look more like some kind of milk soup than a fluffy cloud. Screenshot of our rendered

Volumetric

PRODUCT

cloud with a low max step count and higher step size. Notice how those optimizations have degraded the output quality. You might have encountered the concept of dithering or some images using dithering styles before. This process can create the illusion of more colors or shades in an image than available or purely used for artistic ends. I recommend reading Dithering on the

GPU

ORG

from

Alex Charlton

PERSON

if you want a quick introduction. In

Ray

PERSON

marching fog with blue noise, the author showcases how you can leverage blue noise dithering in your raymarched scene to erase the banding or layering effect due to a lower step count or less granular loop. This technique leverages a blue noise pattern, which has fewer patterns or clumps than other noises and is less visible to the human eye, to obtain a random number each time our fragment shader runs. We then introduce that number as an offset at the beginning of the raymarched loop, moving our sampling start point along our ray for each pixel of our output. Blue noise texture

Diagram

ORG

showcasing the difference between our cloud being sampled without and with blue noise dithering. Notice how each ray is offset when blue noise is introduced and how that ‘erases’ any obvious layering in the final render. Blue noise dithering introducing an offset in our Raymarching loop

1

CARDINAL

uniform sampler2D uBlueNoise ;

2

CARDINAL

3 4 5 vec4 raymarch ( vec3 rayOrigin , vec3 rayDirection , float offset ) {

6

CARDINAL

float depth =

0.0

CARDINAL

;

7

CARDINAL

depth += MARCH_SIZE * offset ;

8

CARDINAL

vec3 p = rayOrigin + depth * rayDirection ;

9 10

CARDINAL

}

11 12

CARDINAL

void ( main ) {

13 14

CARDINAL

float blueNoise = texture2D ( uBlueNoise ,

gl_FragCoord

PERSON

. xy /

1024.0

CARDINAL

) . r ;

15

CARDINAL

float offset = fract (

blueNoise

PERSON

) ;

16

CARDINAL

17

CARDINAL

vec4 res = raymarch ( ro , rd , offset ) ;

18 19

CARDINAL

} By introducing some blue noise dithering in our fragment shader, we can erase those artifacts and get a high-quality output while maintaining the Raymarching step count low! However, under some circumstances, the dithering pattern can be pretty noticeable. By looking at some other

Shadertoy

ORG

examples, I discovered that introducing a temporal aspect to the blue noise can attenuate this issue. Temporal blue noise dithering offset

1

CARDINAL

float offset = fract (

blueNoise

PERSON

+ float (

uFrame %

PERCENT

32

DATE

) / sqrt (

0.5

CARDINAL

) ) ; Here’s a before/after comparison of our single frame of our raymarched cloud. I guess the results speak for themselves here 😄.

Arrow

ORG

An icon representing an arrow

Arrow

ORG

An icon representing an arrow Before/After comparison of our

Volumetric

PRODUCT

cloud with fewer Raymarching steps and without and with

Blue Noise

ORG

dithering. And here’s the demo showcasing our blue noise dithering in action giving us a softer cloud ⛅: Info An icon representing the letter ‘i‘ in a circle In the sandbox above, try to remove the temporal aspect of the blue noise dithering to see how it impacts the visibility of the dithering pattern. Upscaling with

Bicubic

PERSON

filtering This

second

ORDINAL

improvement recommended by @N8Programs aims to fix some remaining noise artifacts that remain following the introduction of the blue noise dithering to our raymarched scene.

Bicubic

PERSON

filtering is used in upscaling and allows smoothing out some noise patterns while retaining details by calculating the value of a new pixel by considering

16

CARDINAL

neighboring pixels through a cubic polynomial (Sources). I was lucky to find an implementation of bicubic filtering on

Shadertoy

ORG

made by

N8Programs

PERSON

himself! Applying it directly to our existing work however, is not that straightforward. We have to add this improvement as its own step or pass in the rendering process, almost as a post-processing effect. I introduced an easy way to build this kind of pipeline in my article titled Beautiful and mind-bending effects with WebGL Render Targets where I showcase how you can use

Frame Buffer Objects

PRODUCT

(

FBO

ORG

) to apply some post-processing effects on an entire scene which we can use for this use case:

Arrow An

ORG

icon representing an arrow We render our main raymarched canvas in a portal.

Arrow

ORG

An icon representing an arrow The default scene only contains a fullscreen triangle.

Arrow

ORG

An icon representing an arrow We render our main scene in a render target.

Arrow

ORG

An icon representing an arrow We pass the texture of the main scene’s render target as a uniform of our bicubic filtering material.

Arrow

ORG

An icon representing an arrow We use the bicubic filtering material as the material for our fullscreen triangle.

Arrow

ORG

An icon representing an arrow Our bicubic filtering will take our noisy raymarched scene as a texture uniform and output the smoothed out scene.

Diagram

ORG

showcasing how the bicubic filtering is applied as a post-processing effect to the original scene using

Render

ORG

Targets. Here’s a quick comparison of our scene before and after applying the bicubic filtering:

Arrow An

ORG

icon representing an arrow

Arrow

ORG

An icon representing an arrow Before/After comparison of our

Volumetric

PRODUCT

cloud with without and with

Bicubic Filtering

PERSON

. Notice the noise around at the edges and less dense region of the cloud being smoothed out when the effect is applied. The full implementation is a bit long, and features concepts I already went through in my render target focused blog post, so I invite you to look at it on your own time in the demo below: Leveraging render targets allowed me to play more with the resolution of the original raymarched scene. You can see a little selector that lets you pick at which resolution we render our raymarched cloud. You can notice that there are not a lot of differences between

1x

CARDINAL

and 0.5x which is great: we can squeeze more

FPS

ORG

without sacrificing the output quality 🎉.

Physically accurate Clouds So far, we’ve managed to build really beautiful cloudscapes with

Volumetric Raymarching

ORG

using some simple techniques and mixing the right colors. The resulting scenes are satisfying enough and give the illusion of large, dense clouds, but what if we wanted a more realistic output? I spent quite some time digging through talks, videos, and articles on how game engines solve the problem of physically accurate clouds and all the techniques involved in them. It’s been a journey, and I wanted to dedicate this last section to this topic because I find the subject fascinating: from a couple of physical principles of actual real-life clouds, we can render clouds in

WebGL

ORG

using

Volumetric Raymarching

PRODUCT

! Shoutout I wouldn’t have been able to build anything related to physically based clouds without @iced_coffee_dev’s video on how

AAA

ORG

game studios use

Volumetric Raymarching

ORG

as well as his guidance on the topic all of which I truly appreciated. Most of the code featured in this part is based on principles introduced in this video which I strongly recommend watching: it’s really really good.

Beer

PERSON

‘s Law I already introduced the concept of

Beer

PERSON

‘s Law in my Raymarching blog post as a way to render fog in the distance of a scene. It states that the intensity of light passing through a transparent medium is exponentially related to the distance it travels. The further to the medium light propagates, the more it is being absorbed. The formula for

Beer’s Law

LAW

is as follows: I = I0 * exp(−α * d) , where α is the absorption or attenuation coefficient describing how "thick" or "dense" the medium is. In our demos, we’ll consider an absorption coefficient of

0.9

CARDINAL

, although I’d invite you to try different values so you can see the impact of this number on the resulting render.

Diagram

ORG

showcasing how

Beer

PERSON

‘s Law can be used to represent how much light gets absorbed through a volume We can use this formula in our GLSL code and modify the Raymarching loop to use it instead of the "hacky" transparency hack we used in the

first

ORDINAL

part: Using Beer’s Law to calculate and return the accumulated light energy going through the cloud 1 # define MAX_STEPS 50 2 # define ABSORPTION_COEFFICIENT

0.9

CARDINAL

3 4 5 6 float

BeersLaw

PERSON

( float dist , float absorption ) {

7

CARDINAL

return exp ( – dist * absorption ) ;

8

CARDINAL

}

9 10

DATE

const vec3 SUN_POSITION = vec3 (

1.0

CARDINAL

,

0.0

CARDINAL

,

0.0

CARDINAL

) ;

11

CARDINAL

const float MARCH_SIZE =

0.16

CARDINAL

;

12 13

CARDINAL

float raymarch ( vec3 rayOrigin , vec3 rayDirection , float offset ) {

14

CARDINAL

float depth =

0.0

CARDINAL

;

15

CARDINAL

depth += MARCH_SIZE * offset ;

16

CARDINAL

vec3 p = rayOrigin + depth * rayDirection ;

17

CARDINAL

vec3 sunDirection

PERSON

= normalize ( SUN_POSITION ) ;

18 19

CARDINAL

float totalTransmittance =

1.0

CARDINAL

;

20

CARDINAL

float lightEnergy =

0.0

CARDINAL

;

21 22

CARDINAL

for ( int i = 0 ; i < MAX_STEPS ; i ++ ) {

23

CARDINAL

float density = scene ( p ) ;

24 25

CARDINAL

if ( density >

0.0

CARDINAL

) {

26

CARDINAL

float transmittance =

BeersLaw

PERSON

( density * MARCH_SIZE , ABSORPTION_COEFFICIENT ) ;

27

CARDINAL

float luminance = density ;

28 29

CARDINAL

totalTransmittance *= transmittance ;

30

CARDINAL

lightEnergy += totalTransmittance * luminance ;

31

CARDINAL

}

32 33

QUANTITY

depth += MARCH_SIZE ;

34

CARDINAL

p = rayOrigin + depth * rayDirection ;

35

CARDINAL

}

36 37

DATE

return lightEnergy ;

38

CARDINAL

} In the code snippet above:

Arrow An

ORG

icon representing an arrow We gutted the raymarching loop, so it now relies on a more physically based property:

Beer

PERSON

‘s Law .

Arrow

ORG

An icon representing an arrow We changed the interface of our function: instead of returning a full color, it now returns a float representing the amount of light or light energy going through the cloud.

Arrow

ORG

An icon representing an arrow As we march through the volume, we accumulate the obtained transmittance . The deeper we go, the less light we add.

Arrow

ORG

An icon representing an arrow We return the resulting lightEnergy The demo below showcases what using

Beers Law

ORG

yields in our Raymarching loop 👀 The resulting cloud is a bit strange:

Arrow An

ORG

icon representing an arrow its edges do indeed behave like a cloud

Arrow

ORG

An icon representing an arrow the center is just a white blob all of which is, once again, due to the lack of a proper lighting model. Sampling light Our new cloud does not interact with light right now. You can try changing the SUN_POSITION vector: the resulting render will remain the same. We not only need a lighting model but also a physically accurate one. For that, we can try to compute how much light has been absorbed for each sample point of our Raymarching loop by:

Arrow An

ORG

icon representing an arrow Start a dedicated nested Raymarching loop that goes from the current sample point to the light source (direction of the light)

Arrow

ORG

An icon representing an arrow Sample the density and apply

Beer

PERSON

‘s Law like we just did The diagram below illustrates this technique to make it a bit easier to understand:

Diagram

ORG

showcasing how we sample multiple points of lights in the direction of the light through our volume for each sampled point in the Raymarching loop. The code snippet below is one of many implementations of this technique. We’ll use this one going forward: Dedicated nested raymarching loop to sample the light received at a given sampled point 1 # define MAX_STEPS 50 2 # define MAX_STEPS_LIGHTS 6 3 # define ABSORPTION_COEFFICIENT

0.9

CARDINAL

4 5 6 7 const vec3 SUN_POSITION = vec3 (

1.0

CARDINAL

,

0.0

CARDINAL

,

0.0

CARDINAL

) ;

8

CARDINAL

const float MARCH_SIZE =

0.16

CARDINAL

;

9 10

CARDINAL

float

lightmarch

ORG

(

vec3 position

PERSON

, vec3 rayDirection ) {

11

CARDINAL

vec3 lightDirection

PERSON

= normalize ( SUN_POSITION ) ;

12

CARDINAL

float totalDensity =

0.0

CARDINAL

;

13

CARDINAL

float marchSize =

0.03

CARDINAL

;

14 15

CARDINAL

for ( int step =

0

CARDINAL

; step < MAX_STEPS_LIGHTS ; step ++ ) {

16

CARDINAL

position += lightDirection * marchSize * float ( step ) ;

17 18

CARDINAL

float lightSample = scene ( position , true ) ;

19

CARDINAL

totalDensity += lightSample ;

20

CARDINAL

}

21 22

CARDINAL

float transmittance =

BeersLaw

PERSON

(

totalDensity

ORG

, ABSORPTION_COEFFICIENT ) ;

23

CARDINAL

return transmittance ;

24

CARDINAL

}

25 26

CARDINAL

float raymarch ( vec3 rayOrigin , vec3 rayDirection , float offset ) {

27

CARDINAL

float depth =

0.0

CARDINAL

;

28

CARDINAL

depth += MARCH_SIZE * offset ;

29

CARDINAL

vec3 p = rayOrigin + depth * rayDirection ;

30

CARDINAL

vec3 sunDirection

PERSON

= normalize ( SUN_POSITION ) ;

31 32

CARDINAL

float totalTransmittance =

1.0

CARDINAL

;

33

CARDINAL

float lightEnergy =

0.0

CARDINAL

;

34 35

CARDINAL

for ( int i = 0 ; i < MAX_STEPS ; i ++ ) {

36

CARDINAL

float density = scene ( p , false ) ;

37 38 39

DATE

if ( density >

0.0

CARDINAL

) {

40

CARDINAL

float lightTransmittance =

lightmarch

ORG

( p , rayDirection ) ;

41

CARDINAL

float luminance = density ;

42 43

CARDINAL

totalTransmittance *= lightTransmittance ;

44

CARDINAL

lightEnergy

PERSON

+= totalTransmittance * luminance ;

45

CARDINAL

}

46 47

QUANTITY

depth += MARCH_SIZE ;

48

CARDINAL

p = rayOrigin + depth * rayDirection ;

49

CARDINAL

}

50 51

CARDINAL

return lightEnergy ;

52

CARDINAL

} Because of this nested loop, the algorithmic complexity of our Raymarching loop just increased, so we’ll need to define a relatively low number of steps to sample our light while also calculating a less precise density by reducing the number of Octaves in our FBM to preserve a decent frame-rate (that’s

one

CARDINAL

easy win I implemented to avoid dropping too many frames). All these little tweaks and performance considerations have been taken into account in the demo below: Info An icon representing the letter ‘i‘ in a circle In the demo above, try to:

Arrow An

ORG

icon representing an arrow Move the light source around and notice how it interacts with our more "physically-based" cloud.

Arrow

ORG

An icon representing an arrow Try to tweak the ABSORPTION_COEFFICIENT and see how it impacts the resulting cloud.

Arrow

ORG

An icon representing an arrow Try to increase or decrease the

MAX_STEPS_LIGHTS

ORG

number and notice how more light is accumulated the more we sample it at a given marchSize Anisotropic scattering and phase function Until now, we assumed that light gets distributed equally in every direction as it propagates through the cloud. In reality, the light gets scattered in different directions with different intensities due to water droplets. This phenomenon is called Anisotropic scattering (vs.

Isotropic

ORG

when light scatters evenly), and to have a realistic cloud, we can try to take this into account within our Raymarching loop.

Diagram

ORG

showcasing the difference between isotropic scattering and anisotropic scattering when sampling our light energy. To simulate Anisotropic scattering in our cloud scene for each sampling point for a given light source, we can use a phase function. A common one is

the Henyey-Greenstein

ORG

phase function, which I encountered in pretty much all the examples I could find on physically accurate

Volumetric Raymarching

PRODUCT

. Phase functions Here’s a more detailed read on the Henyey-Greenstein phase function if you want to learn more about it. It’s also important to note that this phase function is

one

CARDINAL

of many. For example, you might encounter

Mie

PERSON

and

Rayleigh

PERSON

phase functions in many

Shadertoy

ORG

demos for anything related to atmospherical scattering. The GLSL implementation of this phase function looks as follows: Implementation of the Henyey-Greenstein phase function

1

CARDINAL

float

HenyeyGreenstein

FAC

( float g , float mu ) {

2

CARDINAL

float gg = g * g ;

3

CARDINAL

return (

1.0

CARDINAL

/ (

4.0

CARDINAL

* PI ) ) * ( ( 1.0 – gg ) / pow (

1.0

CARDINAL

+ gg – 2.0 * g * mu ,

1.5

CARDINAL

) ) ;

4

CARDINAL

} We now have to introduce the result of this new function in our Raymarching loop by multiplying it by the density at a given sampled point, and what we obtain is more realistic lighting for our cloud, especially if the light source moves around. Introducing the Henyey-Greenstein phase function inside our Raymarching loop

1

CARDINAL

float raymarch ( vec3 rayOrigin , vec3 rayDirection , float offset ) {

2

CARDINAL

float depth =

0.0

CARDINAL

;

3

CARDINAL

depth += MARCH_SIZE * offset ;

4

CARDINAL

vec3 p = rayOrigin + depth * rayDirection ;

5

CARDINAL

vec3 sunDirection = normalize ( SUN_POSITION ) ;

6 7

CARDINAL

float totalTransmittance =

1.0

CARDINAL

;

8

CARDINAL

float lightEnergy =

0.0

CARDINAL

;

9 10

CARDINAL

float phase =

HenyeyGreenstein

PERSON

(

SCATTERING_ANISO

ORG

, dot ( rayDirection , sunDirection ) ) ;

11 12

CARDINAL

for ( int i = 0 ; i < MAX_STEPS ; i ++ ) {

13

CARDINAL

float density = scene ( p , false ) ;

14 15 16

DATE

if ( density >

0.0

CARDINAL

) {

17

CARDINAL

float lightTransmittance =

lightmarch

ORG

( p , rayDirection ) ;

18

CARDINAL

float luminance = density * phase ;

19 20

CARDINAL

totalTransmittance *= lightTransmittance ;

21

CARDINAL

lightEnergy

PERSON

+= totalTransmittance * luminance ;

22

CARDINAL

}

23 24

QUANTITY

depth += MARCH_SIZE ;

25

CARDINAL

p = rayOrigin + depth * rayDirection ;

26

CARDINAL

}

27 28

CARDINAL

return lightEnergy

29

CARDINAL

}

Arrow

ORG

An icon representing an arrow

Arrow

ORG

An icon representing an arrow

Comparison

ORG

of our physically accurate

Volumetric

PRODUCT

cloud with and without applying the

Henyey-Greenstein

ORG

phase function. The final demo of this article below showcases our scene with:

Arrow An

ORG

icon representing an arrow Blue noise dithering

Arrow

ORG

An icon representing an arrow

Bicubic

PERSON

filtering

Arrow

ORG

An icon representing an arrow

Beer

PERSON

‘s law

Arrow

ORG

An icon representing an arrow Our more realistic light sampling

Arrow

ORG

An icon representing an arrow

Henyey-Greenstein

PERSON

phase function The result is looks really good, although I’ll admit I had to add an extra value term to my light energy formula so the cloud wouldn’t simply "fade away" when dense parts would end up in the shade.

Extra

ORG

value added to the luminance formula

1

CARDINAL

float luminance =

0.025

CARDINAL

+ density * phase ; The need for a hack probably highlights some issues with my code, most likely due to how I use the resulting light energy value returned by the Raymarching loop or an absorption coefficient that’s a bit too high. Not sure. If you find any blatantly wrong assumptions in my code, please let me know so I can make the necessary edits. Some other optimizations are possible to make the cloud look fluffier and denser, like using the

Beer

PERSON

‘s Powder approximation (page

64

CARDINAL

), but it was mentioned to me that those are just used for aesthetic reasons and are not actually physically based (I also honestly couldn’t figure out how to apply it without altering significantly my

MAX_STEPS

NORP

,

MAX_STEPS_LIGHTS

ORG

, and marchSize variables 😅 and the result was still not great).

Cody Bennett

PERSON

@

Cody_J_Bennett

PERSON

@MaximeHeckel Note that beer-powder is a non-physical approximation. Maybe see: https://t.co/LlHxW5x5sb

3:17 PM – Oct 7,

TIME

2023 1 2

DATE