Cel Shader

Mini Tutorial: Normal Editing for 3DCG Anime

In order to control the shading for a proper cel-shaded look, custom normals are what we use to solve that issue. To begin with, it is important to have clean topology on the face for this example.

 During the retopo stage, it is important to place these nice edge loops that mask off the front region of the face.

First, duplicate the original face and move it over to the side somewhere. Select all of the polygons that represent the front of the face and scale them in one direction until they are almost flat. Use soft-selection if you want to avoid clipping in delicate areas (interior of eye/mouth cavity)

Depending on the type of cel shading, you can push the face out a bit and rotate it slightly so that it faces upwards. This allows the face to be fully lit even if the light is directly facing down. Otherwise a downwards facing light would cause self shadowing on the face.

Select the edited mesh and then the target face. In the Mesh tab select “Transfer Attributes”.

In this specific case, only “vertex normal” are needed and sample space set to “topology” since we used a duplicate mesh. The method still works if you used a proxy geometry for the normal editing but results may vary.

Once applied, the normals from the edited mesh will link with the target mesh. Delete history to make the change permanent.

In this example, the lighting on the flat face corresponds to the shading on the target face. The original face has shading that would look poor on a cel shader.

The same technique can applied on the hair as well.

In the Transfer Attributes, world space sampling is applied. World Space sampling needs history deleted in order to store the result, otherwise when moving the projected mesh, the normals will change.

Next-Gen Cel Shading in Unity 5

Introduction

With the arrival of Unity 5 we find that it’s never been so easy to get high quality visuals in our game, but we don’t always want realistic graphics for our project.

Cel Shading is a rendering technique used to simulate the lighting in cartoons and comics. In a more traditional diffuse shading, light creates a gradient on an object’s surface using its normals; Cel Shading polarizes the gradient, making a pixel be lit or not lit by light, with no interpolations or interpolated values.

In the previous image we can see the difference between a diffuse lighting and cel shading. The dot product between the light direction and the surface normal (which we’ll refer to as ‘NdotL’) defines the amount of light a pixel receives. Rounding this value to 0 or 1, or defining different ‘grades’ or ‘cuts’ will result in different cel shading styles.

We are going to introduce different Cel Shading approaches in Unity 5 depending on the rendering pipeline (the rendering path) used. You can set your desired rendering path in your Camera component or your Player Settings.

Forward Rendering

With Forward Rendering, the engine goes over every vertex, every pixel and every light in the scene. The rendering complexity is huge when you’ve got multiple light sources in your scene, but when you don’t use many lights it’s a very lightweight rendering pipeline.

The best way to achieve Cel Shading with Forward Rendering is using a Custom Lighting Model in a Surface Shader.

Shader "Custom/CelShadingForward" {
	Properties {
		_Color("Color", Color) = (1, 1, 1, 1)
		_MainTex("Albedo (RGB)", 2D) = "white" {}
	}
	SubShader {
		Tags {
			"RenderType" = "Opaque"
		}
		LOD 200

		CGPROGRAM#pragma surface surf CelShadingForward#pragma target 3.0

		half4 LightingCelShadingForward(SurfaceOutput s, half3 lightDir, half atten) {
			half NdotL = dot(s.Normal, lightDir);
			if (NdotL <= 0.0) NdotL = 0;
			else NdotL = 1;
			half4 c;
			c.rgb = s.Albedo * _LightColor0.rgb * (NdotL * atten * 2);
			c.a = s.Alpha;
			return c;
		}

		sampler2D _MainTex;
		fixed4 _Color;

		struct Input {
			float2 uv_MainTex;
		};

		void surf(Input IN, inout SurfaceOutput o) {
			// Albedo comes from a texture tinted by color
			fixed4 c = tex2D(_MainTex, IN.uv_MainTex) * _Color;
			o.Albedo = c.rgb;
			o.Alpha = c.a;
		}
		ENDCG
	}
	FallBack "Diffuse"
}

The ‘LightingCelShadingForward’ function calculates the NdotL and polarizes it to 0 or 1. We then use this value as a factor to decide how much light the pixel receives, thus getting a cel shading effect.

That same ‘hack’ can be used with other custom shaders that use specular, normal mapping, rim lighting etc.

Light Pre-Pass (Legacy Deferred)

With a Deferred Pipeline the engine goes over every light on the screen just once and stores that info in a buffer. Light Pre-Pass stores only the light info, while Unity 5′s new Deferred pipeline stores info about all of the geometry in the scene. This approach is much more efficient when you’ve got a big quantity of light sources in the scene, but requires a more complex hardware architecture.

With Light Pre-Pass we can’t use a Custom Lighting model to modify lighting info, as the lighting value is already calculated in the Deferred Lighting Pass. What we need to do is modify the internal deferred lighting pass. In order to do this we must download Unity’s Built-In Shaders from the official website.

We extract the files and look for “Internal-PrePassLighting.shader” inside the “DefaultResourcesExtra” folder. We copy this file to our Unity project, preferably in the Resources folder (create it if you haven’t). If we are using Unity 5 we can change its filename.

Now let’s edit the file and look for the line:

half diff = max (0, dot (lightDir, normal));

and modify it to:

half diff = max (0, dot (lightDir, normal));

if (diff > 0.0f) diff = 1.0f;

else diff = 0.0f;

We save the file and it’s done. We now need to tell Unity to use this file instead of the default Pre-Pass Lighting pipeline.

Go to Edit -> Project Settings -> Graphics and, in Legacy Deferred, change the Built-in shader and select our file. If you are using Unity 4, Unity should already recognize the file (if you haven’t changed the filename) and use it with no user action. Please check that your Camera is using the Player Settings or is using Legacy Deferred (Light Pre-Pass), and that our materials use a Legacy shader. If you did everything correctly you should get cel shading working in your game.

Deferred Rendering

But what’s the point of having Unity 5 if we can’t take advantage of its new features and the powerful Standard Shader?

The solution is to use Unity’s new Deferred Rendering pipeline, which stores all scene geometry along lighting in buffers to calculate the final shading of the object.

Following the same steps as Light Pre-Pass, we download the Built-In Shaders but this time we copy into our project the file named “Internal-DeferredShading.shader”. We edit this file looking for the function ‘CalculateLight’ and modify the next line:

light.ndotl = LambertTerm (normalWorld, light.dir);

And add these 2 lines:

if (light.ndotl <= 0.0) light.ndotl = 0;
else light.ndotl = 1;

Like the last time, we go to Edit -> Project Settings -> Graphics and in Deferred we choose our modified file. Check that the Camera component has Rendering Path = Deferred, and that you are using a model with a Standar shader in its material. You should now notice that your model has all the advantages of the Standard Shader, but with Cel Shading applied.

Toon Outline and Other Effects

You’ve got yourself Cel Shading in Unity 5, but we are still not done to get our next-gen toon look. We need a toon outline.

Let’s import the Camera Image Effects by going to Assets -> Import Package -> Effects.

We now go to our camera and add the Edge Detection component. We configure it at our liking (I usually use Robert Cross Depth Normals). We can now add other image effects like Anti Aliasing to soften the lines, or some more advanced effects like Ambient Occlusion, Bloom or Depth of Field to get that AAA look we were going for.

Conclusions

As you can see, in a small amount of time we ‘hacked’ the new Unity Deferred pipeline to completely change our game visual style. 

In Twin Souls: The Path of Shadows (the game I’m working on) we are using this approach with a few other additions to get our unique look:

I hope this article was useful to some of you, and remember to contact me at david@linceworks.com if you have any question regarding this article.