Optimized Rendering and MENTAL RAY FOR MAYA bible:   (in progress, almost weekly update)



“Good models create great render times”



How to model to Render View, which is the main asset of optimized rendering?


01:  Show my initial Poly Cube


02: shows a more defined state


03: this one I smoothes to see what the “EDGES” are doing of the object,  You see the get round and don’t maintain their proportion.  That’s something that’s really bothering me, in maya, unlike nurbs which you can just add an isoparm, and it keeps proportion.


04:  Unsmoothed and we added some faces with the Cut Poly Tool , and with the Split Poly tool.


05: Now we smooth object 4,  and we see if we look in the image below  (the render one) that the middle is kinda tessalated, and doesn’t look smooth.


06: We add some Faces, near the most outer edge of our surface


07: The smoothed final version and see that the corners have more polys than the entire object!!!!..  Most people just grab object 3 and keep moving points and smoothing, until

the object is black, or they keep using the “Poly Cage” and they keep adding poly lines(faces).


With my airplane, I modeled, and then rendered, to check with the amount of polys, I would be satisfied.  My shots were from far away, that’s why my plane is pretty low poly.




The Finish Product, after a sphere and final gather turned on, looks like this:





This modeling will help out, for render techniques like Global Illumination or Final Gather:




Only 8200 polys for the body,  Look in Vector Render section for a wireframe render.


What more can you do?





-01 killer

If you use a pointlight it creates 6 shadow maps  ->  so in other words renders 6 shadows, it’s optional to turn the maps off.


-02 killer

3d Projected textures -> so convert them to 2d textures and then put the image map in the Sourceimages and relocate it to the shader


Multilister/ Hypershade -> Edit -> Convert to File Texture



Hypershade, takes up way to much memory, use the multilister to have a quicker life.



Go to your camera Auto Render Clip PLane (turn it off) ... set the clipping planes manual, increases time for Rays when you render.. (they'll travel less distance)


-05 killer

if you use Shadow maps, you can turn on Reuse Existing Shadow map

First write them -> Rewrite Shadow map


than do Reuse Existing Shadowmap, fill in the names and voila.


Mental Ray, does it automatically, so just file in the name.


!!! Beware this only goes for non moving objects, that are going to keep the same map!!


-06 killer

Optimize Scene, always do this before you render, just turn every option on and check if you didn't loose any animation.




Things that don't reflect or don't have motion blur, just turn it off..


Mental ray will motion blur everything, so use layers. (different render passes)



Memory and Performance Options (maya software)

            -Recursion Depth : don't change, but if your raytrace stops and the computer is out of memory,  turn it to 1 , this will use more CPU and less memory.



Mental Ray:     put Contrast A of sampling quality to 1 if you don't need an alpha channel


                        Jitter: makes it go faster, and gets rid of render errors (like dirt etc)


                        Maya's Filter is set to  Triangle so why not mental rays!  makes it go a lot faster.  Maya's Raytracing,   Max Ray depth is set to 10 so why

Not Mental Ray's , and again we gain speed.





Types of rendering:





-First some thesis Tips


[Go to File->  Save Scene Options-> Incremental Save On]


Turn Incremental Save On, every time you save now it will put a number behind

your shot!!! a new one yes!!


so you won't have to go Save Scene as ,,   01 02 .. nope


[Transform Tools Shortcut]


When you move an object you sometimes want to use a different Orientation,


well just hold   W, E,  or R  down and click with the RMB


choose How you want to move it-> object, world, etc..




Super use, go to Create-> Annotation  (you can make little Post IT notes to your camera, or objects, they'll always face the camera)








Tweaking a million lights -> use attribute spreadsheet (the intensity, shadows on etc is in there)


Glow:  To just have  the highlights glow,  go to Shader Glow and Upper the Threshold

            this only makes the Specular glow!


Shadow light:  just create a spotlight and duplicate it, now turn the 2nd one to a Negative intensity..

                        if you place a 3rd light to light the scene, the   2 spotlights only create shadows for                         the objects they light.!!!


Depth of Field:  To control it easier, go to Connection Editor, and connect Center of interest in the left to Focus Distance on the right.. Where your Center of Interest will be is where it will be the point of focus.


Raytracing:  Only object that have a different Refraction will have a good raytracing aspect to them.

                        glass it's index is: 1.6


                        Reflection Specularity:  Control the Highlights in Reflections


                        Light Absorbance: How hard it is for the light to pass through the object


                        Surface Thickness: You can make a surface look thicker, i think it offsets the                     reflection more


Jitter:  turn it on in the sampling quality, will speed up and will take care of unwanted things in your render





-First of all some explanation bewteen differences in shadows:


What does what?


[Maya Software]


Depth Map shadows:


Dmap resolution:   This basically shows how many pixels it takes to generate the shadow, it creates a little map for it. The resolution goes in map size.  


    128 /  512  /  756 / 1024


Dmap Filter Size   Basically Blurs the shadow  (it creates samples)


Now, the higher the Dmap resolution the softer and better quality the shadow will be.


[Mental Ray]


You have to have Depth Map shadows of maya on


in order to use Mental ray depth map


so turn both on!


ok here comes a smart trick just, change the settings of maya's depth map shadow and just hit, Take Settings From Maya button under the Mental ray SHadow Map Tab:


Resolution :  is the res of the map

Samples: The fading of the blur  (more samples more render time, so optimize this)

Softness:  how much it should blur


[maya software and mental ray]


Raytracing Shadows:


I'm going to suprise you today:  I can achieve the Same effect as a Depth map!


Light Radius:  controls the Blur of the shadow,  numbers starting form 1 / 3 / 6 / 9 will blur the shadow!!


Shadow Rays: does the same thing as with depth  map resolution it   creates more RAYS!!! so creates a softer and higher quality looking shadow...





Maya Vector Rendering


You can find all kinds of plugins, like toonshader or you can buy Wiredata, (great plugin, seriously if I had money) Cambridge’s Software or Cebas FinalToon, but I don’t like buying new stuff. I want maya to do all these things,  I don’t like pluging thing into something that can run on it’s own. Maya is not a  cow where you can just stick your hand in, get some milk out. Leave the plugins, because when you have to render on a farm or for companies, they’re not so into installing the one plugin you like.


[Wire Frame Render]  


Everybody that's paying for the WireData shader,

                                    maya 5 comes with Vector Render, which is free wireframe renders..


                                    -Window -> Preferences-> Plugin Manager

                                    -Turn VectorRender.mll   on

                                    Go to render globals-> open Maya Vector tab


                                    for wireframe render






Render Globals:



File Name Prefix:  "Your_image_name"

Frame/Anim Ext:  name.#.ext

Image format:  Targa



Start and end frame




Maya Vector TAB:


Appearance Options:

Curve Tolerance= 0


Detail Level Preset:

Automatic or High Quality (blurs the lines)


* = Fill Objects

Fill Style: Single Color



  = Show Back Faces (On or Off depends on your goal, I would leave it on)


* = Include Edges


Edge weight=Hair

or  0.200 (any value, makes the wireframe’s line width thicker)


Edge Style = Outline



* = Edge Detail:



(here’s how you render globals should look like)


Ok, now select all your objects



and go to Polygon-> Normals-> Soften/ Harden

(make sure the settings are on 0.0) 




Render in Any File Format of your choice, and voila here you go.




[cartoon rendering]


Here’s a Maya Vector Rendering:



1 color no outline


1 color and an outline

1 color and outline., I didn’t have a separate beak..

1 color and outline, the character has separate pieces of geometry..

Here are the settings, for Maya’s Vector Render:



Hardware Render Buffer


Maya Software RayCast


Maya Software ,  FAKE  GI<  and FAKE HDRI…  (and the every object glows trick)


Raytracing Maya Software


Mental Ray normal



First we're going to activate the Mental Ray Shaders in Maya, this will add a shader tab in the Hypershade.


Go to ../ My Documen/maya/ and you'll find a file called: "Maya.env".



Right Mouse Button click and do "Open With" , now select the Notepad and copy this line into the file.





Save the file and now start maaya, You'll now find the new shaders available.



Cartoon Rendering / Contour Rendering:


My first try gave me this image:

ple Contrast Shader in the Mental Ray globals.

(creates a very nice and hard contrast in your image)


I prefer this image.. I like the look for some reason.  Feels like a Japanese 3d movie..


To achieve the Cartoon look:


A: You turn on Mental ray, IN the plugin manager.


B: You open the Render Globals,  switch Render  to Mental Ray.



C:[Now we have to create and connect some shaders]


-Contour Composite Shader(in the Camera Attribute Editor,   is the Output Shader)




Now the next two shaders are in the Render Globals, under the mental ray tab,   In CONTOURS!!


-Contour Contrast Function Simple   Shader(shader is in the contrast shader, output.. in Render Globals)


-Contour Store Function  Simple  Shader(shader is in the Store Shader, output in Render Globals)



Give me a few more, I’m figuring this out while we speak.

not giving up yet.. This one isn’t it either, however this is a cool outline…

I just did changed the output shader in the perspective camera  to a  CONTOUR ONLY SHADER…

I made the color grey.


Now I put the Contour Composite Shader back in the Perspective camera,  in it’s output shader.


Ofcourse the easy way out is to just make the objects separate, but I want to keep my texture.


After a while and some more research, I discovered how to make it look nice and cartoony.


Big Plus I discovered in Mental Ray’s  Cartoon Shading,   is the image has an ALPHA channel!!..

That’s something I was missing with maya’s vector render.


This is how you do it: 


Turn Mental ray on, and turn the Max samples to 3  (in the sampling quality, under the mental ray tab)


See example below:


First of all we go into the renderglobals and connect to shaders


By clicking on the “checker” box next to Contrast Shader.


We select the shader called: “Contour Contrast Function Levels”


Here is the example:

then we set the settings like this:




Now we go back to the render globals and click on the 2nd shader,

The Store Shader


(in other words we click again on the Checker box)


As soon as we select the shader

The Connection Editor follows.



Now this is the most important thing… We go to   Left Display -> Show Hidden

(now all the hidden Render Globals are turned on, and Visible in the Connection Editor)


Here’s the example:


Now we take the slider on the left all the way down, and at the end of this list we see:


“Contour Store”


We connect Normal_Geom (on the right) to Contour Store (on the left)


Here’s the example:

Now the this Store shader is active and will work!!.


This was the most difficult thing to do.


Now we open the Attribute Editor of the camera that you want to render Cartoony with:


In my case Persp.


Click on Output Shader,  and out of the Output Shaders select   CONTOUR ONLY.

Change the background Color.

Now we’re going to the materials.


Rendering Editors ->  Multilister / Hypershader (multilister has better performance, and shows unused shaders, or lost




We create a Lambert Shader, and then click on the “Little Cube Arrow” to go to the shaders Shading Group.

We see the Mental Ray listed in the Shading Group.

Open it up.


And voila we can connect shaders, and create our own.


Just click on the “checker box”

And select the   “Contour Simple Shader


See example below:





Now Render your image, and it should look like mine.


In order to have it not be an outline, but look like a cartoon.


You can change the Camera’s  Output Shader.


And change it into a Contour_Composite shader


Turn Color and Alpha and Contour on (there’s options where you connect the shader)


Rerender and now your image will look like this:



Now if you turn a shadow on, I used a raytraced shadow of a light.

It will look like this:


Now if you just turn the Ambient up on all your shaders it will

Look like this:

ever heard of that trick, where you create a shader manually, that acts like a Ramp Shader

(You connect a ramp to it’s facing ratio of a sampler info)


Do that and you get this:




Now if you go into the Contour Shader options, and set the option to Diff_Index,

(if you turn Max_level to 2, you’ll create a wire on both sides)

You’ll create a wireframe render:



Now I’m going to tell you something cool, in the Materials.  I mean the shaders you can connect any Mental Ray Contour

Material, and they’ll change line width, or they can create different width according to the camera.


My model wasn’t really meant for cartoon shading, but it still gave a nice effect to it.



Mental Ray Caustics (this can be added to Global Illumination, Final Gather or Both)



In other words the way photons refract true the glass. I’m also going to talk about Dielectric Shader and the DGS shader.

Dielectric Shading:


This is an interesting rendering style, mainly used to create glass with liquid inside of them, or just normal glass.

Ice, DGS material



Ice/ Frosted Glass, DGS material


You create a Phong shader in the multilister/ hypershade, and just leave it default.

Now click on the  box with arrow, to go to the shaderGRP.


After that you  put a DGS material shader in the Material Shader and you drop a photon DGS

Shader in the Photon Shader.

Now these shaders work in relation with eachother, if you change the material one, you have to change

The photon one.


Somebody asked me for my settings, so here they are.


Some explanation real quick for what everything is:


DSG: Diffuse, Glossy and Specular Reflection and Transmission.


Shiny= this blurs the reflection, by giving the material more diffuse (sort of bump to the material)

 Number:    5/  12 / 20 create very Blurry reflections

    100 or higher almost no blur,   and 0.0 just leaves it to be a mirror..


Shiny U and V= great option, makes the reflection Anisotropic. Same values as Shiny, but they blur in one direction.

Think of making  a metal pan to cook with. The reflections on construction pipers, or air conditioning. Those are all,

Reflections that get smeared out.


Transp:  is just how transparent the object should be.   1.0 is totally transparent

And 0 is solid.


IOR= the Index Of Refraction, or in other words, the refraction index of your material

Glass; 1.6    Diamond; 2.6 Water; 1.2;  Frosted glass; 1.4

(you can look these up online)


Mode=  controls the light list (all = 0 , inclusive = 1, exclusive = 2)




Now you have to connect he lights to the photon shaders. I don’t know why you have to, and it slows down the render time.

I haven’t tried not doing this yet, I have a feeling it doesn’t really matter, but I’ll explain how to do it.


Open the Outliner,  Windows-> Outliner


Go  RMB (right mouse button click/ hold down)->  Show DAG nodes Only (turn this option off)


Now scroll down till you see this:

Select the DGS_material_photon.    

Open the Connection Editor, and Reload this one on the right.


Select your spotlight and load it on the left.

Now connect    “Message  to “Lights”


Voila now your done with that.

You have to do this with all your DGS/  DiElectric shaders. 

These are my spotlight settings.


You see a hundred thousand photons.


Now I threw in a Physical_Light to get more accurate Caustics.


Here comes the big secret behind maya and mental ray.

You can up any value great than you’d think it could be.

Especially with HDRI, this is a big deal, cause if you want the HDRI map to be brighter you’ll have to adjust

The V greater than 1.

You see.


Now do your render and you object should look exactly the same like mine.

I added another point light with similar settings to add more light into the glass. (glass depends on light, if you don’t have enough light it will have dark spots.)



Mental Ray Global Illumination:


Now this is a hard one to explain.

2 lights,  Global Illumination


How does it work.


The photons are comparable to shooting paintballs in a room.  From the light source, the photons get shot into the room and

whenever it hits a surface creates light more saturated closer to the distance it was shot from and the farther it will have of course shadow or less saturation.

What’s great about the photons is that when you shoot them, they bounce off, therefore creating indirect light, and that’s what we might refer to as “bounce light”.

Real physical nature has a lot of the ‘bounce’ going on. Mental ray can use Area light to create a physical correct environment, think of the ligthboxes in photography.


The bigger the area light the softer the shadows. You can use raytrace shadows to create the most accurate shadow types.

When you create  a spotlight you can turn Area light on, under Area light. The sampling  and Low Sampling, create the overflow/ resolution of your shadows.

If we set the intensity to Linear Decay Rate, we’ll get a more physical light. Now that we set the light to decay -> linear we have to make the intensity much bigger.


This will just give us light in the room.


After this we can Emit the Photons. Just go into the light attributes and turn ‘Emit Photons’ on.


The Exponent means how the ‘light flows out into the scene’. It’s like decay.


The energy is comparable to light intensity, it just fills the room with it’s light.


Not enough energy is floating in the room, so lets boost it up a knotch, from 20.000

To 100.000


At first we’ll just have a situation like this (one light).



Now we add another light inside the room, let’s make it a pointlight, with 0 intensity, but we want photons to fill the room.

We’re going to put the decay rate to Quadratic.

The photons are this time emitted with the energy of 100, and Exponent 2.


You see the room gets a lot more light.

Now you just have to play around with the settings, but this is the basic principle.


The photon values work as following.




Or in other words:          


            R =  1.0         = equal to the amount of photons

            G =   1.0

            B =    1.0


A easy diagram:


You cross multiply to get the photon color values.

I usually set the color selector of maya to the color I like, and then calculate


The color of your light can be different from the color that gets bounced of into the room.



Mental Ray Final Gather:



This technique is still my favorite, it’s based on lighting with objects. Final Gather uses Ambient lighting. Hang on, does that mean I can just up the ambient

Of any object and the object is being lit?  Yes, even the Camera’s Environment Color, can brighten the scene.


Nowadays, this technique is used to create Industrial Design  and Transportation objects. Final Gather, gives metal the best look, it just looks amazing. What’s also

Great is that you can use normal maya lights in combination with Final Gather. It’s a 2 for one bonus. With Global illumination, things will start looking weird when you use normal lights,

In the realm of Final Gather, things can look awesome.


How to use it?  


-01:  Turn  Mental Ray on;    Window->Settings/ Preferences-> Plugin Manager




02: Go into the Render Global Settings:  Window-> Rendering Editors -> Render Globals

Turn the render using to Mental Ray,   After that Disable -> Enable default light   


Now if you render, everything is Black!.. yes..



(for maya 4.5 with mental ray, you had to create a pointlight, and set the intensity to 0, that’s still possible)



Now after you did this:


You now go to the Mental Ray tab in this Render Globals menu:




I have setup my character with shaders, and I did


Create -> Polygon Primitives-> Sphere


I cut it in half and made it big, so that it’s big enough to light my scene.


Now I add my lambert shader to the Sphere (some call it DOME)  and voila… Done for now.




I opened  the  Render Globals again. Click on the Mental Ray tab,   Put the Quality on,  PRODUCTION QUALITY.

Go to Final Gather, and turn Final Gather on.


Then put Final Gather rays  on 2000

                                                Min Radius:  0.1

                                                Max Radius:  1.0


Done.   We get image one.. 


First image has no light, just a dome around it  (with a shader lambert on it, and the ambient turned up a little)


Second image, has the same  dome, but with a directional light, which we turned the Raytrace Shadow on, in it’s attributes.

(just plain old simple maya light)



Let me explain:


There are 3 settings that relate to each other. We want to go for the best and fastest setting.


  1. Final Gather Rays
  2. Min Radius
  3. Max Radius



Let me start with the Radius, because if you set that one right, your render time will be the fastest, and you won’t have any weird light things going on.

(light artifacts,  or light escaping through the object)


B+ C  together at once:


Use the Distance tool, and just Measure  what the EXACT RADIUS is of your scene. My bird actually has a smaller radius, but I just wanted to show you how to do it.

Later on we’ll see how much this matters.



The radius  is    5.2,    now



The min radius  will be:      5.2  x  0.01   =  0.052  


The Max Radius will be:  5.2 x  0.10 = 0.52  


Now we type that in the render globals.



After that we’re going to set the Final Gather Rays::


I took 4 different settings for showing the difference (most people say the 5000 has the best Shadow Quality)


First image is done with 300 rays,  Second image with 1000 rays

Third image is done with 2000 rays, and the last one with 5000 rays.


Ending up with different artifacts.


Look around the feet.


Now you might think why did his first image in the beginning of my tutorial look better (the one with the shadow of the directional light), well…

I tricked you!. 


The last one, 2000 FG rays with a Diff radius?    That’s the setting I used.


I used  Final Gather Rays 2000 and the Min Radius set to 0.1 and Max Radius 1.0.

Why?  Because I happen to know that below a min radius of 0.1 there’s not any effect anymore, but a low min radius and a high max radius

Is all that matters!!!

You can have Final Gather rays at a 1000 and with a Min radius of 0.7 and a max radius of 7.0, the image can look magnificient.


My new rule of thumb is:


The min radius  will be:           scene radius in units(must be >10)  x 0.01   =  Solution

The max radius will be:            scene radius in units   x 0.1 =   Solution


And most likely Final gather rays  of  1000 or 2000 should be enough..  ( A certain book, from a certain software company talks about  only having to use 600 rays.., so 1000 is pretty high already)


For the best Rendering Speed.


If it still has artifacts  make the range between them much bigger,   instead of  Min Radius 0.1,  and Max Radius 1.0, try  0.1  and 100,    or 1  and 1000… etc..


Now how can we make the render fast?


Go to render globals:


Set the              Max Reflection Rays to 3

                        Max Refraction Rays to 4

                        Max ray Depth to 7


                        Jitter On.

                        And Filter to Triangle


This will improve your rendertimes… I’ll explain later why?



Mental Ray Final Gather-> HDRI


My favorite technique,  I love rendering in HDRI. Anything that’s metal, glass or needs to look extremely real, can look real in a few minutes. New movies use HDRI more often. Somebody that invented the theory behind this is Mr. Paul Debevec. Mr Debevec created a program called HDRI shop, and this is all available on his site  www.debecev.org.


For compositors / animators what’s so interesting about this HDRI map. Well if I have photographs or film, in different angels and I connect theses images or footage together. I can then project these images on planes/ domes and voila the lighting matches exactly up with the character/ object in the environment. This makes compositing CGI into a real life situation super easy.

What is HDRI?


Well, it’s just a very high resolution reflection map. The map has High Dynamic Range Image, or in other words, knows where things are lighter and were things are darker.


Now I discovered we can alter these maps very easily, and therefore decide what gets darker and what doesn’t.


I’m going to explain who things work.


Here’s an image without HDRI, but just final gather.



Below and image with Final Gather, the plane looks more shiny and metal like.



You use the same settings as  the Final Gather above.  HDRI in mental ray, is  just a Final Gather, with an image map added in the dome.


Now, how do I create one of those image maps?


For mine I went online, and I found this website with great Free Skies.


For now we’re just going to use my home made uncopyrighted and fast to use HDRI map.


Online you can find “Light Probes” and use those for testing.


Well my home made HDRI map of today.


Now we’re going to get an amazing piece of software from Mr. Debevec. The software is available on his site.




Let’s use look at his program.


Start the program.


Go to Create-> Assemble HDRI from Image Sequence.


This let’s you load   LDR images  (low dynamic range, in other words just normal Tiffs with less information stored in them)


Now, if you had a camera and you’d photograph a reflection sphere in your environment, or have a film camera film it, will the environment changes.

After this you could assemble the image with HDRI shop or with Realviz Image Modeler to create an HDRI map/ Light Probe.


Then we go   Load Images  (I don’t have one right now with different F-stops and different light intensities),  we load the image

And then we hit  Calculate, which calculates the light intensities in every area of the image.  The more different images the higher the accuracy.

I figured something out, which is that if you change the levels in photoshop and you save out 3 different images.


After that you put them together with HDRI shop and you have a Fake, but functional HDRI image.


Then we hit, Generate Image.


Now we save the image as an .HDR and done.


 In order to get this image working, we have to create a dome (just a polygon sphere), and project the texture on the inside, unfortunately Maya can’t recognize

an HDRI map, and won’t display it. Only when you render with mental ray, will the map be shown, I usually save out a JPG or tiff aswell, next to the HDRI, so I can see where I’m placing the image.


Lately if you hear me say the Ambient is 0.5, I mean that the Ambient of the shader is 0.5 (the  HSV,   the V will be 0.5)


This is the situation, similar to earlier. It’s the exact Final Gather setup, but now we’re going to stick an image on the sphere, that’s going to reflect and create different

Light intensities.


This is without HDRI..


We select the dome (the half poly sphere) and we go to Edit Polygons-> Texture -> Planar Mapping.


Then we eye it out for now,  usually for exact placement I create an extra JPG or TIFF from the HDRI map.



Till it looks like this.  (you see the resemblance with our HDRI map)


Now we go to the shader of the Dome (half poly sphere) and place the texture in the Ambient and Color.


Now we render and here’s the result.



We can also just leave the image in the color, or just the ambient. (for my plane the color has the HDRI only)



That’s the technique now try it with a better image, for a more realistic looking image, or download somebody’s HDRI probe.




Mental Ray FANCY  -> Global Illumination and Final Gather






How can we implement  this knowledge into making a photoreal image, or create a photostudio environment for your objects.

I’ll talk about that now and explain.


If you’re an industrials designer and your showing your product,  a car, a washing machine etc. You want to show this off in a studio environment.

In this case HDRI or plain Final Gather will do the trick.





                                                                        Page 02: Lynx Project  Car Commercial 2005 in Mental Ray




COPYRIGHTED BY ME, Oliver Heijmans©

All Debevec’s Theories and programs are copyrighted to him. Alias/ Wavefront Maya is found online at www.alias.com. Mental Ray has their own site, www.mentalimages.com.




For Questions email me at: [email protected]


My main Website is    www.olivervisualfx.com




Many Thanks to www.deathfall.com

You guys made my site.

Their admins are awesome!