Do you have the same problem?
After some time of working on a scene cycles suddenly stops to render it or crashes while rendering, even if it worked before.
If you are using Blender on windows and get an error that include “Launch failed in cuCtxSynchronize()” then you can eventually save a lot of time by reading my hint below first, before starting to search for an own solution.
First basic informations why the problem appears
Actually it is not a Cycles problem alone. The main reason is found in the hardware, your NVIDIA card. NVIDIA cards offers the CUDA API for applications to calculate complex mathematical routines in a very fast way. If you don’t know CUDA, following a short explanation: CUDA is a kind of interface for programmers to use the graphic card co-processors. Some, but not all calculations, can be outsourced to a graphic card and it computes them very fast because of parallel processing. The caclulations works faster than the most single CPUs can do it today.
The CUDA library (or API) was developed by nvidia and works on special nvidia graphic cards only. OpenCL is similar to CUDA, but not that fast. The advantage of OpenCL is to be compatible to the most modern graphic cards and not bound to an individual graphics hardware.
Enough of basics
Back to our problem. I try to explain it for people who are not coders (me too) and using Blenders Cycles render engine: Cycles sends computing informations to the graphic card by using the CUDA library. Depending on how complex the data is, the library needs a lot of time for the calculations. Sometimes the calculation time can get very long. This is basically not a problem. But meanwhile the graphic card does not respond to the operating system too, in my case: Windows 10.
So if the graphic card needs too long to respond, windows declares this as a driver problem and resets the nvidia drivers for security reasons. But in this case cycles stops or Blender even crash and all information that was sent to the CUDA cores is lost. Your rendering interrupts…
One solution could be to switch cycles to render on your CPU. There you will have no problems, for shure. But honestly: Why using a CPU if you can use your CUDA card? A rendering time example: My recent interior scene needs 12 hours to render on my old i7-3770 CPU. The image resolution is 1920×1080 pixels. For the same image my graphic card needs 45 minutes, a NVIDIA GeForce GTX 1080 Ti. – You got it!? CPU? -> No way!
So there must be another way for a solution. And there is one. We have to tell windows to wait longer for the graphic card response before resetting the graphic card drivers. The default windows waiting time for a driver reset is 2 seconds. In the most cases this value is ok, but if you are starting with complex calculations it could run into a problem.
So how to increase the delay time for the driver reset?
Before you start
Before we start, a WARNING! The solution for our problem is to change/add a system registry key. Please be aware of changing something in your registry. Mistakes could run into real OS problems. All you do is on your own responsibility. I am not responsible for any problems you get after using my hint.
Microsoft has a short explanation about the registry keys we will use. You can find the full documentation here: TDR registry keys.
Before you follow the next steps, you can play safe and check your system for the following event error: Event ID 4101.
If this error appear in your system log, then there is a big chance that my solution can really help you.
So here we go.
Solution (Tested on a Windows 10 Pro 64 Bit System)
- Open your system registry by clicking on the windows icon in the toolbar or pressing the windows key on your keyboard. Type in: regedit and confirm with return. The registry editor should now open.
- Navigate in the registry to the following folder:
- Add or edit a new key called TdrDelay.If the TdrDelay value doesn’t exist yet, right-click in the right pane and choose New > DWORD (32bit) Value. Name it “TdrDelay“. In the right pane, double click on the value TdrDelay. Change the Base setting to Decimal. Set the value to something else than the default 2 (EDIT: I tried 16 seconds first but recommend 30 now). If you still get into render trouble increase this value. This value indicate in seconds how long the operating system will wait before considering that the GPU is unresponsive during a computation. Allegorithmic e.g. recommend even a value of 60 for their application!
- Restart your PC
The right pane should now looks like this and you can restart your PC now.
How to reset the values if needed? There are two ways:
- Set the TdrDelay to a value of 2 which is the default even if TdrDelay is not set.
- Remove the TdrDelay key from the registry.
Now the cycles renderer should render all your images in Blender without any problems. My solution was discussed after I posted the bug in the official developer forums. You can read the complete thread with additional infos here: Read the original developer thread.
There is a additional information by Brecht van Lommel (Coder of Blender Cycles) about this issue. I just post his answer here:
OpenCL (and OpenGL and DirectX) have the same timeout issue by the way on NVIDIA cards, it has nothing to do with CUDA specifically. The solution is clear, we need to implement an optimized split kernel for NVIDIA cards, it will just take time to get there.
If my solution does not help you then visit the official Blender developer forums. Maybe there you will find a solution for your issue.