The solution is to get a graphics card with more VRAM or use another algorythm. DAIN uses A LOT of VRAM. They recommend at least 10GB for 720p interpolating. If it's linear with number of pixels, 4GB should be enough for DVD 16:9 max (ie 853x480) and 2GB for ~ 610 * 343 resolution (if it's 16:9)
Yea, that also sounds like a failure to allocate memory for the CUDA Neural Network model. Try splitting the frames into sections in the "Fix OutOfMemory options" panel. If this is not the case then maybe something is wrong with your CUDA.