I've recently begun learning CUDA, and I've stumbled upon a very strange behavior I can't understand.
My code essentially computes an average execution time for a simple atomicAdd kernel. To accomplish this, I call the kernel in a loop to get a better average. I include the device memory allocation and copies in the loop as I want to include this in my execution time estimate. The problem is, the program often fails with Runtime API error 30 if the number of runs through the loop is too high.
I suspected that I might have an issue with my memory access, so I've run memcheck on the program to no avail. There are apparently no memory errors. Also, if run the kernel only a few times, there are no issues, which would also seem to indicate the kernel isn't exactly the issue. It's only if I call it too frequently in succession that I have problems.
A skeleton of my code follows:
for(int i = 0; i < runs; i++)
{
//////////////////////////////////
// Copy memory from Host to Device
//////////////////////////////////
cutilSafeCallNoSync( cudaMemcpy(dev_waveforms, waveforms, num_wf * wf_length * sizeof(float),
cudaMemcpyHostToDevice) );
cutilSafeCallNoSync( cudaMemcpy(dev_delays, delays, num_wf * sizeof(int),
cudaMemcpyHostToDevice) );
////////////////////////
// Kernel Call
////////////////////////
kernel_wrapper<float>(dev_waveforms, dev_focused, dev_delays,
wf_length, num_wf, threads, blocks, kernel);
//copy back to host memory.
cutilSafeCallNoSync( cudaMemcpy(focused, dev_focused, J * wf_length * sizeof(float),
cudaMemcpyDeviceToHost) );
}
Again, this only fails if runs is sufficiently large. There are other strange things going on, but I'll leave it at this for now.
Oh, I'm developing on Windows 7 using Visual Studio 2010. My GPU is also acting as my video card, and I'm worried this may have strange effects.
Thanks in advance!
cudaDeviceSynchronize();inside your loop. – Roger Dahl Aug 30 '12 at 23:43