cuda illegal memory access on tensor slice

[jonathanasdf]

When I run the following code:

cv = require 'cv'
require 'cv.cudawarping'
require 'cv.imgcodecs'
require 'image'
require 'xlua'

local I = image.lena():permute(2, 3, 1):cuda()
local h = I:size(1) / 2
local w = I:size(2) / 2
local t = (I:size(1) - h) / 2
local l = (I:size(2) - w) / 2
local N = 100000
for i=1,N do
  local slice = I[{{t, t+h-1}, {l, l+w-1}, {}}]
  cv.cuda.resize{slice, {I:size(1), I:size(2)}}
  xlua.progress(i, N)
end

I get

OpenCV Error: Gpu API call (an illegal memory access was encountered) in call_resize_linear_glob, file /opt/opencv/modules/cudawarping/src/cuda/resize.cu, line 220
THCudaCheck FAIL file=/opt/cutorch/lib/THC/generic/THCStorage.c line=147 error=77 : an illegal memory access was encountered
/opt/torch/install/bin/luajit: cuda runtime error (77) : an illegal memory access was encountered at /opt/cutorch/lib/THC/generic/THCStorage.c:147

If I replace resize{slice with resize{slice:clone() then it runs fine without error.

[shrubb]

Thanks for reporting. I confirm this bug, on ordinary Tensors as well. Yet another issue of wrong treating of slices :(

Sorry to say I'm now kind of out of condition, so I'll return to fixing this a bit later...

[shrubb]

@jonathanasdf
I've finally found out why it happens.

Both :transpose() and :permute() do not change any Tensor's data, but only the strides, i.e. these functions only change the way elements are accessed. In the case of a 512x512 Lena, 3rd (channel) dimension stride becomes 512 * 512 * sizeof(double) bytes. That is, to access the next byte within a single pixel, you have to add 2097152 to its address. OpenCV has a constraint that channel dimension stride is ALWAYS sizeof(<mat's element type>), in this case 8.

So Torch's transposing functions are not enough in our case. One must afterwards either call :clone() or :contiguous().

Thank you again for exposing this. I'll have to document it somewhere.

[fmassa]

@shrubb I'm not sure that's the reason for this specific case.
Actually, when you do :cuda() it copies the tensor to be contiguous in memory.
The real problem is that local slice = I[{{t, t+h-1}, {l, l+w-1}, {}}] is not contiguous in memory.

But indeed, the solution here is to call a :clone() or :contiguous() to solve the issue.

I think there should be a check in the TensorWrapper checking that the incoming torch tensor is contiguous. What do you think ?

[shrubb]

@fmassa
Wow. Indeed, :cuda() resets strides to normal values. Honestly, I've never tested @jonathanasdf 's script with cutorch (because I often do not have access to GPU). I've only removed :cuda() and saw the garbage.

The continuousness of the actual used memory should not be a problem. It is OK to take a slice or a sub-tensor. So the behaviour is unexpected. I'll have to investigate that more soon.

I like the idea of checking. Let's implement this.

[jonathanasdf]

Thanks for looking into this issue!

I don't know if this is related, but I've also run into a different crash using resize when working with multiple gpus.

OpenCV Error: Gpu API call (invalid resource handle) in call_resize_linear_glob, file /opt/opencv/modules/cudawarping/src/cuda/resize.cu, line 217

I haven't yet narrowed this down to a small reproducible code segment yet but I'll post it if I get something.

[jonathanasdf]

Well, I think I managed to make it reproducible... and I am extremely confused about what is going on.

It seems to manifest when using multiple threads and slicing a tensor... except the slice doesn't need to actually be used.

require 'cutorch'
require 'image'
require 'xlua'

local nThreads = 20
torch.setnumthreads(nThreads)
local Threads = require 'threads'
Threads.serialization('threads.sharedserialize')
local threads = Threads(nThreads,
  function()
    cv = require 'cv'
    require 'cv.cudawarping'
  end)

local I = image.lena():permute(2, 3, 1):cuda()
local N = 100000
for i=1,N do
  threads:addjob(
    function()
      local slice = I[{{1, 2}, {1, 2}, {}}]:clone()
      cv.cuda.resize{I, {4, 4}}
    end,
    function() xlua.progress(i, N) end)
end

When I run this, it (takes forever to launch then) crashes... BUT if I comment out the local slice line then it runs perfectly fine without crashing.