unittests on Linux GPU often run out of memory

[pmeier]

For example: https://github.com/pytorch/vision/actions/runs/5485581154/attempts/2#summary-14857209658

Looking at the HUD this failure seems to have started around June 23.

Nothing on our side has changed around that time that could cause this. Plus, there are runs where this job succeeds: https://github.com/pytorch/vision/actions/runs/5400569293/attempts/1#summary-14621027485

cc @seemethere