Can't get multi-gpu to work anymore

[EvanZ]

At one point I was able to run my model using 4 GPUs on a single machine but since upgrading to the most recent versions of torch and lightning, I am getting shared memory errors like this:

unable to open shared memory object </torch_4121_699393955_8164> in read-write mode: Too many open files (24)
Is there a tutorial or any docs that explain what changes I need to make to my code to bring it up to date?

[akihironitta]

@EvanZ Currently, we don't have any version update guide at this time. For the time being, I would suggest updating your PL minor version one by one. For example, if you're using 1.5.x, try 1.6.x first instead of jumping to the latest minor version 1.7.x. Please refer to the docs for details about our versioning policy.

I am interested in how the degradation happened to your case. Would it be feasible for you to share your code and environment detail here so that I (or someone) might be able to point out possible causes in your code?

[EvanZ]

Hi @akihironitta. Thanks for the reply. I did figure it out! It turned out I needed to set strategy="ddp". That must have been the change since the version I was using before. But yeah, from now on I learned my lesson and will update more frequently. Thanks!