Export PYTHOPATH inside docker container

[Daniel595]

💡 Is your feature request related to a problem?

I am using your amazing lightly-train package inside your proposed docker container (0.11.0) since the proposed docker usage kind of restrics flexibility (for me):

1. starting a container and
2. using it via CLI

Problem: The python package from conda is used at the moment instead of the copied source. This results in an error because of a missing config file:

FileNotFoundError: [Errno 2] No such file or directory: '/opt/conda/lib/python3.11/site-packages/lightly_train/_models/dinov2_vit/dinov2_vit_src/configs/ssl_default_config.yaml'

🧰 Describe the solution you'd like

Adding the PYTHONPATH in your Dockerfile should fix it. It works for me when I build an image like that:

FROM lightly/train:latest
ENV PYTHONPATH=$PYTHONPATH:/home/lightly_train/src

[guarin]

Hi, thanks for reporting the issue, we'll look into this. Do you mind sharing the command you used to start the container and training run and share the full error? It would help us a lot.

[Daniel595]

Sure. Here are the used files:

1. the Dockerfile to build my custom image:

FROM lightly/train:latest
ENV PYTHONPATH=/home/lightly_train/src
WORKDIR /mnt/development/code/lightly-train # my custom workdir

2. building it:

docker pull lightly/train:latest
docker build -t lightly-train-local . 

3. starting it:

CONTAINER_NAME="lightly-train"

if [ -n "$1" ]; then
    CONTAINER_NAME="$1"
fi

docker run -it --rm --gpus=all --shm-size=16gb \
    --name "$CONTAINER_NAME" \
    --privileged \
    -v /some_mounts:... \
    -v ... \
    lightly-train-local bash

4. training:

python train_small.py.

The train file:

import lightly_train
lightly_train.train(out="out/exp2-224-small", data="small_crops",
                    model="dinov2/vitb14",
                    method="dinov2",
                    batch_size=24,) 
lightly_train.embed(out="my_embeddings.pth", data="small_crops", checkpoint="out/exp2-224-small/checkpoints/last.ckpt", format="torch")

That's my custom example with some extras, but when running a original lightly-train image, there is no PYTHONPATH exported either, so I thought it might be a general problem. But of course I can't say with certainty that the fault is not mine - but I don't hope so :)

[guarin]

Hmm, the LightlyTrain image is built on top of the PyTorch image. We install LightlyTrain from the /home/lightly_train directory. You see the path to conda because the PyTorch docker images install everything into a global conda environment.

When you take the LightlyTrain base images without any changes you should be able to run import lightly_train from any location inside the container. Except if you create a new environment inside the container but this doesn't seem the case. I assume you didn't mount a full virtual env inside the container right?

If you can import lightly_train then it is most likely not a problem related to PYTHONPATH

[Daniel595]

I did not mount a virtual env inside the container. I can import lightly_train but the installation is missing the configs. When listing the paths:
ls /opt/conda/lib/python3.11/site-packages/lightly_train/_models/dinov2_vit/dinov2_vit_src/configs
ls /home/lightly_train/src/lightly_train/_models/dinov2_vit/dinov2_vit_src/configs
the conda path has no ssl_default_config.yaml present (as the Error indicates) while the src path does.

I reinstalled with: pip install lightly-train --force-reinstall - and now it works. The config file is present now. For me it seems like the installation from the src-path is incomplete somehow?

I tried the same with the latest container 0.11.1 too. First the config is missing but after reinstall it's present and the training works as expected.

[Daniel595]

Oops, didn't mean to close it :D

[guarin]

Thanks, will look into this. pip install lightly-train --force-reinstall installs the package from PIP whereas we install the package from the /home/lightly_train directory in the dockerfile with pip install --no-deps /home/lightly_train.