From client Permission denied (publickey)

[Maniumn]

I have configured the docker-compose file on Windows 10, running Docker with WSL.
When trying to connect to the host from the client, I get: "Permission denied (publickey)".
Client detailed:

debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/ubuntu/.ssh/id_ed25519 ED25519 SHA256:XH3/kiJgww9JvNGTQrH/+AUt6/UBlolZ+emID4lXyUQ agent
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Offering public key: /home/ubuntu/.ssh/id_rsa RSA SHA256:t3Keh1h52lNvQ5gveswpDVdMVARWGtUmN4/XVyZutaI agent
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/ubuntu/.ssh/id_ecdsa
debug1: Trying private key: /home/ubuntu/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/ubuntu/.ssh/id_ed25519_sk
debug1: Trying private key: /home/ubuntu/.ssh/id_xmss
debug1: Trying private key: /home/ubuntu/.ssh/id_dsa
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
git@ip-hidden: Permission denied (publickey).

However, the host outputs the following in the console:

Attaching to git-server
git-server  | /docker-entrypoint.sh: Starting container configuration
git-server  | /docker-entrypoint.sh: Running /docker-entrypoint.d/10-setup.sh
git-server  | chpasswd: password for 'git' changed
git-server  | /docker-entrypoint.sh: Container configuration completed
git-server  | Server listening on 0.0.0.0 port 22.
git-server  | Server listening on :: port 22.
git-server  | Could not open user 'git' authorized keys '/home/git/.ssh/authorized_keys': Permission denied
git-server  | Could not open user 'git' authorized keys '/home/git/.ssh/authorized_keys': Permission denied
git-server  | Connection closed by authenticating user git 172.18.0.1 port 58652 [preauth]

docker-compose.yml:

version: "3"

services:
  git-server:
    image: rockstorm/git-server
    container_name: git-server
    restart: unless-stopped
    command: ["/usr/sbin/sshd", "-D", "-e"]
    environment:
      # Password for the git user
      GIT_PASSWORD: "12345"

      # Path where the file with the password for the git user will be
      # mounted in the container in order to replace the default one
      # GIT_PASSWORD_FILE: /run/secrets/git_password

      # Fetch an `authorized_keys` file form an online location
      # SSH_AUTHORIZED_KEYS_URL: https://github.com/username.keys

      # Set the allowed authentication methods
      # SSH_AUTH_METHODS: "publickey keyboard-interactive"
      SSH_AUTH_METHODS: "publickey"

      # Path where the SSH host keys will be mounted in the container
      # in order to replace the default keys
      # SSH_HOST_KEYS_PATH: /tmp/host-keys

      # Setting this variable creates a link in the git user directory
      # to access repositories without absolute paths
      REPOSITORIES_HOME_LINK: /srv/git

      # Set specific UID and GID for the git user
      # GIT_USER_UID: 1001
      # GIT_USER_GID: 1001

    volumes:
      # Folder with git repositories
      - /C/Users/XXX/.ssh/docker_srv_git/:/srv/git

      # File containing the password for the git user
      # - /path/to/password/file:/run/secrets/git_password:ro

      # File containing the SSH keys of clients that will be allowed
      # to use this service through a public key
      - /C/Users/XXX/.ssh/docker_git_authorized_keys/authorized_keys:/home/git/.ssh/authorized_keys

      # A folder with host SSH keys to use instead of the default ones
      # - /path/to/host-keys:/tmp/host-keys:ro

      # Configuration file for the OpenSSH daemon to use instead of
      # the one that is generated by default
      # - ./sshd_config.sample:/etc/ssh/sshd_config:ro

      # Disable interactive SSH login for the git user
      # - /executable/file:/home/git/git-shell-commands/no-interactive-login

    ports:
      - "2222:22"

volumes:
  git-repositories:

On docker image:

/home/git # ls -la
total 28
drwxr-sr-x    1 git      git           4096 Mar 11 15:52 .
drwxr-xr-x    1 root     root          4096 Feb 23 10:27 ..
d---rwxrwx    2 git      git           4096 Mar 11 15:27 .ssh
lrwxrwxrwx    1 root     git              8 Mar 11 15:52 git -> /srv/git
drwxr-sr-x    1 git      git           4096 Feb 23 10:27 git-shell-commands

/home/git/.ssh # ls -la
total 16
d---rwxrwx    2 git      git           4096 Mar 11 15:27 .
drwxr-sr-x    1 git      git           4096 Mar 11 15:52 ..
-rw-r--r--    1 git      git            673 Mar 11 15:40 authorized_keys

What have I done wrong? Seems like issue.

[rockstorm101]

Thanks a lot for such a detailed report.

git-server | Could not open user 'git' authorized keys '/home/git/.ssh/authorized_keys': Permission denied
git-server | Could not open user 'git' authorized keys '/home/git/.ssh/authorized_keys': Permission denied

It definitely looks like a permissions-type of problem.

I would guess it is the permissions on the .ssh folder which in your case is not readable by the git user for some reason.

Here is what a working configuration looks like:

/home/git # ls -al
total 28
drwxr-sr-x    1 git      git           4096 Mar 11 21:06 .
drwxr-xr-x    1 root     root          4096 Feb 23 10:24 ..
drwxr-sr-x    2 git      git           4096 Mar 11 21:06 .ssh
drwxr-sr-x    1 git      git           4096 Feb 23 10:24 git-shell-commands
lrwxrwxrwx    1 root     git             17 Mar 11 21:06 projects -> /srv/git/projects
/home/git # cd .ssh/
/home/git/.ssh # ls -al
total 16
drwxr-sr-x    2 git      git           4096 Mar 11 21:06 .
drwxr-sr-x    1 git      git           4096 Mar 11 21:06 ..
-rw-r--r--    1 git      git             81 Jan  7  2024 authorized_keys

See how the my permissions on the .ssh folder are drwxr-sr-x while yours are d---rwxrwx which interestingly means "not accessible by the owner".

[Maniumn]

Thanks for your feedback.
Sorry for the confusion about file permissions. I had tested different permission rights and forgot to reset them, which is why you saw the incorrect permissions.

In general, I have tested the entire process described in README.md twice, and both times the result was the same as I described earlier. If you want, I can repeat the process again and share the results, I am quite sure it will fail again.

For now, I have resolved the issue. The solution was to set SSH authentication to "publickey password", run the Docker container, connect via SSH (it has connected directly without password), stop the container, and then set SSH authentication back to "publickey". I’m not sure why, but this somehow "unlocked" the issue state.

As I mentioned, I can repeat all the steps and provide the results if needed.

P.S. There is one more issue with sshd config file mounting.

[rockstorm101]

For now, I have resolved the issue. The solution was to set SSH authentication to "publickey password", run the Docker container, connect via SSH (it has connected directly without password), stop the container, and then set SSH authentication back to "publickey". I’m not sure why, but this somehow "unlocked" the issue state.

Thanks for sharing the solution you found! :)

P.S. There is one more issue with sshd config file mounting.

Please elaborate on this to see if we can fix it.

[Maniumn]

Please elaborate on this to see if we can fix it.

In case I add custom sshd_config with or without "ro" flag:

...
    volumes:
...
      # Configuration file for the OpenSSH daemon to use instead of
      # the one that is generated by default
      # - ./sshd_config.sample:/etc/ssh/sshd_config:ro
      - /C/Users/XXX/.ssh/docker_git_sshd_config2/sshd_config:/etc/ssh/sshd_config:ro

this is what container reports:

[+] Running 1/1
 ✔ Container git-server2  Recreated                                                                                0.1s
Attaching to git-server2
git-server2  | /docker-entrypoint.sh: Starting container configuration
git-server2  | /docker-entrypoint.sh: Running /docker-entrypoint.d/10-setup.sh
git-server2  | chpasswd: password for 'git' changed
git-server2  | sed: can't move '/etc/ssh/sshd_configEjpiDb' to '/etc/ssh/sshd_config': Resource busy
git-server2 exited with code 0
git-server2  | sed: can't move '/etc/ssh/sshd_configmeMLfa' to '/etc/ssh/sshd_config': Resource busy
git-server2 exited with code 0
git-server2  | sed: can't move '/etc/ssh/sshd_configGaMdpN' to '/etc/ssh/sshd_config': Resource busy
git-server2 exited with code 0
git-server2  | sed: can't move '/etc/ssh/sshd_configLImPFM' to '/etc/ssh/sshd_config': Resource busy

[rockstorm101]

git-server2 | sed: can't move '/etc/ssh/sshd_configEjpiDb' to '/etc/ssh/sshd_config': Resource busy
git-server2 exited with code 0

This is expected because using option "SSH_AUTH_METHODS" and mounting a custom sshd_config file are not compatible since the first tries to (unsuccessfully) rewrite the second. Current instructions might not be clear on this regard, I'll add a note. Thanks a lot for your feedback here.

[thinkingofmaud]

Had the same issue, figured out it was a permission use. Im a hardhead who decided to never, ever ssh with a password on principle, so here is another fix:

Mount something to /home/git/.ssh

set SSH_AUTHORIZED_KEYS_URL environment pointing to a server (or github)

Start the service. It will get the file and set correct permissions and user.
Stop, remove. Edit the file to your liking.
Then remove SSH_AUTHORIZED_KEYS_URL. Run it again, it should be fine.

[Maniumn]

@rockstorm101 add these steps for the start, or just extra permission changes. It was my problem probably.

[rockstorm101]

Thank you both, I'll add those steps to the README in case other users face the same issue. File permissions are always a headache.