Own startup scripts like /docker-entrypoint-initdb.d regardless of $DATABASE_ALREADY_EXISTS

[esticle]

I'm keen to run startup scripts on container restarts on existing (ie. not new) volumes.

I see that the docker-entrypoint.sh sadly only runs the docker_process_init_files function when $DATABASE_ALREADY_EXISTS=false so putting a script in /docker-entrypoint-initdb.d has no effect if you already have a volume with databases functioning. Is there an alternative to this that I am missing?

I tried looking upstream at the Ubuntu image and do not see a clear way there either.

[wglambert]

This is possible since the entrypoint for MySQL, Postgres, and MariaDB were recently refactored #271

So using docker-library/postgres#661 (comment) as a reference you'll want to make a custom entrypoint to call the docker_process_init_files() function directly

#!/usr/bin/env bash
set -Eeo pipefail

# Example using the functions of the MariaDB entrypoint to customize startup to always run files in /always-initdb.d/

source "$(which docker-entrypoint.sh)"

docker_setup_env "$@"
docker_create_db_directories

if [ -z "$DATABASE_ALREADY_EXISTS" ]; then
        docker_verify_minimum_env
        docker_init_database_dir "$@"

        mysql_note "Starting temporary server"
        docker_temp_server_start "$@"
        mysql_note "Temporary server started."

        docker_setup_db
        docker_process_init_files /docker-entrypoint-initdb.d/*

        mysql_note "Stopping temporary server"
        docker_temp_server_stop
        mysql_note "Temporary server stopped"

        echo
        mysql_note "MySQL init process done. Ready for start up."
        echo
else
        docker_temp_server_start "$@"
        docker_process_init_files /always-initdb.d/*
        docker_temp_server_stop
fi

exec mysqld "$@"

First start is a normal initialization

$ docker run -it --user mysql --name mariadb -e MYSQL_ROOT_PASSWORD=root \
-v "$PWD/custom-entrypoint.sh":/usr/local/bin/custom-entrypoint.sh \
-v "$PWD/docker-entrypoint-initdb.d/":/docker-entrypoint-initdb.d/ \
-v "$PWD/always-initdb.d/":/always-initdb.d/ \
-v "$PWD/mysql/":/var/lib/mysql/ \
--entrypoint /usr/local/bin/custom-entrypoint.sh \
mariadb mysqld

2020-01-16 17:44:26+00:00 [Note] [Entrypoint]: Initializing database files


PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
To do so, start the server, then issue the following commands:

'/usr/bin/mysqladmin' -u root password 'new-password'
'/usr/bin/mysqladmin' -u root -h  password 'new-password'

Alternatively you can run:
'/usr/bin/mysql_secure_installation'

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the MariaDB Knowledgebase at http://mariadb.com/kb or the
MySQL manual for more instructions.

Please report any problems at http://mariadb.org/jira

The latest information about MariaDB is available at http://mariadb.org/.
You can find additional information about the MySQL part at:
http://dev.mysql.com
Consider joining MariaDB's strong and vibrant community:
https://mariadb.org/get-involved/

2020-01-16 17:44:57+00:00 [Note] [Entrypoint]: Database files initialized
2020-01-16 17:44:57+00:00 [Note] [Entrypoint]: Starting temporary server
2020-01-16 17:44:57+00:00 [Note] [Entrypoint]: Waiting for server startup
2020-01-16 17:44:57 0 [Note] mysqld (mysqld 10.4.11-MariaDB-1:10.4.11+maria~bionic) starting as process 90 ...
2020-01-16 17:44:57 0 [Note] InnoDB: Using Linux native AIO
2020-01-16 17:44:57 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-01-16 17:44:57 0 [Note] InnoDB: Uses event mutexes
2020-01-16 17:44:57 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-01-16 17:44:57 0 [Note] InnoDB: Number of pools: 1
2020-01-16 17:44:57 0 [Note] InnoDB: Using SSE2 crc32 instructions
2020-01-16 17:44:57 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2020-01-16 17:44:57 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2020-01-16 17:44:57 0 [Note] InnoDB: Completed initialization of buffer pool
2020-01-16 17:44:57 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-01-16 17:44:58 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2020-01-16 17:44:58 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-01-16 17:44:58 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-01-16 17:44:58 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-01-16 17:44:58 0 [Note] InnoDB: Waiting for purge to start
2020-01-16 17:44:58 0 [Note] InnoDB: 10.4.11 started; log sequence number 61184; transaction id 21
2020-01-16 17:44:58 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-01-16 17:44:58 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-01-16 17:44:58 0 [Warning] 'user' entry 'root@8ec7b275eefd' ignored in --skip-name-resolve mode.
2020-01-16 17:44:58 0 [Warning] 'user' entry '@8ec7b275eefd' ignored in --skip-name-resolve mode.
2020-01-16 17:44:58 0 [Warning] 'proxies_priv' entry '@% root@8ec7b275eefd' ignored in --skip-name-resolve mode.
2020-01-16 17:44:58 0 [Note] InnoDB: Buffer pool(s) load completed at 200116 17:44:58
2020-01-16 17:44:58 0 [Note] Reading of all Master_info entries succeeded
2020-01-16 17:44:58 0 [Note] Added new Master_info '' to hash table
2020-01-16 17:44:58 0 [Note] mysqld: ready for connections.
Version: '10.4.11-MariaDB-1:10.4.11+maria~bionic'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
2020-01-16 17:44:59+00:00 [Note] [Entrypoint]: Temporary server started.

Second start is to try and execute the files in /always-initdb.d/ which is getting access denied at this function https://github.com/docker-library/mariadb/blob/9ddcc2b4b0422a83dafc52e5574474dafc2fa9ca/10.4/docker-entrypoint.sh#L88-L107

$ docker run -it --user mysql --name mariadb -e MYSQL_ROOT_PASSWORD=root \
-v "$PWD/custom-entrypoint.sh":/usr/local/bin/custom-entrypoint.sh \
-v "$PWD/docker-entrypoint-initdb.d/":/docker-entrypoint-initdb.d/ \
-v "$PWD/always-initdb.d/":/always-initdb.d/ \
-v "$PWD/mysql/":/var/lib/mysql/ \
--entrypoint /usr/local/bin/custom-entrypoint.sh \
mariadb mysqld

2020-01-16 17:49:00+00:00 [Note] [Entrypoint]: Waiting for server startup
2020-01-16 17:49:00 0 [Note] mysqld (mysqld 10.4.11-MariaDB-1:10.4.11+maria~bionic) starting as process 21 ...
2020-01-16 17:49:00 0 [Note] InnoDB: Using Linux native AIO
2020-01-16 17:49:00 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-01-16 17:49:00 0 [Note] InnoDB: Uses event mutexes
2020-01-16 17:49:00 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-01-16 17:49:00 0 [Note] InnoDB: Number of pools: 1
2020-01-16 17:49:00 0 [Note] InnoDB: Using SSE2 crc32 instructions
2020-01-16 17:49:00 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2020-01-16 17:49:00 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2020-01-16 17:49:00 0 [Note] InnoDB: Completed initialization of buffer pool
2020-01-16 17:49:00 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-01-16 17:49:03 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1201423
2020-01-16 17:49:05 0 [Note] InnoDB: Starting final batch to recover 22 pages from redo log.
2020-01-16 17:49:07 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2020-01-16 17:49:07 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-01-16 17:49:07 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-01-16 17:49:07 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-01-16 17:49:07 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-01-16 17:49:07 0 [Note] InnoDB: Waiting for purge to start
2020-01-16 17:49:07 0 [Note] InnoDB: 10.4.11 started; log sequence number 1201935; transaction id 1520
2020-01-16 17:49:07 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-01-16 17:49:07 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-01-16 17:49:07 0 [Warning] 'user' entry 'root@8ec7b275eefd' ignored in --skip-name-resolve mode.
2020-01-16 17:49:07 0 [Warning] 'user' entry '@8ec7b275eefd' ignored in --skip-name-resolve mode.
2020-01-16 17:49:07 0 [Warning] 'proxies_priv' entry '@% root@8ec7b275eefd' ignored in --skip-name-resolve mode.
2020-01-16 17:49:07 0 [Note] Reading of all Master_info entries succeeded
2020-01-16 17:49:07 0 [Note] Added new Master_info '' to hash table
2020-01-16 17:49:07 0 [Note] mysqld: ready for connections.
Version: '10.4.11-MariaDB-1:10.4.11+maria~bionic'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
2020-01-16 17:49:07 0 [Note] InnoDB: Buffer pool(s) load completed at 200116 17:49:07
2020-01-16 17:49:08 8 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:09 9 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:10 10 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:11 11 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:12 12 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:13 13 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:14 14 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:15 15 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:16 16 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:17 17 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:18 18 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:19 19 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:20 20 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:21 21 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:22 22 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:23 23 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:24 24 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:25 25 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:26 26 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:27 27 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:28 29 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:29 30 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:30 31 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:31 32 [Warning] Access denied for user 'root'@'localhost' (using password: YES)
2020-01-16 17:49:32+00:00 [ERROR] [Entrypoint]: Unable to start server.

So that's where I'm at so far

[wglambert]

Nevermind it works fine, I just ran into #262 and didn't wait the 22 minutes for it to initialize on my hardware

Using mysql instead:

custom-entrypoint.sh

#!/usr/bin/env bash
set -Eeo pipefail

# Example using the functions of the MariaDB entrypoint to customize startup to always run files in /always-initdb.d/

source "$(which docker-entrypoint.sh)"

docker_setup_env "$@"
docker_create_db_directories

if [ -z "$DATABASE_ALREADY_EXISTS" ]; then
        docker_verify_minimum_env
        docker_init_database_dir "$@"

        mysql_note "Starting temporary server"
        docker_temp_server_start "$@"
        mysql_note "Temporary server started."

        docker_setup_db
        docker_process_init_files /docker-entrypoint-initdb.d/*

        mysql_note "Stopping temporary server"
        docker_temp_server_stop
        mysql_note "Temporary server stopped"

        echo
        mysql_note "MySQL init process done. Ready for start up."
        echo
else
        docker_temp_server_start $@
        docker_process_init_files /always-initdb.d/*
        docker_temp_server_stop
fi

exec mysqld

First start initialize

$ docker run -it --user mysql --name mariadb -e MYSQL_ROOT_PASSWORD=root -v "$PWD/custom-entrypoint.sh":/usr/local/bin/custom-entrypoint.sh -v "$PWD/docker-entrypoint-initdb.d/":/docker-entrypoint-initdb.d/ -v "$PWD/always-initdb.d/":/always-initdb.d/ -v "$PWD/mysql/":/var/lib/mysql/ --entrypoint /usr/local/bin/custom-entrypoint.sh mysql mysqld
2020-01-16 21:03:36+00:00 [Note] [Entrypoint]: Initializing database files
2020-01-16T21:03:36.287097Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2020-01-16T21:03:36.287354Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.19) initializing of server in progress as process 20
2020-01-16T21:04:41.036848Z 5 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2020-01-16 21:05:32+00:00 [Note] [Entrypoint]: Database files initialized
2020-01-16 21:05:33+00:00 [Note] [Entrypoint]: Starting temporary server
mysqld will log errors to /var/lib/mysql/16d5fa727223.err
mysqld is running as pid 72
2020-01-16 21:05:41+00:00 [Note] [Entrypoint]: Temporary server started.
Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.

2020-01-16 21:05:57+00:00 [Note] [Entrypoint]: Stopping temporary server
2020-01-16 21:06:02+00:00 [Note] [Entrypoint]: Temporary server stopped

2020-01-16 21:06:02+00:00 [Note] [Entrypoint]: MySQL init process done. Ready for start up.

2020-01-16T21:06:02.628250Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2020-01-16T21:06:02.628338Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.19) starting as process 1
2020-01-16T21:06:06.367103Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2020-01-16T21:06:06.455073Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2020-01-16T21:06:06.473475Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.19'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
2020-01-16T21:06:06.595058Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '::' port: 33060

Second start executing scripts

$ docker run -it --user mysql --name mariadb -e MYSQL_ROOT_PASSWORD=root -v "$PWD/custom-entrypoint.sh":/usr/local/bin/custom-entrypoint.sh -v "$PWD/docker-entrypoint-initdb.d/":/docker-entrypoint-initdb.d/ -v "$PWD/always-initdb.d/":/always-initdb.d/ -v "$PWD/mysql/":/var/lib/mysql/ --entrypoint /usr/local/bin/custom-entrypoint.sh mysql mysqld
mysqld will log errors to /var/lib/mysql/88ece92d1871.err
mysqld is running as pid 21

2020-01-16 21:08:26+00:00 [Note] [Entrypoint]: /usr/local/bin/custom-entrypoint.sh: running /always-initdb.d/file.sh
k

2020-01-16 21:08:26+00:00 [Note] [Entrypoint]: /usr/local/bin/custom-entrypoint.sh: running /always-initdb.d/test.sh
test.sh file

2020-01-16 21:08:26+00:00 [Note] [Entrypoint]: /usr/local/bin/custom-entrypoint.sh: running /always-initdb.d/test.sql


2020-01-16T21:08:30.423155Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2020-01-16T21:08:30.423271Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.19) starting as process 1
2020-01-16T21:08:33.735840Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2020-01-16T21:08:33.788834Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2020-01-16T21:08:33.829271Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.19'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
2020-01-16T21:08:33.890408Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '::' port: 33060

[esticle]

Thanks @wglambert.

I'm using docker-compose but added the entrypoint to the mapped file and strangely I do not see it ever returning after the 'docker_setup_env "$@"' line is hit.

No arguments are passed right, so presumably $@ is empty? Removing the pipefail and bash -x'ing I see:

++ which docker-entrypoint.sh
+ source /usr/local/bin/docker-entrypoint.sh
++ set -eo pipefail
++ shopt -s nullglob
++ _is_sourced
++ '[' 3 -ge 2 ']'
++ '[' _is_sourced = _is_sourced ']'
++ '[' source = source ']'
+ docker_setup_env
+ declare -g DATADIR SOCKET
++ mysql_get_config datadir
++ local conf=datadir
++ shift
++ awk -v conf=datadir '$1 == conf && /^[^ \t]/ { sub(/^[^ \t]+[ \t]+/, ""); print; exit }'
+++ mktemp -u
++ --verbose --help --log-bin-index=/tmp/tmp.ofYSKXZhMJ
+ DATADIR=

Which is:

DATADIR="$(mysql_get_config 'datadir' "$@")"

Presumably some variables are not passed but I am unsure which should?

Thanks!

[wglambert]

When the entrypoint is overwritten the cmd is as well. So you'd need to re-add that. As in my docker run I just added mysqld on the end.

[esticle]

When the entrypoint is overwritten the cmd is as well. So you'd need to re-add that. As in my docker run I just added mysqld on the end.

I missed that ridiculous one! Thanks!

This of course works:

...<snip>...
    volumes:
      - $PWD/mariadb:/var/lib/mysql
      - $PWD/docker-entrypoint-initdb.d:/always-initdb.d/:ro
      - $PWD/custom-entrypoint.sh:/usr/local/bin/custom-entrypoint.sh:ro
    ports:
      - 127.0.0.1:3306:3306
    entrypoint:
      - /usr/local/bin/custom-entrypoint.sh
      - mysqld

My next problem was that I do not specify "--name" and the always-initdb.d scripts I run need to be run as root (whilst not "docker-like", I like crond in the container for example) - this means that if --name is specified that it naturally runs the entrypoint as that uid and would not be able to execute with root privileges.

To address docker_temp_server_start attempting to start mysqld as root, and the subsequent running of mysqld as the mysql user I changed the last few lines of your custom-entrypoint:

...<snip>...
        echo
        mysql_note "MySQL init process done. Ready for start up."
        echo
else
        #docker_temp_server_start "$@" # I do not need to start mysqld yet
        docker_process_init_files /always-initdb.d/*
        #docker_temp_server_stop # I do not need to stop mysqld yet
fi

su -c mysqld  mysql

Thanks for your help!