[Feature Request]: Update or replace system-docker

[dgiebert]

Currently the version is currently pinned to 17.06

The buildsystem the following forks with the main change linked:

• github.com/rancher/docker-cli changes
• github.com/rancher/docker changes
• github.com/rancher/docker-ce-packaging changes

Do you want to also create a fork or use patch files instead ?

[olljanat]

I have forked those repos under burmilla but I'm not sure if we really want upgrade system-docker because we don't really need those new features on it and binaries sizes has increased a lot of on those new versions.

docker-17.06.2-ce:
total 89M
-rwxr-xr-x 1 pi pi  18M Sep  5  2017 docker
-rwxr-xr-x 1 pi pi 9.5M Sep  5  2017 docker-containerd
-rwxr-xr-x 1 pi pi 8.6M Sep  5  2017 docker-containerd-ctr
-rwxr-xr-x 1 pi pi 3.7M Sep  5  2017 docker-containerd-shim
-rwxr-xr-x 1 pi pi  39M Sep  5  2017 dockerd
-rwxr-xr-x 1 pi pi 755K Sep  5  2017 docker-init
-rwxr-xr-x 1 pi pi 2.4M Sep  5  2017 docker-proxy
-rwxr-xr-x 1 pi pi 7.4M Sep  5  2017 docker-runc

docker-19.03.14:
total 200M
-rwxr-xr-x 1 pi pi  36M Dec  1 21:22 containerd
-rwxr-xr-x 1 pi pi 6.9M Dec  1 21:22 containerd-shim
-rwxr-xr-x 1 pi pi  19M Dec  1 21:22 ctr
-rwxr-xr-x 1 pi pi  59M Dec  1 21:22 docker
-rwxr-xr-x 1 pi pi  69M Dec  1 21:22 dockerd
-rwxr-xr-x 1 pi pi 693K Dec  1 21:22 docker-init
-rwxr-xr-x 1 pi pi 2.8M Dec  1 21:22 docker-proxy
-rwxr-xr-x 1 pi pi 9.2M Dec  1 21:22 runc

docker-20.10.0:
total 219M
-rwxr-xr-x 1 pi pi  38M Dec  8 20:59 containerd
-rwxr-xr-x 1 pi pi 7.0M Dec  8 20:59 containerd-shim
-rwxr-xr-x 1 pi pi 9.5M Dec  8 20:59 containerd-shim-runc-v2
-rwxr-xr-x 1 pi pi  21M Dec  8 20:59 ctr
-rwxr-xr-x 1 pi pi  53M Dec  8 20:59 docker
-rwxr-xr-x 1 pi pi  76M Dec  8 20:59 dockerd
-rwxr-xr-x 1 pi pi 693K Dec  8 20:59 docker-init
-rwxr-xr-x 1 pi pi 2.8M Dec  8 20:59 docker-proxy
-rwxr-xr-x 1 pi pi  13M Dec  8 20:59 runc

Also I can see that system-docker and system-dockerd binaries are smaller than same version from official Docker version so those are most probably patched somehow to only build only needed of the features.

docker-amd64-17.06-ros6:
total 67M
-rwxr-xr-x 1 pi pi 9.9M Feb 11  2019 system-docker
-rwxr-xr-x 1 pi pi 9.5M Feb 11  2019 system-docker-containerd
-rwxr-xr-x 1 pi pi 8.6M Feb 11  2019 system-docker-containerd-ctr
-rwxr-xr-x 1 pi pi 3.7M Feb 11  2019 system-docker-containerd-shim
-rwxr-xr-x 1 pi pi  25M Feb 11  2019 system-dockerd
-rwxr-xr-x 1 pi pi 755K Feb 11  2019 system-docker-init
-rwxr-xr-x 1 pi pi 2.4M Feb 11  2019 system-docker-proxy
-rwxr-xr-x 1 pi pi 7.4M Feb 11  2019 system-docker-runc

So most probably we should update ros command to all containerd directly so we can drop system-docker but I'm not sure if that is worth effort.

Alternative way is stay on 17.06 forever and only cherry-pick fixes from Moby if needed.

[dgiebert]

The reason for this is probably the removal of some features for system-docker:

• rancher/docker@63c882f
• rancher/docker@0e73971
• rancher/docker@1392bbe

I will try to compile and adapt this and post a size for system-docker

[dgiebert]

Actually after a successful built, the size did not shrink that much:

-rwxr-xr-x 1 dgiebert dgiebert  36M Dec 13 14:18 system-containerd*
-rwxr-xr-x 1 dgiebert dgiebert 6.9M Dec 13 14:18 system-containerd-shim*
-rwxr-xr-x 1 dgiebert dgiebert  19M Dec 13 14:18 system-ctr*
-rwxr-xr-x 1 dgiebert dgiebert  59M Dec 13 14:18 system-docker*
-rwxr-xr-x 1 dgiebert dgiebert  55M Dec 13 14:18 system-dockerd*
-rwxr-xr-x 1 dgiebert dgiebert 693K Dec 13 14:18 system-docker-init*
-rwxr-xr-x 1 dgiebert dgiebert 2.8M Dec 13 14:18 system-docker-proxy*
-rwxr-xr-x 1 dgiebert dgiebert 9.2M Dec 13 14:18 system-runc*

Checking the source code and diffs, a bit of google research showed the possibility of running multiple deamons: https://stackoverflow.com/questions/32334167/is-it-possible-to-start-multiple-docker-daemons-on-the-same-machine

This could be a better way to only maintain a single dockerd and just have a small wrapper script for system-docker

[olljanat]

Checking the source code and diffs, a bit of google research showed the possibility of running multiple deamons: https://stackoverflow.com/questions/32334167/is-it-possible-to-start-multiple-docker-daemons-on-the-same-machine

This could be a better way to only maintain a single dockerd and just have a small wrapper script for system-docker

@dgiebert first of all I would like to understand that what you are actually trying to achieve by updating system-docker? Main purpose of it is actually just works as package provider (like apt-get, yum, etc) for BurmillaOS so as long we don't have known bugs on it I don't see reason to change it.

[olljanat]

Go binaries growing looks to be generic problem golang/go#6853

I can see same on https://github.com/burmilla/docker/tree/test/update-build-scripts where I just took latest build scripts from Moby project so those gets build with later version of Go. That issue can be partly fixed by optimizing build parameters but not fully.

[dgiebert]

Also quite visible here in the official stable builds: https://download.docker.com/linux/static/stable/x86_64/
Will try to optimize the build with the compiler flags later today, any tipps ?

Also from the linked issue, seems that in 1.15 the size could be reduced but as of now moby is on 1.13.

[olljanat]

They are working on to get it updated moby/moby#40353

Also if we start working on this then we probably should target directly to 20.10.x (or even 21.03.x) because major refactoring have been done to Moby on that version (backend code have been removed from Moby used from containerd) and afaiu that work still continues.

[dgiebert]

Very good point indeed, changed to 20.10.1 !

Old Size (19.03.14): ~187M
New Size(20.10.1) using 1.15.5: ~177M

-rwxr-xr-x 1 root root  33M Dec 15 12:30 system-containerd*
-rwxr-xr-x 1 root root 6.5M Dec 15 12:30 system-containerd-shim*
-rwxr-xr-x 1 root root  21M Dec 15 12:30 system-ctr*
-rwxr-xr-x 1 root root  49M Dec 15 12:30 system-docker*
-rwxr-xr-x 1 root root  52M Dec 15 12:30 system-dockerd*
-rwxr-xr-x 1 root root 693K Dec 15 12:30 system-docker-init*
-rwxr-xr-x 1 root root 2.8M Dec 15 12:30 system-docker-proxy*
-rwxr-xr-x 1 root root  12M Dec 15 12:30 system-runc*

This was just plain 1.13.15 -> 1.15.5 changes, so I believe more can be achieved here

[dgiebert]

Following this article here: https://boyter.org/posts/trimming-golang-binary-fat/

-rwxr-xr-x 1 root root  11M Dec 15 14:08 system-containerd*
-rwxr-xr-x 1 root root 2.1M Dec 15 14:08 system-containerd-shim*
-rwxr-xr-x 1 root root  21M Dec 15 14:08 system-ctr*
-rwxr-xr-x 1 root root  25M Dec 15 14:08 system-docker*
-rwxr-xr-x 1 root root  15M Dec 15 14:08 system-dockerd*
-rwxr-xr-x 1 root root 277K Dec 15 14:08 system-docker-init*
-rwxr-xr-x 1 root root 1.6M Dec 15 14:08 system-docker-proxy*
-rwxr-xr-x 1 root root 5.0M Dec 15 14:08 system-runc*

Size after UPX compression: ~81M

[olljanat]

Interesting that Docker CLI is bigger than Docker daemon it selves but of course that makes sense as part of logic have been moved to containerd nowadays.

This made me thinking that it might possible actually replace "system-docker" with shell script which call "docker" command with needed parameters because boot process does not need it afaik:

$ sudo docker -H unix:///var/run/system-docker.sock ps
CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS              PORTS               NAMES
19b253c34be2        burmilla/os-docker:19.03.14         "ros user-docker"        36 hours ago        Up 36 hours                             docker
13b35a796613        burmilla/os-console:v1.9.0          "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             console
f3e81544617a        burmilla/os-base:v1.9.0             "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             ntp
0229f9a12949        burmilla/os-base:v1.9.0             "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             network
0f86354032d3        burmilla/os-base:v1.9.0             "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             udev
37ac8ebfb9df        burmilla/container-crontab:v0.5.0   "container-crontab"      36 hours ago        Up 36 hours                             system-cron
742bdbd5c705        burmilla/os-syslog:v1.9.0           "/usr/bin/entrypoint…"   36 hours ago        Up 36 hours                             syslog
2c50b2258fda        burmilla/os-acpid:v1.9.0            "/usr/bin/ros entryp…"   36 hours ago        Up 36 hours                             acpid

Also I think that we do not need "system-docker-containerd-ctr" / "system-ctr" at all and "system-docker-proxy" gets removed during build process on:

os/scripts/layout-initrd

Lines 33 to 34 in 93de612

	# we have diabled the user-proxy so we get rid of system-docker-proxy
	rm -f ${INITRD_DIR}/usr/bin/system-docker-proxy

What comes to idea to use UPX, system-docker is actually packages inside of initrd which is already compressed so we should check how this affect size of it and iso file:

os/scripts/package-initrd

Lines 19 to 27 in 93de612

	if [ "$COMPRESS" == "" ]; then
	COMPRESS="gzip -1"
	fi
	cd ${INITRD_DIR}
	echo Creating ${ARTIFACTS}/${INITRD}
	find | cpio -H newc -o | ${COMPRESS} > ${ARTIFACTS}/${INITRD}
	echo Done creating ${ARTIFACTS}/${INITRD}

Also boot time and memory usage are areas which we need follow when we changing these as there is already comment about high CPU usage with too much compression:

os/scripts/tar-images

Lines 47 to 54 in 93de612

	if [ "$COMPRESS" == "" ]; then
	ARCHIVE_CMD="gzip"
	else
	# system-docker can not load images which compressed by xz with a compression level of 9
	# decompression consumes more memory if using level 9
	# the default compression level for xz is 6
	ARCHIVE_CMD="xz -4 -e"
	fi

[dgiebert]

Yes, I think it is very easy to replace system-docker with a two line bash script in the console image.

You are correct did not think about the initrd compression, but I guess if we strip off all unneeded / duplicated executables the resulting size should not exceed the 17.06 build by much

[olljanat]

This could be a better way to only maintain a single dockerd and just have a small wrapper script for system-docker

Hmm. Maybe that is actually best way because if we decide that system-docker and default user docker versions are same then we can:

• Cache files from https://github.com/burmilla/os-services/tree/master/images/10-docker-20.10.1 + downloaded Docker binaries on build.
• Start system-docker directly from those binaries or with wrapper scripts (I'm not sure if we even need wrapper or maybe it can be simple which denies example swarm and plugin commands).
• Start user docker with busybox image which actually mounts those same binaries inside container (so there is not need import those as Docker image like now).

Only special things needed on that one would be that we need modify settings on here:

os/os-config.tpl.yml

Lines 361 to 385 in 4e5d248

	docker:
	{{if eq "amd64" .ARCH -}}
	image: {{.OS_REPO}}/os-docker:{{.USER_DOCKER_VERSION}}{{.SUFFIX}}
	{{else -}}
	image: {{.OS_REPO}}/os-docker:{{.USER_DOCKER_VERSION}}{{.SUFFIX}}
	{{end -}}
	command: ros user-docker
	environment:
	- HTTP_PROXY
	- HTTPS_PROXY
	- NO_PROXY
	labels:
	io.rancher.os.scope: system
	io.rancher.os.after: console
	net: host
	pid: host
	ipc: host
	uts: host
	privileged: true
	restart: always
	volumes_from:
	- all-volumes
	volumes:
	- /sys:/host/sys
	- /var/lib/system-docker:/var/lib/system-docker:shared

Then it should (on theory) works similarly like consoles was earlier that default settings will be overridden when user uses sudo ros engine switch command (as on that point it need to use version specific configuration from os-service repo) and probably we need also add "default" option to that list which always points to Docker version which is bundled to ISO file.