add possibility to clean untagged manifests #2302

zetaab · 2017-06-06T08:04:29Z

fixes issue #2301

% docker tag alpine:3.4 localhost:5000/alpine:latest
% docker push localhost:5000/alpine:latest
The push refers to a repository [localhost:5000/alpine]
8539d1fe4fab: Pushed
latest: digest: sha256:36fbf18a36d0410de46201a76b7cfb1d69d44a8f407d6a0814517247b896f8e9 size: 528
% docker tag alpine:latest localhost:5000/alpine:latest
% docker push localhost:5000/alpine:latest
The push refers to a repository [localhost:5000/alpine]
3fb66f713c9f: Pushed
latest: digest: sha256:0b94d1d1b5eb130dd0253374552445b39470653fb1a1ec2d81490948876e462c size: 528

now the situation is that there is two manifests, and only one has active tag.

New flag added 'm'

% docker exec f519ebf4cc51 registry garbage-collect
configuration error: configuration path unspecified

Usage:
  registry garbage-collect <config> [flags]
Flags:
  -m, --delete-manifests=false: delete manifests which does not have tag attached
  -d, --dry-run=false: do everything except remove the blobs
  -h, --help=false: help for garbage-collect

old behaviour:

% docker exec f519ebf4cc51 registry garbage-collect /etc/docker/registry/config.yml
alpine
alpine: marking manifest sha256:0b94d1d1b5eb130dd0253374552445b39470653fb1a1ec2d81490948876e462c
alpine: marking blob sha256:a41a7446062d197dd4b21b38122dcc7b2399deb0750c4110925a7dd37c80f118
alpine: marking blob sha256:2aecc7e1714b6fad58d13aedb0639011b37b86f743ba7b6a52d82bd03014b78e
alpine: marking manifest sha256:36fbf18a36d0410de46201a76b7cfb1d69d44a8f407d6a0814517247b896f8e9
alpine: marking blob sha256:6008ce38ddc131d5fc1e35b4a06b78fa1388ca94fd8efc1246fcbc44b45b6b74
alpine: marking blob sha256:486a8e636d6250a74d15cdb3582f4dd198271a80118f5a2f59de3d9cd8433611

6 blobs marked, 0 blobs and 0 manifests eligible for deletion

disk in this point:

 registry % du -h -d 1
4.2M	./v2
4.2M	.

new feature:

% docker exec f519ebf4cc51 registry garbage-collect -m /etc/docker/registry/config.yml
alpine
alpine: marking manifest sha256:0b94d1d1b5eb130dd0253374552445b39470653fb1a1ec2d81490948876e462c
alpine: marking blob sha256:a41a7446062d197dd4b21b38122dcc7b2399deb0750c4110925a7dd37c80f118
alpine: marking blob sha256:2aecc7e1714b6fad58d13aedb0639011b37b86f743ba7b6a52d82bd03014b78e
manifest has no tags alpine@sha256:36fbf18a36d0410de46201a76b7cfb1d69d44a8f407d6a0814517247b896f8e9

3 blobs marked, 3 blobs and 1 manifests eligible for deletion
manifest eligible for deletion: sha256:36fbf18a36d0410de46201a76b7cfb1d69d44a8f407d6a0814517247b896f8e9
time="2017-06-06T07:56:30.226753065Z" level=info msg="Deleting manifest: /docker/registry/v2/repositories/alpine/_manifests/revisions/sha256/36fbf18a36d0410de46201a76b7cfb1d69d44a8f407d6a0814517247b896f8e9" go.version=go1.8.3 instance.id=a36f92ea-3c7a-420b-8fbb-4392722dc61c
blob eligible for deletion: sha256:36fbf18a36d0410de46201a76b7cfb1d69d44a8f407d6a0814517247b896f8e9
time="2017-06-06T07:56:30.236785668Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/36/36fbf18a36d0410de46201a76b7cfb1d69d44a8f407d6a0814517247b896f8e9" go.version=go1.8.3 instance.id=a36f92ea-3c7a-420b-8fbb-4392722dc61c
blob eligible for deletion: sha256:486a8e636d6250a74d15cdb3582f4dd198271a80118f5a2f59de3d9cd8433611
time="2017-06-06T07:56:30.248873403Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/48/486a8e636d6250a74d15cdb3582f4dd198271a80118f5a2f59de3d9cd8433611" go.version=go1.8.3 instance.id=a36f92ea-3c7a-420b-8fbb-4392722dc61c
blob eligible for deletion: sha256:6008ce38ddc131d5fc1e35b4a06b78fa1388ca94fd8efc1246fcbc44b45b6b74
time="2017-06-06T07:56:30.25450166Z" level=info msg="Deleting blob: /docker/registry/v2/blobs/sha256/60/6008ce38ddc131d5fc1e35b4a06b78fa1388ca94fd8efc1246fcbc44b45b6b74" go.version=go1.8.3 instance.id=a36f92ea-3c7a-420b-8fbb-4392722dc61c

disk after this:

 registry % du -h -d 1
1.9M	./v2
1.9M	.

GordonTheTurtle · 2017-06-06T08:04:31Z

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "oma" [email protected]:zetaab/distribution.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

zetaab · 2017-06-06T08:20:26Z

original discussion in #2212

codecov · 2017-06-06T08:25:51Z

Codecov Report

Merging #2302 into master will decrease coverage by 9.81%.
The diff coverage is 53.19%.

@@            Coverage Diff             @@
##           master    #2302      +/-   ##
==========================================
- Coverage   61.42%   51.61%   -9.82%     
==========================================
  Files         128      128              
  Lines       11685    11727      +42     
==========================================
- Hits         7178     6053    -1125     
- Misses       3615     4916    +1301     
+ Partials      892      758     -134

Impacted Files	Coverage Δ
registry/root.go	`14.63% <20%> (+1.12%)`	⬆️
registry/storage/vacuum.go	`40.42% <42.85%> (+1.96%)`	⬆️
registry/storage/garbagecollect.go	`67.56% <71.42%> (+2.65%)`	⬆️
registry/storage/driver/gcs/gcs.go	`0.4% <0%> (-68.94%)`	⬇️
registry/storage/driver/oss/oss.go	`0.56% <0%> (-57.23%)`	⬇️
registry/storage/driver/s3-aws/s3.go	`4.57% <0%> (-56.23%)`	⬇️
registry/storage/driver/s3-goamz/s3.go	`0.5% <0%> (-51.4%)`	⬇️
registry/client/transport/transport.go	`69.69% <0%> (-9.1%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db0bc08...ff87ad8. Read the comment docs.

stevvooe · 2017-06-06T22:56:29Z

registry/root.go

 	RootCmd.AddCommand(ServeCmd)
 	RootCmd.AddCommand(GCCmd)
 	GCCmd.Flags().BoolVarP(&dryRun, "dry-run", "d", false, "do everything except remove the blobs")
+	GCCmd.Flags().BoolVarP(&manifestswithouttags, "delete-manifests", "m", false, "delete manifests which does not have tag attached")


delete-untagged. The manifest still exist. What we are trying to delete is the manifests that are not currently referenced via tag.

Note, that these may not always be manifests.

stevvooe · 2017-06-06T22:57:24Z

registry/root.go

 		}

-		err = storage.MarkAndSweep(ctx, driver, registry, dryRun)
+		err = storage.MarkAndSweep(ctx, driver, registry, dryRun, manifestswithouttags)


Adding a new boolean subsequent to another will get confusing. Let's define an options struct here.

I tried to find way to do that, but could not find anything that works. Can you help me little bit where I should define that struct (it needs to be available in other files where markandsweep is referenced)

// registry/storage/garbagecollect.go type GCOpts struct { DryRun bool ManifestsWithoutTags bool } ... err = storage.MarkAndSweep(ctx, driver, registry, storage.GCOpts{ DryRun: dryRun, ManifestsWithoutTags: manifestsWithoutTags, })

stevvooe · 2017-06-06T23:05:23Z

registry/storage/garbagecollect.go


 	// mark
 	markSet := make(map[digest.Digest]struct{})
+	manifestSet := make(map[digest.Digest]string)


manifestDeleteSet

Make sure to have a comment here that we are storing in the target repository.

stevvooe · 2017-06-06T23:08:18Z

registry/storage/garbagecollect.go

+				}
+				if len(tags) == 0 {
+					emit("manifest has no tags %s@%s", repoName, dgst)
+					manifestSet[dgst] = repoName


Because you are deleting the manifests after a shallow reference test (ie tags), I think you can get away with immediate deletion. Double check in paths.go.

I did not fully understand this. How I should check it with paths.go? And you mean that I could delete manifest here already? (means that I need initialize vacuum in start of function)

@zetaab The file paths.go documents the storage layout, including the relationships that must be maintained when modifying the storage layout. Any assumptions should be checked against the layout there.

I think you might be able to get away with issuing the delete right here, since the reference set is contained in the tags present here and nowhere else. Effectively, this list of tags is acting like a local reference count.

stevvooe · 2017-06-06T23:10:57Z

registry/storage/vacuum.go

 }

+// RemoveManifest removes a manifest from the filesystem
+func (v Vacuum) RemoveManifest(dgst string, name string) error {


These arguments are logically backwards. The name should come first.

stevvooe · 2017-06-06T23:11:14Z

registry/storage/vacuum.go

+		return err
+	}
+
+	context.GetLogger(v.ctx).Infof("Deleting manifest: %s", manifestPath)


Don't capitalize log messages.

stevvooe · 2017-06-06T23:11:32Z

registry/storage/vacuum.go

+	}
+
+	context.GetLogger(v.ctx).Infof("Deleting manifest: %s", manifestPath)
+	err = v.driver.Delete(v.ctx, manifestPath)


return v.driver.Delete(v.ctx, manifestPath)

stevvooe · 2017-06-06T23:13:05Z

registry/storage/vacuum.go

 }

+// RemoveManifest removes a manifest from the filesystem
+func (v Vacuum) RemoveManifest(dgst string, name string) error {


Also, type of dgst should be digest.Digest. No need to re-parse it.

stevvooe · 2017-06-06T23:15:54Z

@zetaab Thanks for the submission!

I have done some review in line. The logic looks correct but there are few nits and I think we can avoid the extra map, since we don't have to actually calculate reachability.

PTAL @dmcgowan @aaronlehmann @hinshun

hinshun

Design looks good! Thanks for the contribution. I've left some comments.

hinshun · 2017-06-07T17:50:33Z

registry/root.go

 }

 var dryRun bool
+var manifestswithouttags bool


Should camelCase local variables

hinshun · 2017-06-07T17:54:06Z

registry/root.go

 		}

-		err = storage.MarkAndSweep(ctx, driver, registry, dryRun)
+		err = storage.MarkAndSweep(ctx, driver, registry, dryRun, manifestswithouttags)


// registry/storage/garbagecollect.go type GCOpts struct { DryRun bool ManifestsWithoutTags bool } ... err = storage.MarkAndSweep(ctx, driver, registry, storage.GCOpts{ DryRun: dryRun, ManifestsWithoutTags: manifestsWithoutTags, })

hinshun · 2017-06-07T17:54:45Z

registry/storage/garbagecollect.go


 		err = manifestEnumerator.Enumerate(ctx, func(dgst digest.Digest) error {
+			if manifestswithouttags {
+				tags, errr := repository.Tags(ctx).Lookup(ctx, distribution.Descriptor{Digest: dgst})


s/errr/err/

zetaab · 2017-06-07T19:05:23Z

is there way to rebuild that circleci check?

Now using version stable
# cd .; git clone https://go.googlesource.com/tools /home/ubuntu/.gvm/pkgsets/stable/global/src/golang.org/x/tools
Cloning into '/home/ubuntu/.gvm/pkgsets/stable/global/src/golang.org/x/tools'...
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
package golang.org/x/tools/cover: exit status 128
package golang.org/x/tools/go/gcexportdata: cannot find package "golang.org/x/tools/go/gcexportdata" in any of:
	/home/ubuntu/.gvm/gos/stable/src/golang.org/x/tools/go/gcexportdata (from $GOROOT)
	/home/ubuntu/.gvm/pkgsets/stable/global/src/golang.org/x/tools/go/gcexportdata (from $GOPATH)

gvm use stable && go get github.com/axw/gocov/gocov github.com/golang/lint/golint
 returned exit code 1

Action failed: gvm use stable && go get github.com/axw/gocov/gocov github.com/golang/lint/golint

does not look like it is because of my code?

stevvooe · 2017-06-07T20:35:58Z

@zetaab We need to restart the build.

Let's make sure to address #2302 (comment) before merging.

zetaab · 2017-06-08T05:39:35Z

@stevvooe not sure is it now how it should be. at least I am checking is pathfor returning error or not.

stevvooe · 2017-06-08T21:50:51Z

registry/storage/garbagecollect.go

+					emit("manifest has no tags %s@%s", repoName, dgst)
+					emit("manifest eligible for deletion: %s", dgst)
+					// counter
+					manifestDeleteSet[dgst] = repoName


This set is no longer needed.

stevvooe · 2017-06-08T21:52:17Z

registry/storage/garbagecollect.go

+					if err != nil {
+						return err
+					}
+					emit("manifest has no tags %s@%s", repoName, dgst)


Only a single message is required here.

m-barthelemy · 2018-06-28T12:59:21Z

It looks like the latest official image for the registry is 2.6.2, released ~1 year(!) ago.
Is there any reason for that? Has the repository changed? Or is docker abandoning the project?
Is there any way to get a recent image of it without building it myself?

dahuether · 2018-09-19T12:22:47Z

Also still waiting for a new release.
Got still problems with it in harbor

boianmihailov · 2018-10-10T11:22:06Z

+1 here

manishtomar · 2018-10-10T16:40:22Z

@dahuether @boianmihailov New pre-release https://github.com/docker/distribution/releases/tag/v2.7.0-rc.0 is out. Please test and let us know if it works.

boianmihailov · 2018-10-11T11:59:07Z

@manishtomar Built an image from that tag, to make it work the entry point had to be changed to ENTRYPOINT ["./bin/registry"] and in my case garbage collector didn't delete anything.

Thanks!

zetaab · 2018-10-11T12:08:00Z

@boianmihailov you need to use -m flag in it. Also you need to have delete enabled in configuration.

Like in above comments:
docker exec xxx registry garbage-collect -m /etc/docker/registry/config.yml

boianmihailov · 2018-10-11T12:44:26Z

@zetaab Thanks for the input! adding -m works!
So I think the RC looks good so far.

branzo · 2018-10-11T13:37:06Z

I can confirm that this is working!

wwyhy · 2018-12-06T06:17:26Z

Where I can find out the docker image of registry 2.7.0?

There is no 2.7.0 in docker hub https://hub.docker.com/r/vmware/registry/tags.

Thanks!

manishtomar · 2018-12-06T06:45:04Z

@wwyhy It should appear in https://hub.docker.com/r/_/registry/ in a few days time.

wwyhy · 2018-12-06T07:10:54Z

@manishtomar

Ok, thank you! hope the it can GC the unused manifest.

johan-smits · 2018-12-14T12:34:03Z

@manishtomar is there an eta of the release to docker hub?

gertvdijk · 2018-12-14T13:12:45Z

@johan-smits The whole docker hub repo got nuked it seems. 😭 Getting a 404 @ https://hub.docker.com/r/_/registry/ currently.

maksym-nazarenko · 2018-12-14T13:21:34Z

@gertvdijk you should use https://hub.docker.com/r/distribution/registry/ instead.
upd: just ignore that ^^^.

gertvdijk · 2018-12-14T13:44:39Z

@Maxim-Nazarenko

@gertvdijk you should use https://hub.docker.com/r/distribution/registry/ instead.

Nope. See the warning there:

WARNING: NOT the registry official image!!! What you LIKELY want is docker pull registry

ghost · 2018-12-14T13:55:08Z

It looks like the paths changed; https://hub.docker.com/r/_/registry/ gives me a 404, but https://hub.docker.com/_/registry works. (Still no 2.7.0, though...)

ghost · 2018-12-14T14:00:01Z

Interestingly, the source repo already has 2.7.0 changes: https://github.com/docker/distribution-library-image/ Maybe some build process has to be triggered?

@dmcgowan @stevvooe do you know when 2.7.0 will be available on docker hub?

manishtomar · 2018-12-14T18:04:46Z

@ffdybuster #2784 (comment)

gertvdijk · 2019-01-04T08:26:50Z

2.7.0 is now available on Docker hub (with this PR being merged).

https://hub.docker.com/_/registry?tab=tags -> lists 2.7.0

zhangsean · 2019-05-19T12:20:34Z

Wonderful work!

scrwr · 2020-01-03T11:20:09Z

What am I overseeing here:

docker run --rm registry --version
registry github.com/docker/distribution v2.7.1

docker run --rm registry --help
registry

Usage:
registry [flags]
registry [command]

Available Commands:
serve serve stores and distributes Docker images
garbage-collect garbage-collect deletes layers not referenced by any manifests
help Help about any command

Flags:
-h, --help=false: help for registry
-v, --version=false: show the version and exit

Neither -m option nor --dry-run option here, although clearly version 2.7.1. Tried the same with 2.7.0 image. Same issue.

johan-smits · 2020-01-03T13:26:25Z

@scrwr inside the docker you can run: registry garbage-collect -m /etc/docker/registry/config.yml

scrwr · 2020-01-03T16:05:56Z

@johan-smits
Gnaa, you are right. I was looking at the global help and expected to find garbage-collect help info.

docker run --rm registry garbage-collect --help
lists the -m and --dry-run params. Quite embarrasing. :(

GordonTheTurtle added the dco/no label Jun 6, 2017

zetaab force-pushed the oma branch from eaf53c6 to bc33ad2 Compare June 6, 2017 08:08

GordonTheTurtle removed the dco/no label Jun 6, 2017

GordonTheTurtle added the dco/no label Jun 6, 2017

zetaab force-pushed the oma branch from 867f054 to 98e1d21 Compare June 6, 2017 10:50

GordonTheTurtle added dco/no and removed dco/no labels Jun 6, 2017

zetaab force-pushed the oma branch from 5cdd6d6 to 2150d49 Compare June 6, 2017 11:29

GordonTheTurtle removed the dco/no label Jun 6, 2017

zetaab force-pushed the oma branch from 2150d49 to f512691 Compare June 6, 2017 11:51

stevvooe reviewed Jun 6, 2017

View reviewed changes

hinshun reviewed Jun 7, 2017

View reviewed changes

zetaab force-pushed the oma branch from 6437edd to 327d910 Compare June 8, 2017 05:38

zetaab force-pushed the oma branch from 327d910 to 3ae18e6 Compare June 8, 2017 05:46

stevvooe reviewed Jun 8, 2017

View reviewed changes

branzo mentioned this pull request Oct 11, 2018

[Question] Test Registry 2.7.0 w/possibility to clean untagged manifests SUSE/Portus#2002

Closed

glensc mentioned this pull request Oct 13, 2018

readme: docker registry 2.7 should have support sciapp/gitlab-registry-cleanup#10

Merged

x-yuri mentioned this pull request Jan 16, 2019

Feature req: garbage-collector should have possibility to clean untagged revisions #2301

Closed

x-yuri mentioned this pull request Mar 29, 2019

Registry garbage-collect does not clean up blobs from overwritten tags #2212

Closed

villevaltonen mentioned this pull request Jan 26, 2020

Adding --delete-untagged parameter description docker/docs#10197

Closed

kuzaxak mentioned this pull request Sep 18, 2021

Garbage collector cronjob twuni/docker-registry.helm#28

Merged

villevaltonen mentioned this pull request Dec 1, 2021

Adding --delete-untagged to registry garbage collection docs docker/docs#13919

Merged

add possibility to clean untagged manifests #2302

add possibility to clean untagged manifests #2302

Conversation

zetaab commented Jun 6, 2017

Uh oh!

GordonTheTurtle commented Jun 6, 2017

Uh oh!

zetaab commented Jun 6, 2017

Uh oh!

codecov bot commented Jun 6, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevvooe commented Jun 6, 2017

Uh oh!

hinshun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zetaab commented Jun 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stevvooe commented Jun 7, 2017

Uh oh!

zetaab commented Jun 8, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

m-barthelemy commented Jun 28, 2018

Uh oh!

dahuether commented Sep 19, 2018

Uh oh!

boianmihailov commented Oct 10, 2018

Uh oh!

manishtomar commented Oct 10, 2018

Uh oh!

boianmihailov commented Oct 11, 2018

Uh oh!

zetaab commented Oct 11, 2018

Uh oh!

boianmihailov commented Oct 11, 2018

Uh oh!

branzo commented Oct 11, 2018

Uh oh!

wwyhy commented Dec 6, 2018

Uh oh!

manishtomar commented Dec 6, 2018

codecov bot commented Jun 6, 2017 •

edited

Loading

zetaab commented Jun 7, 2017 •

edited

Loading

maksym-nazarenko commented Dec 14, 2018 •

edited

Loading

gertvdijk commented Dec 14, 2018 •

edited

Loading