Feature request: support S3 protocol for scanning mirrors #141
Description
The Jenkins project is currently evaluating to use mirrors hosted on Cloudflare R2 bucket which is an S3-compatible system providing direct HTTP access.
Details on jenkins-infra/helpdesk#2649
As such, we would be interested into the support of the S3 protocol to scan mirrors, along with rsync or ftp to support S3 buckets as mirrors (each cloud provider has their compliant).
AWS provides a nifty S3 SDK: https://docs.aws.amazon.com/sdk-for-go/api/service/s3/ that could be used.
I believe the challenge is defining the configuration for credentials (but a default delegation to env vars. or ~/.aws/ could be enough).
Note that Azure bucket scanning is also an option: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-go?tabs=roles-azure-portal (it is not S3)