Skip to content

Object store failing during bulk delete in S3 #277

@Omega359

Description

@Omega359

Describe the bug

This started recently and I'm uncertain as to the cause. I have a rather large S3 directory (10TB) that I delete using the following code:

let object_store_s3_path = &object_store::path::Path::from(s3_path);
let stream = object_store.list(Some(object_store_s3_path)).map_ok(|m| m.location).boxed();
object_store.delete_stream(stream).try_collect::<Vec<Path>>().await?;

The error I started receiving this weekend is:

Got invalid DeleteObjects response: unknown variant `Code`, expected `Deleted` or `Error`

Caused by:
    0: Got invalid DeleteObjects response: unknown variant `Code`, expected `Deleted` or `Error`
    1: unknown variant `Code`, expected `Deleted` or `Error`

Using version 0.11.0.

Deleting the directory via other tooling works as expected.

To Reproduce

For me just attempting to delete a large directory seems to trigger it.

Expected behavior

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions