Skip to content

Default max_buffer_size causes silent failures when streaming #503

Closed
@hauntsaninja

Description

@hauntsaninja

Thanks for a useful library!

Consider the following repro:

import os
import msgpack
import tempfile

original_values = [
    (b"x" * 1024) * (1024 * 40 * i)
    for i in range(1, 5)
]

with tempfile.TemporaryDirectory() as tmpdir:
    fn = os.path.join(tmpdir, "test.msgpack")
    with open(fn, "wb") as f:
        for v in original_values:
            msgpack.pack(v, f)
    
    with open(fn, "rb") as f:
        unpack = msgpack.Unpacker(f, max_buffer_size=0)
        values = list(unpack)
    assert values == original_values

    with open(fn, "rb") as f:
        unpack = msgpack.Unpacker(f)
        values = list(unpack)
    # failed to unpack some of the stream, but we wouldn't know if we didn't have original_values to check against
    assert values == original_values

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions