-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Closed
Labels
needs: approvalNeeds review & approval before work can begin.Needs review & approval before work can begin.needs: more demandNeeds more demand before work can begin, +1 or comment to support.Needs more demand before work can begin, +1 or comment to support.sink: fileAnything `file` sink relatedAnything `file` sink relatedtype: enhancementA value-adding code change that enhances its existing functionality.A value-adding code change that enhances its existing functionality.
Description
When writing log data to a file it's common for users to want to limit files to a maximum size. Currently, this is not possible with Vector's file sink. There are 2 ways I could see approaching this:
1. Batching
Similar to the aws_s3 sink, we could batch data before flushing to a file.
Pros
- We'll write to the file in a single transaction, avoiding the need to deal with files being deleted, etc.
- It should be simpler to implement given that we already have batching support in place.
Cons
- Data is not immediately available in the file.
- It's inefficient. If the destination is the disk it seems strange to buffer data in memory, or on disk, before writing.
2. Streaming / Monitoring
Alternatively, we could monitor the destination file and rotate it when the size exceeds a limit.
Pros
- Much more efficient in terms of resource usage.
- Data is immediately available in the file.
Cons
- Seems more complex.
- How do we deal with files being removed out from under us?
- Is file rotation outside the scope of Vector? There are tools built specifically for this.
crazy-max, ingshtrom, davidrunger, QSummerY and aagor
Metadata
Metadata
Assignees
Labels
needs: approvalNeeds review & approval before work can begin.Needs review & approval before work can begin.needs: more demandNeeds more demand before work can begin, +1 or comment to support.Needs more demand before work can begin, +1 or comment to support.sink: fileAnything `file` sink relatedAnything `file` sink relatedtype: enhancementA value-adding code change that enhances its existing functionality.A value-adding code change that enhances its existing functionality.