Parsing CRI-formatted logs from file source?

[baodrate]

Is it possible to use the kubernetes_log parser with file, without all the kubernetes runtime stuff? Like as a transform, perhaps?

Motivation: I'm using the podman driver with nomad which creates a log file in the CRI logging format. (i.e. <timestamp> <stdout|stderr> <P|F> [data]). This is already implemented within https://github.com/vectordotdev/vector/blob/master/src/sources/kubernetes_logs/parser/cri.rs, but is there a way to use this parser with the standard file source?

Workarounds considered:

• Using the journald log driver for nomad-podman.

  • This is doable, but has additional overhead because logs are parsed and copied/written by journald, vector, and nomad.
  • Not sure of the best way to annotate these journald logs with nomad metadata. For file logging see: https://github.com/mr-karan/nomad-vector-logger and https://github.com/input-output-hk/nomad-follower

• Manual parsing

  • Seems like it should be possible
    • the multiline option + regex is probably insufficient since the two streams and interrupt each other
    • seems like you should be able to carry enough state with lua parsing to do the job
  • I'd prefer to avoid re-implementing parsing since it's already handled very well natively by vector

[jszwedko]

Hey!

Yes, I think using the multiline options and regex parsing would be the most straightforward.

If you felt like contributing, I think CRI parsing could be moved to a codec and more easily used by the file and other sources. The syslog codec you see in there is a fairly similar example. There would also need to be a new framing option that understands how to collapse multiline CRI logs or the existing multiline options would need to continue to be used even if a codec existed since codecs only see one "frame" at a time.

[baodrate]

Thanks for the guidance! I'll give the parsing a shot. Then I'll see about putting together a PR for moving the CRI codec.

For the multiline parsing, do you have any advice on separating the stdout and stderr streams? Lines from one stream can intersperse the lines from the other, so filtering would need to be done before aggregating multiple lines. Would the right approach be:

source:file -> transform:route -> [stdout] transform:reduce -> sink
                               -> [stderr] transform:reduce -> 

[jszwedko]

Aaah, that's a good note. Yeah, in that case, using reduce would be better so that you can split before merging lines.

[jmealo]

@baodrate How did you make out? I'm using the file source with kubernetes to avoid calling the Kubernetes API.