Add YAML config source #402
Conversation
|
After naming it |
| component_0: | | ||
| $yaml: ./templates/component_template | ||
| glob_pattern: /var/**/*.log | ||
| format: json |
There was a problem hiding this comment.
Not directly related to this PR but could we do multi-param like:
$yaml:
selector: ./templates/component_template
glob_pattern: /var/**/*.log
format: json(Downside of having it within text block is the yaml inside text doesn't get syntax highlighted, etc. when editing)
There was a problem hiding this comment.
We had some early interactions with it as actual YAML. That ended up being dropped in favor of consistency when used in the single-line format. Actual YAML on single-line didn't look very good and couldn't be concatenated to other values. That said we could have the multiline format being YAML while the single line was our own but that requires a lot more code.
/cc @tigrannajaryan
There was a problem hiding this comment.
Would include be a better name?
I like
templatebecause all golang template processing is available but no strong opinion in this regard. Let me know if you preferinclude.
I think I prefer include since it may be easier to understand for end users the purpose of it. Not strongly opposed to template but it appears a bit obscure, not self-explanatory what it does.
re: stanza plugins
Interesting stuff: at first glance, it seems that they parse the YAML and extract a template from the comments. It should be doable to convert those to the templates supported here.
Yes, I think we can delete the top portion of their YAMLs, which just declare the parameters (I think we can just replace it by comments, I don't think it enables any functionality) and slightly adjust the pipeline portion to match the operators expected by filelog and it should work.
There was a problem hiding this comment.
(Sorry, replied to a wrong comment thread).
There was a problem hiding this comment.
np, I will rename this config source to include.
|
Got CI failures that seem unrelated to the change, closing and re-opening to kick another run. |
|
close/open doesn't work ... |
pjanotti
force-pushed
the
add-yamlconfigsource
branch
from
83750f5 to
0671c43
Compare
Would config_sources:
include:
receivers:
filelog: |
$include: ./plugins/tomcat.yaml
log_dir: /var/log/tomcat |
|
BTW, stanza has a ton of plugins which use the go templating and which I want to reuse for Otelcol logs: https://github.com/observIQ/stanza-plugins/tree/master/plugins I think we can automate conversion of stanza plugins to filelog-based |
pjanotti
force-pushed
the
add-yamlconfigsource
branch
from
0671c43 to
6c832b1
Compare
| component_0: | | ||
| $template: ./templates/component_template | ||
| my_glob_pattern: /var/**/*.log | ||
| my_format: json |
There was a problem hiding this comment.
Curious if it's possible to use env var expansion here for the template value
I like
Interesting stuff: at first glance, it seems that they parse the YAML and extract a template from the comments. It should be doable to convert those to the templates supported here. |
Adds a config source that can be templatized via https://pkg.go.dev/text/template
Since changing the template typically requires also changing the parameters passed to the template this config source doesn't watch for the template files, i.e., changes to the template file won't trigger reload of the configuration.