|
| 1 | +# K8slog Receiver |
| 2 | + |
| 3 | +<!-- status autogenerated section --> |
| 4 | +| Status | | |
| 5 | +| ------------- |-----------| |
| 6 | +| Stability | [development]: logs | |
| 7 | +| Distributions | [] | |
| 8 | +| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Areceiver%2Fk8slog) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Areceiver%2Fk8slog) | |
| 9 | +| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@h0cheung](https://www.github.com/h0cheung), [@TylerHelmuth](https://www.github.com/TylerHelmuth) | |
| 10 | + |
| 11 | +[development]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#development |
| 12 | +<!-- end autogenerated section --> |
| 13 | + |
| 14 | +Tails and parses logs in k8s environment. |
| 15 | + |
| 16 | +There only one mode of discovery as for now, it's specified by the `discovery.mode` configuration option: |
| 17 | +- `daemonset-stdout`: (default) Deployed as a DaemonSet, the receiver will read logs from the stdout of pods in the same node. |
| 18 | + |
| 19 | +Two modes of discovery are planned to be supported in the future: |
| 20 | + |
| 21 | +- `daemonset-file`: Deployed as a DaemonSet, the receiver will read logs from files inside pods in the same node. |
| 22 | +- `sidecar`: Deployed as a sidecar container, the receiver will read logs from files. |
| 23 | + |
| 24 | +## Configuration |
| 25 | + |
| 26 | +The following settings are common to all discovery modes: |
| 27 | + |
| 28 | +| Field | Default | Description | |
| 29 | +|------------------|--------------------|------------------------------------------------------------------------------------------------------------------| |
| 30 | +| `discovery.mode` | `daemonset-stdout` | The mode of discovery. Only `daemonset-stdout` is supported now. `daemonset-file` and `sidecar` are coming soon. | |
| 31 | +| `extract` | | The rules to extract metadata from pods and containers. TODO default values. | |
| 32 | +| TODO: add fields for reading files similar to filelogreceiver | |
| 33 | + |
| 34 | +When `discovery.mode` is not `sidecar`, there are additional configuration options: |
| 35 | + |
| 36 | +| Field | Default | Description | |
| 37 | +|-------------------------------|------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| |
| 38 | +| `discovery.k8s_api.auth_type` | `serviceAccount` | The authentication type of k8s api. Options are `serviceAccount` or `kubeConfig`. | |
| 39 | +| `discovery.host_root` | `/host-root` | The directory which the root of host is mounted on. | |
| 40 | +| `discovery.runtime_apis` | | The runtime apis used to get log file paths. docker and cri-containerd are supported now. By default, it will try to automatically detect the cri-containerd. | |
| 41 | +| `discovery.node_from_env` | `KUBE_NODE_NAME` | The environment variable name of node name. | |
| 42 | +| `discovery.filter` | [] | The filter used to filter pods and containers. By default, all pods and containers will be collected. | |
| 43 | + |
| 44 | +### Operators |
| 45 | + |
| 46 | +Each operator performs a simple responsibility, such as parsing a timestamp or JSON. Chain together operators to process logs into a desired format. |
| 47 | + |
| 48 | +- Every operator has a `type`. |
| 49 | +- Every operator can be given a unique `id`. If you use the same type of operator more than once in a pipeline, you must specify an `id`. Otherwise, the `id` defaults to the value of `type`. |
| 50 | +- Operators will output to the next operator in the pipeline. The last operator in the pipeline will emit from the receiver. Optionally, the `output` parameter can be used to specify the `id` of another operator to which logs will be passed directly. |
| 51 | +- Only parsers and general purpose operators should be used. |
| 52 | + |
| 53 | +### Filters |
| 54 | + |
| 55 | +When `discovery.mode` is not `sidecar`, the `discovery.filter` field can be used to filter pods and containers. The filter is a list of rules. Each rule is a map with the following fields: |
| 56 | + |
| 57 | +| Field | Description | |
| 58 | +|---------------|--------------------------------------------------------------| |
| 59 | +| `annotations` | MapFilters that filters pods by annotations. | |
| 60 | +| `labels` | MapFilters that filters pods by labels. | |
| 61 | +| `env` | MapFilters that filters containers by environment variables. | |
| 62 | +| `containers` | ValueFilters that filters containers by name. | |
| 63 | +| `namespaces` | ValueFilters that filters pods by namespace. | |
| 64 | +| `pods` | ValueFilters that filters pods by name. | |
| 65 | + |
| 66 | +#### MapFilter |
| 67 | + |
| 68 | +A MapFilter can be used to filter pods by maps, such as annotations or labels. It has the following fields: |
| 69 | + |
| 70 | +| Field | Description | |
| 71 | +|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
| 72 | +| `op` | The operation to perform. Options are: <br> - "equals": (default) the value must be equal to the specified value. <br>- "not-equals": the value must not be equal to the specified value. <br> - "exists": the value must exist. <br> - "not-exists": the value must not exist. <br> - "matches": the value must match the specified regular expression. <br> - "not-matches": the value must not match the specified regular expression. | |
| 73 | +| `key` | The key of the map. | |
| 74 | +| `value` | The value to match. Only used for "equals", "not-equals", "matches", and "not-matches" operations. | |
| 75 | + |
| 76 | +#### ValueFilter |
| 77 | + |
| 78 | +A ValueFilter can be used to filter pods by string values, such as container names or namespaces. It has the following fields: |
| 79 | + |
| 80 | +| Field | Description | |
| 81 | +|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
| 82 | +| `op` | The operation to perform. Options are: <br> - "equals": (default) the value must be equal to the specified value. <br>- "not-equals": the value must not be equal to the specified value. <br> - "matches": the value must match the specified regular expression. <br> - "not-matches": the value must not match the specified regular expression. | |
| 83 | +| `value` | The value to match. | |
| 84 | + |
| 85 | +### Extract |
| 86 | + |
| 87 | +The `extract` field can be used to extract fields from the log file path. It has the following fields: |
| 88 | + |
| 89 | +| Field | Description | |
| 90 | +|---------------|--------------------------------------------------------------------------------------| |
| 91 | +| `metadata` | A string slice of metadata to extract from the pods and containers. | |
| 92 | +| `env` | A FieldExtractConfig that extracts fields from environment variables of containers. | |
| 93 | +| `annotations` | A FieldExtractConfig that extracts fields from annotations of pods. | |
| 94 | +| `labels` | A FieldExtractConfig that extracts fields from labels of pods. | |
| 95 | + |
| 96 | +#### FieldExtractConfig |
| 97 | + |
| 98 | +A FieldExtractConfig can be used to extract fields from maps, such as annotations or labels. It has the following fields: |
| 99 | + |
| 100 | +| Field | Description | |
| 101 | +|-------------|------------------------------------------------------------------------------------------------------| |
| 102 | +| `tag_name` | Required. The name of the extracted attributes. | |
| 103 | +| `key` | The key of the map (annotation, label or etc).Exactly one of `key` or `key_regex` must be specified. | |
| 104 | +| `key_regex` | The regular expression of the key. Exactly one of `key` or `key_regex` must be specified. | |
| 105 | +| `regex` | Optional. The regular expression to extract a submatch from the value. | |
| 106 | + |
| 107 | +### Supported encodings |
| 108 | + |
| 109 | +| Key | Description | |
| 110 | +|------------|------------------------------------------------------------------| |
| 111 | +| `nop` | No encoding validation. Treats the file as a stream of raw bytes | |
| 112 | +| `utf-8` | UTF-8 encoding | |
| 113 | +| `utf-16le` | UTF-16 encoding with little-endian byte order | |
| 114 | +| `utf-16be` | UTF-16 encoding with big-endian byte order | |
| 115 | +| `ascii` | ASCII encoding | |
| 116 | +| `big5` | The Big5 Chinese character encoding | |
| 117 | + |
| 118 | +Other less common encodings are supported on a best-effort basis. See [https://www.iana.org/assignments/character-sets/character-sets.xhtml](https://www.iana.org/assignments/character-sets/character-sets.xhtml) for other encodings available. |
| 119 | + |
| 120 | +## Additional Terminology and Features |
| 121 | + |
| 122 | +- An [entry](../../pkg/stanza/docs/types/entry.md) is the base representation of log data as it moves through a pipeline. All operators either create, modify, or consume entries. |
| 123 | +- A [field](../../pkg/stanza/docs/types/field.md) is used to reference values in an entry. |
| 124 | +- A common [expression](../../pkg/stanza/docs/types/expression.md) syntax is used in several operators. For example, expressions can be used to [filter](../../pkg/stanza/docs/operators/filter.md) or [route](../../pkg/stanza/docs/operators/router.md) entries. |
| 125 | + |
| 126 | +### Parsers with Embedded Operations |
| 127 | + |
| 128 | +Many parsers operators can be configured to embed certain followup operations such as timestamp and severity parsing. For more information, see [complex parsers](../../pkg/stanza/docs/types/parsers.md#complex-parsers). |
| 129 | + |
| 130 | +## Example - Collect logs from stdout of all containers |
| 131 | + |
| 132 | +Receiver Configuration |
| 133 | +```yaml |
| 134 | +receivers: |
| 135 | + k8slog: |
| 136 | + discovery: |
| 137 | + mode: daemonset-stdout |
| 138 | + operators: |
| 139 | + - type: recombine |
| 140 | + combine_field: body |
| 141 | + is_first_entry: body matches "^\\d{4}-\\d{2}-\\d{2}" |
| 142 | + max_log_size: 128kb |
| 143 | + source_identifier: attributes["k8s.pod.uid"] |
| 144 | +``` |
0 commit comments