nomad-vector-logger

A daemon which continuously watches for deployments in a Nomad cluster and generates a Vector configuration file, which can be used to collect logs enriched with Nomad metadata.

Each log event is annotated with the following metadata:

• Namespace of the application
• Node where the deployment is running
• Job name
• Group name
• Task name
• Allocation ID

Why

Problem

Currently, Nomad stores all application logs inside $NOMAD_DATA_DIR/$NOMAD_ALLOC_DIR/logs/ directory. The limitation is that these logs don't have any information about the task/job/allocation etc. Suppose there are multiple deployments on the same host. In that case, no central log collecting agent can distinguish and process these logs uniquely.

For the docker driver, this is a non-issue since logging of tasks with the docker driver is configured with logging stanza.

Users running deployments with raw_exec and exec as the task driver will find that no such configuration exists as mentioned in this GitHub Issue.

Solution

• nomad-vector-logger is a daemon that runs in the background, periodically polling for Allocations on the node.
• It then generates a vector configuration to collect logs from the allocation's log directory. It enriches the log event with relevant metadata.
• vector is started with a --watch-config flag, which automatically live-reloads vector whenever config changes. A config change can happen whenever an allocation is created/stopped/restarted.

You can see a sample config file that is generated by this daemon. This config file can be used in addition to other vector config files to provide the config for the rest of the pipeline (additional transformations, sinks etc.).

Before

Logs without any metdata on /opt/nomad/data/alloc/$ALLOC_ID/alloc/logs:

==> proxy.stdout.0 <==
192.168.29.76 - - [03/Sep/2022:17:30:36 +0000] "GET / HTTP/1.1" 200 27 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0" "-"

After

This is an example JSON log collected from nginx task running with raw_exec task driver on Nomad, collected using vector:

{
    "file": "/opt/nomad/data/alloc/64a2f9fd-e003-0bb3-b5cd-838125283a06/alloc/logs/proxy.stdout.0",
    "host": "pop-os",
    "message": "192.168.29.76 - - [03/Sep/2022:17:30:36 +0000] \"GET / HTTP/1.1\" 200 27 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0\" \"-\"",
    "nomad": {
        "alloc_id": "64a2f9fd-e003-0bb3-b5cd-838125283a06",
        "group_name": "nginx",
        "job_name": "nginx",
        "namespace": "default",
        "node_name": "pop-os",
        "task_name": "proxy"
    },
    "source_type": "file",
    "timestamp": "2022-09-03T17:30:42.569487273Z"
}

Dev Setup

make dev

You can refer to a local dev suite which runs this program in a Nomad cluster. The jobspec can also be used as a reference for production deployment.

Deployment Notes

• This program is meant to be run inside a Nomad cluster and should have proper ACL to fetch Allocation:* events. You can use this ACL policy to generate a token:

namespace "*" {
  policy = "read"
}

node {
  policy = "read"
}

agent {
  policy = "read"
}

• It's preferable to run it as a system job. Each program allocation will be responsible for configuring vector to collect logs from that particular log directory on the host.

You can choose one of the various deployment options:

Binary

Grab the latest release from Releases.

To run:

$ ./nomad-vector-logger.bin --config config.toml

Nomad

View a sample deployment file at dev/deployment.nomad.

Docker

Docker images are available on GitHub.

Configuration

Refer to config.sample.toml for a list of configurable values.

Environment Variables

All config variables can also be populated as env variables by prefixing NOMAD_VECTOR_LOGGER_ and replacing . with __.

For eg: app.data_dir becomes NOMAD_VECTOR_LOGGER_app__data_dir.

Contribution

Please feel free to open a new issue for bugs, feedback etc.

LICENSE

LICENSE