Metric names are very unusual and inconvenient.

[Monnoroch]

Metrics exported from the "production" images (not all-in-one though) are not "namespaced". If you have a Prometheus that monitors dozens of apps, you can have thousands of metrics. The names of Jaeger metrics (i.e. IndexCreate:attempts) are very not friendly to this situation. Normally I would expect a binary names jaeger-collector to export metrics starting with jaeger_collector_. This is the case with the all-in-one binary, but not the other ones.

Other less important issues:

• It is common to export metrics with underscores, not CamelCase. Also not an issue in all-in-one binary afaik.
• The : in metric names is unique to Jaeger.

[yurishkuro]

+1 to add service name as a prefix

the camelcase should be easy to fix - it just refers to table names

The : is used as namespace separator, e.g. cassandra:Read:ReadTraces:latency_err_count. With your first suggestion it would become jaeger_query:cassandra:Read:ReadTraces:latency_err_count, however this is because we are still using hierarchical names (#395) instead of tags for dimensions. I am not a fan of going all the way to tags, e.g.

latency{service="jaeger_query",storage="cassandra",path="read",action="ReadTraces",result="err"}

because the metric name can easily clash with something else that uses different dimensions.

Suggestions are welcome.

[Monnoroch]

because the metric name can easily clash with something else that uses different dimensions.

That was my original point: metrics should be "namespaced". I would expect the collector to export jaeger_latency{service="collector",storage="cassandra",path="read",action="ReadTraces",result="err"} and the agent to export jaeger_latency{service="agent",storage="cassandra",path="read",action="ReadTraces",result="err"} (btw I can't find the URL to get agent metrics on 1.2.0, but that's a whole other story).

That way metrics don't clash with other monitored software. For example you can look at metrics exported by official Prometheus products: Prometheus itself, node_exporter and a few other "official" exporters from this page. My suggestion is to follow the standard prometheus metric naming style because most software does and this will allow jaeger to coexist with it very nicely. Also, the all-in-one binary already has nice metrics (except for use of : instead of standard _).

[yurishkuro]

These two examples jaeger_latency{service="collector"... and jaeger_latency{service="agent"... are insufficient to dedupe the metrics. The collector can very easily have another jaeger_latency metric with completely different dimensions (e.g. something about http endpoints, nothing to do with storage), which will blow up prometheus client. So we'd instead do something like jaeger_collector:storage:latency{...} and jaeger_collector:http_server:latency{...}

[Monnoroch]

That's totally fine, it's easy to ensure metrics are called differently within the same project. Right now the bigger issue is ensuring they're unique between projects to scrape different software with a single prometheus instance. I literally have a metric called IndexCreate:attempts now and this is not very user-friendly.

[yurishkuro]

I literally have a metric called IndexCreate:attempts now and this is not very user-friendly.

doh, that's just a bug - #733

[Monnoroch]

Nice bug fixing latency, thanks! But still, you might want to think about aligning with standard prometheus metric naming conventions.