Skip to content

Commit 194009c

Browse files
feat: Add OTel Kafka Docs pages
1 parent fd0a8dd commit 194009c

19 files changed

+7292
-1080
lines changed
Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
---
2+
title: Kafka monitoring overview
3+
tags:
4+
- Integrations
5+
- On-host integrations
6+
- On-host integrations list
7+
metaDescription: "Choose the right Kafka monitoring approach: New Relic Native agent, OpenTelemetry Collector, AWS MSK, or Confluent Cloud."
8+
freshnessValidatedDate: never
9+
---
10+
11+
Monitor your Apache Kafka clusters, brokers, producers, consumers, and topics with New Relic. We offer multiple approaches to instrument and collect Kafka metrics based on your environment and preferences.
12+
13+
## Choose your monitoring approach [#choose-approach]
14+
15+
Select the monitoring solution that best fits your Kafka deployment:
16+
17+
<table>
18+
<thead>
19+
<tr>
20+
<th style={{ width: "200px" }}>
21+
Integration Type
22+
</th>
23+
<th>
24+
Best For
25+
</th>
26+
<th>
27+
Key Features
28+
</th>
29+
</tr>
30+
</thead>
31+
32+
<tbody>
33+
<tr>
34+
<td>
35+
**[NRI-Kafka](/docs/infrastructure/host-integrations/host-integrations-list/kafka/nri-kafka/kafka-integration)**<br/>
36+
(New Relic Native)
37+
</td>
38+
<td>
39+
Traditional on-host deployments for existing New Relic Infrastructure users or new users
40+
</td>
41+
<td>
42+
* Native New Relic agent
43+
* Automatic broker discovery (Zookeeper/Bootstrap)
44+
* Out-of-the-box dashboards
45+
</td>
46+
</tr>
47+
48+
<tr>
49+
<td>
50+
**[AWS MSK Integration](/docs/message-queues-streaming/installation/aws-managed-kafka-msk-integration)**<br/>
51+
(New Relic Native)
52+
</td>
53+
<td>
54+
Amazon Managed Streaming for Apache Kafka (MSK) users
55+
</td>
56+
<td>
57+
* CloudWatch-based metrics collection
58+
* AWS MSK-specific metrics
59+
* Managed service integration
60+
* No agent installation required
61+
</td>
62+
</tr>
63+
64+
<tr>
65+
<td>
66+
**[Confluent Cloud Integration](/docs/message-queues-streaming/installation/confluent-cloud-integration)**<br/>
67+
(New Relic Native)
68+
</td>
69+
<td>
70+
Confluent Cloud managed Kafka users
71+
</td>
72+
<td>
73+
* API-based metrics collection
74+
* Confluent Cloud-specific metrics
75+
* Fully managed integration
76+
* No infrastructure access needed
77+
</td>
78+
</tr>
79+
80+
<tr>
81+
<td>
82+
**[OpenTelemetry Kafka](/docs/opentelemetry/integrations/kafka/overview)**<br/>
83+
(OTel Collector)
84+
</td>
85+
<td>
86+
Cloud-native environments, Kubernetes, multi-vendor observability
87+
</td>
88+
<td>
89+
* Vendor-neutral telemetry collection
90+
* Flexible deployment (host, Kubernetes, sidecar)
91+
* Extensive metric collection via JMX
92+
* Support for self-hosted and Kubernetes (Strimzi)
93+
</td>
94+
</tr>
95+
</tbody>
96+
</table>
97+
98+
## Platform-specific guides [#platform-guides]
99+
100+
### New Relic Native Kafka integration
101+
102+
* **[NRI-Kafka installation](/docs/infrastructure/host-integrations/host-integrations-list/kafka/nri-kafka/kafka-integration)**: Traditional on-host integration
103+
* **[AWS MSK integration](/docs/message-queues-streaming/installation/aws-managed-kafka-msk-integration)**: Monitor Amazon MSK clusters
104+
* **[Confluent Cloud integration](/docs/message-queues-streaming/installation/confluent-cloud-integration)**: Monitor Confluent Cloud Kafka
105+
106+
### OpenTelemetry Kafka monitoring
107+
108+
OpenTelemetry provides flexible instrumentation for various Kafka deployments:
109+
110+
* **[Self-hosted Kafka](/docs/opentelemetry/integrations/kafka/self-hosted)**: Install on Linux hosts with direct Kafka access
111+
* **[Kubernetes with Strimzi](/docs/opentelemetry/integrations/kafka/kubernetes-strimzi)**: Deploy on Kubernetes using kubectl
112+
113+
## What's collected? [#whats-collected]
114+
115+
All Kafka monitoring approaches collect key metrics including:
116+
117+
* **Cluster metrics**: Topic count, partition distribution, controller status
118+
* **Broker metrics**: Request rates, byte rates, replica lag, under-replicated partitions
119+
* **Topic metrics**: Messages per second, bytes in/out per topic
120+
* **Consumer group metrics**: Lag, commit rates, group membership
121+
122+
For detailed metric definitions:
123+
* NRI-Kafka metrics (documented in the [NRI-Kafka integration page](/docs/infrastructure/host-integrations/host-integrations-list/kafka/nri-kafka/kafka-integration#metrics))
124+
* [OpenTelemetry metrics reference](/docs/opentelemetry/integrations/kafka/metrics-reference)
125+
126+
## Next steps [#next-steps]
127+
128+
After setting up Kafka monitoring:
129+
130+
* **Explore your data**: View Kafka metrics in multiple ways:
131+
* Navigate to <DNT>**[one.newrelic.com](https://one.newrelic.com) > All Capabilities > Queues & Streams**</DNT>
132+
* Check All entities for a list of Kafka entities that were created
133+
* Go to <DNT>**[one.newrelic.com](https://one.newrelic.com) > All Capabilities > Infrastructure > Third-party services**</DNT>
134+
135+
For more details, see:
136+
* [Set up alerts](/docs/alerts/create-alert/create-alert-condition/alert-conditions)
137+
* [Build custom dashboards](/docs/query-your-data/explore-query-data/dashboards/introduction-dashboards)
138+
* [Query metric data](/docs/data-apis/understand-data/metric-data/query-metric-data-type)
139+
140+
## Get help [#get-help]
141+
142+
For troubleshooting and support:
143+
144+
* **Community support**: Visit the [New Relic Community](https://discuss.newrelic.com/) for community help and discussions
145+
* **Apache Kafka documentation**: See [Apache Kafka official docs](https://kafka.apache.org/documentation/) for Kafka-specific issues
146+
* **OpenTelemetry Collector**: Check [OpenTelemetry Collector documentation](https://opentelemetry.io/docs/collector/) for OTel-specific configuration
147+
* **New Relic Support**: Contact [New Relic Support](https://support.newrelic.com/) for account and integration issues

src/content/docs/infrastructure/host-integrations/host-integrations-list/kafka/kafka-config.mdx renamed to src/content/docs/infrastructure/host-integrations/host-integrations-list/kafka/nri-kafka/kafka-config.mdx

File renamed without changes.

src/content/docs/infrastructure/host-integrations/host-integrations-list/kafka/kafka-integration.mdx renamed to src/content/docs/infrastructure/host-integrations/host-integrations-list/kafka/nri-kafka/kafka-integration.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ To install the Kafka monitoring integration, you must run through the following
2020
2. [Install and activate the integration](#install).
2121
3. [Configure the integration](#config).
2222
4. [Find and use data](#find-and-use).
23-
5. Optionally, see [Kafka's configuration settings](/docs/infrastructure/host-integrations/host-integrations-list/kafka/kafka-config).
23+
5. Optionally, see [Configuration reference: Advanced NRI-Kafka settings](/docs/infrastructure/host-integrations/host-integrations-list/kafka/nri-kafka/kafka-config).
2424

2525
<Callout variant="tip">
2626
To read about best practices when monitoring Kafka, check [this blogpost](https://newrelic.com/blog/best-practices/new-relic-kafkapocalypse).

0 commit comments

Comments
 (0)