|
| 1 | +--- |
| 2 | +title: Kafka monitoring overview |
| 3 | +tags: |
| 4 | + - Integrations |
| 5 | + - On-host integrations |
| 6 | + - On-host integrations list |
| 7 | +metaDescription: "Choose the right Kafka monitoring approach: New Relic Native agent, OpenTelemetry Collector, AWS MSK, or Confluent Cloud." |
| 8 | +freshnessValidatedDate: never |
| 9 | +--- |
| 10 | + |
| 11 | +Monitor your Apache Kafka clusters, brokers, producers, consumers, and topics with New Relic. We offer multiple approaches to instrument and collect Kafka metrics based on your environment and preferences. |
| 12 | + |
| 13 | +## Choose your monitoring approach [#choose-approach] |
| 14 | + |
| 15 | +Select the monitoring solution that best fits your Kafka deployment: |
| 16 | + |
| 17 | +<table> |
| 18 | + <thead> |
| 19 | + <tr> |
| 20 | + <th style={{ width: "200px" }}> |
| 21 | + Integration Type |
| 22 | + </th> |
| 23 | + <th> |
| 24 | + Best For |
| 25 | + </th> |
| 26 | + <th> |
| 27 | + Key Features |
| 28 | + </th> |
| 29 | + </tr> |
| 30 | + </thead> |
| 31 | + |
| 32 | + <tbody> |
| 33 | + <tr> |
| 34 | + <td> |
| 35 | + **[NRI-Kafka](/docs/infrastructure/host-integrations/host-integrations-list/kafka/nri-kafka/kafka-integration)**<br/> |
| 36 | + (New Relic Native) |
| 37 | + </td> |
| 38 | + <td> |
| 39 | + Traditional on-host deployments for existing New Relic Infrastructure users or new users |
| 40 | + </td> |
| 41 | + <td> |
| 42 | + * Native New Relic agent |
| 43 | + * Automatic broker discovery (Zookeeper/Bootstrap) |
| 44 | + * Out-of-the-box dashboards |
| 45 | + </td> |
| 46 | + </tr> |
| 47 | + |
| 48 | + <tr> |
| 49 | + <td> |
| 50 | + **[AWS MSK Integration](/docs/message-queues-streaming/installation/aws-managed-kafka-msk-integration)**<br/> |
| 51 | + (New Relic Native) |
| 52 | + </td> |
| 53 | + <td> |
| 54 | + Amazon Managed Streaming for Apache Kafka (MSK) users |
| 55 | + </td> |
| 56 | + <td> |
| 57 | + * CloudWatch-based metrics collection |
| 58 | + * AWS MSK-specific metrics |
| 59 | + * Managed service integration |
| 60 | + * No agent installation required |
| 61 | + </td> |
| 62 | + </tr> |
| 63 | + |
| 64 | + <tr> |
| 65 | + <td> |
| 66 | + **[Confluent Cloud Integration](/docs/message-queues-streaming/installation/confluent-cloud-integration)**<br/> |
| 67 | + (New Relic Native) |
| 68 | + </td> |
| 69 | + <td> |
| 70 | + Confluent Cloud managed Kafka users |
| 71 | + </td> |
| 72 | + <td> |
| 73 | + * API-based metrics collection |
| 74 | + * Confluent Cloud-specific metrics |
| 75 | + * Fully managed integration |
| 76 | + * No infrastructure access needed |
| 77 | + </td> |
| 78 | + </tr> |
| 79 | + |
| 80 | + <tr> |
| 81 | + <td> |
| 82 | + **[OpenTelemetry Kafka](/docs/opentelemetry/integrations/kafka/overview)**<br/> |
| 83 | + (OTel Collector) |
| 84 | + </td> |
| 85 | + <td> |
| 86 | + Cloud-native environments, Kubernetes, multi-vendor observability |
| 87 | + </td> |
| 88 | + <td> |
| 89 | + * Vendor-neutral telemetry collection |
| 90 | + * Flexible deployment (host, Kubernetes, sidecar) |
| 91 | + * Extensive metric collection via JMX |
| 92 | + * Support for self-hosted and Kubernetes (Strimzi) |
| 93 | + </td> |
| 94 | + </tr> |
| 95 | + </tbody> |
| 96 | +</table> |
| 97 | + |
| 98 | +## Platform-specific guides [#platform-guides] |
| 99 | + |
| 100 | +### New Relic Native Kafka integration |
| 101 | + |
| 102 | +* **[NRI-Kafka installation](/docs/infrastructure/host-integrations/host-integrations-list/kafka/nri-kafka/kafka-integration)**: Traditional on-host integration |
| 103 | +* **[AWS MSK integration](/docs/message-queues-streaming/installation/aws-managed-kafka-msk-integration)**: Monitor Amazon MSK clusters |
| 104 | +* **[Confluent Cloud integration](/docs/message-queues-streaming/installation/confluent-cloud-integration)**: Monitor Confluent Cloud Kafka |
| 105 | + |
| 106 | +### OpenTelemetry Kafka monitoring |
| 107 | + |
| 108 | +OpenTelemetry provides flexible instrumentation for various Kafka deployments: |
| 109 | + |
| 110 | +* **[Self-hosted Kafka](/docs/opentelemetry/integrations/kafka/self-hosted)**: Install on Linux hosts with direct Kafka access |
| 111 | +* **[Kubernetes with Strimzi](/docs/opentelemetry/integrations/kafka/kubernetes-strimzi)**: Deploy on Kubernetes using kubectl |
| 112 | + |
| 113 | +## What's collected? [#whats-collected] |
| 114 | + |
| 115 | +All Kafka monitoring approaches collect key metrics including: |
| 116 | + |
| 117 | +* **Cluster metrics**: Topic count, partition distribution, controller status |
| 118 | +* **Broker metrics**: Request rates, byte rates, replica lag, under-replicated partitions |
| 119 | +* **Topic metrics**: Messages per second, bytes in/out per topic |
| 120 | +* **Consumer group metrics**: Lag, commit rates, group membership |
| 121 | + |
| 122 | +For detailed metric definitions: |
| 123 | +* NRI-Kafka metrics (documented in the [NRI-Kafka integration page](/docs/infrastructure/host-integrations/host-integrations-list/kafka/nri-kafka/kafka-integration#metrics)) |
| 124 | +* [OpenTelemetry metrics reference](/docs/opentelemetry/integrations/kafka/metrics-reference) |
| 125 | + |
| 126 | +## Next steps [#next-steps] |
| 127 | + |
| 128 | +After setting up Kafka monitoring: |
| 129 | + |
| 130 | +* **Explore your data**: View Kafka metrics in multiple ways: |
| 131 | + * Navigate to <DNT>**[one.newrelic.com](https://one.newrelic.com) > All Capabilities > Queues & Streams**</DNT> |
| 132 | + * Check All entities for a list of Kafka entities that were created |
| 133 | + * Go to <DNT>**[one.newrelic.com](https://one.newrelic.com) > All Capabilities > Infrastructure > Third-party services**</DNT> |
| 134 | + |
| 135 | +For more details, see: |
| 136 | +* [Set up alerts](/docs/alerts/create-alert/create-alert-condition/alert-conditions) |
| 137 | +* [Build custom dashboards](/docs/query-your-data/explore-query-data/dashboards/introduction-dashboards) |
| 138 | +* [Query metric data](/docs/data-apis/understand-data/metric-data/query-metric-data-type) |
| 139 | + |
| 140 | +## Get help [#get-help] |
| 141 | + |
| 142 | +For troubleshooting and support: |
| 143 | + |
| 144 | +* **Community support**: Visit the [New Relic Community](https://discuss.newrelic.com/) for community help and discussions |
| 145 | +* **Apache Kafka documentation**: See [Apache Kafka official docs](https://kafka.apache.org/documentation/) for Kafka-specific issues |
| 146 | +* **OpenTelemetry Collector**: Check [OpenTelemetry Collector documentation](https://opentelemetry.io/docs/collector/) for OTel-specific configuration |
| 147 | +* **New Relic Support**: Contact [New Relic Support](https://support.newrelic.com/) for account and integration issues |
0 commit comments