Skip to content

Commit 8edf88f

Browse files
Milvus-doc-botMilvus-doc-bot
authored andcommitted
Release new docs to master
1 parent d4aec87 commit 8edf88f

File tree

3 files changed

+96
-14
lines changed

3 files changed

+96
-14
lines changed
893 KB
Loading
976 KB
Loading

v2.6.x/site/en/adminGuide/monitor/configure_grafana_loki.md

Lines changed: 96 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,36 @@ This guide provides instructions on how to configure Loki to collect logs and Gr
1010

1111
In this guide, you will learn how to:
1212

13-
- Deploy [Loki](https://grafana.com/docs/loki/latest/get-started/overview/) and [Promtail](https://grafana.com/docs/loki/latest/send-data/promtail/) on a Milvus cluster using Helm.
13+
- Deploy [Loki](https://grafana.com/docs/loki/latest/get-started/overview/) and [Alloy](https://grafana.com/docs/alloy/latest/) on a Milvus cluster using Helm.
1414
- Configure object storage for Loki.
1515
- Query logs using Grafana.
1616

17+
For reference, [Promtail](https://grafana.com/docs/loki/latest/send-data/promtail/#promtail-agent) will be deprecated.
18+
So we instead introduce Alloy, which has been officially suggested by Grafana Labs as the new agent.
19+
20+
# Introduction
21+
22+
Before diving into how to build a logging system with Milvus, we’d like to first introduce the mechanisms of the logging system being used.
23+
Broadly speaking, there are two main structures you can apply.
24+
Please note that the mechanism to be introduced can be applied regardless of whether the [log functionality](https://milvus.io/docs/configure_log.md) in Milvus is enabled.
25+
26+
## 1. Using host volumes of kubernetes worker node
27+
28+
kubernetes worker nodes periodically write stream logs generated from pods scheduled on those nodes to a specific path in the node’s file system as files with a `.log` extension, we will leverage this feature.
29+
Next, we will deploy Alloy, which acts as an agent, as a DaemonSet on the worker nodes.
30+
This Alloy will share the path where the log files are stored on the worker nodes via a host volume.
31+
As a result, the log files from the Milvus pods will be visible inside the Alloy pod, and Alloy will read these files and send them to Loki.
32+
33+
![Logging with k8s worker node host volume](../../../../assets/monitoring/logging_HostVolume.png "logging with host volume sharing.")
34+
35+
## 2. Using kubernetes API server
36+
37+
kubernetes API server is one of the control plane components. Alloy doesn't necessarily need to be deployed as a DaemonSet. It works well as a Deployment.
38+
Instead, Alloy must request to kubernetes API server for fetching stream logs of milvus pods and get them.
39+
Finally, Alloy will send the stream logs to Loki.
40+
41+
![Logging with k8s API Server](../../../../assets/monitoring/logging_K8sApi.png "logging with k8s api server.")
42+
1743
## Prerequisites
1844

1945
- You have [installed a Milvus cluster on K8s](install_cluster-helm.md).
@@ -79,26 +105,82 @@ kubectl create ns loki
79105
helm install --values loki.yaml loki grafana/loki -n loki
80106
```
81107

82-
## Deploy Promtail
83-
84-
Promtail is a log collection agent for Loki. It reads logs from Milvus pods and sends them to Loki.
108+
## Deploy Alloy
85109

86-
### 1. Create Promtail Configuration
87-
88-
Create a `promtail.yaml` configuration file:
110+
You can configure alloy and deploy alloy based on Helm chart. Refer to the official Alloy [documentation](https://grafana.com/docs/alloy/latest/set-up/install/) for more installation options.
111+
We will show you Alloy [configuration](https://grafana.com/docs/alloy/latest/configure/).
89112

113+
### Create Alloy Configuration
114+
#### 1. Using host volumes of kubernetes worker node
115+
`alloy.yaml`:
90116
```yaml
91-
config:
92-
clients:
93-
- url: http://loki-gateway/loki/api/v1/push
117+
alloy:
118+
enableReporting: false
119+
resources: {}
120+
configMap:
121+
create: true
122+
content: |-
123+
loki.write "remote_loki" {
124+
endpoint {
125+
url = "http://loki-gateway/loki/api/v1/push"
126+
}
127+
}
128+
129+
loki.source.file "milvus_logs" {
130+
targets = local.file_match.milvus_log_files.targets
131+
forward_to = [loki.write.remote_loki.receiver]
132+
}
133+
134+
local.file_match "milvus_log_files" {
135+
path_targets = [
136+
{"__path__" = "/your/worker/node/var/log/pods/milvus_milvus-*/**/*.log"},
137+
]
138+
}
139+
# mount to pods with host volume
140+
mounts:
141+
extra:
142+
- name: log-pods
143+
mountPath: /host/var/log/pods
144+
readOnly: true
145+
controller:
146+
type: 'daemonset'
147+
# make volume that use host volume in worker node
148+
volumes:
149+
extra:
150+
- name: log-pods
151+
hostPath:
152+
path: /var/log/pods
94153
```
95154

96-
### 2. Install Promtail
155+
#### 2. Using kubernetes API server
156+
`alloy.yaml`:
157+
```yaml
158+
alloy:
159+
enableReporting: false
160+
resources: {}
161+
configMap:
162+
create: true
163+
content: |-
164+
loki.write "remote_loki" {
165+
endpoint {
166+
url = "http://loki-gateway/loki/api/v1/push"
167+
}
168+
}
169+
170+
discovery.kubernetes "milvus_pod" {
171+
role = "pod"
172+
}
173+
174+
loki.source.kubernetes "milvus_pod_logs" {
175+
targets = discovery.kubernetes.milvus_pod.output
176+
forward_to = [loki.write.remote_loki.receiver]
177+
}
178+
```
97179

98-
Install Promtail using Helm:
180+
### Install Alloy
99181

100182
```shell
101-
helm install --values promtail.yaml promtail grafana/promtail -n loki
183+
helm install --values alloy.yaml alloy grafana/alloy -n loki
102184
```
103185

104186
## Query Logs with Grafana
@@ -146,4 +228,4 @@ After adding Loki as a data source, query Milvus logs in Grafana:
146228
2. In the upper-left corner of the page, choose the loki data source.
147229
3. Use __Label browser__ to select labels and query logs.
148230

149-
![Query](../../../../assets/milvuslog.jpg "Query Milvus logs in Grafana.")
231+
![Query](../../../../assets/milvuslog.jpg "Query Milvus logs in Grafana.")

0 commit comments

Comments
 (0)