Skip to content

Commit 4a56589

Browse files
EZ4BRUCEEZ4BRUCE
authored
feat(): support alertmanager basic auth (update opensource and ruler chart to 1.0.4) (#20)
## What type of PR is this? * [x] Feature * [ ] BUG * [ ] Alerts * [x] Improvement * [ ] Documentation * [x] Test and CI ## Which issue(s) this PR related: issue # https://github.com/matrixorigin/MO-Cloud/issues/3817 ## What this PR does / why we need it: 1. alertmanager 添加 basic auth鉴权 2. alertmanager 支持 ha 集群部署 3. 添加基于 tilt 的本地集成测试环境部署脚本
1 parent 28a45d2 commit 4a56589

18 files changed

+737
-126
lines changed

‎README.md

Lines changed: 61 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,6 @@ GRAFANA_USER=<your-admin-user>
3636
GRAFANA_PWD=<your-grafana-pwd>
3737
MO_RULER_STACK_VERSION=<helm version>
3838
MO_OB_OPENSOURCE_VERSION=<helm version>
39-
CONTROLPLANE_RESOURCE_CHART_VERSION=<helm version>
4039
```
4140

4241
### 部署 mo-ruler-stack
@@ -136,6 +135,67 @@ kubectl get secret -n ${OBNS} grafana-admin-secret -o jsonpath="{.data['admin-u
136135
kubectl get secret -n ${OBNS} grafana-admin-secret -o jsonpath="{.data['admin-password']}" | base64 -d
137136
```
138137

138+
139+
# 进阶配置
140+
141+
142+
## alertmanger 打开 web 鉴权
143+
144+
1.在 `charts/mo-ruler-stack/values.yaml` 下设置 secretValue.alertmanager，alertmanager_web_auth_password_bcrypted 是 alertmanager_web_auth_password 的 bcrypt 加密
145+
146+
```
147+
# secret value to create secret automatically
148+
secretValue:
149+
alertmanager:
150+
# see: https://prometheus.io/docs/alerting/0.25/https
151+
alertmanager_web_auth_user: admin
152+
alertmanager_web_auth_password: admin
153+
# need to be bcrypted, in bash: htpasswd -bnBC 10 "" <alertmanager_web_auth_password> | tr -d ':\n'
154+
alertmanager_web_auth_password_bcrypted: $2y$10$Z3zgfm2IIeQqNmGWeqsrSecRuRmo/EAh4Srn0Mi0fG98dJZMn7RTS
155+
```
156+
157+
2.在 `charts/mo-ruler-stack/values.yaml` 下启用 web.config.file:
158+
```
159+
alertmanager:
160+
extraArgs:
161+
web.config.file: /tmp/alertmanager-web-config/alertmanager-web-config.yaml
162+
```
163+
164+
165+
## 开启 alertmanager 鉴权与 alertmanager ha集群模式
166+
167+
需要修改以下配置：
168+
169+
1.在 `charts/mo-ruler-stack/values.yaml` 下修改 replicaCount:
170+
```
171+
alertmanager:
172+
replicaCount: 3
173+
```
174+
175+
176+
2.在 `charts/mo-ob-opensource/values.yaml` 下修改 prometheus 的 alertingEndpoints 启用多个 alertmanager
177+
178+
```
179+
kube-prometheus-stack:
180+
prometheus:
181+
pometheusSpec:
182+
alertingEndpoints:
183+
- name: "mo-ob-alertmanager-0"
184+
- name: "mo-ob-alertmanager-1"
185+
- name: "mo-ob-alertmanager-2"
186+
```
187+
188+
3.在 `charts/mo-ob-opensource/values.yaml` 下修改 loki 的 alertmanager_url 启用多个 alertmanager
189+
190+
```
191+
loki:
192+
loki:
193+
rulerConfig:
194+
alertmanager_url: http://mo-ob-alertmanager-0.mo-ob:9093,http://mo-ob-alertmanager-1.mo-ob:9093,http://mo-ob-alertmanager-2.mo-ob:9093
195+
```
196+
197+
即可启用 alertmanager ha集群
198+
139199
# Scrape
140200

141201
[Scrape List](./docs/scrape/README.md)

‎Tiltfile

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
load('ext://helm_remote', 'helm_remote')
2+
helm_remote(
3+
'operator',
4+
repo_url='https://operator.min.io',
5+
release_name='minio-operator',
6+
namespace='minio-operator',
7+
version='6.0.2',
8+
create_namespace=True,
9+
)
10+
11+
helm_remote(
12+
'tenant',
13+
repo_url='https://operator.min.io',
14+
release_name='loki-tenant',
15+
namespace='loki-tenant',
16+
version='6.0.2',
17+
values=['./dev/loki-tenant.yaml'],
18+
create_namespace=True,
19+
)
20+
21+
# 设置 Helm Chart 的本地路径
22+
mo_ob_opensource_chart = './charts/mo-ob-opensource'
23+
mo_ruler_stack_chart = './charts/mo-ruler-stack'
24+
25+
local('kubectl get ns mo-ob || kubectl create ns mo-ob')
26+
27+
k8s_yaml(
28+
helm(
29+
mo_ruler_stack_chart,
30+
name='mo-ruler-stack',
31+
namespace='mo-ob',
32+
values=['./dev/mo-ruler-stack.dev.yaml'],
33+
)
34+
)
35+
36+
k8s_yaml(
37+
helm(
38+
mo_ob_opensource_chart,
39+
name='mo-ob-opensource',
40+
namespace='mo-ob',
41+
values=['./dev/mo-ob-opensource.dev.yaml'],
42+
)
43+
)
44+
45+
k8s_yaml("./dev/loki_test_rule.yaml")

‎charts/mo-ob-opensource/Chart.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ apiVersion: v2
22
name: mo-ob-opensource
33
description: mo-ob-opensource's Helm chart for Kubernetes
44
type: application
5-
version: 1.0.3-alpha.2
5+
version: 1.0.4
66
appVersion: 0.9.0
77
dependencies:
88
- condition: kube-prometheus-stack.enabled

‎charts/mo-ob-opensource/values.yaml

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,7 +156,17 @@ loki:
156156
type: local
157157
local:
158158
directory: /rules
159+
# alertmanager 单节点
159160
alertmanager_url: http://mo-ob-alertmanager.mo-ob:9093
161+
# alertmanager 3节点集群使用：
162+
# alertmanager_url: http://mo-ob-alertmanager-0.mo-ob:9093,http://mo-ob-alertmanager-1.mo-ob:9093,http://mo-ob-alertmanager-2.mo-ob:9093
163+
enable_alertmanager_v2: true
164+
enable_api: true
165+
alertmanager_client:
166+
type: "Basic"
167+
credentials_file: "/tmp/loki/alertmanager-loki-credentials"
168+
169+
160170
# -- Additional query scheduler config
161171
### use default config, old config not work in loki 3.0.0
162172
#storage_config:
@@ -233,6 +243,15 @@ loki:
233243
limits:
234244
memory: "2Gi"
235245
cpu: "2000m"
246+
# -- Volume mounts to add to the backend pods
247+
extraVolumeMounts:
248+
- name: alertmanager-credentials
249+
mountPath: /tmp/loki
250+
readOnly: true
251+
extraVolumes:
252+
- name: alertmanager-credentials
253+
secret:
254+
secretName: alertmanager-loki-credentials
236255

237256
# Configuration for the gateway
238257
gateway:
@@ -570,13 +589,65 @@ kube-prometheus-stack:
570589
target_label: pod
571590

572591
alertingEndpoints:
592+
# 单节点部署
573593
- name: "mo-ob-alertmanager"
574594
# 如果跨ns，需要修改
575595
namespace: "mo-ob"
576596
port: 9093
577597
scheme: http
578598
pathPrefix: ""
579599
apiVersion: v2
600+
basicAuth:
601+
username:
602+
key: username
603+
name: alertmanager-auth-secret
604+
password:
605+
key: password
606+
name: alertmanager-auth-secret
607+
#集群部署
608+
# - name: "mo-ob-alertmanager-0"
609+
# # 如果跨ns，需要修改
610+
# namespace: "mo-ob"
611+
# port: 9093
612+
# scheme: http
613+
# pathPrefix: ""
614+
# apiVersion: v2
615+
# basicAuth:
616+
# username:
617+
# key: username
618+
# name: alertmanager-auth-secret
619+
# password:
620+
# key: password
621+
# name: alertmanager-auth-secret
622+
# - name: "mo-ob-alertmanager-1"
623+
# # 如果跨ns，需要修改
624+
# namespace: "mo-ob"
625+
# port: 9093
626+
# scheme: http
627+
# pathPrefix: ""
628+
# apiVersion: v2
629+
# basicAuth:
630+
# username:
631+
# key: username
632+
# name: alertmanager-auth-secret
633+
# password:
634+
# key: password
635+
# name: alertmanager-auth-secret
636+
# - name: "mo-ob-alertmanager-2"
637+
# # 如果跨ns，需要修改
638+
# namespace: "mo-ob"
639+
# port: 9093
640+
# scheme: http
641+
# pathPrefix: ""
642+
# apiVersion: v2
643+
# basicAuth:
644+
# username:
645+
# key: username
646+
# name: alertmanager-auth-secret
647+
# password:
648+
# key: password
649+
# name: alertmanager-auth-secret
650+
580651

581652
thanosRuler:
582653
enabled: false

‎charts/mo-ruler-stack/Chart.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,13 @@ apiVersion: v2
22
name: mo-ruler-stack
33
description: mo-ruler's Helm chart for Kubernetes
44
type: application
5-
version: 1.0.3-alpha.1
5+
version: 1.0.4
66
appVersion: 0.9.0
77
dependencies:
88
- condition: alertmanager.enabled
99
name: alertmanager
1010
repository: https://prometheus-community.github.io/helm-charts
11-
version: 1.2.0
11+
version: 1.12.0
1212
- condition: grafana.enabled
1313
name: grafana
1414
repository: https://grafana.github.io/helm-charts

‎charts/mo-ruler-stack/templates/_helpers.tpl

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -277,3 +277,12 @@ global:
277277
{{- end }}
278278
{{- end }}
279279
{{- end -}}
280+
281+
{{ define "alertmanager.web" }}
282+
basic_auth_users:
283+
{{ .Values.secretValue.alertmanager.alertmanager_web_auth_user }}: {{ .Values.secretValue.alertmanager.alertmanager_web_auth_password_bcrypted }}
284+
{{ end }}
285+
286+
{{- define "alertmanager.web-digest" -}}
287+
{{ include "alertmanager.web" . | sha256sum | trunc 8 }}
288+
{{- end -}}

‎charts/mo-ruler-stack/templates/alertmanager-auth-secret.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
apiVersion: v1
2+
kind: Secret
3+
metadata:
4+
name: alertmanager-auth-secret
5+
namespace: {{ .Release.Namespace }}
6+
labels:
7+
app.kubernetes.io/name: {{ .Release.Name }}
8+
data:
9+
username: {{ .Values.secretValue.alertmanager.alertmanager_web_auth_user | b64enc }}
10+
password: {{ .Values.secretValue.alertmanager.alertmanager_web_auth_password | b64enc }}
11+
type: Opaque

‎charts/mo-ruler-stack/templates/alertmanager-datasource.yaml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,8 @@ data:
2222
# Whether or not Grafana should send alert instances to this Alertmanager
2323
handleGrafanaManagedAlerts: false
2424
# optionally
25-
# basicAuth: true
26-
# basicAuthUser: my_user
27-
# secureJsonData:
28-
# basicAuthPassword: test_password
25+
basicAuth: true
26+
basicAuthUser: {{ .Values.secretValue.alertmanager.alertmanager_web_auth_user }}
27+
secureJsonData:
28+
basicAuthPassword: {{ .Values.secretValue.alertmanager.alertmanager_web_auth_password }}
2929
{{- end -}}

‎charts/mo-ruler-stack/templates/alertmanager-loki-credentials.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
apiVersion: v1
2+
kind: Secret
3+
metadata:
4+
name: alertmanager-loki-credentials
5+
data:
6+
alertmanager-loki-credentials: {{ printf "%s:%s" .Values.secretValue.alertmanager.alertmanager_web_auth_user .Values.secretValue.alertmanager.alertmanager_web_auth_password | b64enc | b64enc }}
7+
type: Opaque

‎charts/mo-ruler-stack/templates/alertmanager-web-config.yaml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
apiVersion: v1
2+
kind: Secret
3+
metadata:
4+
name: alertmanager-web-config
5+
namespace: {{ .Release.Namespace }}
6+
labels:
7+
app.kubernetes.io/name: {{ .Release.Name }}
8+
data:
9+
alertmanager-web-config.yaml: {{ include "alertmanager.web" . | b64enc }}

0 commit comments

Comments
 (0)