-
Notifications
You must be signed in to change notification settings - Fork 0
mo ob deploy doc v0.7.0
xzxiong edited this page Apr 20, 2023
·
6 revisions
-
Kubernetes 1.18 or later;
本文档的安装环境是一个正常运行的k8s集群并装有MO-Opeatror，我们需要安装以下组件：
- 一套专门用于可观测的分布式 MO
- MO-Agent Helm Chart ( mo agent, prometheus agent, fluentbit, node exporter, cadvisor included )
- MO-Ruler Helm Chart ( mo ruler, grafana, alertmanager included )
首先，创建用于可观测的命名空间 mo-ob
$ kubectl create ns mo-ob由于我们已经预先安装了 MO-Operator 我们可以直接创建一个k8s的资源文件并apply即可安装
$ curl https://raw.githubusercontent.com/matrixorigin/matrixone-operator/main/examples/mo-cluster.yaml > mo.yaml根据需要，调整集群的配置)，并安装
$ vim ./mo.yaml
# install
$ kubectl -n mo-ob apply -f ./mo.yaml
# uninstall
$ kubectl -n mo-ob delete matrixonecluster mo推荐的配置在附录中，另外需要确认的是是否有在命名空间中配置aws s3的密钥（本地则是minio），s3默认的存储桶是 mo-ob/ob-data
kubectl -n mo-ob create secret generic aws --from-literal=AWS_REGION=us-west-2 --from-literal=AWS_ACCESS_KEY_ID=<ID> --from-literal=AWS_SECRET_ACCESS_KEY=<KEY>需要稍微等待一段时间，输入以下命令验证，见到 PHASE 为 Ready 说明 MO-Cluster 已经安装成功
$ kubectl -n mo-ob get matrixonecluster
NAME LOG DN TP AP VERSION PHASE AGE
mo 3 1 2 nightly-2ba279bb Ready 102s开放端口用于mysql连接(测试用，可跳过）
$ nohup kubectl -n mo-ob port-forward svc/mo-tp-cn 6001:6001 &
$ mysql -h 127.0.0.1 -P6001 -udump -p111- 获取chart包
$ git clone [email protected]:matrixone-cloud/observability.git
$ cd observability/charts/mo-agent-stack- 目录结构
mo-agent-stack
├── Chart.lock
├── Chart.yaml
├── charts
├── mo.yaml # 配置管理
├── templates
└── values.yaml
安装MO-Agent之前只需要在chart包目录下的mo.yaml
- 指定好写入的DB相关的 hostname，password 即可：
moAgent:
agentSpec:
mo:
- host: mo-tp-cn.<OB_NAMESPACE>
- port: 6001
- user: dump
- password: 111
+ host: host
+ port: 6001
+ user: user
+ password: pwd
connectTimeout: 15s
maxOpens: 5
maxIdles: -1
initPingCnt: 60默认配置是以命名空间为mo-ob前提的，如果使用别的命名空间，还需要修改相应配置项：
- fluent-bit写入mo-agent的http output信息：
fluent-bit:
config:
outputs: |
[OUTPUT]
Name http
Match *
- Host mo-agent-service.mo-ob
+ Host mo-agent-service.<OTHER_NAMESPACE>
Port 8000
URI /v1/logs
Format json- prometheus-agent写入mo-agent的 remote write信息：
prometheus:
remoteWrite:
- - url: "http://mo-agent-service.<OB_NAMESPACE>:8000/v1/metrics"
+ - url: "http://mo-agent-service.mo-ob:8000/v1/metrics"- (可选) 修改镜像
moAgent:
...
image:
containerName: mo-agent
pullPolicy: IfNotPresent
- repository: matrixorigin/observability
- tag: mo-agent-0.7.0
+ repository: your_hub/observability
+ tag: your_image_tag# install
$ helm install mo-agent-stack ./charts/mo-agent-stack -n mo-ob -f ./charts/mo-agent-stack/mo.yaml
# uninstall
$ helm uninstall mo-agent-stack -n mo-ob
# upgrade
$ helm upgrade mo-agent-stack ./charts/mo-agent-stack -n mo-ob -f ./charts/mo-agent-stack/mo.yaml注：如果只需要采集日志/指标，可以将mo.yaml里的 promtheus-enabled / fluentbit-enabled 设为 false
- 获取chart包
$ git clone [email protected]:matrixone-cloud/observability.git
$ cd observability/charts/mo-ruler-stack- 目录结构
mo-ruler-stack
├── Chart.lock
├── Chart.yaml
├── charts
├── grafana # grafana dashboard 配置
├── mo.yaml # MO 配置
├── templates
└── values.yaml
安装MO-Ruler 之前需要修改 chart包目录下的 mo.yaml 配置
- 指定好写入的DB相关的hostname，password：
moRuler:
...
rulerSpec:
alerting:
alertingEndpoint:
scheme: http
targets:
- - "mo-ruler-stack-alertmanager.<OB_NAMESPACE>:9093"
+ - "mo-ruler-stack-alertmanager.mo-ob:9093"
...
ruler:
...
mo:
- host: mo-tp-cn.<OB_NAMESPACE>
- port: 6001
- user: dump
- password: 111
+ host: host
+ port: 6001
+ user: user
+ password: pwd
connectTimeout: 15s
maxOpens: 5
maxIdles: -1
initPingCnt: 60
- 以及告警相关的 alertmanager配置，为此需要创建一个secret存储smtp密码：
kubectl -n mo-ob create secret generic alertmanager-secret --from-literal=smtp-password=<password>- 还需要填入告警邮件接受者邮箱以及smtp信息，也可以自行修改相应的mount信息（以下为默认值）：
alertmanager:
config:
global:
smtp_from: '[email protected]'
smtp_hello: 'smtp.exmail.qq.com:465'
smtp_smarthost: 'smtp.exmail.qq.com:465'
smtp_auth_username: '[email protected]'
smtp_auth_password_file: '/tmp/smtp-password/smtp-password'
smtp_require_tls: false
receivers:
- name: default-receiver
email_configs:
- - to: '[email protected]'
+ - to: 'email_receivce_addr'
# send_resolved: true
extraSecretMounts:
- name: smtp-files
mountPath: /tmp/smtp-password
subPath: ""
secretName: alertmanager-secret
readOnly: true- （可选）修改镜像
moRuler:
containerName: mo-ruler
replicaCount: 1
image:
- repository: matrixorigin/observability
- tag: mo-ruler-0.7.0
+ repository: your_hub/observability
+ tag: your_image_tag
pullPolicy: Always
...- 可选配置：为grafana添加数据源（也可以部署以后在UI页面天添加）
TODO：grafana的额外Auth方式
grafana:
adminUser: admin
adminPassword: admin
datasources:
datasources.yaml:
apiVersion: 1
datasources:
- name: MO-OB-QA-Control-Plane-Metrics
type: prometheus
url: http://mo-ruler-service.mo-ob:9199
access: proxy
- name: MO-OB-QA-Unit-Metrics
type: prometheus
url: http://aws-ap-southeast-1-mo-ruler-service.cos:9199
access: proxy
- name: MO-OB-QA-Control-Plane-Logs
type: mysql
# no http://
url: host:3306
user: mo_ob
jsonData:
database: mo_ob_logs
secureJsonData:
password: pwd可选配置：如果需要添加dashboard，将json文件拷贝至 charts/mo-ruler-stack/grafana/dashboards/matrixone-cloud 下即可
可选配置：如果需要增加告警规则，使用prom规则语法将告警规则文件放在 charts/mo-ruler-stack/rules 下即可
# install
$ helm install mo-ruler-stack ./charts/mo-ruler-stack -n mo-ob -f ./charts/mo-ruler-stack/mo.yaml
# uninstall
$ helm uninstall mo-ruler-stack -n mo-ob
# upgrade
$ helm upgrade mo-ruler-stack ./charts/mo-ruler-stack -n mo-ob -f ./charts/mo-ruler-stack/mo.yaml