一、监控体系架构在 LVS-DR 高可用集群之上基础架构见《LVS_DR高可用集群实战》。本篇记录监控层如何用 Ansible 自动化搭建。┌─────────────────────────────────────────┐ │ 监控机 prometheus01 (.91) │ │ Prometheus :9090 │ Grafana :3000 │ │ Alertmanager :9093 │ blackbox :9115 │ └───────────────────┬─────────────────────┘ │ 抓取 (scrape) ┌────────────┬───────────────┼──────────────┬──────────────┐ ▼ ▼ ▼ ▼ ▼ node-exporter blackbox探活 keepalived-exp blackbox探活 node-exporter 7台主机:9100 web/dns:9115 LVS×2:9165 LVS:22 (同上)监控目标与采集方式目标采集器指标7 台主机资源CPU/内存/磁盘node-exporter :9100node_cpu/memory/filesystemWeb 服务存活VIP/nginx×2blackbox http_2xxprobe_successDNS 服务存活VIP/named×2blackbox dns_udpprobe_successLVS 主机存活blackbox tcp_connect :22probe_successkeepalived 主备/脑裂keepalived-exporter :9165keepalived_vrrp_state / keepalived_up二、Ansible 控制端与 Inventory控制端workstation (.21)# /home/devops/ansible/ansible.cfg [defaults] remote_userroot inventory/home/devops/ansible/inventory host_key_checking False collections_path/home/devops/ansible/mycollections roles_pathroles:/home/devops/ansible/roles# /home/devops/ansible/inventory [web] web01 ansible_host192.168.211.81 web02 ansible_host192.168.211.82 ​ [dns] dns01 ansible_host192.168.211.61 dns02 ansible_host192.168.211.62 ​ [lvs] lb-master ansible_host192.168.211.71 lb-backup ansible_host192.168.211.72 ​ [nfs] nfs01 ansible_host192.168.211.51 ​ [prometheus] prometheus01 ansible_host192.168.211.91软件包管理本地存放避免国内下载超时/home/devops/ansible/ ├── exporter/ │ └── node_exporter-1.11.1.linux-amd64.tar.gz ├── myapp/ │ ├── prometheus-3.5.3.linux-amd64.tar.gz │ └── grafana-10.4.13-1.x86_64.rpm ├── template/ │ ├── node_exporter.service.j2 │ ├── prometheus.service.j2 │ └── prometheus.conf.j2 ├── inventory ├── ansible.cfg ├── node_exporter.yml └── prometheus.yml三、node-exporter 部署 Playbook完整记录部署到 7 台主机用statwhen实现幂等handler实现配置变更自动重启。# /home/devops/ansible/node_exporter.yml --- - name: node-exporter hosts: prometheus become: true tasks: - name: 创建监控用户 ansible.builtin.user: name: prometheus state: present ​ - name: 创建监控目录 ansible.builtin.file: path: /opt/monitor/ state: directory owner: prometheus mode: 2755 ​ - name: 检查是否已安装(目录是否存在) ansible.builtin.stat: path: /opt/monitor/node_exporter register: ne_dir ​ - name: 解压node-exporter到被控节点(仅首次) ansible.builtin.unarchive: src: /home/devops/ansible/exporter/node_exporter-1.11.1.linux-amd64.tar.gz dest: /opt/monitor/ when: not ne_dir.stat.exists ​ - name: 目录改名(幂等) ansible.builtin.command: cmd: mv /opt/monitor/node_exporter-1.11.1.linux-amd64 /opt/monitor/node_exporter creates: /opt/monitor/node_exporter ​ - name: 修改目录所有者为prometheus(递归) ansible.builtin.file: path: /opt/monitor/node_exporter owner: prometheus recurse: yes ​ - name: 使用模板创建服务文件 ansible.builtin.template: src: /home/devops/ansible/template/node_exporter.service.j2 dest: /usr/lib/systemd/system/node_exporter.service notify: reload systemd and restart node_exporter ​ - name: 启动node_exporter并设置开机启动 ansible.builtin.service: name: node_exporter.service state: started enabled: yes ​ handlers: - name: reload systemd and restart node_exporter ansible.builtin.systemd: name: node_exporter.service state: restarted daemon_reload: true模板 node_exporter.service.j2[Unit] Descriptionnode_exporter Documentationhttps://prometheus.io/ Afternetwork.target [Service] Userprometheus Groupprometheus ExecStart/opt/monitor/node_exporter/node_exporter Restarton-failure [Install] WantedBymulti-user.target幂等设计要点stat判断二进制目录是否已存在 →when: not exists控制只解压一次mv用creates保证已改名就跳过模板变更通过notify触发 handler 重启。四、Prometheus Grafana 部署 Playbook完整记录# /home/devops/ansible/prometheus.yml --- - name: prometheus hosts: prometheus01 become: true tasks: - name: 创建监控用户 ansible.builtin.user: name: prometheus state: present ​ - name: 创建监控目录 ansible.builtin.file: path: /opt/monitor/ state: directory owner: prometheus mode: 2755 ​ - name: 检查prometheus二进制是否存在(幂等) ansible.builtin.stat: path: /opt/monitor/prometheus/prometheus register: prom_bin ​ - name: 创建数据目录 ansible.builtin.file: path: /opt/monitor/prometheus/data state: directory owner: prometheus group: prometheus ​ - name: 解压prometheus到被控节点(仅首次) ansible.builtin.unarchive: src: /home/devops/ansible/myapp/prometheus-3.5.3.linux-amd64.tar.gz dest: /opt/monitor/prometheus extra_opts: [--strip-components1] # 去掉顶层目录, 直接解到目标 when: not prom_bin.stat.exists ​ - name: 修改目录权限(递归) ansible.builtin.file: path: /opt/monitor/prometheus state: directory recurse: yes owner: prometheus group: prometheus ​ - name: 创建服务文件 ansible.builtin.template: src: /home/devops/ansible/template/prometheus.service.j2 dest: /usr/lib/systemd/system/prometheus.service notify: reload systemd and restart prometheus ​ - name: 创建prometheus配置文件(变更时重启) ansible.builtin.template: src: /home/devops/ansible/template/prometheus.conf.j2 dest: /opt/monitor/prometheus/prometheus.yml owner: prometheus group: prometheus notify: reload systemd and restart prometheus ​ - name: 启动prometheus ansible.builtin.service: name: prometheus.service state: started enabled: yes ​ handlers: - name: reload systemd and restart prometheus ansible.builtin.systemd: name: prometheus.service state: restarted daemon_reload: true ​ - name: grafana hosts: prometheus01 become: true tasks: - name: 复制grafana rpm到目标节点 ansible.builtin.copy: src: /home/devops/ansible/myapp/grafana-10.4.13-1.x86_64.rpm dest: /tmp/grafana.rpm ​ - name: 安装grafana ansible.builtin.dnf: name: /tmp/grafana.rpm state: present disable_gpg_check: yes # 本地rpm跳过GPG校验 ​ - name: 启动grafana并开机自启 ansible.builtin.service: name: grafana-server state: started enabled: yesprometheus.service.j2关键ExecStart 单行别拆行[Unit] DescriptionPrometheus Server Documentationhttps://prometheus.io/docs/introduction/overview/ Afternetwork-online.target [Service] Typesimple Userprometheus Groupprometheus ExecStart/opt/monitor/prometheus/prometheus --config.file/opt/monitor/prometheus/prometheus.yml --storage.tsdb.path/opt/monitor/prometheus/data --storage.tsdb.retention.time60d --web.enable-lifecycle Restarton-failure [Install] WantedBymulti-user.target五、Prometheus 抓取配置含探活prometheus.conf.j2 初始版node-exporterglobal: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: node-exporter static_configs: - targets: [192.168.211.71:9100] labels: { instance: lb-master } - targets: [192.168.211.72:9100] labels: { instance: lb-backup } - targets: [192.168.211.81:9100] labels: { instance: web01 } - targets: [192.168.211.82:9100] labels: { instance: web02 } - targets: [192.168.211.61:9100] labels: { instance: dns01 } - targets: [192.168.211.62:9100] labels: { instance: dns02 } - targets: [192.168.211.91:9100] labels: { instance: prometheus }后续手工补充的探活 jobblackbox- job_name: blackbox-web metrics_path: /probe params: { module: [http_2xx] } static_configs: - targets: [http://192.168.211.100/, http://192.168.211.81/, http://192.168.211.82/] relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 127.0.0.1:9115 - job_name: blackbox-dns metrics_path: /probe params: { module: [dns_udp] } static_configs: - targets: [192.168.211.200:53, 192.168.211.61:53, 192.168.211.62:53] # ⚠️ 注意: dns_udp 的 target 不要带 udp:// 前缀! relabel_configs: #blackbox-exporter的固定写法 - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 127.0.0.1:9115 - job_name: blackbox-tcp metrics_path: /probe params: { module: [tcp_connect] } static_configs: - targets: [192.168.211.71:22, 192.168.211.72:22] # LVS探22不探80! relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 127.0.0.1:9115 - job_name: keepalived-exporter static_configs: - targets: [192.168.211.71:9165] labels: { instance: lb-master } - targets: [192.168.211.72:9165] labels: { instance: lb-backup }六、blackbox-exporter 模块配置# /opt/monitor/blackbox_exporter/blackbox.yml modules: http_2xx: prober: http http: preferred_ip_protocol: ip4 tcp_connect: prober: tcp dns_udp: # 需要自己加 prober: dns dns: transport_protocol: udp query_name: www.chengke.com query_type: A preferred_ip_protocol: ip4七、keepalived-exporterLVS 主机 .71/.72# 下载 rpm (cafebazaar 有预编译包) curl -L -o keepalived-exporter.rpm \ https://github.com/cafebazaar/keepalived-exporter/releases/download/v1.7.1/keepalived-exporter_1.7.1_linux_amd64.rpm rpm -ivh keepalived-exporter.rpm # rpm 不自带 systemd 服务, 手动创建 cat /usr/lib/systemd/system/keepalived-exporter.service EOF [Unit] DescriptionKeepalived Exporter Afternetwork.target keepalived.service [Service] Typesimple Userroot ExecStart/usr/bin/keepalived-exporter --ka.json --web.listen-address:9165 Restarton-failure [Install] WantedBymulti-user.target EOF systemctl enable --now keepalived-exporter关键指标指标含义keepalived_upkeepalived 进程是否运行keepalived_vrrp_state1BACKUP 2MASTERkeepalived_become_master_total成为 MASTER 次数检测切换/脑裂八、告警规则rules 目录# /opt/monitor/prometheus/rules/keepalived_alerts.yml groups: - name: keepalived-alerts rules: - alert: KeepalivedDown expr: keepalived_up 0 for: 5m labels: { severity: critical } annotations: summary: keepalived 进程挂了: {{ $labels.instance }} - alert: LVSHostDown expr: probe_success{jobblackbox-tcp} 0 for: 5m labels: { severity: critical } - alert: KeepalivedSplitBrain expr: (keepalived_vrrp_state{instancelb-master} 2) and on (iname) (keepalived_vrrp_state{instancelb-backup} 2) for: 1m labels: { severity: critical } - alert: HighMemoryUsage expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 90 for: 5m labels: { severity: critical }#内存告警和主机探活失败告警# alertmanager.yml (邮件告警) global: smtp_smarthost: smtp.163.com:465 smtp_from: 15320239818163.com smtp_auth_username: 15320239818163.com smtp_auth_password: SMTP授权码 # 163用授权码, 不是登录密码! smtp_require_tls: true route: group_by: [alertname] group_wait: 30s group_interval: 60s repeat_interval: 10m receiver: email receivers: - name: email email_configs: - to: 收件邮箱qq.com send_resolved: true九、踩坑记录亲测#坑根因解决1prometheus 203/EXECunarchivemv 顺序导致二进制没到位extra_opts: [--strip-components1]直接解到目标2prometheus.yml 解析失败scrape_interval没缩进到global:下YAML 缩进严格 2 空格3grafana 登录 500 readonly databaserpm 重装没重启服务, 旧进程还在跑systemctl restart grafana-server4grafana 下载超时官方源国内慢本地下载 rpm 再dnf install ./xxx.rpm5GPG 校验失败本地 rpm 无公钥disable_gpg_check: yes6dns_udp 探活失败target 带了udp://前缀target 写IP:端口7LVS 探活误报探 :80(Director 不监听 80)探:228blackbox 服务启动失败ExecStart 路径错 参数拆行单行全路径9keepalived-exporter 无 systemdrpm 不自带手动建 service十、最终验证与全景# 1. 所有 target 应 up curl -s localhost:9090/api/v1/targets | python3 -m json.tool | grep health # 2. keepalived 主备 (master2 backup1) curl -s localhost:9090/api/v1/query?querykeepalived_vrrp_state # 3. 规则全部 healthok curl -s localhost:9090/api/v1/rules | python3 -m json.tool # 4. 邮件告警实测: 临时调低阈值触发, 收邮件后恢复组件端口状态Prometheus9090✅Grafana3000✅ (仪表盘 11074/9965)Alertmanager9093✅ 邮件链路实测通blackbox-exporter9115✅node-exporter ×79100✅keepalived-exporter ×29165✅十一、一句话记忆分工Prometheus 采 / Grafana 看 / Alertmanager 喊 / exporter 传感器 / blackbox 探针 / Ansible 铺幂等三件套stat判断 when控制 creates防重探活probe_success 1活 0挂dns 别带 udp://LVS 探 22 不探 80rpm 升级必须重启服务告警链路rules → Prometheus 评估 → Alertmanager 路由 → 邮箱十二、后续规划MySQL MGR 高可用 HAProxy数据层高可用mysql_exporter / haproxy_exporter 纳入监控整套监控用 Ansible role 化参数化版本、规则模板