一、各软件功能简介

prometheus：Prometheus(是由go语言(golang)开发)是一套开源的监控&报警&时间序列数 据库的组合。主要优点：外部依赖安装使用超简单、系统集成 多等

grafana：Grafana 是一款采用 go 语言编写的开源应用，主要用于大规模指标数据的 可视化展现，是网络架构和应用分析中最流行的时序数据展示工具，目前已经支 持绝大部分常用的时序数据库。主要优点：展示方便、数据源种类多、内置通知提醒功能

altermanager:AlterManager是一个基于开源框架Prometheus和Grafana的告警管理系统。它可以帮助我们轻松地实现监控告警功能，并支持多种告警方式。主要优点：告警方式多样

prometheus-webhook-dingtalk：prometheus-webhook-dingtalk 是一个用于将 Prometheus 告警通知发送到钉钉群组的 webhook 模块。它提供了一种与钉钉无缝集成的方式，使监控团队能够及时接收和处理告警通知，并进行有效的团队协作。

二、基础准备工作

docker安装：这个太简单了我就不介绍了，后期有时间再出一篇专门安装docker的

服务器：由于我是测试就是用了一台机器centos7 ip：10.10.30.34(后期中配置文件需要用到)

安装目录：创建目录<code>mkdir -p /data/prometheus 切换目录cd /data/prometheus/，主要用于各种软件的配置和数据文件存放

三、prometheus准备工作

3.1、prometheus配置文件和数据文件

数据文件准备：创建目录mkdir -p prometheus/data 修改权限chmod 777 prometheus/data实际应用可以根据具体使用情况设置权限

配置文件准备：vim prometheus/prometheus.yml

global:

scrape_interval: 15s # 多久收集一次数据

evaluation_interval: 15s # 多久评估一次规则

scrape_timeout: 10s # 每次收集数据的超时时间

scrape_configs: #收集数据配置列表

- job_name: prometheus # 必须配置, 自动附加的job labels, 必须唯一

static_configs:

- targets: ['10.10.30.34:9090'] # 指定prometheusip端口

labels:

instance: prometheus #标签

- job_name: ehospital-exploit-database #监控客户端

static_configs:

- targets: ['10.10.30.34:9100']

labels:

instance: eehospital-exploit-database

alerting: #Alertmanager相关的配置

alertmanagers:

- static_configs:

- targets:

- 10.10.30.34:9093 #指定告警模块

rule_files: #告警规则文件, 可以使用通配符

- "/etc/prometheus/rules/*.yml"

3.2、prometheus告警配置

创建目录：mkdir rules 用于存放告警和触发文件

通用规则：vim rules/alert-rules.yml

groups:

- name: prometheus-alert

rules:

- alert: prometheus-down

expr: prometheus:up == 0

for: 1m

labels:

severity: 'critical'

annotations:

summary: "instance: { -- -->{ $labels.instance }} 宕机了"

description: "instance: { { $labels.instance }} \n- job: { { $labels.job }} 关机了， 时间已经1分钟了。"

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-cpu-high

expr: prometheus:cpu:total:percent > 80

for: 3m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} cpu 使用率高于 { { $value }}"

description: "instance: { { $labels.instance }} \n- job: { { $labels.job }} CPU使用率已经持续一分钟高过80% 。"

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-cpu-iowait-high

expr: prometheus:cpu:iowait:percent >= 12

for: 3m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} cpu iowait 使用率高于 { { $value }}"

description: "instance: { { $labels.instance }} \n- job: { { $labels.job }} cpu iowait使用率已经持续三分钟高过12%"

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-load-load1-high

expr: (prometheus:load:load1) > (prometheus:cpu:count) * 1.2

for: 3m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} load1 使用率高于 { { $value }}"

description: ""

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-memory-high

expr: prometheus:memory:used:percent > 85

for: 3m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} memory 使用率高于 { { $value }}"

description: ""

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-disk-high

expr: prometheus:disk:used:percent > 80

for: 10m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} disk 使用率高于 { { $value }}"

description: ""

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-disk-read:count-high

expr: prometheus:disk:read:count:rate > 2000

for: 2m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} iops read 使用率高于 { { $value }}"

description: ""

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-disk-write-count-high

expr: prometheus:disk:write:count:rate > 2000

for: 2m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} iops write 使用率高于 { { $value }}"

description: ""

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-disk-read-mb-high

expr: prometheus:disk:read:mb:rate > 60

for: 2m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} 读取字节数 高于 { { $value }}"

description: ""

instance: "{ { $labels.instance }}"

value: "{ { $value }}"

- alert: prometheus-disk-write-mb-high

expr: prometheus:disk:write:mb:rate > 60

for: 2m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} 写入字节数 高于 { { $value }}"

description: ""

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-filefd-allocated-percent-high

expr: prometheus:filefd_allocated:percent > 80

for: 10m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} 打开文件描述符 高于 { { $value }}"

description: ""

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-network-netin-error-rate-high

expr: prometheus:network:netin:error:rate > 4

for: 1m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} 包进入的错误速率 高于 { { $value }}"

description: ""

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-network-netin-packet-rate-high

expr: prometheus:network:netin:packet:rate > 35000

for: 1m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} 包进入速率 高于 { { $value }}"

description: ""

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-network-netout-packet-rate-high

expr: prometheus:network:netout:packet:rate > 35000

for: 1m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} 包流出速率 高于 { { $value }}"

description: ""

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-network-tcp-total-count-high

expr: prometheus:network:tcp:total:count > 40000

for: 1m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} tcp连接数量 高于 { { $value }}"

description: ""

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-process-zoom-total-count-high

expr: prometheus:process:zoom:total:count > 10

for: 10m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} 僵死进程数量 高于 { { $value }}"

description: ""

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

- alert: prometheus-time-offset-high

expr: prometheus:time:offset > 0.03

for: 2m

labels:

severity: info

annotations:

summary: "instance: { { $labels.instance }} { { $labels.desc }} { { $value }} { { $labels.unit }}"

description: ""

value: "{ { $value }}"

instance: "{ { $labels.instance }}"

细化规则：vim rules/record-rules.yml

groups:

- name: prometheus-record

rules:

- expr: up{ job!="prometheus"}

record: prometheus:up

labels:

desc: "节点是否在线, 在线1,不在线0"

unit: " "

job: "prometheus"

- expr: time() - node_boot_time_seconds{ }

record: prometheus:node_uptime

labels:

desc: "节点的运行时间"

unit: "s"

job: "prometheus"

- expr: (1 - avg by (environment,instance) (irate(node_cpu_seconds_total{ job!="prometheus",mode="idle"}[5m]))) * 100code>

record: prometheus:cpu:total:percent

labels:

desc: "节点的cpu总消耗百分比"

unit: "%"

job: "prometheus"

- expr: (avg by (environment,instance) (irate(node_cpu_seconds_total{ -- -->job!="prometheus",mode="idle"}[5m]))) * 100code>

record: prometheus:cpu:idle:percent

labels:

desc: "节点的cpu idle百分比"

unit: "%"

job: "prometheus"

- expr: (avg by (environment,instance) (irate(node_cpu_seconds_total{ -- -->job!="prometheus",mode="iowait"}[5m]))) * 100code>

record: prometheus:cpu:iowait:percent

labels:

desc: "节点的cpu iowait百分比"

unit: "%"

job: "prometheus"

- expr: (avg by (environment,instance) (irate(node_cpu_seconds_total{ -- -->job!="prometheus",mode="system"}[5m]))) * 100code>

record: prometheus:cpu:system:percent

labels:

desc: "节点的cpu system百分比"

unit: "%"

job: "prometheus"

- expr: (avg by (environment,instance) (irate(node_cpu_seconds_total{ -- -->job!="prometheus",mode="user"}[5m]))) * 100code>

record: prometheus:cpu:user:percent

labels:

desc: "节点的cpu user百分比"

unit: "%"

job: "prometheus"

- expr: (avg by (environment,instance) (irate(node_cpu_seconds_total{ -- -->job!="prometheus",mode=~"softirq|nice|irq|steal"}[5m]))) * 100

record: prometheus:cpu:other:percent

labels:

desc: "节点的cpu 其他的百分比"

unit: "%"

job: "prometheus"

- expr: node_memory_MemTotal_bytes{ job!="prometheus"}

record: prometheus:memory:total

labels:

desc: "节点的内存总量"

unit: byte

job: "prometheus"

- expr: node_memory_MemFree_bytes{ job!="prometheus"}

record: prometheus:memory:free

labels:

desc: "节点的剩余内存量"

unit: byte

job: "prometheus"

- expr: node_memory_MemTotal_bytes{ job!="prometheus"} - node_memory_MemFree_bytes{ job!="prometheus"}

record: prometheus:memory:used

labels:

desc: "节点的已使用内存量"

unit: byte

job: "prometheus"

- expr: node_memory_MemTotal_bytes{ job!="prometheus"} - node_memory_MemAvailable_bytes{ job!="prometheus"}

record: prometheus:memory:actualused

labels:

desc: "节点用户实际使用的内存量"

unit: byte

job: "prometheus"

- expr: (1-(node_memory_MemAvailable_bytes{ job!="prometheus"} / (node_memory_MemTotal_bytes{ job!="prometheus"})))* 100

record: prometheus:memory:used:percent

labels:

desc: "节点的内存使用百分比"

unit: "%"

job: "prometheus"

- expr: ((node_memory_MemAvailable_bytes{job!="prometheus"} / (node_memory_MemTotal_bytes{job!="prometheus"})))* 100

record: prometheus:memory:free:percent

labels:

desc: "节点的内存剩余百分比"

unit: "%"

job: "prometheus"

- expr: sum by (instance) (node_load1{ job!="prometheus"})

record: prometheus:load:load1

labels:

desc: "系统1分钟负载"

unit: " "

job: "prometheus"

- expr: sum by (instance) (node_load5{ job!="prometheus"})

record: prometheus:load:load5

labels:

desc: "系统5分钟负载"

unit: " "

job: "prometheus"

- expr: sum by (instance) (node_load15{ job!="prometheus"})

record: prometheus:load:load15

labels:

desc: "系统15分钟负载"

unit: " "

job: "prometheus"

- expr: node_filesystem_size_bytes{ job!="prometheus" ,fstype=~"ext4|xfs"}

record: prometheus:disk:usage:total

labels:

desc: "节点的磁盘总量"

unit: byte

job: "prometheus"

- expr: node_filesystem_avail_bytes{ job!="prometheus",fstype=~"ext4|xfs"}

record: prometheus:disk:usage:free

labels:

desc: "节点的磁盘剩余空间"

unit: byte

job: "prometheus"

- expr: node_filesystem_size_bytes{ job!="prometheus",fstype=~"ext4|xfs"} - node_filesystem_avail_bytes{ job!="prometheus",fstype=~"ext4|xfs"}

record: prometheus:disk:usage:used

labels:

desc: "节点的磁盘使用的空间"

unit: byte

job: "prometheus"

- expr: (1 - node_filesystem_avail_bytes{ job!="prometheus",fstype=~"ext4|xfs"} / node_filesystem_size_bytes{ job!="prometheus",fstype=~"ext4|xfs"}) * 100

record: prometheus:disk:used:percent

labels:

desc: "节点的磁盘的使用百分比"

unit: "%"

job: "prometheus"

- expr: irate(node_disk_reads_completed_total{ job!="prometheus"}[1m])

record: prometheus:disk:read:count:rate

labels:

desc: "节点的磁盘读取速率"

unit: "次/秒"

job: "prometheus"

- expr: irate(node_disk_writes_completed_total{ job!="prometheus"}[1m])

record: prometheus:disk:write:count:rate

labels:

desc: "节点的磁盘写入速率"

unit: "次/秒"

job: "prometheus"

- expr: (irate(node_disk_written_bytes_total{ job!="prometheus"}[1m]))/1024/1024

record: prometheus:disk:read:mb:rate

labels:

desc: "节点的设备读取MB速率"

unit: "MB/s"

job: "prometheus"

- expr: (irate(node_disk_read_bytes_total{ job!="prometheus"}[1m]))/1024/1024

record: prometheus:disk:write:mb:rate

labels:

desc: "节点的设备写入MB速率"

unit: "MB/s"

job: "prometheus"

- expr: (1 -node_filesystem_files_free{ job!="prometheus",fstype=~"ext4|xfs"} / node_filesystem_files{ job!="prometheus",fstype=~"ext4|xfs"}) * 100

record: prometheus:filesystem:used:percent

labels:

desc: "节点的inode的剩余可用的百分比"

unit: "%"

job: "prometheus"

- expr: node_filefd_allocated{ job!="prometheus"}

record: prometheus:filefd_allocated:count

labels:

desc: "节点的文件描述符打开个数"

unit: "%"

job: "prometheus"

- expr: node_filefd_allocated{ job!="prometheus"}/node_filefd_maximum{ job!="prometheus"} * 100

record: prometheus:filefd_allocated:percent

labels:

desc: "节点的文件描述符打开百分比"

unit: "%"

job: "prometheus"

- expr: avg by (environment,instance,device) (irate(node_network_receive_bytes_total{ device=~"eth0|eth1|ens33|ens37"}[1m]))

record: prometheus:network:netin:bit:rate

labels:

desc: "节点网卡eth0每秒接收的比特数"

unit: "bit/s"

job: "prometheus"

- expr: avg by (environment,instance,device) (irate(node_network_transmit_bytes_total{ device=~"eth0|eth1|ens33|ens37"}[1m]))

record: prometheus:network:netout:bit:rate

labels:

desc: "节点网卡eth0每秒发送的比特数"

unit: "bit/s"

job: "prometheus"

- expr: avg by (environment,instance,device) (irate(node_network_receive_packets_total{ device=~"eth0|eth1|ens33|ens37"}[1m]))

record: prometheus:network:netin:packet:rate

labels:

desc: "节点网卡每秒接收的数据包个数"

unit: "个/秒"

job: "prometheus"

- expr: avg by (environment,instance,device) (irate(node_network_transmit_packets_total{ device=~"eth0|eth1|ens33|ens37"}[1m]))

record: prometheus:network:netout:packet:rate

labels:

desc: "节点网卡发送的数据包个数"

unit: "个/秒"

job: "prometheus"

- expr: avg by (environment,instance,device) (irate(node_network_receive_errs_total{ device=~"eth0|eth1|ens33|ens37"}[1m]))

record: prometheus:network:netin:error:rate

labels:

desc: "节点设备驱动器检测到的接收错误包的数量"

unit: "个/秒"

job: "prometheus"

- expr: avg by (environment,instance,device) (irate(node_network_transmit_errs_total{ device=~"eth0|eth1|ens33|ens37"}[1m]))

record: prometheus:network:netout:error:rate

labels:

desc: "节点设备驱动器检测到的发送错误包的数量"

unit: "个/秒"

job: "prometheus"

- expr: node_tcp_connection_states{ job!="prometheus", state="established"}code>

record: prometheus:network:tcp:established:count

labels:

desc: "节点当前established的个数"

unit: "个"

job: "prometheus"

- expr: node_tcp_connection_states{ -- -->job!="prometheus", state="time_wait"}code>

record: prometheus:network:tcp:timewait:count

labels:

desc: "节点timewait的连接数"

unit: "个"

job: "prometheus"

- expr: sum by (environment,instance) (node_tcp_connection_states{ -- -->job!="prometheus"})

record: prometheus:network:tcp:total:count

labels:

desc: "节点tcp连接总数"

unit: "个"

job: "prometheus"

四、grafana配置