ELK钉钉机器人告警

elk 服务器安装钉钉插件

创建个文件夹存放插件

mkdir -p /data/prometheus/dingtalk

下载prometheus-webhook-dingtalk

wget https://github.com/timonwong/prometheus-webhook-dingtalk/releases/download/v1.4.0/prometheus-webhook-dingtalk-1.4.0.linux-amd64.tar.gz

解压

tar -vxzf prometheus-webhook-dingtalk-1.4.0.linux-amd64.tar.gz

移动dingtalk下

mv prometheus-webhook-dingtalk-1.4.0.linux-amd64 /usr/local/prometheus-webhook-dingtalk

配置服务

进入system下

cd /usr/lib/systemd/system

创建服务文件

vim prometheus-webhook-dingtalk.service

添加下面内容

[Unit]
Description=https://github.com/timonwong/prometheus-webhook-dingtalk/releases/
After=network-online.target
[Service]
Restart=on-failure
ExecStart=/usr/local/prometheus-webhook-dingtalk/prometheus-webhook-dingtalk --config.file=/usr/local/prometheus-webhook-dingtalk/config.example.yml
 
[Install]
WantedBy=multi-user.target

重新加载服务配置

systemctl daemon-reload

启动服务

systemctl start prometheus-webhook-dingtalk.service

查看服务状态

systemctl status prometheus-webhook-dingtalk

设置开机自启

systemctl enable prometheus-webhook-dingtalk

钉钉添加机器人

取个名字，选择安全设置，这里我选择IP地址

不知道填什么IP地址？先随便填个保存，会生成个Webhook地址，复制下来

回到 ELK 服务器输入下面测试命令
把后面的地址换成上面钉钉机器人生成的地址

curl -H "Content-Type: application/json" -d '{"msgtype":"text","text":{"content":"hello，我是告警机器人！！！"}}' https://oapi.dingtalk.com/robot/send?access_token=ff488d860cc615bd8d56aa967bb5eabf64564c3d76451819e78c7387e0198917

会看见报错了，然后将提示的ip添加到钉钉机器人里面

再执行上面的测试命令，可以看见机器人成功了

ELK Watcher告警设置

创建个Watcher

Watch JSON配置如下

每分钟执行一次

size:1,保存一条查到的数据

查询过去一分钟里日志带ERROR的记录大于等于1条，就钉钉机器人告警

并输出报错日志的_index、source、traceId、spanId、tpye字段的信息

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "easyspeed-cloud-logs-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 1,
          "query": {
            "bool": {
              "must": [
                {
                  "match": {
                    "level": "ERROR"
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-1m",
                      "lte": "now"
                    }
                  }
                }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 1
      }
    }
  },
  "actions": {
    "Watcher的ID": {
      "webhook": {
        "scheme": "https",
        "host": "oapi.dingtalk.com",
        "port": 443,
        "method": "post",
        "path": "/robot/send",
        "params": {
          "access_token": "钉钉生成的"
        },
        "headers": {},
        "body": """{"msgtype":"text","text":{"content":"TEST发现ERROR，错误信息如下：
        _index：{{ctx.payload.hits.hits.0._index}}，
        source：{{ctx.payload.hits.hits.0._source.source}}, 
        traceId：{{ctx.payload.hits.hits.0._source.traceId}} ,
        spanId：{{ctx.payload.hits.hits.0._source.spanId}}, 
        type：{{ctx.payload.hits.hits.0._source.type}}
        详情信息请登录查看：http://logs.hkeasyspeed.com/ "}}"""
      }
    }
  }
}