Blackbox_exporter
blackbox_exporter is a black box monitoring tool used by Prometheus to monitor http/https, tcp, icmp, and dns.
What is black box surveillance? The following introduction is copied from zhangguanzhang’s prometheus black box monitoring
Conventional exporters are installed together with the machine that needs to be monitored. If you need to monitor some TCP ports and the status of the seven-layer application layer, you need black box monitoring at this time. You don’t need to install it on the target machine to start from External monitoring.
Installation
Binary installation
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.16.0/blackbox_exporter-0.16.0.linux-amd64.tar.gz tar zxvf blackbox_exporter-0.16.0.linux-amd64.tar.gz cd blackbox_exporter-0.16.0.linux-amd64 ./blackbox_exporter <flags>
The useful parameters of blackbox_exporter are probably as follows:
# ./blackbox_exporter --help
usage: blackbox_exporter [<flags>]
Flags:
-h, --help Show context-sensitive help (also try --help-long and --help-man).
--config.file="blackbox.yml"
Blackbox exporter configuration file.
--web.listen-address=":9115"
The address to listen on for HTTP requests.
--log.level=info Only log messages with the given severity or above. One of: [debug, info, warn, error]
start up
# The default port is 9115 nohup ./blackbox_exporter --config.file="blackbox.yml" & amp;
docker installation
Since all Prometheus components use UTC as the default time zone, it is recommended to use Docker to run it.
# If you do not need to enable debug, please remove the last --log.level=debug docker run --rm -d -p 9115:9115 --name blackbox_exporter -v /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime:ro -v /data/prometheus/blackbox_exporter/blackbox.yml:/config /blackbox.yml prom/blackbox-exporter:master --config.file=/config/blackbox.yml --log.level=debug
blackbox.yml
# Official default configuration file
modules:
http_2xx:
prober: http
http_post_2xx:
prober: http
http:
method: POST
tcp_connect:
prober: tcp
pop3s_banner:
prober: tcp
tcp:
query_response:
- expect: "^ + OK"
tls: true
tls_config:
insecure_skip_verify: false
ssh_banner:
prober: tcp
tcp:
query_response:
- expect: "^SSH-2.0-"
irc_banner:
prober: tcp
tcp:
query_response:
- send: "NICK prober"
- send: "USER prober prober prober :prober"
- expect: "PING :([^ ] + )"
send: "PONG ${1}"
- expect: "^:[^ ] + 001"
icmp:
prober: icmp
Configuration
prometheus.yml
HTTP configuration
scrape_configs:
- job_name: 'blackbox'
metrics_path: /probe
params:
module: [http_2xx] # The module corresponds to blackbox.yml
static_configs:
- targets:
- http://baidu.com # http
- https://baidu.com #https
- http://xx.com:8080 # Domain name of port 8080
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 127.0.0.1:9115 # Which machine is blackbox installed on?
TCP configuration
- job_name: blackbox_tcp
metrics_path: /probe
params:
module: [tcp_connect]
static_configs:
- targets:
- 192.168.1.2:280
- 192.168.1.2:7013
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 192.168.1.99:9115 # Blackbox exporter.
Restart Prometheus, open targets, and you can see it.
If you want to see what indicators there are, click the URL below Endpoint: 
The following items are worthy of attention in HTTP:
# Returned http status code probe_http_status_code 200 # https certificate expiration time, unixtime is used by default probe_ssl_earliest_cert_expiry 1.637745419e + 09 # If the detection is successful, it is 1, otherwise it is 0 probe_success 1
Alarm Rules
# Choose one of the following two
groups:
- name: http
rules:
- alert: xxx domain name resolution failed
expr: probe_success{instance="https://xx.com"} == 0
for: 1m
labels:
severity: "error"
annotations:
summary: "xxx domain name resolution failed"
- alert: xxx domain name resolution failed
expr: probe_http_status_code{instance="https://xx.com"} != 200
for: 5m
labels:
severity: "error"
annotations:
summary: "xxx domain name resolution failed"
Custom module
Sometimes some URLs may need to have parameters, such as header, body, etc., so you need to customize a module, official example.
Edit blackbox.yml
http_2xx_wxjj:
prober: http
timeout: 5s
http:
method: GET
headers:
Cookie: JSESSIONID=C123455dfdf
sid: 41c912344555-24rkjkffd
appid: 1221kj2h1k3hjk13hk
body: '{}'
Edit Prometheus.yml
- job_name: 'blackbox_wxjl'
metrics_path: /probe
params:
module: [http_2xx_wxjj] # Look for a HTTP 200 response.
static_configs:
- targets:
- http://192.168.201.173:808/byxxxxx/41234456661f-4357c9?head=APP_GeList &user=黄?
# Target to probe with http.
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 172.18.11.154:9115 # The blackbox exporter's real hostname:port.
Turn on debug
When you think your settings are correct, but the http status code returned is incorrect, and you want to debug it, you need to turn on debug.
- Specify –log.level=debug when starting
- Targets are followed by & debug=true, i.e. http://172.18.11.154:9115/probe?module=http_2xx_wxjj & target=http://192.168.201.173:808/byxxxxx/41234456661f-4357c9?head= APP_GeList &user=黄 &debug=true
Targets turning on debug will output more information than normal links.
Module configuration:
prober: http
timeout: 5s
http:
ip_protocol_fallback: true
method: GET
headers:
Cookie: JSESSIONID=C123455dfdf
appid: 41c912344555-24rkjkffd
sid: 1221kj2h1k3hjk13hk
body: '{}'
tcp:
ip_protocol_fallback: true
icmp:
ip_protocol_fallback: true
dns:
ip_protocol_fallback: true
FAQ
How many blackbox_exporters need to be installed?
Theoretically, you only need to install one. In a special network environment, such as a government cloud, when connecting to a third party, only a few machines are connected to the network. Then you need to deploy blackbox_exporter on one of those machines, and at the same time Fill in the corresponding ip for replacement in Prometheus.yml