This is not a complete plug and go HOW-TO for using Prometheus for scaling a Docker Swarm, but it does contain the building blocks for how to do it.
Orchestration
Orchestration is the ability to deploy and manage systems. If we want to manage Docker, we would typically require an orchestration tool such as Kubernetes or Hashicorp Nomad.
Kubernetes has a steep learning curve, and to automatically scale services with Nomad you need an enterprise licence.
Monitoring
Prometheus
Prometheus is a time series database that is able to pull key value pairs (metrics) from systems that export the data using a web service.
Some software does this as a feature, others require additional services. If you install the prometheus-node-exporter service on Linux, you can gather a whole raft of metrics by visiting
http://localhost:9100/metrics
You then point Prometheus at the URL, and it will periodically scrape the data from the URL and process it into its time series database.
Our usage example is to monitor Nginx for the number of active connections. If it goes above 100, then we use AlertManager to trigger a message.
We scrape metrics from the nginx-node-exporter – published on port 9113, which collects the metrics from the Nginx stub_status. This location is enabled by adding the following directive into the default.conf
location = /stub_status {
stub_status;
access_log off;
}
prometheus.yml
global:
scrape_interval: 15s
scrape_timeout: 10s
scrape_protocols:
- OpenMetricsText1.0.0
- OpenMetricsText0.0.1
- PrometheusText0.0.4
evaluation_interval: 1m
rule_files:
- "rules.yml"
scrape_configs:
- job_name: nginx
static_configs:
- targets: ["192.168.121.174:9113"]
alerting:
alertmanagers:
- static_configs:
- targets: ['192.168.121.174:9093']
Under the alerting: stanza, we add in the target IP address and port (9093) for our AlertManager instance.
rules.yml
# rules.yml
groups:
- name: nginx
rules:
- alert: Nginx 100 active connections
for: 1m
expr: nginx_connections_active{job="nginx"} >= 100
labels:
severity: critical
annotations:
title: Nginx 100 active connections on {{ $labels.instance }}
description: The Nginx on instance {{ $labels.instance }} has seen >100 active connections for the past 1 minute.
AlertManager
AlertManager is a Prometheus product that can be leveraged by Prometheus to send alerts when conditions are met for specified rules about the metrics it received.
The messages it sends out can be of many types SMTP, web chat, discord, etc. and web hooks.
If we use a web hook, we can configure AlertManager with a simple config:
alertmanager.yml
global:
resolve_timeout: 5m
route:
group_by: ['alertname']
group_wait: 10s
group_interval: 5m
repeat_interval: 3h
receiver: 'webhook'
receivers:
- name: 'webhook'
webhook_configs:
- url: 'http://192.168.121.174:3000'
send_resolved: true
The receivers: stanza contains the webhook URL for our custom web service that will handle the data that is passed to it.
Python
Using python, we have a simple script that listed on port 3000 for our posted web data from the web hook call.
from flask import Flask, request, jsonify
app = Flask(__name__)
@app.route('/', methods=['POST'])
def process_webhook():
try:
alert_data = request.json
# Process the alert data here (e.g., extract labels, annotations, etc.)
# Implement your scaling logic based on the alert information
# ...
print("What we do to process the data goes here")
# Return a response (optional)
return jsonify({'message': 'Webhook received successfully'}), 200
except Exception as e:
return jsonify({'error': str(e)}), 500
if __name__ == '__main__':
app.run(host='0.0.0.0', port=3000)
The data that comes in from the AlertManager webhook is JSON, and when formatted looks like this:
When firing
{
"receiver": "webhook",
"status": "firing",
"alerts": [
{
"status": "firing",
"labels": {
"alertname": "Nginx 100 active connections",
"instance": "192.168.121.174:9113",
"job": "nginx",
"severity": "critical"
},
"annotations": {
"description": "The Nginx on instance 192.168.121.174:9113 has seen >100 active connections for the past 1 minute.",
"title": "Nginx 100 active connections on 192.168.121.174:9113"
},
"startsAt": "2024-04-26T17:21:05.311Z",
"endsAt": "0001-01-01T00:00:00Z",
"generatorURL": "http: //prometheus:9090/graph?g0.expr=nginx_connections_active%7Bjob%3D%22nginx%22%7D+%3E%3D+100&g0.tab=1",
"fingerprint": "f19572f660b24b61"
}
],
"groupLabels": {
"alertname": "Nginx 100 active connections"
},
"commonLabels": {
"alertname": "Nginx 100 active connections",
"instance": "192.168.121.174:9113",
"job": "nginx",
"severity": "critical"
},
"commonAnnotations": {
"description": "The Nginx on instance 192.168.121.174:9113 has seen >100 active connections for the past 1 minute.",
"title": "Nginx 100 active connections on 192.168.121.174:9113"
},
"externalURL": "http://alertmanager:9093",
"version": "4",
"groupKey": "{}:{alertname=\"Nginx 100 active connections\"}",
"truncatedAlerts": 0
}
When resolved
{
"receiver": "webhook",
"status": "resolved",
"alerts": [
{
"status": "resolved",
"labels": {
"alertname": "Nginx 100 active connections",
"instance": "192.168.121.174:9113",
"job": "nginx",
"severity": "critical"
},
"annotations": {
"description": "The Nginx on instance 192.168.121.174:9113 has seen >100 active connections for the past 1 minute.",
"title": "Nginx 100 active connections on 192.168.121.174:9113"
},
"startsAt": "2024-04-26T17:21:05.311Z",
"endsAt": "2024-04-26T17:23:05.311Z",
"generatorURL": "http://prometheus:9090/graph?g0.expr=nginx_connections_active%7Bjob%3D%22nginx%22%7D+%3E%3D+100&g0.tab=1",
"fingerprint": "f19572f660b24b61"
}
],
"groupLabels": {
"alertname": "Nginx 100 active connections"
},
"commonLabels": {
"alertname": "Nginx 100 active connections",
"instance": "192.168.121.174:9113",
"job": "nginx",
"severity": "critical"
},
"commonAnnotations": {
"description": "The Nginx on instance 192.168.121.174:9113 has seen >100 active connections for the past 1 minute.",
"title": "Nginx 100 active connections on 192.168.121.174:9113"
},
"externalURL": "http://alertmanager:9093",
"version": "4",
"groupKey": "{}:{alertname=\"Nginx 100 active connections\"}",
"truncatedAlerts": 0
}
We can then develop our python script to respond to the data received.
Docker
We can include a very simple method in our webhook service, the ability to manage a service in Docker.
import docker
client = docker.from_env()
service = client.services.get('helloworld')
desired_replicas = 3 # Set your desired replica count
service.scale(desired_replicas)
Stuff I'm Up To 