Monitoring

Cetacean runs without Prometheus. Without it you lose CPU and memory charts, resource gauges, capacity bars, per-task sparklines, stack drill-downs, and every sizing recommendation.

Component	Provides	Unlocks
Prometheus	Stores and queries metrics	Anything metrics-related at all
node-exporter	Host metrics (CPU, memory, disk)	Node gauges, cluster capacity bars
cAdvisor	Container metrics	Per-service charts, per-task sparklines, stack drill-downs

Setup

Deploy the bundled monitoring stack first. It reads prometheus.yml from the directory you deploy it from, and creates the monitoring overlay network Cetacean joins:

curl -O https://cetacean.mazetti.me/dist/compose.monitoring.yaml
curl -O https://cetacean.mazetti.me/dist/prometheus.yml
docker stack deploy -c compose.monitoring.yaml monitoring

Prometheus publishes no port, because it has no authentication of its own; Cetacean reaches it over the overlay. To open its UI, put it behind a proxy that authenticates, or publish it for a while on the node running it. Swarm can’t bind a published port to one address, so this opens it on every interface of that node; the second command closes it again:

docker service update --publish-add published=9090,target=9090,mode=host monitoring_prometheus
docker service update --publish-rm published=9090,target=9090,mode=host monitoring_prometheus

compose.prometheus.yaml is a small overlay for compose.yaml that joins that network and sets prometheus.url. Pass both files, on this deploy and every later one:

curl -O https://cetacean.mazetti.me/dist/compose.prometheus.yaml
docker stack deploy -c compose.yaml -c compose.prometheus.yaml cetacean

Pointing at a Prometheus you already run is the same setting by another route—set prometheus.url through the environment or a config file, and make sure Cetacean shares a network with it.

The stack runs Prometheus on a manager node, with node-exporter and cAdvisor on every node.

What Cetacean needs from the scrape

If you already run Prometheus, set prometheus.url and check the scrape against the three requirements that follow. The bundled prometheus.yml and compose.monitoring.yaml already satisfy all three.

① The instance label must identify the node

Cetacean matches metrics to nodes by the instance label. Scraping over an overlay network leaves it set to the task’s overlay IP, which matches no node: Prometheus looks healthy and every node chart is empty.

Rewrite instance to the Swarm node address, as the bundled config does:

relabel_configs:
  - source_labels: [__meta_dockerswarm_node_address]
    target_label: instance
    replacement: ${1}:9100

Relabeling instance is required, not optional, whenever the scrape goes over an overlay network.

The dashboard also maps a node to its instance through node_uname_info. Run node-exporter with hostname: "{{.Node.Hostname}}" so its nodename label reports the Swarm node hostname.

② cAdvisor must emit the Swarm service label

Container metrics are attributed to services through container_label_com_docker_swarm_service_name. cAdvisor emits it when started with --store_container_labels=true.

cAdvisor also needs containerd’s socket, not just Docker’s. Without it Prometheus looks healthy and container_* metrics exist, but none carries the service label, so per-service charts stay empty and no sizing recommendation is ever produced. Docker Engine keeps its containerd under /run/docker/containerd, so mount:

- type: bind
  source: /run/docker/containerd
  target: /run/containerd
  read_only: true

If your Docker uses a system containerd instead of its own, use /run/containerd as the source. To check which case you are in, count the labeled series:

count(container_cpu_usage_seconds_total{container_label_com_docker_swarm_service_name!=""})

One or zero means the factory didn’t register. docker service logs monitoring_cadvisor names the socket it couldn’t reach.

③ The cAdvisor scrape job must be named cadvisor

Cetacean detects cAdvisor with up{job="cadvisor"}. Under any other job name the dashboard treats container metrics as unavailable and skips them on the service, task, and node pages. node-exporter is detected by the presence of node_uname_info and doesn’t depend on its job name.

Status banner

The cluster overview and the metrics console show a banner when something is missing:

State	Shown when	Banner
Healthy	Prometheus answers and both exporters cover every node	Nothing
Not configured	prometheus.url is unset	Deploy instructions. Dismissible
Unreachable	prometheus.url is set but queries fail	Warning. The cause is in Cetacean’s log. Not dismissible
Partially configured	Prometheus answers, but an exporter covers no nodes or only some	One line per exporter naming what’s missing and on how many nodes. Dismissible

The MCP get_metrics tool reports the same gap rather than charting zeros. When every series comes back empty it probes for the exporter and answers “cAdvisor isn’t reporting per-container metrics for this cluster” or the node-exporter equivalent.

Prometheus proxy

Browsers query Prometheus through Cetacean rather than directly, so Prometheus needs no CORS configuration and never has to be reachable from outside the swarm network. See the API reference for the proxy endpoints.

Under an ACL policy, only a caller with read on the bare resource * may use the proxy. Anyone else sees the dashboard without its charts and the query console without a query field.

Self-metrics

Cetacean exposes Prometheus metrics about its own operation at /-/metrics: HTTP requests and latency, SSE connections and dropped events, cache size and sync duration, Prometheus proxy latency, and recommendation check duration and counts. Scrape it as a standard target:

scrape_configs:
  - job_name: cetacean
    static_configs:
      - targets: ["cetacean:9000"]
    metrics_path: /-/metrics

Turn it off with server.self_metrics.

Three of these metrics say whether Cetacean is still tracking the cluster, which is what the dashboard’s Stale indicator reports:

cetacean_watcher_connected

1 while the Docker event stream is up, 0 while retrying

cetacean_cache_last_sync_timestamp_seconds

When the last full sync with Docker succeeded

cetacean_cache_sync_failures_total

Full syncs that failed

Alert on the first, since a frozen dashboard is worse than a missing one:

- alert: CetaceanLostDocker
  expr: cetacean_watcher_connected == 0
  for: 5m

The same state is available without Prometheus as the body of GET /-/health; see the API reference.