ScyllaDB Monitoring on datadog

Monitoring stack details on single instance:

Scylla-Monitoring Stack version: 4.4.5

  • OS: Ubuntu 22.04.2 LTS
  • Prometheus : v2.44.0
  • Grafana: 9.5.5
  • datadog agent : v7.50.3

Scylla node details: :

  • OS: Ubuntu 20.04.6 LTS
  • node_exporter: version 1.0.1

I have allowed required port on security group.

Issue:

  1. Not getting some system /os metrics value on datadog like node_network_receive_bytes_total, scylla.node_uname_info , scylla.node_load1, etc . Node exporter is running and able to communicate.
  2. DB metrics

Can someone suggest here.

The Scylla Monitoring stack holds far too many metrics for the Datadog to handle.
Therefore, we are marking a small subset of metrics with a label (which used to be level and now is dd) and using that for the Datadog integration.

For the node_exported metrics, we are using the Prometheus relabel config to add the label.

I suggest you update to the latest 4.6. There are a few changes we made.

If there are more metrics you need, you can edit prometheus/prometheus.yml.template

Look for the node_exporter section’s relabel config, it adds a label, you can add more metrics over there.