In Part 1 of A Ceph guide for Kubernetes and Openshift users, we covered:
- Quick Introduction to Ceph and alternatives
- How to Deploy Ceph on AWS
- Other Ceph deployment strategies for Kubernetes and Openshift
- How to Monitor Ceph with built-in tools
- How to Monitor Ceph with Sysdig Monitor
- Top 5 metrics to monitor in your Ceph cluster
How to Monitor Ceph with built-in tools
Ceph provides built-in monitoring through the command line interface. Our first stop is to check if Ceph health is ok. Running thehealth command gives us an OK, a warning or error on the overall cluster health status:
$ sudo ceph ceph> health HEALTH_OKbut we could also get something like:
ceph> health HEALTH_WARN 1/3 in osds are downor
ceph> health HEALTH_ERR 1 nearfull osds, 1 full osds osd.2 is near full at 85% osd.3 is full at 97%More detailed information can be retrieved with ceph
status that will give us a few lines about the monitor, storage nodes and placement groups:
ceph> status
cluster 3e13cbbe-d9c5-4cd0-b18f-8b349b04ba45
health HEALTH_OK
monmap e1: 3 mons at {ip-172-31-17-113=172.31.17.113:6789/0,ip-172-31-17-75=172.31.17.75:6789/0,ip-172-31-20-164=172.31.20.164:6789/0}
election epoch 4, quorum 0,1,2 ip-172-31-17-75,ip-172-31-17-113,ip-172-31-20-164
osdmap e15: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
pgmap v65: 104 pgs, 6 pools, 1588 bytes data, 171 objects
106 MB used, 76653 MB / 76759 MB avail
104 active+clean
We can see here (for each line): 1) cluster ID 2) cluster health status 3-4) monitor map and status of monitor quorum 5-6) OSD map and status of OSD nodes 7-9) placement groups, pools, objects and capacity usage used is the raw storage used (including replicas and snapshots) while xxx MB / xxx MB is the available storage from the total capacity.
When looking at your placement groups status, you want them to be active and clean. You can see more details of PG with the pg dump command.
To know what the OSD nodes are doing, we have a few different options:
ceph> osd stat
osdmap e15: 3 osds: 3 up, 3 in
flags sortbitwise,require_jewel_osds
and
ceph> osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 0.07320 root default -2 0.02440 host ip-172-31-24-236 0 0.02440 osd.0 up 1.00000 1.00000 -3 0.02440 host ip-172-31-29-248 1 0.02440 osd.1 up 1.00000 1.00000 -4 0.02440 host ip-172-31-30-149 2 0.02440 osd.2 up 1.00000 1.00000An OSD (object storage daemon) is a node with storage capacity. If you have configured some redundancy (probably you have) multiples copies will be stored across different nodes for data resilience. OSD server status can be either:
- Up: the node (daemon) is running and responding
- Down: the node is either stopped or not responding, other OSD or MON nodes didn’t receive any heartbeats from that node
- In: the node participates in data placement
- Out: the node does NOT participate in data placement
ceph> mon stat
e1: 3 mons at {ip-172-31-17-113=172.31.17.113:6789/0,ip-172-31-17-75=172.31.17.75:6789/0,ip-172-31-20-164=172.31.20.164:6789/0}, election epoch 4, quorum 0,1,2 ip-172-31-17-75,ip-172-31-17-113,ip-172-31-20-164
Monitoring the cluster capacity and data distribution across the different pools we can use the df command:
ceph> df [detail]
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
76759M 76653M 106M 0.14
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd 0 0 0 25551M 0
.rgw.root 1 1588 0 25551M 4
default.rgw.control 2 0 0 25551M 8
default.rgw.data.root 3 0 0 25551M 0
default.rgw.gc 4 0 0 25551M 32
default.rgw.log 5 0 0 25551M 127
Or by OSD node:
ceph> osd df
ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
0 0.02440 1.00000 25586M 35380k 25551M 0.14 1.00 104
2 0.02440 1.00000 25586M 35256k 25552M 0.13 1.00 104
1 0.02440 1.00000 25586M 35252k 25552M 0.13 1.00 104
TOTAL 76759M 103M 76656M 0.13
MIN/MAX VAR: 1.00/1.00 STDDEV: 0
We cannot miss looking at the I/O metrics, latency and reads/writes both in ops per second and bandwidth using osd perf:
ceph> osd perf osd fs_commit_latency(ms) fs_apply_latency(ms) 2 41 55 1 41 58 0 732 739
ceph> osd pool stats pool rbd id 0 nothing is going on pool .rgw.root id 1 nothing is going on pool default.rgw.control id 2 nothing is going on pool default.rgw.data.root id 3 nothing is going on pool default.rgw.gc id 4 nothing is going on pool default.rgw.log id 5 nothing is going on pool scbench id 6 client io 33649 kB/s wr, 0 op/s rd, 8 op/s wrIf you are interested in reading more about troubleshooting and performance in Ceph we recommend checking out Troubleshooting OSDs in the Ceph documentation, and to check out setup limits read Benchmark Ceph Cluster Performance.
How to Monitor Ceph with Sysdig Monitor
To retrieve Ceph metrics and send them to Sysdig Monitor you just need to have a Sysdig Monitor agent running in one of the monitor nodes but since any node can go down at any point in time in a highly available cluster, we recommend installing Sysdig Monitor agent in all of them, as will also help to collect system level metrics specific of that host. Sysdig uses an auto-discovery mechanism to figure out that Ceph is running, and extract the correct metrics with no human intervention. The default configuration will check if aceph-mon process (the Ceph Monitor software) is running and will collect the following metrics:
Ceph Cluster Performance Metrics
- ceph.commit_latency_ms: Time in milliseconds to commit an operation
- ceph.apply_latency_ms: Time in milliseconds to sync to disk
- ceph.read_bytes_sec: bytes/second read
- ceph.write_bytes_sec: bytes/second written
Ceph Cluster Availability Metrics
- ceph.num_osds: number of discovered OSD nodes
- ceph.num_in_osds: number of participating OSD nodes
- ceph.num_up_osds: number of reachable OSD nodes
- ceph.num_near_full_osds: number of OSD nodes near full storage capacity
- ceph.num_full_osds: number of OSD nodes in full storage capacity
- ceph.osd.pct_used: percentage of OSD nodes in near full or full storage capacity
- ceph.num_pgs: number of placement groups available
- ceph.num_mons: number of monitor nodes available
- ceph.aggregate_pct_used: percentage of storage capacity used
- ceph.num_pools: number of pools
- ceph.total_objects: number of objects
Per pool metrics
- ceph.op_per_sec: Operations per second
- ceph.read_bytes: Counter of read bytes (to be used with Rate function)
- ceph.write_bytes: Counter of written bytes (to be used with Rate function)
- ceph.read_op_per_sec: bytes/second read
- ceph.write_op_per_sec: bytes/second written
- ceph.num_objects: number of objects
In addition, Sysdig automatically applies metadata tags from Kubernetes, OpenShift, AWS and more to each and every metric. This means you can aggregate and segment data across deployments, pods, datacenters, and more with a couple clicks. For example, if you want to see operations per second for each pool you just need to select ceph.ops_per_second and segment by ceph_pool.
To complete the monitoring setup, we recommend creating at a minimum the 5 alerts we recommend below. Here’s a quick example of creating a health != OK across all Ceph storage nodes in the system:
This example allows you to set one alert and have it apply across each Ceph node you’re running. This is particularly useful in orchestrated environments, where you system may be spinning up additional Ceph nodes as needed. In this case, we collect the ceph.overall_status across all the Monitor nodes as these can scale up or down. If the alert triggers, we will only send one alert and not an alert for each of the node.