On this tutorial, we will focus on how to install Sensu Monitoring on RHEL 9/ CentOS Stream 9. Monitoring is the process of implementing monitoring tools to analyze system resources, such as application servers and databases. To evaluate performance and fix problems of a system, it entails acquiring telemetry data on how an IT environment functions. Monitoring enables the observation of how and why certain systems act in a particular way and provides knowledge to deliver sophisticated analytics.

Monitoring stacks provide a good base for system performance tuning like keeping production workloads healthy and debugging issues. Some monitoring tools provide visual functionalities like graphing. Amazon CloudWatch, Nagios, Sensu, Datadog, New Relic, Telegraf, Prometheus, InfluxDB, and other monitoring tools are a few examples.

Sensu Monitoring

Sensu monitoring tool is an observability pipeline tool that is designed to work on cloud platforms and web platforms. Sensu consolidates existing monitoring tools, including Nagios, StatsD, Telegraf, Prometheus, & more to remove data redundancy, improve observability, and reduce multi-cloud tool sprawl. Sensu collects data from your system, sends it to a database (of your choice), automates alerts with your incident management platform, then automates monitoring and remediation with your existing configuration management tools.

Sensu Go is an agent-based observability tool with an agent that is a lightweight client that runs on the infrastructure components you want to monitor and a backend service that provides a customizable, automated pipeline for filtering, transforming, and processing alerts and metrics. The agent entities create metrics and events that are sent to the backend which has an integrated structure for scheduling the checks according to the subscription defined in the agent configuration.

Sensu can be configured to correspond to workflow with the right interface and right data for individual problems which let you define the monitoring insights that matter most. These can be;

  • Health checks, custom metrics, & logs that let you collect custom metrics for applications and integrate them with external databases.
  • Alerts and incident management to customize alert policies with event filters and contact routing to reduce alert fatigue.
  • Monitoring plugins and pre-configured templates offer all the flexibility of monitoring-as-code workflows with no coding required.
  • Automate repetitive operations by configuring and executing custom scripts when a problem is identified.
  • A centralized management interface for monitoring highly complex observability workflows in an automated and repeatable fashion.
  • Easily automates service discovery and registers VMs, public cloud compute instances, containers, and the services running on them.

The architecture of the Sensu Observality Pipeline flow of events can be seen below.

source: Sensu Docs

Sensu Go is packaged for Linux, Windows (agent and sensuctl only), macOS (sensuctl only), and Docker. To install it we will;

  • Install the Backend server that receives the resulting status and metric events from the agent.
  • Install the Sensuctl which is a command line tool for managing resources within Sensu.
  • Install the Agent entities in the client systems that you want to monitor. This is installed on the infrastructure (client) that is being monitored by Sensu, You are only required to update with the Sensu Backed server IP address and it is automatically registered as an entity.

1. Install Sensu Backend on RHEL 9/ CentOS Stream 9

Update your system packages

sudo dnf update -y

Add the Sensu repository to your system.

curl -s https://packagecloud.io/install/repositories/sensu/stable/script.rpm.sh | sudo bash

Install the Sensu Go backend package with the following command.

sudo dnf install sensu-go-backend

Configure the Sensu backend to start by creating a YAML configuration file with the following command. This copies the configuration template from the official docs.

sudo curl -k https://docs.sensu.io/sensu-go/latest/files/backend.yml -o /etc/sensu/backend.yml

Start and enable the Sensu backend service.

sudo systemctl enable --now sensu-backend

Verify that the service is running.

$ sudo systemctl status sensu-backend
sensu-backend.service - The Sensu Backend service.
     Loaded: loaded (/usr/lib/systemd/system/sensu-backend.service; disabled; vendor>
     Active: active (running) since Fri 2022-09-02 23:42:30 EAT; 8s ago
   Main PID: 4365 (sensu-backend)
      Tasks: 8 (limit: 48809)
     Memory: 33.2M
        CPU: 157ms
     CGroup: /system.slice/sensu-backend.service
             └─4365 /usr/sbin/sensu-backend start -c /etc/sensu/backend.yml

Setup Sensu administrator username and password by running the sensu-backend init. But first, specify the admin credential with the following syntax.

export SENSU_BACKEND_CLUSTER_ADMIN_USERNAME=technixleo
export [email protected]

Replace the username and password with your preferred details and run the initialization command

sensu-backend init

Configure the firewall to allow the Sensu port.

sudo firewall-cmd --permanent --zone=public --add-port=3000/tcp
sudo firewall-cmd --permanent --add-port=8080/tcp 
sudo firewall-cmd --permanent --add-port=8081/tcp 
sudo firewall-cmd --reload

Verify that the backend is running.

$ curl http://127.0.0.1:8080/health
{"Alarms":null,"ClusterHealth":[{"MemberID":13195394291058371180,"MemberIDHex":"b71f75320dc06a6c","Name":"default","Err":"","Healthy":true}],"Header":{"cluster_id":2037210783374497686,"member_id":13195394291058371180,"raft_term":2}}

Go to your browser to access the Sensu dashboard at http://localhost:3000 . Then use the Sensu administrator username and password to sign in, just like as shown below;

Below is how the dashboard looks like, once you are signed in;

Select the default namespace to view the events, entities, and other properties;

2. Install Sensuctl on RHEL 9/ CentOS Stream 9

We can now proceed to install the Sensuctl on RHEL 9/ CentOS Stream 9. Before then, first add the Sensu repository to your system by running the command shown below on your terminal;

[[email protected] ~]$ curl https://packagecloud.io/install/repositories/sensu/stable/script.rpm.sh | sudo bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0[s100  6796    0  6796    0     0   7985      0 --:--:-- --:--:-- --:--:--  7976

[sudo] password for annalianie:
Detected operating system as centos/9.
Checking for curl...
Detected curl...
Downloading repository file: https://packagecloud.io/install/repositories/sensu/stable/config_file.repo?os=centos&dist=9&source=script
done.
Installing pygpgme to verify GPG signatures...
sensu_stable-source                             383  B/s | 819  B     00:02
No match for argument: pygpgme
Error: Unable to find a match: pygpgme

WARNING:
The pygpgme package could not be installed. This means GPG verification is not possible for any RPM installed on your system.
To fix this, add a repository with pygpgme. Usualy, the EPEL repository for your system will have this.
More information: https://fedoraproject.org/wiki/EPEL#How_can_I_use_these_extra_packages.3F

Installing yum-utils...
sensu_stable-source                             478  B/s | 951  B     00:01
Package yum-utils-4.3.0-1.el9.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!
Generating yum cache for sensu_stable...
Generating yum cache for sensu_stable-source...

The repository is setup! You can now install packages.

Once the repository has successfully been added, you can then continue to install the Sensu Go CLI package, by running the command shown below;

sudo dnf install sensu-go-cli

Configure the sensuctl to start by setting up using the prompts for interactive setup with the following command. Once you make a choice, tap Enter/Return to continue for each;

$ sensuctl configure
? Authentication method: username/password
?Sensu Backend API URL: http://127.0.0.1:8080
? Namespace: default
? Preferred output format: tabular
? Username: technixleo
? Password: ********

Alternatively, you can configure it in a non-interactive mode with the following syntax.

sensuctl configure -n \
--username 'technixleo' \
--password '[email protected]' \
--namespace default \
--url 'http://127.0.0.1:8080'

3. Install Sensu Agent on RHEL 9/ CentOS Stream 9

Install Sensu Agent on a client machine that will be monitored. First, add the Sensu repository with the following command.

curl -s https://packagecloud.io/install/repositories/sensu/stable/script.rpm.sh | sudo bash

Then install the Sensu Go Agent package with the following command.

sudo dnf install sensu-go-agent

Configure the Sensu Go agent with a YAML configuration file. Download it from the official docs with the following command.

sudo curl -L https://docs.sensu.io/sensu-go/latest/files/agent.yml -o /etc/sensu/agent.yml

Configure Sensu agent to send requests to Sensu backend server

$ sudo vi /etc/sensu/agent.yml
name: "node1"
namespace: "default"
backend-url:
  - "ws://192.168.200.40:8081"

Start and enable the Sensu Go agent service.

sudo systemctl enable --now sensu-agent

Verify it is running with the following command.

$ sudo systemctl status sensu-agent
● sensu-agent.service - The Sensu Agent process.
     Loaded: loaded (/usr/lib/systemd/system/sensu-agent.service; disabled; ven>
     Active: active (running) since Sat 2022-09-03 00:01:30 EAT; 7s ago
   Main PID: 5021 (sensu-agent)
      Tasks: 8 (limit: 48771)
     Memory: 10.6M
        CPU: 29ms
     CGroup: /system.slice/sensu-agent.service
             └─5021 /usr/sbin/sensu-agent start -c /etc/sensu/agent.yml

4. Verify Keepalive events

Keepalives ensure the Sensu agents are running and reach the Sensu backend. To confirm the agent is registered with Sensu and is sending keepalive events, open the entity page on the web UI.

You can also run the following command on the CLI

$ sensuctl entity list
   ID     Class    OS     Subscriptions             Last Seen            
──────── ─────── ─────── ─────────────── ────────────────────────────────
  Node1   agent   linux   entity:Node1    2022-09-03 00:34:01 +0300 EAT

5. Verify an example event

With both the agent and backend still running, send this request to the Sensu core/v2/events API from the Sensu agent.

curl -X POST \
-H 'Content-Type: application/json' \
-d '{
  "check": {
    "metadata": {
      "name": "check-mysql-status"
    },
    "status": 1,
    "output": "could not connect to mysql"
  }
}' \
http://127.0.0.1:3031/events

This will create a warning event that can be seen on the web UI events page.

To get more information about the warning, click on it.

To send an OK status we will send the following request

curl -X POST \
-H 'Content-Type: application/json' \
-d '{
  "check": {
    "metadata": {
      "name": "check-mysql-status"
    },
    "status": 0,
    "output": "Connected to mysql"
  }
}' \
http://127.0.0.1:3031/events

The OK event will not show an alert as it is successful. You will instead have an OK page that the events are passing.

To see the event change the filter to event.check.state != “failing”.

Then click on the event for more information.

Conclusion

Sensu is a complete monitoring solution designed to give you visibility into your whole system infrastructure that revolves around traditional server closets, containers, applications, the cloud, and more. It integrates with the Cloud Platforms (AWS, Kubernetes), Data Platforms (Elastic search, MySQL), Web Platforms (Nginx, Apache), Automation platforms (Ansible, Terraform), Incident management (PagerDuty, Slack), and Monitoring tools (Nagios, Prometheus) to improve the system’s infrastructure observability. The following include some of our other articles;

LEAVE A REPLY

Please enter your comment!
Please enter your name here