Monitor Linux Server using Prometheus Node Exporter

Prerequisites

Ubuntu 16.04
Kubernetes Cluster already created
Already Prometheus Server is up and Running
Port 9100 opened in server firewall as Prometheus reads metrics on this port.

Setup Node Exporter Binary

Step 1: Download the latest node exporter package. You should check the Prometheus downloads section for the latest version and update this command to get that package.

cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.0.1/node_exporter-1.0.1.linux-amd64.tar.gz

Step 2: Unpack the tarball

tar -xvf node_exporter-1.0.1.linux-amd64.tar.gz

Step 3: Move the node export binary to /usr/local/bin

sudo mv node_exporter-1.0.1.linux-amd64/node_exporter /usr/local/bin/

Step 4: Create a node_exporter user to run the node exporter service

sudo useradd -rs /bin/false node_exporter

Step 5: Create a node_exporter service file under systemd.

sudo vi /etc/systemd/system/node_exporter.service

Step 6: Add the following service file content to the service file and save it.

[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target

Step 4: Reload the system daemon and star the node exporter service.

sudo systemctl daemon-reload
sudo systemctl start node_exporter

Step 7: check the node exporter status to make sure it is running in the active state.

sudo systemctl status node_exporter

Step 8: Enable the node exporter service to the system startup.

sudo systemctl enable node_exporter

Step 9: As node exporter would be exporting metrics on port 9100, open the port in the security groups with the following values

http://localhost:9100/metrics

You can see all the server metrics by visiting your server URL on /metrics as shown below

Step 10: Configure the Server as Target on Prometheus Server

Now that we have the node exporter up and running on the server, we have to add this server a target on the Prometheus server configuration. Note: This configuration should be done on the Prometheus server.

Note: Login to the Prometheus server and open the prometheus.yml file

sudo vi /etc/prometheus/prometheus.yml

Note: Under the scrape config section add the node exporter target as shown below. Change localhost with your server IP where you have setup node exporter. Job name can be your server hostname or IP for identification purposes.

- job_name: ‘node_exporter_metrics’
 scrape_interval: 5s
 static_configs:
 — targets: [‘localhost:9100’]

Step11: Restart the prometheus service for the configuration changes to take place.

Step12 :Now, if you check the target in prometheus web UI (http://localhost :9090/targets) , you will be able to see the status as shown below.

http://localhost :9090/targets

Step13 :Click Graphs in the web UI and monitor.

Following are the few key node metrics you can use to find its statistics.

1.node_memory_MemFree_bytes
2.node_cpu_seconds_total
3.node_filesystem_avail_bytes
4.rate(node_cpu_seconds_total{mode=”system”}[1m])
5.rate(node_network_receive_bytes_total[1m])