later edit
I couldn’t elaborate more on this setup sorry. I do frequently use this to throw up a basic Grafana/Prometheus/Node_Exporter setup for simple monitoring jobs, but because I’ve stopped using this stack professionally, I don’t really have the personal time to elaborate on how to set these up further.
This does provide a really good stub for getting you started though. Just keep in mind that the wget’s will pull older versions.
Even better would be if you just used Docker images.
I had to do this for work, and I wanted to simplify the deployment as much as possible. You can very easily achieve what we’re chasing here with various other options, but I believe it’s important to lay out the basics before going crazy.
It’s highly likely that you would be better off configuring repos for this instead of installing the binaries. I personally did this because I wanted a quick, agnostic method of running these services without falling into a trap of repos not being available for certain operating systems.
Typically I would use containers for things, however in the circumstances around building this document as well as for as simplistic of an overview as possible, I have done all of this in systemd.
If you’re containerizing this then you could very happily run the ExecStart scripts from the CLI; with a bit of love, of course.
This will be your web-based endpoint to see all of the things.
# Get latest version from: https://grafana.com/grafana/download
wget https://dl.grafana.com/enterprise/release/grafana-enterprise-8.3.4.linux-amd64.tar.gz
tar -zxvf grafana-enterprise-8.3.4.linux-amd64.tar.gz
# Create dir to run Grafana from
mkdir /opt/grafana
# Create a user to run Grafana with
useradd grafana
# Move what you extracted into the running folder
rsync -av grafana-8.3.4/ /opt/grafana/
# Change ownership over grafana dir
chown -R grafana:grafana /opt/grafana
# Create a service file for systemd
cat > /etc/systemd/system/grafana.service << EOF
[Unit]
Description=Grafanas Awesome Web Display
After=network.target
[Service]
User=grafana
Group=grafana
ExecStart=/opt/grafana/bin/grafana-server --homepath /opt/grafana/
Restart=always
[Install]
WantedBy=default.target
EOF
# Reload systemctl conf and enable/start grafana
systemctl daemon-reload
systemctl enable --now grafana
Prometheus is an endpoint that will scrape your hosts and collect the metrics.
You will need some kind of exporter for the scraped host to publish something to scrape. We’ll install Node Exporter for this later.
# Get latest version from: https://prometheus.io/download/
wget https://github.com/prometheus/prometheus/releases/download/v2.33.0/prometheus-2.33.0.linux-amd64.tar.gz
tar -zxvf prometheus-2.33.0.linux-amd64.tar.gz
# Create dir to run Grafana from
mkdir /opt/prometheus
# Create a user to run Grafana with
useradd prometheus
# Move what you extracted into the running folder
rsync -av prometheus-2.33.0.linux-amd64/ /opt/prometheus/
# Change ownership over grafana dir
chown -R prometheus:prometheus /opt/prometheus
# Create a service file for systemd
cat > /etc/systemd/system/prometheus.service << EOF
[Unit]
Description=Prometheus is better than Zabbix imo
After=network.target
[Service]
User=prometheus
Group=prometheus
WorkingDirectory=/opt/prometheus/
ExecStart=/opt/prometheus/prometheus
[Install]
WantedBy=default.target
EOF
# Reload systemctl conf and enable/start prometheus
systemctl daemon-reload
systemctl enable --now prometheus
Loki is a logging endpoint similar to what rsyslogd can do. It’s a central place where other servers can ship their logs to, but it’s a little easier to use programmatically than something like rsyslogd.
Your hosts will require something like (promtail)[https://grafana.com/docs/loki/latest/getting-started/get-logs-into-loki/] for exporting logs to Loki. I’ve done a really basic explanation of this below
# Get latest version from: https://github.com/grafana/loki/releases/
wget https://github.com/grafana/loki/releases/download/v2.4.2/loki-linux-amd64.zip
unzip loki-linux-amd64.zip
# Create dir to run Grafana from
mkdir /opt/loki
# Create a user to run Grafana with
useradd loki
# Move binary to the new loki folder, then download a sample config
mv loki-linux-amd64 /opt/loki/loki
wget https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml -O /opt/loki/loki.yaml
# Change ownership over grafana dir
chown -R loki:loki /opt/loki
# Create a service file for systemd
cat > /etc/systemd/system/loki.service << EOF
[Unit]
Description=All Hail Loki
After=network.target
[Service]
User=loki
Group=loki
WorkingDirectory=/opt/loki/
ExecStart=/opt/loki/loki -config.file=/opt/loki/loki.yaml
[Install]
WantedBy=default.target
EOF
# Reload systemctl conf and enable/start loki
systemctl daemon-reload
systemctl enable --now loki
Easiest way to try out if this is working is to install promtail on your server.
# Create dir to run Grafana from
mkdir /opt/promtail
# Go into new dir and extract promtail exe
cd /opt/promtail
wget https://github.com/grafana/loki/releases/download/v2.4.2/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip && rm -f promtail-linux-amd64.zip
mv promtail-linux-amd64 promtail
# Download sample config
wget https://raw.githubusercontent.com/grafana/loki/main/clients/cmd/promtail/promtail-local-config.yaml -O /opt/promtail/promtail.yaml
# Create a service file for systemd
cat > /etc/systemd/system/promtail.service << EOF
[Unit]
Description=Send gifts to Loki, god of stress
After=network.target
[Service]
User=root
Group=root
WorkingDirectory=/opt/promtail/
ExecStart=/opt/promtail/promtail -config.file=/opt/promtail/promtail.yaml
[Install]
WantedBy=default.target
EOF
# Reload systemctl conf and enable/start promtail
systemctl daemon-reload
systemctl enable --now promtail
Like I said, (node_exporter)[https://prometheus.io/download/#node_exporter] is probably the easiest way to do this. I’ll just show you how to do a single instance for testings sake, but there’s a lot of options in this section: https://prometheus.io/docs/prometheus/latest/configuration/configuration/
# Basically all the steps from before, consolidated.
useradd node_exporter
mkdir /opt/node_exporter/
wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
tar xvfs node_exporter-1.3.1.linux-amd64.tar.gz && rm -f node_exporter-1.3.1.linux-amd64.tar.gz
mv node_exporter-1.3.1.linux-amd64/node_exporter /opt/node_exporter/node_exporter
rm -rf node_exporter-1.3.1.linux-amd64
chown -R node_exporter:node_exporter /opt/node_exporter
# Create a service file for systemd
cat > /etc/systemd/system/node_exporter.service << EOF
[Unit]
Description=Beam statistics to Prometheus
After=network.target
[Service]
User=node_exporter
Group=node_exporter
WorkingDirectory=/opt/node_exporter/
ExecStart=/opt/node_exporter/node_exporter
[Install]
WantedBy=default.target
EOF
# Reload systemctl conf and enable/start promtail
systemctl daemon-reload
systemctl enable --now node_exporter
Simplest way to check if node_exporter and prometheus can cluck like a couple of hens is to jump onto your prometheus server and update it’s prometheus.yaml file.
Open /opt/prometheus/prometheus.yml and find the line with targets: ["localhost:9090"] inside of it. All you have to do is add your new host:
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: "prometheus"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ["localhost:9090","10.69.69.69:9100"]
Then issue a systemctl restart prometheus to have the server detect the changes and you’ll be able to browse to your prometheus install and execute a go_info to see your instance appear in the table.