In this article, I am going to walk through installing some monitoring software on a CentOS 7 box.<\/p>\n\n\n\n
Start with the the base install of CentOS is described here<\/p>\n\n\n\n
https:\/\/my.apolonio.tech\/?p=78<\/a> <\/p>\n\n\n\n but upgraded with yum update so you would end up with a CentOS 7.9.2009 instance.<\/p>\n\n\n\n All commands are run with a user that has sudo permissions unless otherwise noted.<\/p>\n\n\n\n Add the Grafana repo in to your yum repo list<\/p>\n\n\n\n Install it<\/p>\n\n\n\n Enable it and start it up, this will make it available on reboot<\/p>\n\n\n\n At this point you should be able to log in to Grafana with a web browser using port 3000<\/p>\n\n\n\n http:\/\/grafana.makernet.local:3000\/login<\/a><\/p>\n\n\n\n Username: admin Password: admin<\/p>\n\n\n\n You will be prompted to change it.<\/p>\n\n\n\n Note: If you ever forget the admin password for Grafana use the following command to reset the password to newpassword<\/p>\n\n\n\n Add the InfluxDB repo to your yum repo list<\/p>\n\n\n\n Install it<\/p>\n\n\n\n Start it up<\/p>\n\n\n\n Next run the influx command<\/p>\n\n\n\n And create a the telegraf database with the appropriate usernames, passwords, and permissions<\/p>\n\n\n\n Go to your home directory and download the Prometheus package.<\/p>\n\n\n\n Go to opt and unroll the package, rename to prometheus<\/p>\n\n\n\n Create a user account for Prometheus to run as and give it ownership of the Prometheus install<\/p>\n\n\n\n Create a system service<\/p>\n\n\n\n With the following contents<\/p>\n\n\n\n And reload the daemon, enable and start Prometheus<\/p>\n\n\n\n Go back to your home directory and download the Loki package<\/p>\n\n\n\n Go to opt, create the loki folder and unzip the package in to it<\/p>\n\n\n\n While you are in the \/opt\/loki folder, create a loki config file<\/p>\n\n\n\n With the following<\/p>\n\n\n\n Create a user account for loki and give it ownership of the loki directory<\/p>\n\n\n\n Create a systemd service<\/p>\n\n\n\n With the following content<\/p>\n\n\n\n Nagios is in the epel repo, install the repo first then install nagios and plugins<\/p>\n\n\n\n Nagios uses basic auth for authentication, create password file and users, the first command creates the password file and adds the user larry, the second, simply adds the user standard to that same file.<\/p>\n\n\n\n Create a file in the root of web server so there are no errors then restart apache<\/p>\n\n\n\n Add the users as contacts in the contacts file<\/p>\n\n\n\n Here is a sample of how the users look<\/p>\n\n\n\n Then add modify the group to add new users as administrators<\/p>\n\n\n\n Verify that the nagios config file is correct, you should always verify after you make some configuration changes.<\/p>\n\n\n\n Enable nagios and start the daemon<\/p>\n\n\n\n At this point run the following commands to check if all the services are running correctly<\/p>\n\n\n\n Furthermore use a browser to check that these services are OK<\/p>\n\n\n\n The following two services should give you a 404 error which is OK, it means something is reponding on those ports, it is a problem if you get The site can\u2019t be reached and\/or refused to connect.<\/p>\n\n\n\n It is highly recommended you protect your services with SSL\/TLS encryption. Since there are a variety of ways that an SSL certificate gets created, the steps in creating a certificate and key pair will not be outlined here.<\/p>\n\n\n\n It is assumed the reader know about certificates and keys and if there is an intermediate, how to add them as a chain file or combine them.<\/p>\n\n\n\n Certs here are PEM encoded and the examples used do not have an intermediate.<\/p>\n\n\n\n Apache is the web server used by nagios and will serve as a proxy for loki and Prometheus. So make sure mod ssl is installed, and the auto generated cert and keys are replaced with signed ones.<\/p>\n\n\n\n Use cat to paste the cert and keys in to the locations below, or scp\/sftp to those locations.<\/p>\n\n\n\n Copy the certs the locations used by Grafana and influxdb which will use the cert and keys directly<\/p>\n\n\n\n Edit the Grafana config file<\/p>\n\n\n\n Look for the server section, change the protocol to https and point the cert_file and cert_key parameters to the appropriate location<\/p>\n\n\n\n Restart Grafana<\/p>\n\n\n\n Edit the InfluxDB File<\/p>\n\n\n\n In the http section, enable https and point to the appropriate cert and key files<\/p>\n\n\n\n Restart InfluxDB<\/p>\n\n\n\n uncomment out the SSLRequireSSL (Remove the # from 2 locations)<\/p>\n\n\n\n Restart the web service<\/p>\n\n\n\n Create a Prometheus apache config file<\/p>\n\n\n\n Here are the contents<\/p>\n\n\n\n Restart the web service<\/p>\n\n\n\n Create a loki apache config file<\/p>\n\n\n\n Here are the contents<\/p>\n\n\n\n Restart the web service<\/p>\n\n\n\n Here are example URLs to test<\/p>\n\n\n\n Note that the unencrypted ports for Loki and Prometheus are still accessible. You can either block those ports with firewalld or change the config to only listen to localhost<\/p>\n","protected":false},"excerpt":{"rendered":" In this article, I am going to walk through installing some monitoring software on a CentOS 7 box. Start with the the base install of CentOS is described here https:\/\/my.apolonio.tech\/?p=78 but upgraded with yum...<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_crdt_document":"","footnotes":""},"categories":[6],"tags":[],"class_list":["post-194","post","type-post","status-publish","format-standard","hentry","category-linux-server"],"_links":{"self":[{"href":"https:\/\/my.apolonio.tech\/index.php?rest_route=\/wp\/v2\/posts\/194","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/my.apolonio.tech\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/my.apolonio.tech\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/my.apolonio.tech\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/my.apolonio.tech\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=194"}],"version-history":[{"count":3,"href":"https:\/\/my.apolonio.tech\/index.php?rest_route=\/wp\/v2\/posts\/194\/revisions"}],"predecessor-version":[{"id":197,"href":"https:\/\/my.apolonio.tech\/index.php?rest_route=\/wp\/v2\/posts\/194\/revisions\/197"}],"wp:attachment":[{"href":"https:\/\/my.apolonio.tech\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=194"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/my.apolonio.tech\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=194"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/my.apolonio.tech\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=194"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Install Grafana<\/h1>\n\n\n\n
cd \/etc\/yum.repos.d<\/code>sudo wget https:\/\/my.apolonio.tech\/linux\/grafana.repo<\/a><\/code><\/p>\n\n\n\nsudo yum -y install grafana<\/code><\/p>\n\n\n\nsudo systemctl enable --now grafana-server<\/code><\/p>\n\n\n\ngrafana-cli admin reset-admin-password newpassword<\/code><\/p>\n\n\n\nInstall InfluxDB<\/h1>\n\n\n\n
cd \/etc\/yum.repos.d
sudo wget https:\/\/my.apolonio.tech\/linux\/influxdb.repo<\/a><\/code><\/p>\n\n\n\nsudo yum install influxdb<\/code><\/p>\n\n\n\nsudo systemctl enable --now influxdb<\/code><\/p>\n\n\n\ninflux<\/code><\/p>\n\n\n\nCREATE DATABASE \"telegraf\"
CREATE USER \"admin\" WITH PASSWORD 'supersecretpassword'
CREATE USER \"telegraf\" WITH PASSWORD 'secretpassword'
GRANT ALL ON \"telegraf\" TO \"admin\"
GRANT READ ON \"telegraf\" TO \"telegraf\"
GRANT WRITE ON \"telegraf\" TO \"telegraf\"<\/code>EXIT<\/code><\/p>\n\n\n\nInstall Prometheus<\/h1>\n\n\n\n
cd
wget https:\/\/github.com\/prometheus\/prometheus\/releases\/download\/v2.22.1\/prometheus-2.22.1.linux-amd64.tar.gz<\/a><\/code><\/p>\n\n\n\ncd \/opt
sudo tar zxpvf ~\/prometheus-2.22.1.linux-amd64.tar.gz
sudo mv prometheus-2.22.1.linux-amd64 prometheus<\/code><\/p>\n\n\n\nsudo mkdir -p \/opt\/prometheus\/var\/lib\/
sudo useradd -rs \/bin\/false prometheus
sudo chown -R prometheus:prometheus \/opt\/prometheus<\/code><\/p>\n\n\n\nsudo vi \/etc\/systemd\/system\/prometheus.service<\/code><\/p>\n\n\n\n[Unit]
Description=Prometheus
After=network.target
<\/code>[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=\/opt\/prometheus\/prometheus \\
--config.file \/opt\/prometheus\/prometheus.yml \\
--storage.tsdb.path \/opt\/prometheus\/var\/lib\/ \\
--web.console.templates=\/opt\/prometheus\/console \\
--web.console.libraries=\/opt\/prometheus\/console_libraries<\/code>
[Install]
WantedBy=multi-user.target<\/code><\/p>\n\n\n\nsudo systemctl daemon-reload
sudo systemctl enable --now prometheus<\/code><\/p>\n\n\n\nInstall Loki<\/h1>\n\n\n\n
cd
wget https:\/\/github.com\/grafana\/loki\/releases\/download\/v1.6.1\/loki-linux-amd64.zip<\/a><\/code><\/p>\n\n\n\nsudo mkdir -p \/opt\/loki
cd \/opt\/loki
sudo unzip ~\/loki-linux-amd64.zip<\/code><\/p>\n\n\n\nsudo vi \/opt\/loki\/config-loki.yml<\/code> <\/p>\n\n\n\nauth_enabled: false<\/code><\/p>\n\n\n\nserver:
http_listen_port: 3100<\/code>
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 5m
chunk_retain_period: 30s
max_transfer_retries: 0<\/code>
schema_config:
configs:
- from: 2018-04-15
store: boltdb
object_store: filesystem
schema: v11
index:
prefix: index_
period: 168h<\/code>
storage_config:
boltdb:
directory: \/tmp\/loki\/index<\/code>
filesystem:
directory: \/tmp\/loki\/chunks<\/code>
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h<\/code>
chunk_store_config:
max_look_back_period: 0s<\/code>
table_manager:
retention_deletes_enabled: false
retention_period: 0s<\/code><\/p>\n\n\n\nsudo useradd -rs \/bin\/false loki
sudo chown -R loki.loki \/opt\/loki<\/code><\/p>\n\n\n\nsudo vi \/etc\/systemd\/system\/loki.service<\/code><\/p>\n\n\n\n[Unit]
Description=Loki service
After=network.target<\/code>
[Service]
Type=simple
User=loki
Group=loki
ExecStart=\/opt\/loki\/loki-linux-amd64 -config.file \/opt\/loki\/config-loki.yml<\/code>
[Install]
WantedBy=multi-user.target<\/code><\/p>\n\n\n\n
And reload the daemon, enable and start Prometheus<\/p>\n\n\n\nsudo systemctl daemon-reload
sudo systemctl enable --now loki<\/code><\/p>\n\n\n\nInstall Nagios<\/h1>\n\n\n\n
sudo yum -y install epel-release
sudo yum -y install nagios nagios-plugins-all<\/code><\/p>\n\n\n\nsudo htpasswd -c \/etc\/nagios\/passwd larry
sudo htpasswd \/etc\/nagios\/passwd <\/code>standard<\/p>\n\n\n\nsudo touch \/var\/www\/html\/index.html
sudo systemctl restart httpd<\/code><\/p>\n\n\n\nsudo vi \/etc\/nagios\/objects\/contacts.cfg<\/code><\/p>\n\n\n\ndefine contact {
contact_name larry
use generic-contact
alias Larry Apolonio
email larry@makernet.local
}<\/code><\/p>\n\n\n\ndefine contact {
contact_name standard
use generic-contact
alias Standard Account
email standard@makernet.local
}<\/code><\/p>\n\n\n\ndefine contactgroup {
contactgroup_name admins
alias Nagios Administrators
members nagiosadmin,larry,standard
}<\/code><\/p>\n\n\n\nsudo nagios -v \/etc\/nagios\/nagios.cfg<\/code><\/p>\n\n\n\nsudo systemctl enable --now nagios<\/code><\/p>\n\n\n\nValidation<\/h1>\n\n\n\n
sudo systemctl status nagios
sudo systemctl status loki
sudo systemctl status influxdb
sudo systemctl status prometheus
sudo systemctl status grafana-server
sudo systemctl status httpd<\/code><\/p>\n\n\n\nProtect Service with SSL<\/h1>\n\n\n\n
Apache<\/h2>\n\n\n\n
sudo yum -y install mod_ssl<\/code><\/p>\n\n\n\nsudo su -
cat > \/etc\/pki\/tls\/certs\/localhost.crt
cat > \/etc\/pki\/tls\/private\/localhost.key
exit<\/code><\/p>\n\n\n\nsudo cp -p \/etc\/pki\/tls\/certs\/localhost.crt \/etc\/grafana\/server.crt
sudo cp -p \/etc\/pki\/tls\/private\/localhost.key \/etc\/grafana\/server.key
sudo chown grafana:root \/etc\/grafana\/server.crt
sudo chown grafana:root \/etc\/grafana\/server.key
sudo cp -p \/etc\/pki\/tls\/certs\/localhost.crt \/etc\/influxdb\/server.crt
sudo cp -p \/etc\/pki\/tls\/private\/localhost.key \/etc\/influxdb\/server.key
sudo chown influxdb:root \/etc\/influxdb\/server.crt
sudo chown influxdb:root \/etc\/influxdb\/server.key<\/code><\/p>\n\n\n\nGrafana<\/h1>\n\n\n\n
sudo vi \/etc\/grafana\/grafana.ini<\/code><\/p>\n\n\n\n[server]
# Protocol (http, https, h2, socket)
protocol = https
cert_file = \/etc\/grafana\/server.crt
cert_key = \/etc\/grafana\/server.key<\/code><\/p>\n\n\n\nsudo systemctl restart grafana-server<\/code><\/p>\n\n\n\nInfluxDB<\/h1>\n\n\n\n
sudo vi \/etc\/influxdb\/influxdb.conf<\/code><\/p>\n\n\n\n[http]
https-enabled = true
https-certificate = \"\/etc\/influxdb\/server.crt\"
https-private-key = \"\/etc\/influxdb\/server.key\"<\/code><\/p>\n\n\n\nsudo systemctl restart influxdb<\/code><\/p>\n\n\n\nNagios<\/h1>\n\n\n\n
sudo vi \/etc\/httpd\/conf.d\/nagios.conf<\/code><\/p>\n\n\n\nsudo systemctl restart httpd<\/code><\/p>\n\n\n\nPrometheus<\/h1>\n\n\n\n
sudo vi \/etc\/httpd\/conf.d\/prometheus.conf<\/code><\/p>\n\n\n\nListen 9443
NameVirtualHost *:9443
<VirtualHost *:9443>
SSLEngine on
SSLProxyEngine On
ProxyRequests Off
ProxyPreserveHost On
SSLCertificateFile \/etc\/pki\/tls\/certs\/localhost.crt
SSLCertificateKeyFile \/etc\/pki\/tls\/private\/localhost.key
ProxyPass \/ http:\/\/grafana.makernet.local:9090\/
ProxyPassReverse \/ http:\/\/grafana.makernet.local:9090\/
<\/VirtualHost><\/code><\/p>\n\n\n\nsudo systemctl restart httpd<\/code><\/p>\n\n\n\nLoki<\/h1>\n\n\n\n
sudo vi \/etc\/httpd\/conf.d\/loki.conf<\/code><\/p>\n\n\n\nListen 9100
NameVirtualHost *:9100
<VirtualHost *:9100>
SSLEngine on
SSLProxyEngine On
ProxyRequests Off
ProxyPreserveHost On
SSLCertificateFile \/etc\/pki\/tls\/certs\/localhost.crt
SSLCertificateKeyFile \/etc\/pki\/tls\/private\/localhost.key
ProxyPass \/ http:\/\/localhost:3100\/
ProxyPassReverse \/ http:\/\/localhost:3100\/
<\/VirtualHost><\/code><\/p>\n\n\n\nsudo systemctl restart httpd<\/code><\/p>\n\n\n\nSSL Validation<\/h1>\n\n\n\n