Alertmanger manger helps to Send the Notification for any of the Alert generated for your server to either, email, Slack, Hipchat, etc. It is Configured from the command line and has certain configuration files.

In one of the Previous blog, we have seen step by step process of setup and configurations of Prometheus, Node Exporter, and CAdvisor. We will be using the same docker compose file and will be appending it with the Alert manager Service. You Can refer the Previous blog here. We will be doing the following,

1. Appending the Alert Manager Service in the Docker compose file
2. Writing an alertmanager.yml configuration file which will have all the necessary details of the receiver, like slack channel, api_url, Aler Message, etc.
3. Writing alert.rule file: Here we will be defining our alerts expression/conditions and defining the threshold, So once the Server Statistics is identified beyond this, the Alarm will be triggered to the Slack.

Let’s get Started with the Steps now,

You need to have a slack account, If not Just make it from the Slack Website. Create a channel of your preferred name. Now once the channel is created we have to Create an Incoming Webhook for that, Open below link,
https://your_workspace.slack.com/apps

Click On Add Configurations

Now, Select the channel from the dropdown list where you wanted to post your Alert Notification and ClickAdd incoming Webhooks integrations.

Once you Clicked on Add incoming Webhooks integrations, You will get your Webhook Url which you need to use in order to push Notification to that particular channel. There will also be a Setup Instructions.

We're sorry but levi-frontend doesn't work properly without JavaScript enabled. Please enable it to continue. Click Account Settings in the left hand navigation, and scroll down to the Notifications section. Click the plug icon next to Slack. The Docker Hub page refreshes to show a Slack authorization screen. On the page that appears, double check that you’re signed in to the correct Slack team.

Just make a copy of webhook Url to be used in the alertmanager.yml file. After you add the Integration, there will also be a notification you’ll receive in the channel, “added integration to this channel: “

Join the #docker room in the Splunk Slack channel. If you're a new Splunk customer you can register for Slack here; If you are a Splunk Enterprise customer with a valid support entitlement contract and have a Splunk-related question, you can also open a support case on the https.

Now, Let’s Spin up the Alert Manager Container, Add below piece of yaml code in the docker compose file. Get the Complete set of code from the devopsage official repository.

Now Create a file named, alertmanager.yml and paste the below yaml content.

Append prometheus.yml file and paste the belowyml code at the last.

Eventually, create Alert rule now, A sample Alert Rule you can find it from the DevOpsAge Github Repository. Entire Alert along with the threshold need to written in alert.rules. Based on your requirement it could be very lengthy. A sample format is as below,

The last thing we have to do is to add the rule_file to the Prometheus Configuration file. Add below piece of code to the prometheus.yml. Check here for reference.

Also, don’t forget to make changes to the docker compose file for the Prometheus service under the volume section. Add,

– ./alert.rules:/etc/prometheus/alert.rules

Check the complete docker compose file here for your reference.

Now, recreate the entire containers from the docker container file and you should be good to go.

Verify, whether all the containers are running or not, If by any chance any container fails or get exited, just check the logs and try to troubleshoot it.

docker logs container_id

Now check whether the rules are set and configured correctly or not. Just open Prometheus (Ip:9090)

From the Status tab Click Rules and you will be able to see the alert rules which you have set as shown in the screenshot below

Now, Intentionally bring down the CAdvisor Container to check whether it identifies the Alert and Sends Notification to the Slack Channel or Not.
Stop CAdvisor Container.

Wait for Some time, and If you configured everything correctly then you will get Notified to Slack for Sure.

Here, We are getting Alert Notification first on Slack Channel and Once We start the Container Again We will be notified as Resolved.

Here once the Alert is firing, we can not see any Summary and Description, It’s Probably because we have stopped the CAdvisor Container itself.
Let’s test It With a test container and see whether the notification comes along with the Summary and Description or not.

So We Can See that the Notification Comes with the Summary and Description for the Test Container.
Alert may take up to 5 minsto Notify, So just have some patience.

Also, Check the Alert Section in the Prometheus Dashboard.

So here we can see that the Alert is in firing mode. That’s all for this particular blog, do some hands-on and test.

If you Like Our Content here at Devopsage, then please support us by sharing this post.

Please Like and follow us at, LinkedIn, Facebook, Twitter, and GitHub

Also, Please comment on the post with your views and let us know if any changes need to be done.

Thanks!!

You May Also need to look into,

Monitoring Servers and Docker Containers using Prometheus and Grafana.

Creating Dashboard in Grafana

How to Add Nodes to be Monitored in Grafana

Watchtower can send notifications when containers are updated. Notifications are sent via hooks in the logging system, logrus.The types of notifications to send are set by passing a comma-separated list of values to the --notifications option (or corresponding environment variable WATCHTOWER_NOTIFICATIONS), which has the following valid values:

• email to send notifications via e-mail
• slack to send notifications through a Slack webhook
• msteams to send notifications via MSTeams webhook
• gotify to send notifications via Gotify
• shoutrrr to send notifications via containrrr/shoutrrr

There is currently a bug in Viper, which prevents comma-separated slices to be used when using the environment variable. A workaround is available where we instead put quotes around the environment variable value and replace the commas with spaces, as WATCHTOWER_NOTIFICATIONS='slack msteams'

If you're a docker-compose user, make sure to specify environment variables' values in your .yml file without double quotes (').

This prevents unexpected errors when watchtower starts.

Settings¶

• --notifications-level (env. WATCHTOWER_NOTIFICATIONS_LEVEL): Controls the log level which is used for the notifications. If omitted, the default log level is info. Possible values are: panic, fatal, error, warn, info, debug or trace.
• Watchtower will post a notification every time it is started. This behavior can be changed with an argument.

Available services¶

Email¶

To receive notifications by email, the following command-line options, or their corresponding environment variables, can be set:

• --notification-email-from (env. WATCHTOWER_NOTIFICATION_EMAIL_FROM): The e-mail address from which notifications will be sent.
• --notification-email-to (env. WATCHTOWER_NOTIFICATION_EMAIL_TO): The e-mail address to which notifications will be sent.
• --notification-email-server (env. WATCHTOWER_NOTIFICATION_EMAIL_SERVER): The SMTP server to send e-mails through.
• --notification-email-server-tls-skip-verify (env. WATCHTOWER_NOTIFICATION_EMAIL_SERVER_TLS_SKIP_VERIFY): Do not verify the TLS certificate of the mail server. This should be used only for testing.
• --notification-email-server-port (env. WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT): The port used to connect to the SMTP server to send e-mails through. Defaults to 25.
• --notification-email-server-user (env. WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER): The username to authenticate with the SMTP server with.
• --notification-email-server-password (env. WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD): The password to authenticate with the SMTP server with. Can also reference a file, in which case the contents of the file are used.
• --notification-email-delay (env. WATCHTOWER_NOTIFICATION_EMAIL_DELAY): Delay before sending notifications expressed in seconds.
• --notification-email-subjecttag (env. WATCHTOWER_NOTIFICATION_EMAIL_SUBJECTTAG): Prefix to include in the subject tag. Useful when running multiple watchtowers.

Example:

The previous example assumes, that you already have an SMTP server up and running you can connect to. If you don't or you want to bring up watchtower with your own simple SMTP relay the following docker-compose.yml might be a good start for you.

The following example assumes, that your domain is called your-domain.com and that you are going to use a certificate valid for smtp.your-domain.com. This hostname has to be used as WATCHTOWER_NOTIFICATION_EMAIL_SERVER otherwise the TLS connection is going to fail with Failed to send notification email or connect: connection refused. We also have to add a network for this setup in order to add an alias to it. If you also want to enable DKIM or other features on the SMTP server, you will find more information at freinet/postfix-relay.

Example including an SMTP relay:

Slack¶

If watchtower is monitoring the same Docker daemon under which the watchtower container itself is running (i.e. if you volume-mounted /var/run/docker.sock into the watchtower container) then it has the ability to update itself. If a new version of the containrrr/watchtower image is pushed to the Docker Hub, your watchtower will pull down the new image and restart itself automatically.

To receive notifications in Slack, add slack to the --notifications option or the WATCHTOWER_NOTIFICATIONS environment variable.

Additionally, you should set the Slack webhook URL using the --notification-slack-hook-url option or the WATCHTOWER_NOTIFICATION_SLACK_HOOK_URL environment variable. This option can also reference a file, in which case the contents of the file are used.

By default, watchtower will send messages under the name watchtower, you can customize this string through the --notification-slack-identifier option or the WATCHTOWER_NOTIFICATION_SLACK_IDENTIFIER environment variable.

Other, optional, variables include:

Docker Slacks Sale

• --notification-slack-channel (env. WATCHTOWER_NOTIFICATION_SLACK_CHANNEL): A string which overrides the webhook's default channel. Example: #my-custom-channel.
• --notification-slack-icon-emoji (env. WATCHTOWER_NOTIFICATION_SLACK_ICON_EMOJI): An emoji code string to use in place of the default icon.
• --notification-slack-icon-url (env. WATCHTOWER_NOTIFICATION_SLACK_ICON_URL): An icon image URL string to use in place of the default icon.

Example:

Microsoft Teams¶

To receive notifications in MSTeams channel, add msteams to the --notifications option or the WATCHTOWER_NOTIFICATIONS environment variable.

Additionally, you should set the MSTeams webhook URL using the --notification-msteams-hook option or the WATCHTOWER_NOTIFICATION_MSTEAMS_HOOK_URL environment variable. This option can also reference a file, in which case the contents of the file are used.

Docker Slack Channel

MSTeams notifier could send keys/values filled by log.WithField or log.WithFields as MSTeams message facts. To enable this feature add --notification-msteams-data flag or set WATCHTOWER_NOTIFICATION_MSTEAMS_USE_LOG_DATA=true environment variable.

Example:

Gotify¶

To push a notification to your Gotify instance, register a Gotify app and specify the Gotify URL and app token:

-e WATCHTOWER_NOTIFICATION_GOTIFY_TOKEN or --notification-gotify-token can also reference a file, in which case the contents of the file are used.

If you want to disable TLS verification for the Gotify instance, you can use either -e WATCHTOWER_NOTIFICATION_GOTIFY_TLS_SKIP_VERIFY=true or --notification-gotify-tls-skip-verify.

To send notifications via shoutrrr, the following command-line options, or their corresponding environment variables, can be set:

• --notification-url (env. WATCHTOWER_NOTIFICATION_URL): The shoutrrr service URL to be used.

Go to containrrr.github.io/shoutrrr/services/overview to learn more about the different service URLs you can use.You can define multiple services by space separating the URLs. (See example below)

You can customize the message posted by setting a template.

• --notification-template (env. WATCHTOWER_NOTIFICATION_TEMPLATE): The template used for the message.

The template is a Go template and the you format a list of log entries.

The default value if not set is {{range .}}{{.Message}}{{println}}{{end}}. The example below uses a template that also outputs timestamp and log level.

Example: