First I apologize for the brevity of this post. I hadn’t planned on doing a post but this turned into a lot of steps that are not put together in one place. These are more or less notes for me to jog my memory if I ever need to do this again. If I have to do this again, I will definitely do a more detailed post.
So I have been setting up a new monitoring solution (prometheus/alertmanager/grafana) and I wanted to relay alerts through Gmail. This wasn’t as straight forward a process as I had hoped.
I wanted to do things as securely as possible so I have it locked down to IP address and I created a "Service Account" email account for relaying mail. So there were a couple of gotchas.
First, you need to log into webmail and accept the End User License agreement. This hung me up for a spell.
Second, you need to enable 2 Factor authentication so you can create app passwords. Rightfully Google sees the SMTP agent as an unauthorized device if you attempt to use the same password you do to log into the web client.
Lastly I have been experiencing some TLS/SSL issues communicating with Postfix locally. For now AlertManager is connecting to the local Postfix without TLS. To relay mail to Google, Postfix connects securely though. This was acceptable for us at this time.
I also had to make modifications to the startup scripts for prometheus and alertmanager to make use of the NGINX reverse proxy so I can get authentication along with a Let’s Encrypt certificate that encrypts everything. This in turn required a change to prometheus for metrics collection and the Grafana data source so I could graph the collected metrics.
Again I apologize for the vagueness here, this is a basic guideline to help kick start things for me.
Links that where helpful: