Automated Health Checks

We have 200+ notifications and send 1M+ emails every month. Notifications are a crucial part of our business. That’s why, we want to make sure they get delivered.

What is this check?

There is an automated health check for the notification module. It creates a test email, queues and sends it through our notification system, and then checks if that email is delivered to the recipient address using the mailbox reader.

The system goes through this check a few times every hour, apx. once every 8 min. If it fails, it notifies the developers via Email and Slack. If the problem persists long enough, it notifies the developers via SMS.

How does it check?

The system sends the dummy email to support-box@interaction-design.org. It also connects to this email’s mailbox using Gmail API to check if the email is delivered.

Why does it check?

This check behaves like an end-to-end test. It allows us to make sure our emails get delivered without thinking much about the state of the notification system or the transactional email provider.

It executes based on a reasoning: If developers don’t get notified about a failure, then it means we don’t have problems delivering our 1M+ emails every month.

Note about support-box@interaction-design.org

This mailbox gets a copy of every email sent to rikke.friis.dam@interaction-design.org which is our support email address. It is also used to check whether certain people responded to certain notifications we sent them. Based on this check, some notifications are not triggered.

This fact shows that this mailbox is used for two different domain logic, and perhaps it’s wise to separate them into two different addresses in the future.