HetrixTools can deliver your alerts to ntfy, either through the public ntfy.sh server or through a ntfy server that you host yourself.

This article explains why notifications sent through the public ntfy.sh server may occasionally arrive late or not arrive at all, and what you can do about it.

Why notifications can be missed on the public ntfy.sh server

The public ntfy.sh server is a free, shared service. To keep it usable for everyone, it applies strict rate limits to every sending source: a cap on how many messages a source can publish within a short period, and a cap on how many it can publish per day. Once a limit is reached, ntfy.sh rejects any further messages from that source until the limit resets.

HetrixTools delivers ntfy notifications on behalf of all of our users who chose ntfy.sh, from a shared infrastructure. From the point of view of ntfy.sh, all of those notifications come from the same few sending sources, so the limits are shared across all HetrixTools users publishing to ntfy.sh, not applied per user. During busy periods, for example, when a widespread outage triggers many alerts at once, or later in the day once the daily allowance has been used up, ntfy.sh can reject notifications regardless of how few alerts your own account generates.

We retry rejected notifications for a short while, but if ntfy.sh keeps refusing them, the notification is eventually dropped and will not be delivered. These limits are enforced by ntfy.sh itself and are outside of our control. You can read more about them in the ntfy documentation: https://docs.ntfy.sh/publish/#limitations

What you can do

  1. Host your own ntfy server (recommended)
    ntfy is open source and can be self-hosted. A self-hosted server shares no limits with other HetrixTools users: its limits are yours to set, and by default it does not cap the number of messages per day. Once your server is running, simply use its URL as the ntfy URL in your HetrixTools contact list. Installation instructions are available at https://docs.ntfy.sh/install/
  2. Use a paid ntfy.sh plan with an access token
    If you prefer to stay on ntfy.sh, a paid ntfy.sh plan comes with its own message allowance. Add the access token from your paid ntfy.sh account to the ntfy settings of your HetrixTools contact list, and ntfy.sh will count your notifications against your own allowance instead of the shared limits. Please note that an access token from a free ntfy.sh account does not change how the limits are applied.
  3. Add a backup notification channel
    For critical alerts, we recommend adding at least one more notification channel to your contact list, such as email, SMS, Telegram, Slack, or any of our other integrations, so that an important alert still reaches you if a ntfy.sh notification is dropped.

We have taken steps on our side to reduce how often this happens, but as long as notifications go through the public ntfy.sh server, missed notifications cannot be ruled out entirely.

Tagged: