Retries
Automatic delivery retries with exponential backoff, per-channel throttling, delivery statuses, the 24-hour delivery window, and failure notifications for failed deliveries.
Alphorn automatically retries failed notification deliveries with exponential backoff. No configuration needed - retries are built in.
Retry Behavior
Each delivery (one notification to one channel) gets up to 5 attempts: the first send plus up to 4 retries. The delay doubles each time, with random jitter:
| Retry | Delay after the previous attempt |
|---|---|
| 1st | 30-60 seconds |
| 2nd | 1-2 minutes |
| 3rd | 2-4 minutes |
| 4th | 4-8 minutes |
In practice, a delivery is either delivered or marked as failed within about 15 minutes.
What Counts as Failure
Retried (temporary failures):
- HTTP 5xx responses
- HTTP 429 (rate limited by the receiving service)
- Network errors, connection refused, DNS failures
- Timeouts - each HTTP request to a channel times out after 15 seconds (the Email (SMTP) channel: 15 seconds to connect, 30 seconds of inactivity)
Not retried (permanent failures):
- HTTP 4xx responses other than 429 - they indicate a configuration problem (wrong API key, invalid URL, etc.) that retrying won't fix
- A stored channel configuration that is no longer valid
- Email (SMTP): rejected credentials, invalid sender or recipient address, and any SMTP 5xx response
- Some provider-specific errors, e.g. Vonage reporting invalid credentials, missing or invalid parameters, or an invalid sender
A permanent failure stops immediately and triggers a failure notification.
Per-Channel Throttling
Some services limit how fast you can post. Alphorn caps deliveries per channel to stay under those limits:
| Channel | Limit |
|---|---|
| Slack | 60 per minute |
| Discord | 50 per minute |
| Telegram | 20 per minute |
When a channel is over its limit, the delivery is simply rescheduled for the next window. Throttling does not use up a retry. The limit applies per channel and is shared across all workers.
Delivery Window
Alphorn never sends a delivery that is more than 24 hours old. This protects against flooding a channel with stale alerts after a long outage.
A background sweep runs every minute and recovers deliveries that got stuck - for example after a worker crash or restart:
- Deliveries that were never picked up are queued again, at most every 15 minutes per delivery - in practice the first time about 20 minutes after they were created
- Failed deliveries with attempts left whose retry got lost (e.g. after a crash) are queued again after 10 minutes
- Deliveries not picked up after 5 minutes are marked Stale. They are still sent if a worker picks them up within 24 hours
- Deliveries stuck in Processing for 10 minutes are marked Failed and queued again
- After 24 hours, undelivered Stale deliveries get the error "Expired: undelivered after 24 hours" and are not sent anymore
If a channel is disabled or deleted while deliveries are still waiting for a retry, those deliveries are set to Failed with the error "Channel disabled" and not retried. No failure notification is sent for them.
Deliveries that expire after 24 hours don't trigger a failure notification either.
Delivery Status
Each delivery has a status, shown on the message detail page:
| Status | Description |
|---|---|
| Pending | Waiting to be sent, or waiting for the next retry after being throttled |
| Processing | A worker is sending it right now |
| Delivered | Successfully accepted by the channel |
| Failed | The last attempt failed. Retried automatically while attempts remain; final when all 5 attempts are used or the error is permanent |
| Stale | Not picked up by a worker in time. Still sent if a worker picks it up within 24 hours |
Delivery Logs
Open a message under Messages to see one row per channel with:
- Status
- Number of attempts
- The last error message (error text from the channel is capped at 500 characters)
Admins and owners can click Resend on a delivery that is not pending or processing to queue it again. Deliveries older than 24 hours can't be resent. Resend doesn't reset the attempt counter: a delivery that already used all 5 attempts gets exactly one more try.
Failure Notifications
You can get alerted when a delivery fails for good - for example, send a notification to a backup channel when your primary Slack channel stops working.
- Go to Settings → Failure Notifications
- Pick the channel that should receive the alerts
- Click Save settings
A failure notification is sent when a delivery fails permanently or uses up all 5 attempts. It contains the webhook name, the channel name and type, the message title, the error, and the number of attempts. Note: after a permanent failure the attempt counter is set to 5 (so it isn't retried), so it shows 5 even if only one attempt was made.
If the failure notification channel itself is the one failing, no alert is sent (to avoid loops). The same happens silently when the failure notification channel is disabled or deleted. Pick a channel on a different service than your main channels.
Only admins and owners can change this setting.
Filtering Rules
Filter and route notifications per webhook and channel by priority, tags, title, message, or JSON payload fields - operators, AND/OR groups, regex, and testing filters.
Real-time Streaming
Stream real-time notifications via Server-Sent Events (SSE) with the Live Stream channel - stream URL, JSON and text event formats, connection limits, and standalone SSE server setup.