Alerting
The server records what happens to it as events, and a rule decides which of them leave the machine through which channel. Nothing has to be polled and nothing has to be asked: a failed job, a container that stopped on its own, a filesystem filling up and a license about to lapse are all events the moment they happen.
The three pages are under Alerting in the console and are Administrator only - a rule names channels, and a channel holds a credential.
Events happen whether or not you can send them
Recording events is never licensed. Neither is the console's own bell, nor uptime monitoring. A server with no channel at all still shows everything that happened to it.
What is licensed is delivery to somewhere else.
So a Community, Developer or unregistered server still records and shows everything; only sending it elsewhere needs the edition. What happens below Professional is further down this page - it is not a refusal.
What raises an event
Thirty-eight event types across thirteen areas. The authoritative list is the rule editor itself, and on the command line:
pelmo$ ohm server alerts types --server pelmo| Area | What it covers |
|---|---|
| Jobs | A job that failed, a job interrupted by a restart, a backup or restore that finished or failed, backup encryption switched off on an instance that arrived here, a certificate renewal that failed, a server update that finished, an image signature refused, image signing not set up, and the weekly Backup storage report |
| Containers | An instance container that stopped or went unhealthy outside a job, and a core service that stopped |
| Uptime | An instance unreachable, reachable again, or answering with a gateway error, and a certificate close to expiry |
| Resources | Sustained CPU, memory or I/O pressure, a filesystem below its free-space boundary - one alert per filesystem, naming the server's directories on it - and an instance over its storage threshold |
| Health | An instance whose health check found something wanting attention, resolved when every check passes again, and the full health report every run |
| Security | An account locked out after repeated failed sign-ins, a user account created, changed or deleted, and an untrusted proxy in front of the console, the API or an instance |
| License | A license close to expiry, every license change - features lost or gained, a lower limit, the license server out of reach - and an instance set up for more than the license carries |
| Executor | The container event relay silent past its grace period |
| Database | A slow statement, while it stays slow and when it is back under the threshold |
| Git | A run of the automatic update of an instance's add-ons from git: the pins moved, or why it failed |
| A failed mail-test verdict. In the catalogue, but nothing raises it yet: a rule may select it, and it will never match | |
| Alerting | A channel paused after too many failures, a notification a channel could not deliver (under the bell only), and the synthetic event a channel test sends |
| Backup storage | A backup storage account's periodic check failing or passing again (under the bell only) |
The two marked "under the bell only" are never matched by a rule.
The dotted identifiers (job.failed, instance.http.down) are what a webhook receiver switches on.
A name is added to the catalogue but never rewritten, so a receiver written today keeps working.
An untrusted proxy
When a proxy, load balancer or tunnel agent forwards requests to the console or the management API without being listed under Configuration > Network > Trusted proxies, everybody behind it signs in, is rate-limited and is recorded as the proxy. A few mistyped passwords by anyone then refuse every sign-in for everyone, and new-device mails cannot say where a sign-in came from.
The same goes for the instances. A CDN, a cloud load balancer or a tunnel in front of an instance's site that is not listed makes Odoo, the instance's access rules, the rate limits, Fail2Ban and CrowdSec see every visitor as the proxy: Odoo's own sign-in throttle and a ban then lock out everybody behind it at once, or a ban never comes.
The server raises the Untrusted proxy event (security.proxy.untrusted) when it sees this:
- Critical, once, when the proxy is a program on the server itself (a loopback address, or the address the server's web server sees for a program on the server that reaches port 443). It is not sent again for that proxy, a restart included.
- A warning for any other proxy on your network.
- A warning for a proxy on the internet in front of an instance, once it forwarded at least 50 requests within a day and at least 90% of its requests carried a forwarding header. A visitor that sends the header now and then is not a proxy and raises nothing.
- A warning once a day while the proxy is still seen.
- Resolved a day after the proxy was last seen.
Administrators also see a notice on every console page while a proxy was seen in the last 24 hours: which part saw it (the console, the management API or an instance), its address, how many requests it forwarded, and when it was first and last seen. An instance's proxy is also shown on that instance's Configuration tab beside its Trusted proxies field, and on the Fail2Ban and CrowdSec tabs of Security > Threat protection, where a ban of that address would lock out everybody behind it.
Trust this proxy on the notice adds the address to Configuration > Network > Trusted proxies for the whole server and applies it, the same as typing it there. Nothing is trusted on its own. An instance with its own Trusted proxies list keeps that list; add the proxy there instead. The notice cannot be dismissed; it goes away once the proxy is trusted, or a day after it stops.
A server with no alert channel gets the event in the bell and in the alert list, and the notice; nothing is mailed around the rules.
Severity, and what resolves
| Severity | Meaning |
|---|---|
info | Something happened worth recording |
warning | Something needs attention before it becomes an outage |
critical | Something is broken now |
resolved | An earlier opening event no longer holds |
Many events are paired: the one that opens a condition is matched later by a resolution through a correlation key. An instance that goes unreachable and comes back produces an opening and a resolution, not two unrelated lines. A resolution always follows the rule that delivered the outage it closes, so you cannot be told about a failure on one channel and its recovery on another.
Channels
A channel is where messages go. Four kinds:
| Kind | What it needs |
|---|---|
Nothing of its own - it sends through the SMTP profile already on the server. Name recipients with --to, or leave it and the profile's own default is used | |
| Slack | An incoming webhook URL |
| Discord | A webhook URL |
| Webhook | Your own HTTPS endpoint, with a signing secret and any extra headers |
pelmo$ ohm server alerts channels list --server pelmopelmo$ ohm server alerts channels test --server pelmo --name "Ops room"A test delivers the synthetic alert.test event, which no rule ever matches, so it proves the
channel without waiting for something to go wrong.
Credentials are write-only
The URL, the signing secret and any extra headers are stored encrypted and are never returned - not by the API, not on the page. A read tells you only that each is set, plus the scheme and host of the target.
That covers the whole URL rather than a password field, because an incoming-hook URL is the credential: anyone holding a Slack hook URL can post to that channel.
Typing --url or --secret with no value prompts for it without echo, so a credential need not go
into your shell history:
pelmo$ ohm server alerts channels add --server pelmo --kind slack --name "Ops room" --urlA webhook target is checked, twice
Before it is stored and again before every attempt:
- HTTPS only
- No user name or password in the URL
- A fully qualified public name
- No address that is private, loopback, link-local or otherwise not routable from the internet
The server resolves the name itself and then pins the connection to the address it checked. A name that answers with a public address during validation and a private one at send time cannot be used to reach inside your network. A refusal names the reason it refused.
Proving a request came from your server
Every generic webhook carries three headers:
| Header | What it holds |
|---|---|
X-Antelao-Event | The event type |
X-Antelao-Timestamp | When the request was signed |
X-Antelao-Signature | sha256= followed by an HMAC-SHA256 over the timestamp and the exact body sent |
The key is the channel's signing secret, which only you and the server hold. Verify over the raw body, not a re-serialised copy of it - a JSON library that reorders keys or changes spacing produces a different signature for the same content.
Slack and Discord channels are not signed: their hook URLs are unguessable secrets, which is the scheme those services provide.
Rules
A rule says which events, above which severity, for which instances, go to which channels.
pelmo$ ohm server alerts rules list --server pelmopelmo$ ohm server alerts rules add --server pelmo --name "Outages to ops" --channel "Ops room" --type instance.http.down --min-severity critical --cooldown 15| Field | Effect when left empty |
|---|---|
| Event types | Every type matches |
| Minimum severity | Every severity matches |
| Instances | Every instance matches, server-wide events included |
| Channels | Nothing is delivered - a rule with no channel matches and goes nowhere |
| Cooldown | No suppression |
The cooldown is per correlation key, not per rule. An instance that flaps does not send the same message every minute, while a different instance failing in the same minute still gets through. The ledger is held in memory, so a restart of the API costs at most one extra notification per subject.
A rule is the only thing that sends. Until one exists, events accumulate and the bell shows them, and nothing leaves the server.
You do not have to write the first rules
Creating the first e-mail channel on a server that has no rules installs the recommended ones: failures and outages, capacity and expiry, security (an untrusted proxy included, once a day while it lasts), backups that ran, and a net that catches anything critical.
The cooldowns carry the judgement rather than being uniform - none at all on a failure, hourly or daily on a condition that persists. That is the part worth copying if you go on to write your own: a rule that repeats a standing condition every minute gets muted by whoever receives it within a day.
Add the same set whenever you want it, from Add the recommended rules on the Rules page or from the command line, against any channel you name:
pelmo$ ohm server alerts rules defaults --server pelmo --channel "Ops room"Only the names you do not already have are added. A rule you deleted stays deleted, and a server that already has rules is never seeded behind your back.
When the far end stops answering
A delivery gets five attempts on a widening back-off: one minute, five, fifteen, an hour, four
hours. A target that asks for a delay with Retry-After gets it, capped at four hours - Slack and
Discord answer 429 that way, and an uncapped value from a misbehaving target would park a delivery
indefinitely.
Only failures worth retrying are retried: 429, any 5xx, and a transport failure where nothing answered at all. Any other 4xx is definitive - a 404 or a 401 will answer the same way in four hours.
A delivery that has failed for good says so under the bell, naming the channel, the event it could not deliver and why - a mail that could not be sent included, so a failed backup whose mail never left is still seen. One entry per channel, replaced by the next failure. It is never matched by a rule: delivering it to the channel that just failed would never end.
After ten consecutive failures the channel is paused. An alert says so, and nothing further is attempted until you resume it:
pelmo$ ohm server alerts channels resume --server pelmo --name "Ops room"Resuming queues the last hour of paused messages again and leaves older ones alone. Replaying a weekend of accumulated alerts into a chat room helps nobody.
The delivery log
Every attempt: what answered, how long it took, and when the next one is due.
pelmo$ ohm server alerts deliveries --server pelmo| Status | What it means |
|---|---|
queued | Waiting for its next attempt |
delivered | The sink accepted it |
failed | Every attempt was spent, or the failure was not worth retrying |
paused | The channel was paused before this row could be attempted again |
skipped | Never attempted, deliberately - the error says why, and an unlicensed server is the common case |
A row you have fixed the cause of can be sent again rather than waited out:
pelmo$ ohm server alerts deliveries --server pelmo --status failed --retry <id>skipped is the one to know. A server without alert delivery in its license does not fail a
delivery silently and does not pretend it sent: it records the row with the reason. Nothing about
why you received no message is left to be guessed at.
Below Professional, nothing is sent and the log says so
A Community, Developer or unregistered server therefore does not go quiet and does not pretend.
Every message it would have sent is a skipped row naming the license as the reason, so the answer
to "why did nobody get paged" is on the page rather than in a support ticket.
That includes the License changed alert itself. When a license that carried delivery is replaced by one that does not, a subscription that ended above all, the alert naming what was switched off reaches the bell only, and the banner over every page is what tells you. See When the license changes.
Reading the events themselves
pelmo$ ohm server alerts events --server pelmoEvents are kept for 90 days and delivery rows for 30. Both are settings rather than constants; see Configuration and state.
The bell
The bell lists completed and failed jobs as well as alerts, and it fills itself from the server on connect - so a console opened after an overnight outage shows what happened during it, rather than nothing because no browser was listening at the time.
While a job you started is running, the bell shows a spinner; hover it for what is running. Its unread count rises for a failure, an alert and the outcome of a job you started yourself - a job that succeeds somewhere else on the server is listed, but does not ask for your attention. The success notice of your own job appears under the bell for the time set in Success notices on the Interface tab of My account.
The notices under the bell are coloured by what they say, each with its own icon and word so the colour is never the only sign:
| Notice | Colour and icon | Stays |
|---|---|---|
| Success | green, a check mark in a circle | the time set in Success notices |
| Information | blue, an i in a circle | six seconds |
| Warning | amber, a warning triangle: a job that finished with a warning, or was cancelled, a high usage | six seconds |
| Error | red, a cross in a circle: a job that failed, a critical alert | until you close it |
A failed job you started is also shown on the page, where you started it; the red notice is in addition to that message, not instead of it.