Troubleshooting
Organised by what you see, not by what is broken.
First moves
$ sudo ohm server info$ sudo docker ps$ sudo docker logs ohm-api --tail 200For anything that was started from the console or by a schedule, the Jobs page is a better starting point than any log: every operation captures its own.
Core services is the second place to look. It reports the agent, the API, and the GeoIP and fail2ban services, with the build each one is running - which is where a dispatch that refuses to start explains itself.
Recording what you did
A bug report describes a problem from memory. A session trace records it.
Start one on the Support page, reproduce the problem, and stop it with the recording button that appears beside your name. What you did is stored on the server and can be attached to a bug report.
A trace lists the pages you opened, the controls you used, and anything that went wrong. It can also save a copy of each page as it looked, which is the only thing that answers it rendered wrong.
A trace records which field you filled, never what you put in it, and the places that show a secret are blanked before anything is written down. A copy of a page still shows whatever that page showed - names, host names, addresses - so read a trace before you send it, as you would a log or a support bundle.
A trace belongs to the tab it was started in, and it names the user it belongs to, so a file can be read back to a person.
What an administrator decides
The Support page carries a Configuration card, and what is set there is what every user gets: whether the recording button sits beside your name at all, how much a trace writes down, and how many traces and bundles the server keeps. Lowering a keep count deletes the files above it as soon as the card is saved.
The page itself is for administrators. Every user, a read-only one included, can still record a trace with the recording button, without opening the page.
Attaching it
Download the trace from the Support page and send it, with a support bundle, to whoever asked for it. Never attach either to a public post: see Where to ask, and where a bundle goes.
The support bundle
Support collects the facts an investigation needs into one archive, which you download and attach where support asks for it.
Where to ask, and where a bundle goes
| Your server runs | Ask on | A bundle or a trace |
|---|---|---|
| Community, Developer, or no license (the free editions) | The community forum on antelao.io, like any free product. Describe what you did and what you saw, and quote the error | Never on the forum. It is public |
| Professional or Provider | Submit from the Support page, or write to support@antelao.io | Attach it to the submission, which keeps it private, or send it privately to support@antelao.io |
A bundle and a session trace carry your configuration, your instances' host names and log lines with your visitors' addresses. Never post a bundle or a trace publicly, not on the forum, not in a public issue, whatever your edition.
Read a bundle before you send it. It takes out what it recognises as a secret or a person (see What a bundle carries), but that redaction is best effort and not a guarantee: what is left can still hold personal data, configuration and identifiers, and removing anything more before you send it is yours to do. Article 10.3 of the license agreement says the same.
Bundles are listed on the same page and can be deleted from it: each row has a delete action behind a confirmation that names the file and its size. The removal runs as a job like every other change, so the row goes when the job reports the file is actually gone rather than when the click lands.
Several bundles, or several traces, are handled at once by selecting them: click a row, Ctrl-click another, Shift-click to take a range. The bar above the list then offers Download, which saves each selected file in turn - your browser may ask once whether this site may download several files - and Delete, which names every selected file and their total size before anything goes. Selected bundles are deleted in one job, so the audit trail shows one deletion of several bundles.
Worth doing. A bundle is a large archive of your own configuration, and there is no reason to keep last quarter's on the server. Each one names who asked for it; one collected by a ticket rather than by hand names nobody, and says so.
A bundle also describes the machine, not just the product: the distribution and kernel, uptime and load, processors and memory, whether it is physical or virtual and of what kind, every installed package, the file and watch limits, which of the product's services are set to start at boot, and how the container engine is configured. It records the difference between the two clocks involved, which explains a class of problem nothing else in a bundle would.
The time window
A bundle covers a time window. Covers beside Create bundle offers the last 24 hours, the last 48 hours (the default) and the last 7 days, or A window I choose with a start and an end on your own clock, up to 31 days long. Pick the hours of the incident and a bundle pulled the day after still reaches it. A bundle a ticket collects covers the last 48 hours.
From the command line, ohm server support-bundle --server <name> does the same: --hours 72, or
--from 2026-10-07T16:30Z --to 2026-10-07T17:00Z, and --output <file or folder> to download the
finished bundle.
What a bundle carries
Everything a bundle needs to answer why was this visitor refused at this time on its own:
| Part | For the window | What it holds |
|---|---|---|
| Web server logs | Yes | Every instance's access and error log, the access and error log of the consoles published on port 443, and the server-wide error log: the status, whether the request reached Odoo or the console, and whether the firewall check could be reached at all |
| Web server configuration | As it is now | The web server's main configuration, its log format, every instance's site with its access rules and the site of the consoles published on port 443, with when each was last written and the jobs in the window that rewrote them |
| Firewall decisions | Yes | The firewall log's rows in the same columns as its Export, so a bundle row and an export row can be laid side by side |
| Firewall configuration | As it is now | Every instance's rules as its Rules tab shows them, the host names in them with the addresses they resolve to, its Pre-authentication settings and exemptions, the management API's and console's own rules, and the GeoIP database's status and build date |
| Audit trail | Yes | Who saved or applied what, and when, in the same columns as the audit trail's export, newest first by time; each person is a stand-in that stays the same within the bundle |
| Bans | In force and expired | CrowdSec's decisions and alerts, the Fail2Ban jails and their bans, and the pre-authentication sign-in locks in force, each with where it came from, its scenario or jail, its start and its expiry |
| Container logs | Yes | The agent's, the API's and the console's logs and those of every service the server runs - the web server, PostgreSQL with its standby and connection pooler, the mail relay, GeoIP, Fail2Ban, CrowdSec, pgAdmin, the analytics service and the SSH gateway - with the part written before an update replaced a container, and the agent's log from before its last restart; a service the server does not run is listed as absent, with why |
| Instance logs | Yes | Each instance's Odoo log, yesterday's part included, and the logs of the backups, restores and shell sessions that ran during the window |
| Instance health | The latest run | Each instance's latest health report as its Health tab shows it: every finding, what it observed and why it matters |
| Server state | As it is now | The configuration, the containers, the jobs, the agent's tasks, the versions with the exact build of every container image, and the size and free space of the disks the product uses |
An update replaces containers, and a replaced container's log would go with it. The server keeps a
compressed copy of the log of each container it replaces - the newest ten per container, for 31
days - so a bundle for the hours of an update still has them. A kept copy holds only the newest
16 MiB of that container's log, so a busy container's copy can start well after the container did.
When that leaves part of the window out, the bundle names those hours ("from ... to ... not kept")
and why, in bundle.json and in errors.txt, rather than letting the log look complete.
The host agent starts afresh on every update and every restart of its service, and no copy is kept of its old log. Instead, its log in a bundle reaches back across those restarts through the host's system journal, which holds everything the agent wrote. When the journal cannot be read, or the host keeps it in memory only and a reboot emptied it, the bundle says from when the agent's log is missing and why. The product does not change how the host keeps its journal; a journal kept on disk is what makes the agent's earlier log available after a reboot.
The web server's configuration has no history: every apply rewrites it. A bundle carries it as it is when the bundle is written, and says so, rather than guessing what it held at an earlier moment.
Every part is capped in size, and so is the bundle as a whole. When a log does not fit, its newest
part is kept and its first line says what was left out. A file named bundle.json inside the bundle
lists every part, the window it covers, its size, whether it was cut and what was redacted from it,
and every part that could not be collected and why; those are also listed in errors.txt.
Passwords, keys and tokens are replaced by *** wherever the bundle recognises them: in the configuration, in the
web server's site configuration (the firewall check's own key included), and in the query strings of
logged requests, whose paths and parameter names stay. An address with a login in it, such as an
outbound proxy given as http://user:password@proxy:3128, keeps its scheme and host and loses the
user name and password. Certificate paths stay; the keys do not.
Email addresses lose everything before the @ and keep their domain, which is what a mail problem
needs. The names and accounts of people - the console's users, their display names, git identities
and email addresses, the server's own login accounts - are taken out wherever they appear, and where
a person did something (the audit trail, who asked for a job or for the bundle) they are a stand-in
such as person-3f9a1c07 that stays the same within one bundle, so you can still see that one person
did three things. The server's own actors, such as the scheduler, stay as they are. Visitors'
addresses stay, because they are the evidence.
This is redaction by name and by pattern, so it is best effort: read the bundle before you send it.
When the product itself is broken
For a host too broken to run the product's own job, a script collects the same ground with the
operating system's own tools only - no ohm, no API, no console. It is published beside the
installer; download it, check its signature, and run it as
root on the server:
curl -fsSLO https://packages.antelao.io/raw/antelao-ohm-support-bundle.sh
sudo bash antelao-ohm-support-bundle.sh
It writes one archive under /var/tmp/antelao, readable by root alone, and prints its path. It takes
out secrets by name and the same people the product's bundle does: email addresses keep only their
domain, and the server's own login accounts are replaced. Like the product's bundle, it also masks
the user name and password in an address and the tokens in the query strings of the Odoo logs (a
password reset or sign-up link, a portal link's access token). The jobs list and the API's own view of the
configuration are in the product's bundle only.
Submitting a bug report or a feature request
On a Professional or Provider license, Submit bug report and Submit feature request on the Support page open a ticket in Antelao's helpdesk, without leaving the console. The ticket is opened for the license holder: the form names the address, and answers come by mail to it and on the customer portal at antelao.io, where you sign in with that address or the license key.
What goes with it:
- the text you write, with the product version, the server's host name and who submitted it;
- files of your own, 25 MiB together;
- for a bug report, a support bundle (a fresh one, or one you pick) and a session trace if you choose one, 45 MiB each. A bigger one stays on the server, and the ticket says so; send it when asked.
The bundle and the trace are attached privately: Antelao's support sees them, the portal does not show them. The same redaction as always applies, so read them first (see Where to ask, and where a bundle goes).
Every submission is listed under Submissions on the same page, with its ticket number and a link to it on the portal. One that failed can be sent again from its row; a resubmit goes to the same ticket rather than opening a second one.
The server proves its license with its latest check-in, so the helpdesk needs a recent one that names
the license holder's address. When the page says the check-in is missing or too old, use Check in
now on the License page and try again. The server must reach antelao.io over HTTPS, through
the outbound proxy when one is set. A server
that cannot reach it keeps the submission: send it again later, or download the bundle and send it
to support@antelao.io.
Installation
| Symptom | Cause and fix |
|---|---|
| The installer refuses to run at all | Not root, no systemd, not a Debian-family host, or not amd64. All four are requirements |
| The install stopped part-way | The log names the step. Fix the cause and run ohm install again; it repairs whatever it finds, and it is never wedged by a half-finished run |
| It stops at "Waiting for the agent" | systemctl status ohm-agent, then journalctl -u ohm-agent -n 100. A failed image pull is the usual cause: docker pull the agent image by hand and read what it says |
| An image pull is refused | Whether this host can reach the registry at all, then the registry credential in /etc/antelao/agent.env - the only place it lives after the install |
server update or server upgrade refuses to run | You ran it inside a container. It recreates the API's own container, so it must run on the host |
| Installation succeeds but no mail is sent | No SMTP profile was configured, so the Postfix container was not installed. This is a warning, not a failure |
A newer ohm behaves like the older one | Upgrading the package does not upgrade the installation. Every local command says so and names both versions; ohm install closes the gap |
The console and the API
| Symptom | Cause and fix |
|---|---|
| Every API call returns 401 | The console's key is not the one the API adopted. ohm server apikey rotate, or ohm install, then update remote clients |
| Every API call returns 503 | The API could not compose its configuration. Check GET /health for serverConfigError, then systemctl status ohm-agent and docker logs ohm-api. The composition is retried every ten seconds, so fixing the cause is enough |
| The console says the API is unreachable | ohm-api is not running or not listening on 5311 |
| The console says the server configuration is not composed | The agent is down. systemctl restart ohm-agent |
| Core services shows the agent Unreachable | /run/antelao/ohm-agent.sock is missing, or ohm-api was created without the directory mounted. Restart the agent; if it persists, ohm install recreates the management containers |
| Core services shows the agent in red with two builds | The agent and the API are different builds, and the API refuses to dispatch until they match. ohm server upgrade, or systemctl restart ohm-agent if the image was already pulled |
| A configuration change had no effect | Check Pending changes: a failed apply is listed there with its error and a retry. Some settings, bind mounts and addons paths in particular, need the container recreated rather than restarted |
| Passkey registration fails, password sign-in works | You reached the console by IP address. Use the host name: a relying-party id must be a domain |
| Nobody can sign in | Reset from the host with ohm user password-reset, user totp-disable or user unlock |
| Every sign-in says "The sign-in could not be confirmed. Try again." | Turnstile is refusing. See Locked out of the console by Turnstile |
Instances
| Symptom | Cause and fix |
|---|---|
| Certificate request fails | DNS does not point here, or port 80 is not reachable. Check with ohm server config validate-dns |
| A fresh 15.0 or 16.0 instance will not initialise a database | Those versions reject an empty addons path. Antelao omits the path entirely until a module exists; if you added one by hand, remove it |
| An instance runs but cannot write | Try instance repair filesystem |
| PDF reports come out unstyled or fail on an instance with firewall rules or pre-authentication | The database's report address points at the public address instead of the instance's own container. The instance's Health tab lists it; Auto repair sets it. See Instance health |
| Something about an instance is wrong and you do not know what | The Health tab runs every check and says what was found, why it matters and what to do; ohm instance health --check does the same from the command line |
Odoo container image for version X was not found | The image the instance runs on is not on this server. ohm instance image list --instance fanes lists the images it has: activate one with ohm instance image activate, or build it again on the instance's Docker image card |
| An instance will not start, or its visitors get the maintenance page or a 502 | Odoo is not answering. The start job on the Jobs page and the instance's Odoo log on the Logs page say why; the Health tab, or ohm instance health --check --instance fanes, checks the container, the site and the image. A start refused because as many instances run as the license allows is under Licensing. Once the cause is fixed, ohm instance container start --instance fanes |
| An instance's Let's Encrypt certificate is not renewed, or its certificate alert fires | The Certificate renewal job on the Maintenance page renews every certificate nearing expiry, and its log on the Jobs page says why one failed. The instance's names must point at this server and port 80 must be open to everyone: ohm instance config validate-dns --instance fanes checks the names. Renew one by hand with ohm instance certificate renew --instance fanes. See Certificates |
The server
| Symptom | Cause and fix |
|---|---|
| The disk is full, or a running low alert names it | The Storage page shows where the space went: instance data, backups, archives, Docker images, volumes and the build cache. Remove images an instance no longer uses (ohm instance image remove --instance fanes), run Odoo image cleanup and Docker build cache on the Maintenance page, and shorten a local backup target's retention. ohm system maintenance removes stale temporary files and old logs. On the host, df -h and docker system df say which filesystem is full and how much Docker holds. PostgreSQL stops writing when its disk is full, so treat this one first |
| Mail is not delivered, or the test message fails | Send test on the E-mail tab of Configuration says whether the relay and the credentials work. The relay sends only over TLS unless Allow unencrypted relay is on, and it does not use the outbound proxy: the server needs its own way out to the SMTP host and port. docker logs ohm-postfix --tail 200 shows what the SMTP host answered. When the test arrives and a notification does not, the license does not carry that notification: see Mail |
| Location rules place addresses wrongly, or the GeoIP databases are old | Security > GeoIP shows each database's build date and the MaxMind account. The GeoIP service downloads them every 24 hours from download.maxmind.com with that account, through the outbound proxy when one is set; a refused account shows MaxMind's own words there. GeoLite2 database sync on the Maintenance page copies them to the rules daily: run it now after a fix. A license that no longer carries IP location data stops the refresh: see GeoIP |
| The console's certificate is close to expiry, or has expired | The bootstrap agent renews it every day. Its Maintenance page, on port 5202, shows how the last Certificate renewal went, and its Run now runs it again; Request certificate on its Certificates page issues one at once. A Let's Encrypt certificate needs the server's host name to point here and port 80 to be open. See The console's certificate |
Locked out of the console by Turnstile
When Cloudflare Turnstile on the console's sign-in refuses everybody - wrong keys, Cloudflare unreachable from the server, the console opened by a name the Turnstile widget does not list - turn it off from the server itself. It needs neither the console nor the management API:
$ sudo ohm server turnstile disableThe next sign-in is not checked, and the keys stay. Put the cause right on Security > Threat protection > Turnstile, then switch Turnstile back on:
$ sudo ohm server turnstile enable| Symptom | Cause and fix |
|---|---|
| The widget never appears or shows an error | The console is opened by a host name the widget does not list, or the browser cannot reach challenges.cloudflare.com |
| The widget passes but the sign-in is still refused | The server cannot reach Cloudflare, or the secret key is wrong. The management API's log says which. A host name set on the Turnstile tab that is not the one in the address bar refuses too |
enable ran but Turnstile stays off | ANTELAO_BOOT_TURNSTILE_BYPASS=true is in /etc/antelao/agent.env and wins. Remove it and run sudo ohm install |
Locked out of an Odoo portal
When an Odoo portal's Turnstile check refuses everybody - wrong keys, a Cloudflare outage, a test key left in place - nobody can sign in to the portal to fix it. Switch the portal bypass on from the server itself. It needs neither the management API nor the console, only the instance's PostgreSQL:
sudo ohm --local instance database turnstile -i fanes -d fanes --bypass on --restart
The instance restarts, and the portal sign-in skips the Turnstile check while the keys stay in
place. Put the keys right - --site-key, and --secret-key to be prompted for the secret - check
them with the command without options, and switch the check back on:
sudo ohm --local instance database turnstile -i fanes -d fanes --bypass off --restart
The same switch is on the database's Sign-in protection tab, when the console is
reachable. Each change made with --local is recorded in the audit trail. See
Sign-in protection.
| Symptom | Cause and fix |
|---|---|
| The command says Turnstile is not active in this database | Odoo's Turnstile module is not installed there, so it cannot be what keeps people out. Nothing was changed |
| The command says the bypass has no effect | The portal sign-in check is not installed in that database. Nothing was changed |
| New keys are refused as parked | A non-production environment preset switched Turnstile off and keeps the keys aside. Change the preset, or apply the database options, first |
| The change was saved but nothing changed in Odoo | The instance was not restarted: Odoo reads these settings when it starts. Run the command again with --restart, or restart the instance |
Firewall
An instance's firewall log says in its Reason column which kind of refusal a row is. What each one means is in Reading an instance's refusal log.
| Symptom | Cause and fix |
|---|---|
Rows read refused closed: the access check could not answer (HTTP 502) or (HTTP 504) | The management API was not answering, for example while an update restarted it. The rules were not consulted: the caller would have been refused whatever they said. Outside an update, Core services says why the API is down |
Allowed rows read admitted; Odoo could not be reached (HTTP 502) (or 503, 504) | Not the firewall: it let the caller in and Odoo did not answer, while it restarted, was replaced during an update, or was in maintenance. The visitor saw the maintenance page. The instance's Odoo log for the same minute says why |
Allowed rows read admitted; Odoo itself answered HTTP 500 (or 503, 403) | Not the firewall: it let the caller in and Odoo answered with an error of its own. The instance's Odoo log for the same minute says why |
| Visitors your rules admit get the instance's refusal instead of the maintenance page while Odoo restarts | The instance's site was written by an earlier release. Apply the instance's configuration, or update the server, and its site is rewritten |
| Addresses your rules admit are refused at the same moment, then admitted again | Not the rules: the management API was unavailable for those seconds. The rows' reason says so; a support bundle for those hours carries the web server's own lines and the update that caused it |
A payment provider reports failed webhooks, and the firewall log refuses /payment/<provider>/webhook with the surface blocks by default and nothing admits it | The instance has no Payments rule, so its webhooks follow the Odoo rule, which admits only the addresses you listed. Add a Payments rule, open or narrowed to the providers you use, and apply. A path exempt on the Pre-authentication tab is still judged by the firewall. See Payment providers' webhooks |
Jobs
| Symptom | Cause and fix |
|---|---|
| A job is stuck at Queued | Another job holds the instance, or a server-level job is draining the queue. Both are by design |
A job is Interrupted | The API restarted while it ran. It is not retried automatically because half-done work can be destructive. Read its log and decide |
| A job failed naming an edition | A licensing denial. It names the capability, not the code. See Editions |
| A generated password is not shown any more | It is revealed once, briefly, and then gone. Reset it |
| A new database's Show the password is gone | A database administrator password generated at creation is shown once, to whoever created the database, within an hour. Reset it on the database's Users tab. See The generated administrator password |
Backups
| Symptom | Cause and fix |
|---|---|
| Backups run but nothing reaches S3 or Azure | Either the credentials are wrong, or the license no longer carries that target. A target the license does not carry is switched off, with a notice on the Backup page and an entry on the License page; a backup that reached it first records a failed result for it. Check the job result per target |
| Scheduled backups stopped | The license no longer carries scheduled backups, so the schedules were disabled. The License page lists them, and they are enabled again by themselves when a license carries them. A subscription that ended runs unregistered and takes no backups at all |
| A schedule did not fire after an outage | The scheduler does not catch up. Run it by hand for the missed window |
| An instance is not covered by the server-wide sweep | It holds its own schedule of that type, which excludes it from the sweep. Remove the per-instance schedule to return it |
Licensing
| Symptom | Cause and fix |
|---|---|
| A license worked for two weeks and then stopped | It was never activated. An unactivated license runs on a non-renewable 14-day window from first sight. Activate it against this server's challenge |
| A license installs and is refused immediately | It is bound to a different machine. The activation window is only for a license never activated anywhere |
| A previously working license stopped | The hardware changed, the subscription ended, the license server refused the license, the clock was rolled back, the license does not cover this product version, or the check-in grace window of a trial or perpetual license ended. The License page names which. Unless the license server refused it or its term ended, a server that held a valid license first ran on it for 14 days as a fault |
| The License page reports a fault, then the server unregisters, after a hardware or hypervisor change | The challenge changed, so the license no longer matches this machine. It ran on for 14 days as a fault first. Reactivate, or move the license to this server |
| The License page reports the state file is unwritable | The license directory is read-only. The grace window still ends, but the anchor is only in memory |
| A feature, an instance or an account stopped after a license change | The license no longer carries it, or a limit is lower. The License page's Switched off by the license card names each item, when and why. It comes back by itself when a license carries it again; Enterprise add-ons and databases moved back from an external PostgreSQL are yours to re-do. See When the license changes |
| The banner says the license cannot be read or verified | The server can no longer read or verify its license: after an update, a corrupt file, a key it does not know. Nothing is switched off: it keeps running on the last valid license until the date the banner names, 14 days after the fault began. Install a newly issued license on the License page. See When the license cannot be read or verified |
| A change is refused naming the unreachable license server | The server has not reached the license server since its last receipt ran out. Everything keeps running; only changes that need a feature beyond Community wait. Fix outbound connectivity, then Check in now |
| Starting an instance is refused although it ran before | As many instances run as the license allows. Stop another one, or install a license that allows more; the instances the license stopped start again by themselves when it does |
Fail2Ban
| Symptom | Cause and fix |
|---|---|
| Bans are listed but attacks continue | Almost certainly a reverse proxy in front of nginx. See the trusted-proxy warning in Intrusion prevention. Check the jail's packet counters, not the ban list: sudo iptables -L f2b-ohm-login -n -v for ohm-login. Counters that stay at zero mean the bans stop nothing |
| A ban has no instance | Jails are server-wide. The instance is derived from the database named in the match, and not every jail carries one |
| You banned yourself | ohm server fail2ban unban works at every edition, including unregistered, precisely for this |
Upgrades
| Symptom | Cause and fix |
|---|---|
| An older binary refuses to start after a rollback | The master schema upgrade is forward-only. Restore the master database from before the upgrade |
| The API refuses the master database after an update | It carries a schema this build does not own. ohm database upgrade brings it forward; ohm install does the same as part of a repair, before it rolls any container |
Diagnostics worth keeping to hand
$ sudo docker logs ohm-web --tail 200sudo ls -t /root/.local/share/ohm/logs/ | head$ sudo ohm server info --show-secrets