Product
A 17-minute stall on 19 October: cause fixed, watchdog added
From 22:24 to 22:41 UTC the server accepted no requests. A model-routing loop caused it. It is fixed, and a watchdog now restarts a frozen server after 60 seconds.
ZeroLog did not answer for 17 minutes on 19 October; nothing for you to do, and a watchdog now restarts a frozen server after about a minute.
On 19 October between 22:24 and 22:41 UTC, ZeroLog did not answer. The main process spent 17 minutes in an endless loop.
The cause was the Free Auto router, an internal router that customers cannot pick, but its loop froze the whole server for every user. It listed itself, under a second name, as one of its own fallback models. After a cancelled request each fallback called the router again, every attempt failed at once, and the chain never ended. In a test it made 17,851 attempts in 20 seconds.
Fixes: the router is never part of its own chain, a cancel ends the fallback run immediately, each model is tried once with a cap of 40 steps, and the model lists are read once instead of per model (3,023 ms down to 63 ms).
New: a watchdog thread checks a heartbeat of the main process. If it is silent for 60 seconds, the process is stopped and systemd starts it again. The next start reports the event to our telemetry. A stall like this one should now last about a minute, not 17.