Zero-downtime deploys

A release starts the new code next to the old code and then moves traffic over. You are not sent through a restart: a reply that is already streaming, or a file upload that is already under way, finishes on the old copy.

How a deploy switches ZeroLog to new code without downtimeVisitors, the tunnel and worker machines all connect to port 8080. An nftables table redirects that port to the active front-door proxy, either instance a on port 8084 or instance b on port 8085. The proxy sends traffic to the one active app slot, blue on port 8082 or green on port 8083. A deploy tests the new code, takes a backup, starts the idle slot, health-checks and activates it, reloads the proxy, and restarts the background worker with a drain.restart,with drainone instance at a timeactive slot onlystart idle slot,activatereload proxyApp slots, one activezerolog@blueport 8082zerolog@greenport 8083Front-door proxy (Node.js)zerolog-proxy@aport 8084zerolog-proxy@bport 8085Visitors, tunnel andworker machines(all use port 8080)nftables table zl_zdtredirects port 8080to the active proxydeploy.shtests, backup, switchzerolog-workerbackground jobs
Traffic always reaches one active slot through one active proxy; a deploy prepares the idle slot, then moves the proxy over.
Diagram as text
  1. Visitors, the tunnel and worker machines connect to port 8080. An nftables table named zl_zdt redirects that port to the active proxy.
  2. The front-door proxy runs as two instances, zerolog-proxy@a on port 8084 and zerolog-proxy@b on port 8085. Only one receives traffic at a time.
  3. The proxy forwards HTTP, server-sent events, WebSocket upgrades and CONNECT requests to the active app slot: zerolog@blue on port 8082 or zerolog@green on port 8083.
  4. deploy.sh tests the new code, takes a backup, then starts the idle slot, health-checks it and activates it. It reloads the proxy so the new slot takes traffic. If the proxy code itself changed, it starts the other proxy instance, health-checks it, moves the nftables redirect over and stops the old instance.
  5. The old slot gets SIGTERM and drains. The proxy keeps follow-up requests of chats and uploads that began on it on that slot until they finish.
  6. deploy.sh restarts zerolog-worker, which drains its running jobs first.

How a release goes

  1. Test in isolation. The new code is tested in an isolated copy first: lint and the test suite. Only green code moves on.
  2. Back up. A backup is taken before anything on the live system changes.
  3. Check the database changes. Old and new copy share one database for a few moments, so a release that drops or renames a table or column, or adds a required column without a default, is refused.
  4. Start the idle slot. It starts with the new code. If it does not pass its health check, the active slot keeps serving and the release stops there.
  5. Move traffic. The new slot is activated and the front-door proxy is told to send traffic to it. If the proxy's own code changed, a second proxy instance starts, is health-checked, and the port redirect is moved to it before the old instance stops.
  6. Drain the old slot. Follow-up requests of chats and uploads that began on it are kept on it until they finish; it is stopped after at most ten minutes.
  7. Restart the worker. The background worker restarts and finishes its running jobs first, for at most five minutes.
  8. Smoke test. A smoke test runs against the slot that is now active.

What it does not do

A reply that is still running when the old copy's ten minutes are up is cut off. A release that removes or renames database columns is refused in the normal path and needs a planned two-step release instead. None of this is something you configure.

Going back

The operator can return to the single-process setup. The old process starts and is health-checked while the proxy keeps serving; only then does traffic move to it, and the proxies and slots are removed last.

Related: Isolation, Trust and security, Status.