# Reliable backend deploys: plan

Written 8 Nov 2026, after production deploys sometimes failed with **error 137**
while building.

## Status (8 Nov 2026)

**Phases 1 and 2 are built.** They take effect once these files are merged into `deploy`:
`.github/workflows/deploy.yml`, `scripts/deploy/activate-release.sh`,
`ecosystem.config.js`, `.nvmrc` (Node 24), `package.json` (the `prisma` CLI is now a dependency)
and `prisma/schema.prisma` (`binaryTargets`).

Tested on a pretend server with real pm2, the real API and a throwaway database:

| Test | Result |
|---|---|
| First deploy from today's layout | pm2 moved from the old checkout to `current` once; healthy |
| Normal deploy | switched and healthy |
| Broken release (crashes on start) | switched back to the last good release; broken one moved to `failed/` |
| Broken first deploy | pm2 put back on the old checkout; healthy |
| Failing migration | stopped before switching; running release untouched; moved to `failed/` |
| Roll back to previous / to a named release | correct release, healthy; a failed rollback returns to where it was |
| Keep five releases | older releases deleted, never the running one |

Not tested here: the GitHub job itself (it runs on the first push to `deploy`), and the atomic
`mv -T` switch, which is Linux-only (the Mac test used the fallback).

### How it works now

- **Deploy:** merge to `deploy` as before. GitHub builds the release (80 MB) and uploads it. The server
  unpacks, migrates, switches and checks health. The Actions summary names the release.
- **Roll back:** Actions → "Deploy (production)" → Run workflow. Set `rollback_to` to `previous` or a
  release name such as `20261108101500-4c108b0`. On the server:
  `bash /home/ubuntu/hostelcare-app/current/scripts/deploy/activate-release.sh --list`
  shows the releases, and `--rollback` does the same as the button.
- **Settings:** `.env` stays where it is, at `/home/ubuntu/hostelcare-backend/.env`. Every release links
  to it, so editing it and running `sudo pm2 restart hostelcare-api --update-env` works as before.
- **Where things live:**
  - `/home/ubuntu/hostelcare-app/releases/` holds the last five releases.
  - `current` points at the live one.
  - `failed/` keeps the last failed release, for a look.
  - The old checkout `/home/ubuntu/hostelcare-backend` is no longer updated by deploys. Its `dist/` is only
    the fallback if the very first deploy fails.
- **Admin scripts** (`npm run enterprise:harden-data`, the seeds) now run from
  `/home/ubuntu/hostelcare-app/current`, which has the same `scripts/` folder.
- **pm2** runs the API from `ecosystem.config.js`: one process, restarted if it passes 1 GB of memory.
  The old workflow saved pm2's list as the wrong user (`pm2 save` without `sudo`); the script now uses
  `sudo pm2` throughout.

### First deploy after merging

The first run moves pm2 from the old checkout to the release folder. The API is down for a few seconds,
as on any deploy today. If that first release isn't healthy, the script puts pm2 back on the old checkout
by itself. Nothing needs doing by hand on the server beforehand. Phase 0's swap is still worth adding as
a safety net.

**Live on production since 8 Nov 2026, 04:36 UTC** (release `20261108043650-e7092c9`, confirmed on
the server: pm2 runs from `/home/ubuntu/hostelcare-app/current`).

## Phase 4: live on production since 8 Nov 2026, 05:08 UTC (`7d62442`)

- **Health names the running commit:** `/api/health` now replies with `revision` (the short commit
  from the release's `REVISION` file), additive and absent when run from a checkout. After switching, the deploy
  checks that the API is up, that `/api/health/db` reaches the database, and that `revision` is the new
  release's. So another process holding the port can't pass for the new release.
- **Pre-checks before anything changes:**
  - free disk and memory (as before);
  - the database answers a `SELECT 1` (otherwise it stops with "The database can't be reached; nothing was
    changed" and parks the release in `failed/`);
  - a warning if the server's Node major differs from the release's `.nvmrc`;
  - uploads older than an hour left in `/tmp` by a failed run are removed.
- **Log rotation:** the first successful deploy installs `pm2-logrotate` once: 20 MB per file, 14 files
  kept, compressed. Later deploys see it and skip. If the install fails, the deploy carries on with a warning.
- **Deploy summary:** the server step prints `SUMMARY|key|value` lines, and the workflow turns them into a
  table on the run's page, whether the deploy passed, failed or rolled back:
  - release, previous release and revision;
  - migrations applied, or "none pending";
  - health and the time taken;
  - "rolled back to …" after a rollback, or the failure reason.

  The rollback job writes its own summary.
- **Manual rollback:** unchanged; `rollback_to` on Run workflow.

Tested on the pretend server with releases built from this code:

| Test | Result |
|---|---|
| First deploy from today's layout | healthy, `revision` checked, log rotation set up once, summary complete |
| Second deploy | healthy; log rotation skipped; summary shows the previous release |
| Database unreachable | stopped before any change; release in `failed/`; live API untouched |
| Manual rollback | correct release; its own summary |
| Health with a wrong revision | refused; right revision accepted |

## Phase 3: built 8 Nov 2026 (takes effect with the next merge to `deploy`)

The `hostelcare` database (PostgreSQL 16, about 37 MB) runs on the production server, so its
backups must also leave the server. The other databases on that server (`hrms`, `gympro`,
`ruleexpert`, `hostelcare-ws`) are not covered here.

- **`scripts/deploy/backup-db.sh`** (ships in every release):
  - `backup-db.sh [label]` dumps the database (`pg_dump --format=custom`, the schema from `DATABASE_URL`)
    at low priority into `/home/ubuntu/hostelcare-app/hostelcare-backups/`. It checks the dump reads back, keeps
    the newest five, and copies it to `s3://eia-postgres-backups/hostelcare/`. It fails early when the disk can't
    hold another dump.
  - `backup-db.sh --check [file]` restores a dump (default: the newest) into a scratch database
    `hostelcare_restore_check`, reports its tables and applied migrations, and drops the scratch database.
    It never touches the live database.
  - `backup-db.sh --list` shows the dumps; `--install-cron` adds the nightly backup (below).
  - Exit codes: 0 done, 3 saved here but the S3 copy failed, anything else failed.
- **Before migrations:** a deploy with migrations pending takes a backup named `before-<release>` first.
  If it fails, the deploy stops with nothing changed. If only the S3 copy fails, the deploy carries on and
  the summary warns. Deploys with nothing to migrate skip it (`BACKUP_ON_DEPLOY=always|pending|never`;
  `pending` is the default).
- **Copy off the server:** `scripts/deploy/upload-backup.mjs` uses the AWS SDK already in the release,
  the app's own `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_REGION` from `.env`, and server-side
  encryption. It follows the bucket to its own region if that differs from `AWS_REGION`.
  - `BACKUP_S3_URI` in `.env` overrides the location; `BACKUP_S3_URI=none` keeps backups on the server only
    (with a warning).
  - S3 creates the `hostelcare/` folder on the first upload.
- **The deploy summary** gains "backup" and "backup copy" rows.

Tested against a local PostgreSQL and the pretend server:

| Test | Result |
|---|---|
| Backup with `BACKUP_S3_URI=none` | saved and verified; warns "on this server only" |
| Four backups, keep two | the two newest kept |
| S3 upload refused (bad key) | backup kept here; exit code 3; summary warns |
| Restore check | 142 tables, 79 migrations restored into the scratch database, then dropped |
| Damaged dump | refused; scratch database dropped |
| Deploy with nothing to migrate | backup skipped; deployed |
| Deploy with a migration pending | backed up first, then migrated and healthy |

Not tested here: an upload to the real `eia-postgres-backups` bucket (only with a fake key against a
test bucket name), and `sudo -u postgres` for `--check`, which the Mac test replaced with a local superuser.

### One-time setup (owner)

1. **Let the app's AWS key write backups.** The key in `AWS_ACCESS_KEY_ID` needs `s3:PutObject` on
   `arn:aws:s3:::eia-postgres-backups/hostelcare/*`. If its IAM policy only covers the uploads bucket, add:

   ```json
   { "Effect": "Allow", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::eia-postgres-backups/hostelcare/*" }
   ```

   Optional but recommended: a lifecycle rule on the bucket for the `hostelcare/` prefix that deletes
   objects after 30 days.
2. **Nothing to add to `.env`**: the bucket, folder and keys are already the defaults.
3. **After the deploy that brings Phase 3**, run once on the server:

   ```bash
   bash /home/ubuntu/hostelcare-app/current/scripts/deploy/backup-db.sh first
   bash /home/ubuntu/hostelcare-app/current/scripts/deploy/backup-db.sh --check
   bash /home/ubuntu/hostelcare-app/current/scripts/deploy/backup-db.sh --install-cron
   ```

   - The first command takes a backup and copies it to S3. If it prints "S3 upload failed", step 1 is
     missing; the backup is still kept on the server.
   - The second command proves the backup restores.
   - The third adds the nightly backup at 21:30 UTC (03:00 IST), logging to
     `/home/ubuntu/hostelcare-app/hostelcare-backups/backup.log`.

### Restoring for real

Only when the live data is actually lost or damaged:
1. Stop the API (`sudo pm2 stop hostelcare-api`).
2. Restore the chosen dump with `pg_restore --clean --if-exists --no-owner -d hostelcare <file>`.
3. Start the API again.

Try `--check` on that file first.

## What goes wrong today

Exit code 137 means the process was killed with signal 9 (137 = 128 + 9). On
Linux that is almost always the kernel's out-of-memory killer.

`.github/workflows/deploy.yml` builds on the production server itself, next to
the live API:

1. `npm ci` deletes `node_modules` and installs every package, dev tools included,
   while the running API still uses those files.
2. `npm run build` runs `prisma generate` and then `tsc` over the whole source.
3. `npx prisma migrate deploy`, then `pm2 restart`.

Measured on this code (246 TypeScript files):

| Step | Peak memory |
|---|---|
| `tsc` (the build) | about 1.4 GB |
| `prisma generate` | about 0.6 GB |
| The live API (typical Node process) | 0.2–0.4 GB |

On a 1–2 GB server, with Postgres or nginx also running, the build doesn't fit, so
the kernel kills it. How often that happens depends on what else is using memory
at the time, which is why it's only "sometimes".

The same design has three more risks, even when memory holds:

- **No rollback.** `npm ci` wipes `node_modules` first, and `tsc` overwrites `dist/` file by
  file. If either stops halfway, the server holds a half-old, half-new copy. The
  API keeps running from memory until the next restart or crash, then fails.
- **Downtime on every deploy.** `pm2 restart` stops the process before the new one is up.
- **Migrations before a known-good build.** A migration can run, then a later step fails,
  leaving the database ahead of the code.

The GitHub job already builds and type-checks the code once (`verify`). The server then
builds it a second time. The web app avoids all of this: it builds on GitHub and only
copies the finished files to the server (`rsync`).

## The plan

### Phase 0: stop the failures today (owner, about 10 minutes, no code change)

Add swap to the server, so a memory peak slows the build down instead of killing it.
This is a safety net, not the fix.

```bash
free -h
sudo fallocate -l 2G /swapfile && sudo chmod 600 /swapfile
sudo mkswap /swapfile && sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo sysctl vm.swappiness=10 && echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-swappiness.conf
free -h
```

### Phase 1: build once, on GitHub, and ship the result (code change)

The server never compiles again.

- **On the GitHub job, which has 7 GB of memory:**
  - `npm ci`, `prisma generate`, `typecheck`, `build` (already done today in `verify`).
  - Then `npm ci --omit=dev` into a clean folder, to get production packages only.
  - Pack a release: `dist/`, `prisma/` (schema and migrations), `package.json`, `package-lock.json`,
    the production `node_modules` with the generated Prisma client, and a `REVISION` file holding the
    commit.
- **Packages it needs:**
  - Move the `prisma` CLI from devDependencies to dependencies, so the release can run
    `migrate deploy` without dev tools.
  - Set `binaryTargets = ["native", "debian-openssl-3.0.x"]` in `schema.prisma`, so the engine built
    on GitHub's Ubuntu runs on the server. Phase 1 confirms the server's target first.
- **Same Node version on both sides:** pin it in `.nvmrc` and in the workflow, matching the server
  (`node -v` there). GitHub uses 22 today.
- **Upload:** `rsync` the release to the server.

On the server, the only work left is unpacking files and running Node. Peak memory during a deploy
drops from about 2 GB to almost nothing beyond the API itself.

### Phase 2: release folders, instant switch, automatic rollback (code + one-time server setup)

```
/home/ubuntu/hostelcare-backend/
  releases/<timestamp>-<commit>/   one folder per deploy, last 5 kept
  current -> releases/…            symlink pm2 runs from
  shared/.env                      linked into each release
```

Deploy steps on the server:

1. Upload into a new `releases/<timestamp>-<commit>/` and link `shared/.env` into it.
2. Run `npx prisma migrate deploy` from the new release (see Phase 3).
3. Point `current` at the new release in one step (`ln -sfn`), then `pm2 reload`.
4. Health check `/api/health` for 30 seconds. If it fails, point `current` back at
   the previous release, reload, and fail the job with the logs.
5. Delete releases beyond the newest five.

A one-time `ecosystem.config.js` (in the repo) runs `current/dist/server.js` with
`cwd: current` and `max_memory_restart` as a guard. It optionally runs in cluster mode with
two instances, so `pm2 reload` swaps them one at a time with no downtime. That needs at least
2 GB of memory, and the background jobs inside the API (reminders, late-return runs) must
then run in one instance only. Phase 2 checks those before turning cluster mode on.

The one-time cutover from today's folder to this layout is a short, scripted step the owner
runs once, during a quiet time. It is written out in full when Phase 2 is built.

### Phase 3: safer migrations

- **The database runs on the same server** (owner, 8 Nov 2026). The earlier check only looked
  for `localhost` in `DATABASE_URL`, so a `127.0.0.1` or private-address URL read as "elsewhere".
  So Postgres shares the 2 GB of memory and the 29 GB disk with the API. It also means losing the
  server would lose the database, so a copy off the server matters more than anywhere else in this plan.
- **Back up first, on every deploy that has migrations to apply.** Before `migrate deploy`,
  `pg_dump --format=custom` into `/home/ubuntu/hostelcare-app/hostelcare-backups/`, named after the release.
  - Run it at low CPU and I/O priority (`nice` and `ionice`), so the live API keeps its share of the
    two CPUs.
  - If the dump fails, stop with nothing changed.
  - Keep the last 5 on the server, and fail early if the disk can't hold another (estimated from
    the last dump's size).
  - Skip it when no migrations are pending (`prisma migrate status`), so ordinary deploys stay quick.
- **A copy off the server.** Upload each dump to a private S3 bucket (or a private prefix of the
  existing one), with an expiry rule keeping 30 days. The server's own AWS keys in `.env` already
  reach S3; a separate key limited to that prefix is safer.
- **A nightly backup too**, not only at deploy time: a cron job with the same script, so a
  bad day without a deploy is covered.
- **Restore, written and tried.** A `restore` command loads a chosen dump into a scratch database
  to prove it's good. Restoring over the live database stays a step a person takes knowingly,
  with the API stopped first.
- **Only additive migrations go out with code.** Add a column or table now; remove the
  old one in a later deploy, once no running code reads it. Already the rule here for
  store apps; this makes it explicit for the database.
- **Big indexes.** Prisma runs a migration inside a transaction, where
  `CREATE INDEX CONCURRENTLY` isn't allowed. For a large table, create the index by hand
  with `CONCURRENTLY` first, then let the migration's `CREATE INDEX IF NOT EXISTS` pass. The
  L4 migration (`20261217090000_l4_list_indexes`) is small today, but the rule applies from
  now on.
- **Failed migration.** The deploy stops before the switch. The old release keeps serving,
  and `prisma migrate resolve` plus the backup handle the fix.

### Phase 4: visibility and housekeeping

- **pm2 logs:** log rotation (`pm2 install pm2-logrotate`), so logs can't fill the disk.
- **Disk checks:** the deploy checks free disk space and memory before starting, and fails with a clear
  message instead of dying halfway.
- **Deploy message:** a short summary in the Actions run: commit, migrations applied, health result,
  previous release kept for rollback.
- **Manual rollback button:** a `workflow_dispatch` input that points `current` at a chosen
  earlier release without rebuilding.

## Order and size

| Phase | What | Who | Size |
|---|---|---|---|
| 0 | Swap on the server | Owner, on the server | 10 minutes |
| 1 | Build on GitHub, ship the release | Code (workflow, package.json, schema.prisma) | half a day |
| 2 | Release folders, switch, auto-rollback, pm2 config | Code + one-time cutover by the owner | 1 day |
| 3 | Backup before migrating, migration rules | Code + backup bucket | half a day |
| 4 | Log rotation, pre-checks, rollback button | Code + one server command | half a day |

Phase 1 alone ends the error 137 failures. Phase 2 makes a bad deploy undo itself.

## What's needed from the owner first

Run this on the production server and share the output. It prints no secrets.

```bash
free -h; nproc; lsb_release -ds; node -v; npm -v; pm2 -v
sudo pm2 describe hostelcare-api | grep -E "exec mode|script path|exec cwd|node.js version|instances"
df -h /home/ubuntu
cd /home/ubuntu/hostelcare-backend && grep -c '^DATABASE_URL=.*localhost' .env
openssl version
```

- **Memory and CPU** size the swap, and decide whether cluster mode fits.
- **Ubuntu, OpenSSL and Node versions** set the Prisma binary target and the pinned Node version.
- **pm2 mode and paths** shape the cutover.
- **The `DATABASE_URL` line** was meant to say whether Postgres runs on this server. It counted only
  `localhost`; the owner confirmed on 8 Nov 2026 that the database is on this server.
- **The web app's deploy** already builds on GitHub; it gets only Phase 4's pre-checks.
