phase-8: run it on Postgres, and find out what that was hiding
Six phases claimed the product runs on SQLite and on Postgres. Nothing had ever run it on Postgres. TEST_DATABASE_URL now points the whole suite at a real server and CI runs both arms. The first run found a bug that would have shipped. better-auth's banned flag is integer 0/1 on SQLite and a real boolean on Postgres, and the code read it as `banned === 1`, so on Postgres an account someone asked us to freeze went on receiving email. Both of those flags are now typed for either dialect and read through isFlagSet. It also found the migration advisory lock being taken on a pool. An advisory lock belongs to the session that took it, so a lock on one pooled connection and an unlock on another leaves it held. Two replicas migrating at once is the ordinary case in k8s and is precisely what it was there to protect. New for the scaled mode: k8s manifests with migrations as an initContainer, one poller in its own worker Deployment rather than one per API replica, and an Ingress that exposes the web app only. The storage drivers finally have tests, S3 included, since scaled mode requires it and it had never been exercised. Two acceptance tests, both checked against a deliberately broken build first: two workers claiming a hundred jobs report 188 claims with SKIP LOCKED removed, and the in-flight request is cut off with the drain wait removed. 340 tests on SQLite, 341 on Postgres, 70 Playwright, rules coverage 100%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
8c8463924f
commit
37370dd079
@@ -7,12 +7,13 @@ and ready to file yourself.
|
||||
Working name. See `docs/` for the specifications, `DECISIONS.md` for choices made along the
|
||||
way and the gaps that still need answers.
|
||||
|
||||
> **Status: phase 7 of 8.** The whole taxpayer path works: scan a comprobante, confirm it,
|
||||
> watch the position move, and take the resulting Formulario 120 or 515 from review to
|
||||
> **Status: phase 8 of 8, feature complete.** The whole taxpayer path works: scan a
|
||||
> comprobante, confirm it, watch the position move, and take the resulting Formulario 120 or 515 from review to
|
||||
> approved to a PDF you file yourself in Marangatu. Staff have a console: look an account
|
||||
> up, work the ingestion error queue, read and export the audit log. It installs to a home
|
||||
> screen, keeps a capture taken with no signal and sends it later, and can send a push when
|
||||
> a deadline is close. The scale-out work is the last phase.
|
||||
> a deadline is close. It runs on SQLite on one box or on Postgres and S3 across as many
|
||||
> replicas as you like, and the whole suite is run against both.
|
||||
|
||||
---
|
||||
|
||||
@@ -131,6 +132,40 @@ S3. The k8s manifests assume both.
|
||||
Rate limiting is in memory and therefore per replica. At this scale that is deliberate;
|
||||
the limiter is behind an interface for when it is not.
|
||||
|
||||
### Running it on Kubernetes
|
||||
|
||||
`deploy/k8s/` holds the manifests, and its README explains what each one is for. The order
|
||||
is namespace, config, secret, api, worker, web, ingress, hpa. Three things about the shape
|
||||
are worth knowing before you read them:
|
||||
|
||||
**Migrations run as an initContainer on every API pod.** Two pods starting together is the
|
||||
normal case, not the exception, and it is safe: the migration takes a Postgres advisory
|
||||
lock on one pinned connection, so the second waits for the first and then finds nothing to
|
||||
do.
|
||||
|
||||
**Exactly one poller per job, not one per replica.** The API Deployment sets
|
||||
`JOBS_INLINE=false` and the polling lives in its own worker Deployment. The worker is safe
|
||||
at any replica count on Postgres: claiming uses `FOR UPDATE SKIP LOCKED`, which
|
||||
`apps/api/src/modules/jobs/scale.test.ts` holds to a hundred jobs and two workers with no
|
||||
job claimed twice.
|
||||
|
||||
**Only the web app is exposed.** The Ingress routes to the web Service and the API has no
|
||||
route in from outside. The browser talks to one origin and `/api` is forwarded inside the
|
||||
cluster, which is why there is no CORS configuration anywhere in this repository.
|
||||
|
||||
Sizing: `DATABASE_POOL_MAX` is per pod. Multiply it by (api replicas + worker replicas) and
|
||||
keep the total under the Postgres `max_connections`, or the tenth pod to start is the one
|
||||
that cannot connect.
|
||||
|
||||
### What has not been run here
|
||||
|
||||
The container images, the compose stack and the Kubernetes manifests have never been built
|
||||
or applied on the machine this was written on: there is no Docker daemon it can reach and
|
||||
no cluster. The manifests parse, their configuration is checked against the env schema by
|
||||
`apps/api/src/deploy.test.ts`, and the Dockerfiles are ordinary multi stage Node builds,
|
||||
but none of that is the same as having watched a pod come up. Treat the first deploy as the
|
||||
first real test of them.
|
||||
|
||||
## Installing it, offline and push
|
||||
|
||||
`app/manifest.ts` and the icons in `apps/web/public` make the web app installable; the
|
||||
@@ -196,9 +231,20 @@ old definition in place: it is what old declarations still render through.
|
||||
|
||||
## Testing
|
||||
|
||||
Vitest covers the rules package, the API services against SQLite in memory, and catalog
|
||||
parity. Playwright covers the golden paths against a running stack. CI runs the suite
|
||||
against both SQLite and Postgres.
|
||||
Vitest covers the rules package, the API services, and catalog parity. Playwright covers
|
||||
the golden paths against a running stack.
|
||||
|
||||
The same suite runs against both dialects. With no `TEST_DATABASE_URL` it uses SQLite in
|
||||
memory; point that at a Postgres and every harness builds a database of its own inside it,
|
||||
which is how CI covers the claim that the product runs on either:
|
||||
|
||||
```bash
|
||||
TEST_DATABASE_URL=postgres://user:pass@localhost:5432/postgres pnpm test
|
||||
```
|
||||
|
||||
Two tests only mean something on Postgres and skip themselves without it: the two-worker
|
||||
hundred-job claim test, and anything that depends on `FOR UPDATE SKIP LOCKED`. The S3
|
||||
driver behaves the same way through `TEST_S3_ENDPOINT`.
|
||||
|
||||
`pnpm test:e2e` does not start anything: bring the stack up first, then point it at the
|
||||
right origin. The suite signs in as the demo accounts and confirms, rejects and scans
|
||||
|
||||
Reference in New Issue
Block a user