Add photo content-hash dedup and local↔S3 library sync.
Uploads skip per-gallery duplicates, checksums can be backfilled, and STORAGE_BACKEND plus sync tooling make switching storage backends safe.
This commit is contained in:
Michilis
+81
-4
@@ -14,9 +14,13 @@ Design/decisions: [PLAN.md](./PLAN.md).
|
||||
- Originals are stored byte-identical for download; a worker generates JPEG
|
||||
variants (thumb 512px q78, preview 2048px q85, EXIF stripped/orientation
|
||||
applied) queued in the DB with retries.
|
||||
- Uploads are deduplicated per gallery: see
|
||||
[Duplicate detection](#duplicate-detection).
|
||||
- Storage: local disk (`STORAGE_PATH`) or S3/Garage/MinIO (set `S3_ENDPOINT`
|
||||
+ `S3_BUCKET`), same selection convention as the backend. S3 downloads use
|
||||
short-lived presigned URLs; nothing in the bucket is public.
|
||||
+ `S3_BUCKET`), same selection convention as the backend, overridable with
|
||||
`STORAGE_BACKEND`. S3 downloads use short-lived presigned URLs; nothing in
|
||||
the bucket is public. Switching backends later: see
|
||||
[Move the library between backends](#move-the-library-between-backends).
|
||||
- Auth: validates the backend's HS256 JWTs with the shared `JWT_SECRET`
|
||||
(issuer `spanglish`, audience `spanglish-app`) including the DB-backed
|
||||
tokenVersion/account-status revocation check. Admin surface is
|
||||
@@ -37,7 +41,8 @@ go test ./... # SQLite; add PHOTO_TEST_PG=<url> to also run on Post
|
||||
```
|
||||
|
||||
From the repo root: `npm run dev:photos`, `npm run build:photos`,
|
||||
`npm run test:photos`. The Next dev server rewrites `/api/photos/*` to
|
||||
`npm run test:photos`, `npm run migrate:photos`, `npm run sync:photos`,
|
||||
`npm run backfill:photos:checksums`. The Next dev server rewrites `/api/photos/*` to
|
||||
`PHOTO_API_URL` (default `http://localhost:3003`), so the frontend needs no
|
||||
extra config in dev.
|
||||
|
||||
@@ -45,6 +50,78 @@ HEIC uploads require a converter CLI on the host: `apt install libvips-tools`
|
||||
(or `libheif-examples`). Without one, HEIC uploads are rejected with a clear
|
||||
message and a startup warning is logged.
|
||||
|
||||
## Move the library between backends
|
||||
|
||||
`photo-api sync` copies the whole photo library one way between local disk and
|
||||
S3, so local↔S3 is a config switch rather than a migration project. **Both
|
||||
backends must be configured in `photo-api/.env`** (`STORAGE_PATH` *and* the
|
||||
`S3_*` values); `STORAGE_BACKEND` decides which one actually serves requests,
|
||||
so filling in S3 does not switch anything by itself.
|
||||
|
||||
Local disk → S3:
|
||||
|
||||
```bash
|
||||
npm run sync:photos -- to-s3 --dry-run # see what would be copied
|
||||
npm run sync:photos:to-s3 # copy it
|
||||
# then set STORAGE_BACKEND=s3 in photo-api/.env and restart the service
|
||||
```
|
||||
|
||||
S3 → local disk is the same with `to-local` / `STORAGE_BACKEND=local`
|
||||
(`npm run sync:photos:to-local`). From `photo-api/`: `make sync-to-s3`,
|
||||
`make sync-to-local`.
|
||||
|
||||
Flags (`npm run sync:photos -- to-s3 --overwrite`, or after the direction on
|
||||
the direct scripts): `--dry-run`, `--overwrite` (re-copy objects already
|
||||
present with the same size), `--concurrency=N` (default 4), `--gallery=<id>`.
|
||||
|
||||
How it behaves:
|
||||
|
||||
- The `photos_photos` rows are the inventory — for each photo the original
|
||||
plus, once processed, the thumb and preview. Objects with no row (worker
|
||||
scratch files, leftovers of deleted galleries) are not copied.
|
||||
- Only the destination is written. The source stays as a fallback; delete it
|
||||
by hand once the switch is verified.
|
||||
- Reruns are cheap and safe: objects already on the destination with the same
|
||||
size are skipped, so an interrupted or partly failed sync just needs
|
||||
rerunning. A failed object is logged and the exit code is non-zero.
|
||||
- Keys are identical on both backends, so nothing in the database changes and
|
||||
no re-processing is triggered.
|
||||
- Photos uploaded *after* the copy but *before* the restart land on the old
|
||||
backend. For a clean cutover, stop the service, sync, flip
|
||||
`STORAGE_BACKEND`, start again — or sync a second time after the switch to
|
||||
pick up stragglers.
|
||||
|
||||
## Duplicate detection
|
||||
|
||||
Every upload is hashed (sha256 of the original bytes) into
|
||||
`photos_photos.checksum`, unique per `(gallery_id, checksum)`. If a gallery
|
||||
already holds those exact bytes, the incoming copy is **discarded**: no object
|
||||
is stored, no row is inserted, and the upload response echoes the existing
|
||||
photo with `"duplicate": true`. The rest of the batch continues normally — a
|
||||
duplicate is not an error and does not consume a position.
|
||||
|
||||
Scope is one gallery. The same image can still live in several galleries, each
|
||||
with its own row and its own stored object, so deleting a gallery never orphans
|
||||
another one's photos.
|
||||
|
||||
The admin uploader panel shows those rows as *"Already in this gallery"*; no
|
||||
second tile appears in the grid.
|
||||
|
||||
Photos uploaded before this existed have no checksum, so they are not matched
|
||||
until hashed once:
|
||||
|
||||
```bash
|
||||
npm run backfill:photos:checksums -- --dry-run # what would be hashed
|
||||
npm run backfill:photos:checksums # hash it
|
||||
```
|
||||
|
||||
From `photo-api/`: `make backfill-checksums`. Flags: `--dry-run`,
|
||||
`--concurrency=N` (default 4), `--gallery=<id>`. It reads from the active
|
||||
`STORAGE_BACKEND`, only ever writes the checksum column, and is idempotent —
|
||||
rerun it after a sync or a restore. Photos whose content already matches an
|
||||
earlier one in the same gallery are **reported and left unhashed**; the command
|
||||
never deletes anything, so removing the extras is an admin's call.
|
||||
|
||||
## API
|
||||
|
||||
Everything under `/api/photos`. Errors are `{"error": string}`.
|
||||
@@ -58,7 +135,7 @@ Admin (Bearer token, role admin/organizer):
|
||||
| GET | `/api/photos/galleries/:id` | gallery + photos (all statuses) |
|
||||
| PATCH | `/api/photos/galleries/:id` | update title/visibility/event/cover |
|
||||
| DELETE | `/api/photos/galleries/:id` | delete gallery + objects |
|
||||
| POST | `/api/photos/galleries/:id/photos` | multipart upload (`files`) |
|
||||
| POST | `/api/photos/galleries/:id/photos` | multipart upload (`files`), deduplicated |
|
||||
| PATCH | `/api/photos/galleries/:id/order` | reorder (`{photoIds}`) |
|
||||
| POST | `/api/photos/galleries/:id/share-token` | rotate share token |
|
||||
| DELETE | `/api/photos/photos/:photoId` | delete photo |
|
||||
|
||||
Reference in New Issue
Block a user