A memory-as-document exhibit: a fictional AI coding-assistant's project memory for the made-up Lantern notification-dispatch service, in conformant doc.html v0.3 format.
This file is a memory-as-document exhibit — a worked example in the doc.html v0.3 format showing how an AI coding-assistant might store project knowledge as a witnessed, addressable document rather than a prose blob or a database.
It holds the fictional Lantern service's conventions, architecture decisions, preferences, and gotchas. Every fact is a <section> with a data-witness (SHA-256 over its raw inner bytes) and a data-char-count. A reader can verify any single section in isolation: fetch #the-id, recompute the hash over what sits between the opening tag's > and </section>, and compare.
How to hydrate efficiently: read the manifest below to find the section you need, load only that anchor (#id), verify the witness, and inject it into your context. You do not need the whole file. This is the point.
Supersession: this document follows append-and-supersede discipline. When a fact changes, the original section remains (permanently addressable) and a new section carries data-supersedes="#old-id". Both are readable; the newer one is authoritative. See #datastore and #datastore-revised for a live example.
All content is obviously fictional and illustrative. No real service, person, or decision is described.
Lantern is a fictional internal notification-dispatch service. Callers POST a structured alert to /dispatch; Lantern fans it out to one or more channels (email via Postmark, webhook, SMS via Fictitious Telecom) and returns a receipt with per-channel delivery status.
It is a single Go binary deployed on two instances behind a load-balancer. There is no persistent HTTP session state; all state lives in the job queue and the delivery log.
github.example/acme/lantern (fictional)These are the team's standing agreements. New contributors are expected to follow them; deviations need a comment explaining why.
cmd/ for binaries, internal/ for all application code, pkg/ for code that other services might vendor. No lib/.fmt.Errorf("context: %w", err) at every boundary. Never discard an error silently. No sentinel errors exported from internal/.slog (Go 1.21+). Fields: level, ts (RFC 3339 UTC), msg, plus a req_id on every request-scoped log line. No log.Printf in production paths._test.go per package, no global state. Integration tests live under internal/integration/ and require a -tags integration build tag.init() functions are banned. Initialisation is explicit in main().Note: this section has been superseded by #datastore-revised. It is retained here as an append-only audit record. The current authoritative decision is in the newer section.
We use SQLite for the job queue. The schema lives in internal/queue/schema.sql. Jobs are inserted by the HTTP handler, polled by a background worker goroutine, and deleted on successful delivery or moved to a dead-letter table after three retries.
Rationale at the time: zero external dependencies, trivial local dev setup, acceptable throughput for the projected load (<500 jobs/min at peak). The file is placed on a local SSD mount; no replication.
Known limitation noted at decision time: SQLite write serialisation will become a bottleneck if concurrent dispatch volume exceeds ~1000 writes/s. That threshold was believed to be years away.
We migrated the job queue from SQLite to PostgreSQL 15 (managed instance, pg.lantern.example.internal) following the Q3 billing-alerts spike that hit 1,200 jobs/min and caused SQLite write-serialisation stalls of up to 800 ms.
The schema is unchanged in structure; the migration added SKIP LOCKED on the polling query so that multiple worker goroutines can dequeue concurrently without stepping on each other. Connection pooling via pgxpool (max 20 conns per instance).
internal/queue/schema.sql (unchanged; migration script at internal/queue/migrate_20251104.sql)/var/lantern/queue.db — decommissioned and archivedThe original SQLite decision is preserved in #datastore for audit purposes.
Email is dispatched via the Postmark transactional API (fictional account). The integration lives in internal/channel/email/postmark.go.
lantern-alerts (outbound, transactional)internal/ratelimit/). Postmark's own limit is 100/s; we stay under half to leave headroom for retries.#lantern-oncall.lantern@alerts.acme.example — do not change without coordinating with the Postmark account owner (domain SPF/DKIM records).Webhook delivery POSTs the alert payload as JSON to the caller-supplied URL. The integration is in internal/channel/webhook/deliver.go.
Signature: every request carries an X-Lantern-Signature header — HMAC-SHA256 of the raw request body, keyed with the caller's webhook secret (stored encrypted in the secrets manager). Receivers should verify this before processing.
Timeout: 5 s. We give webhooks a slightly longer window than email because caller endpoints vary wildly.
Retries: same 3-attempt exponential backoff as email. A non-2xx response is treated as a failure. A timeout is a failure. A connection refused is a failure and also fires a warning log so oncall can investigate stale webhook URLs.
Gotcha: redirects are not followed. If a caller's endpoint returns 301/302, delivery will fail. This is intentional — following redirects with a signed body is unsafe because the signature was computed for the original URL's receiver.
Local dev requires Docker (for Postgres) and Go 1.22+. There is no external dependency on real SMS or email services in the local environment.
git clone github.example/acme/lantern # fictional
cd lantern
docker compose up -d # starts postgres on :5432
make dev # builds + runs with .env.local
make test # unit tests (no -tags integration)
make test-integration # requires compose upEnvironment variables for local dev live in .env.local (not committed). Copy .env.example and fill in:
DATABASE_URL — defaults to postgres://lantern:lantern@localhost:5432/lantern?sslmode=disablePOSTMARK_SERVER_TOKEN — use the sandbox token from the team 1Password vault (fictional)WEBHOOK_SIGNING_KEY — any 32-byte hex string is fine locallySMS is a no-op stub in local and staging environments; it only activates when ENV=production.
This bit us in staging (fictional, 2025-11-18). Do not use SERIALIZABLE isolation on the dequeue transaction.
The polling query uses SELECT ... FOR UPDATE SKIP LOCKED. Under SERIALIZABLE isolation, Postgres can abort the transaction with a serialisation failure even when SKIP LOCKED is active, because the isolation level tracks the read set, not just the locked rows. Under high concurrent load this produced a storm of serialisation errors that looked like a Postgres outage.
The fix: the dequeue transaction must use READ COMMITTED (Go: pgx.TxOptions{IsoLevel: pgx.ReadCommitted}). SKIP LOCKED is safe and correct under READ COMMITTED; each worker grabs a non-overlapping set of rows with no phantom reads possible in this pattern.
The connection pool default is READ COMMITTED, but anyone calling pool.BeginTx with a custom options struct must not inadvertently set a higher isolation level.
Lantern does not use an ORM (no GORM, no Ent, no Bun). This is a deliberate team preference, not a gap.
Rationale: the query surface is small and stable (5 tables, ~15 queries). Hand-written SQL is readable by anyone who can read SQL, debuggable by pasting into psql, and optimisable without fighting a query builder. ORMs earn their keep at scale or when the schema is highly dynamic; neither applies here.
What we do use: sqlc to generate type-safe Go bindings from the hand-written .sql files in internal/queue/queries/. The generated code is committed; regenerate with make sqlc after editing queries.
Corollary: do not add an ORM as a dependency. If you find yourself reaching for one, the right move is to add a query to the .sql file and re-run make sqlc.
A job that fails all three delivery attempts is moved to the dead_letter table with a failure_reason column recording the last error message and a failed_at timestamp. The job is never deleted automatically.
Oncall alert: the worker posts to the #lantern-oncall Slack channel (fictional) via an incoming webhook when a job lands in dead-letter. The alert includes the job ID, channel, alert type, and failure reason.
Manual re-queue: to retry a dead-lettered job, use the admin script:
go run ./cmd/admin requeue --job-id=<uuid>This copies the job back into the main queue with a fresh attempt counter. Do not modify the dead_letter row directly — the audit trail must be intact.
Bulk re-queue after an outage: if a downstream channel had an extended outage and many jobs piled up, use --since=<RFC3339> and --channel=email flags to scope the re-queue. Always run in --dry-run first.
The /dispatch endpoint is currently unversioned. This is intentional: Lantern has one internal consumer cluster and changes are coordinated. We do not yet need a versioned surface.
When to introduce versioning: when an external team onboards as a consumer (outside the Platform Reliability perimeter), or when a breaking field change is needed. At that point, introduce /v2/dispatch and keep /dispatch (= v1) stable for a deprecation window of at least 90 days.
What counts as breaking: removing a required request field, changing the type of an existing field, removing a response field that consumers read, changing error codes. Adding optional request fields or adding response fields is non-breaking.
Do not add a version prefix pre-emptively. It adds ceremony with no current benefit and makes the API harder to curl in local dev. The policy is here so the decision, when it comes, is deliberate rather than improvised.