docs: design postgres admin state module
This commit is contained in:
youfak
parent
63bc56be27
commit
08800cb7a5
142
docs/adr/006-postgresql-admin-state.md
Normal file
142
docs/adr/006-postgresql-admin-state.md
Normal file
@ -0,0 +1,142 @@
|
||||
# ADR-006：PostgreSQL 管理面采用事务深模块
|
||||
|
||||
## 状态
|
||||
|
||||
接受，2026-07-29。
|
||||
|
||||
## 背景
|
||||
|
||||
Admin HTTP Handler 已定义 Upstream 启停、Routing 手工切换、配置重载和状态查询，
|
||||
但权威管理状态、审计与 Outbox 还没有生产存储实现。PostgreSQL 只属于控制面；
|
||||
Proxy 地址、凭据、生命周期、Worker 所有权、逐次提取记录和短期幂等结果都不能
|
||||
进入 PostgreSQL。
|
||||
|
||||
管理写入存在一个不可拆分的不变量：业务状态、Admin 审计和 Outbox 必须在同一
|
||||
PostgreSQL 事务内提交。若把它们暴露成多个 Repository，调用方容易漏写审计、
|
||||
漏发事件或在失败时留下部分状态。
|
||||
|
||||
## 决策
|
||||
|
||||
### 模块与 seam
|
||||
|
||||
新增 `domain/adminstate` 公用契约。调用方按用途依赖窄接口，一个 Adapter 可以
|
||||
同时实现全部接口：
|
||||
|
||||
```go
|
||||
type Mutator interface {
|
||||
SetUpstreamEnabled(context.Context, SetUpstreamCommand) (MutationResult, error)
|
||||
SwitchRouting(context.Context, SwitchRoutingCommand) (MutationResult, error)
|
||||
CommitConfig(context.Context, CommitConfigCommand) (MutationResult, error)
|
||||
}
|
||||
|
||||
type SnapshotReader interface {
|
||||
Snapshot(context.Context) (Snapshot, error)
|
||||
}
|
||||
|
||||
type Outbox interface {
|
||||
Claim(context.Context, ClaimCommand) ([]Event, error)
|
||||
Acknowledge(context.Context, AcknowledgeCommand) error
|
||||
}
|
||||
```
|
||||
|
||||
三个 mutation 方法保留现有 Admin 语义，事务、修订号、CAS、审计和 Outbox 都
|
||||
隐藏在模块实现内部。不会暴露 `BeginTx`、SQL executor 或五个可被错误组合的浅
|
||||
Repository。
|
||||
|
||||
`MemoryStore` 是并发安全参考 Adapter；PostgreSQL Adapter 必须运行同一套公用
|
||||
行为契约。Controller Admin 应用层只负责 DTO 映射、配置解析/校验和运行态发布，
|
||||
不自行拼装数据库事务。
|
||||
|
||||
### 全局修订
|
||||
|
||||
每次产生管理状态变化时分配单调递增的全局 `revision`：
|
||||
|
||||
- Upstream 目标状态已满足时返回 `changed=false`，不增加修订。
|
||||
- Routing 的 `expectedCurrent` 不匹配时返回冲突，不写任何记录。
|
||||
- 相同配置版本和校验和再次提交时返回 `changed=false`。
|
||||
- 相同配置版本对应不同校验和时返回冲突。
|
||||
- `MutationResult.version` 是已提交的全局管理修订，不是 YAML 格式版本或 Worker
|
||||
Snapshot 版本。
|
||||
|
||||
每次合法 Admin mutation 都写审计，包括 `changed=false`；只有真实状态变化写
|
||||
Outbox。状态变化、审计和 Outbox 在同一事务提交。
|
||||
|
||||
### 配置修订
|
||||
|
||||
配置提交只保存管理面恢复所需的非敏感事实：配置版本、SHA-256 校验和、来源、
|
||||
Upstream 启用状态和 Routing 候选/当前选择。已解析 Secret、Provider Token、
|
||||
Proxy 凭据和完整运行时对象不进入 PostgreSQL。
|
||||
|
||||
配置重载以一个事务替换管理快照。新 Routing 的当前 Upstream 必须属于其候选集，
|
||||
所有引用的 Upstream 必须存在，名称与列表必须非空且唯一。校验失败发生在事务前，
|
||||
旧修订继续生效。
|
||||
|
||||
### Routing CAS
|
||||
|
||||
`SwitchRouting` 在单条事务中锁定目标 Routing，并同时校验：
|
||||
|
||||
1. Routing 存在且启用；
|
||||
2. `expectedCurrent` 等于权威当前值；
|
||||
3. `target` 属于该 Routing 的候选 Upstream；
|
||||
4. 目标与当前值不同。
|
||||
|
||||
并发使用同一 expected 值时最多一个请求成功。目标等于当前值且 expected 匹配时
|
||||
返回 `changed=false`，仍写审计但不写 Outbox。
|
||||
|
||||
### Outbox 消费
|
||||
|
||||
Outbox 使用有界 claim/ack，而不是无界全表扫描：
|
||||
|
||||
- `Claim` 要求稳定 consumer ID、当前时间、正租期和有界 limit。
|
||||
- 未发布且未被有效租约占用的事件按序 claim。
|
||||
- `Acknowledge` 只允许当前 consumer 在租期内确认自己 claim 的事件。
|
||||
- 发布失败不 ACK，租期过后可由其他 consumer 重试。
|
||||
|
||||
事件 payload 只包含管理资源名、目标状态、修订和必要原因，不包含 Secret、
|
||||
Proxy、Client 或完整配置正文。
|
||||
|
||||
### Schema 边界
|
||||
|
||||
首个迁移只创建：
|
||||
|
||||
```text
|
||||
control_revisions
|
||||
config_revisions
|
||||
upstream_admin_state
|
||||
routing_admin_state
|
||||
admin_audit_log
|
||||
admin_outbox
|
||||
```
|
||||
|
||||
迁移和集成测试必须断言不存在 Proxy 明细、Worker ownership、提取记录或短期
|
||||
幂等表。PostgreSQL 故障只使管理写入失败关闭，不改变 Redis Extract 的可用性。
|
||||
|
||||
## 测试与验收
|
||||
|
||||
MemoryStore 与 PostgreSQL Adapter 运行相同契约，至少覆盖：
|
||||
|
||||
- 配置首次提交、相同重放、校验和冲突和非法引用零写入。
|
||||
- Upstream enable/disable 幂等、修订单调、审计必写、Outbox 仅在变更时写。
|
||||
- Routing CAS、目标校验和 100 个并发请求最多一个成功。
|
||||
- 任一审计/Outbox 写故障导致状态完全回滚。
|
||||
- Outbox 有界 claim、租约到期重试、错误 consumer ACK 拒绝和顺序稳定。
|
||||
- 上下文取消、错误脱敏和 Snapshot 防止调用方修改内部状态。
|
||||
- 真实 PostgreSQL 重复迁移、事务回滚和禁止数据表边界。
|
||||
|
||||
所有测试命令最长 60 秒。100,000 QPS 属于 Gateway 集群目标，不以管理面数据库
|
||||
测试推导吞吐结论。
|
||||
|
||||
## 后果
|
||||
|
||||
收益：调用方无法绕过事务不变量；Memory/PostgreSQL 行为一致；管理数据边界可
|
||||
通过 Schema 自动验证；Outbox 重试有界且可观测。
|
||||
|
||||
代价：PostgreSQL Adapter 内部实现较深；配置提交需要完整管理快照；Outbox
|
||||
dispatcher 需要租约续期或确保单批发布时间小于 claim TTL。
|
||||
|
||||
## 不采用的方案
|
||||
|
||||
- 五个公开 Repository 加公开事务管理器：接口浅，调用方容易产生部分提交。
|
||||
- 将所有命令塞入一个弱类型 `Command`：入口最少，但 Go 调用方需要运行时判断
|
||||
联合字段，错误更晚暴露。
|
||||
- 把 Proxy 或提取事实写入 PostgreSQL：违反短效活动池与数据最小化边界。
|
||||
@ -47,3 +47,15 @@ Provider 入池、健康状态、Distribution 独占提取、Worker 所有权、
|
||||
|
||||
完整决策、键空间、原子操作和测试门禁见
|
||||
[ADR-005](005-redis-activity-pool.md)。
|
||||
|
||||
## ADR-006：PostgreSQL 管理面采用事务深模块
|
||||
|
||||
**状态：** 接受。
|
||||
|
||||
Upstream/Routing/配置修订、Admin 审计和 Outbox 由一个深 Adapter 在同一事务
|
||||
提交。调用方只依赖 mutation、snapshot 和有界 outbox 接口，不接触数据库事务
|
||||
或细粒度 Repository。PostgreSQL Schema 明确排除 Proxy、Worker ownership、
|
||||
逐次提取记录和短期幂等结果。
|
||||
|
||||
完整接口、事务不变量、Schema 与公用契约见
|
||||
[ADR-006](006-postgresql-admin-state.md)。
|
||||
|
||||
149
docs/superpowers/plans/2026-07-29-postgresql-admin-state.md
Normal file
149
docs/superpowers/plans/2026-07-29-postgresql-admin-state.md
Normal file
@ -0,0 +1,149 @@
|
||||
# PostgreSQL Admin State Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use
|
||||
> superpowers:subagent-driven-development (recommended) or superpowers:executing-plans
|
||||
> to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Build the authoritative PostgreSQL management-state module for config revisions,
|
||||
Upstream/Routing mutations, Admin audit and reliable Outbox without storing Proxy or
|
||||
per-extraction data.
|
||||
|
||||
**Architecture:** `domain/adminstate` defines narrow capability interfaces implemented by a
|
||||
concurrency-safe MemoryStore and one deep PostgreSQL Adapter. Every mutation hides a single
|
||||
transaction that commits state, audit and Outbox together. Both adapters run the same public
|
||||
contract; Controller Admin maps its existing typed commands onto this seam.
|
||||
|
||||
**Tech Stack:** Go 1.26, standard library, pgx/v5, PostgreSQL 18, Docker Compose.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Domain Contracts and Immutable Values
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/domain/adminstate/adminstate.go`
|
||||
- Create: `internal/domain/adminstate/validation_test.go`
|
||||
|
||||
- [ ] Define `Mutator`, `SnapshotReader` and `Outbox` interfaces from ADR-006.
|
||||
- [ ] Define typed config, Upstream, Routing, actor, audit, event and mutation values.
|
||||
- [ ] Define stable errors for invalid input, missing resources, CAS conflict and unavailable
|
||||
storage.
|
||||
- [ ] Validate non-empty bounded identifiers, UTC timestamps, unique lists, config checksum,
|
||||
Routing references, claim limits and claim TTL.
|
||||
- [ ] Clone every slice/map/JSON value at the seam so callers cannot mutate stored state.
|
||||
- [ ] Run `go test -count=1 -timeout 60s ./internal/domain/adminstate` and verify the tests
|
||||
fail before implementation, then pass after implementation.
|
||||
- [ ] Commit with `feat: define admin state transaction contracts`.
|
||||
|
||||
### Task 2: Memory Reference Adapter and Shared Contract
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/domain/adminstate/memory.go`
|
||||
- Create: `internal/domain/adminstate/contracttest/contract.go`
|
||||
- Create: `internal/domain/adminstate/contract_external_test.go`
|
||||
|
||||
- [ ] Write a public contract factory that can create an isolated `Mutator + SnapshotReader +
|
||||
Outbox` implementation.
|
||||
- [ ] Cover config commit/replay/conflict/invalid references and zero-write rollback.
|
||||
- [ ] Cover Upstream idempotency, monotonic revisions, mandatory audit and changed-only events.
|
||||
- [ ] Cover Routing CAS and 100 concurrent switches with at most one success.
|
||||
- [ ] Cover bounded claim, exclusive claim, lease expiry, ACK ownership and stable ordering.
|
||||
- [ ] Cover context cancellation and immutable Snapshot/output values.
|
||||
- [ ] Implement MemoryStore behind the seam with one mutex per atomic management state.
|
||||
- [ ] Run `go test -count=1 -timeout 60s ./internal/domain/adminstate/...`.
|
||||
- [ ] Commit with `feat: add transactional admin state reference store`.
|
||||
|
||||
### Task 3: Schema Migration and Static Data-Boundary Test
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/adapters/postgresadmin/migrations/0001_admin_state.sql`
|
||||
- Create: `internal/adapters/postgresadmin/migrations.go`
|
||||
- Create: `internal/adapters/postgresadmin/migrations_test.go`
|
||||
|
||||
- [ ] Embed ordered migrations and expose one `Migrations() []Migration` read-only accessor.
|
||||
- [ ] Create only the six ADR-006 tables with primary/foreign keys, UTC timestamps, indexes,
|
||||
outbox claim fields and bounded checks.
|
||||
- [ ] Add a parser-backed/static test that asserts required tables/columns are present and
|
||||
forbidden Proxy/extraction/ownership/idempotency tables or columns are absent.
|
||||
- [ ] Test migration IDs are unique, strictly ordered and statements are transactional.
|
||||
- [ ] Run `go test -count=1 -timeout 60s ./internal/adapters/postgresadmin`.
|
||||
- [ ] Commit with `feat: add postgres admin state schema`.
|
||||
|
||||
### Task 4: PostgreSQL Deep Adapter
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/adapters/postgresadmin/adapter.go`
|
||||
- Create: `internal/adapters/postgresadmin/mutate.go`
|
||||
- Create: `internal/adapters/postgresadmin/snapshot.go`
|
||||
- Create: `internal/adapters/postgresadmin/outbox.go`
|
||||
- Create: `internal/adapters/postgresadmin/codec.go`
|
||||
- Modify: `go.mod`
|
||||
- Modify: `go.sum`
|
||||
|
||||
- [ ] Add the approved pinned pgx/v5 dependency without changing unrelated modules.
|
||||
- [ ] Accept a narrow pgx pool interface and options; reject nil pools and invalid namespaces.
|
||||
- [ ] Implement Config/Upstream/Routing mutations with SQL validation, row locks/CAS and one
|
||||
transaction for revision, state, audit and Outbox.
|
||||
- [ ] Map PostgreSQL constraint/CAS/connection errors to domain errors without leaking DSNs,
|
||||
SQL or values.
|
||||
- [ ] Implement immutable Snapshot reads from one repeatable-read transaction.
|
||||
- [ ] Implement bounded Outbox claim using `FOR UPDATE SKIP LOCKED` and guarded ACK.
|
||||
- [ ] Keep SQL, tx retries, codecs and driver types private to the Adapter.
|
||||
- [ ] Run focused unit tests and `go vet ./internal/adapters/postgresadmin/...`.
|
||||
- [ ] Commit with `feat: implement postgres admin state adapter`.
|
||||
|
||||
### Task 5: Real PostgreSQL Contract Fixture
|
||||
|
||||
**Files:**
|
||||
- Create: `deploy/docker-compose.postgres-test.yml`
|
||||
- Create: `scripts/test-postgres.ps1`
|
||||
- Create: `internal/adapters/postgresadmin/testpostgres_test.go`
|
||||
- Create: `internal/adapters/postgresadmin/contract_integration_test.go`
|
||||
- Create: `internal/adapters/postgresadmin/rollback_integration_test.go`
|
||||
|
||||
- [ ] Start an isolated PostgreSQL 18 fixture on a dedicated loopback port and database.
|
||||
- [ ] Apply migrations through the same migration runner used by the production Adapter.
|
||||
- [ ] Run the public adminstate contract against a unique schema per test.
|
||||
- [ ] Inject audit and Outbox constraint failures and prove state/revision rollback.
|
||||
- [ ] Query `information_schema` and prove no Proxy, extraction, ownership or idempotency
|
||||
detail tables/columns exist.
|
||||
- [ ] Clean only the unique test schema; do not drop shared databases or use broad cleanup.
|
||||
- [ ] Run `.\scripts\test-postgres.ps1` with every Go test timeout set to 60 seconds.
|
||||
- [ ] Commit with `test: add postgres admin state contract fixture`.
|
||||
|
||||
### Task 6: Admin Application Integration
|
||||
|
||||
**Files:**
|
||||
- Create: `internal/controller/admin/service.go`
|
||||
- Create: `internal/controller/admin/service_test.go`
|
||||
- Modify: `internal/controller/admin/handler.go`
|
||||
- Modify: `internal/controller/admin/handler_test.go`
|
||||
|
||||
- [ ] Change Admin protection to resolve `httpsecurity.Identity` once and add actor/source IP
|
||||
to mutation commands without exposing credentials.
|
||||
- [ ] Map typed Handler commands to `adminstate.Mutator`; map domain conflict/not-found/
|
||||
invalid/unavailable errors to the existing HTTP contract.
|
||||
- [ ] Build Status from one adminstate Snapshot plus injected activity/worker aggregate readers.
|
||||
- [ ] Reload configuration with existing strict loader/validator, submit a secret-free management
|
||||
snapshot, then atomically publish runtime config only after persistence succeeds.
|
||||
- [ ] Test persistence failure leaves the current runtime config unchanged and Redis Extract is
|
||||
not referenced by the Admin module.
|
||||
- [ ] Run `go test -count=1 -timeout 60s ./internal/controller/admin/...`.
|
||||
- [ ] Commit with `feat: connect admin API to management state`.
|
||||
|
||||
### Task 7: Documentation and Delivery Verification
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/development/implementation-plan.md`
|
||||
- Modify: `docs/requirements/completion-audit.md`
|
||||
- Modify: `docs/testing/test-strategy.md`
|
||||
- Modify: `docs/operations/runbook.md`
|
||||
- Modify: `progress.md`
|
||||
|
||||
- [ ] Mark only verified PostgreSQL capabilities complete and retain command/runtime/load gaps.
|
||||
- [ ] Document migration, backup, Outbox backlog/replay and data-boundary checks.
|
||||
- [ ] Run `.\scripts\verify.ps1`, `.\scripts\test-redis.ps1`,
|
||||
`.\scripts\test-postgres.ps1` and `git diff --check`.
|
||||
- [ ] Audit that Gateway has no PostgreSQL/Redis dependency and Distribution has no PostgreSQL
|
||||
dependency.
|
||||
- [ ] Confirm the user-owned deletion of `proxy-pool-docs-v1.0.zip` is not staged.
|
||||
- [ ] Commit with `docs: record postgres admin state delivery` and push the feature branch.
|
||||
Loading…
Reference in New Issue
Block a user