proxy-pool/docs/development/implementation-plan.md

# Proxy Pool Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use
> `superpowers:subagent-driven-development` or `superpowers:executing-plans`.
> Every step is tracked with checkbox syntax and must preserve the requirement IDs in
> `docs/requirements/traceability.md`.

**Goal:** Build a production-oriented Go repository whose domain behavior, interfaces,
configuration, contracts, documentation, and deployment layout implement the final
semantics in `对话内容.md`.

**Architecture:** Separate Gateway, Controller, Checker, and Loadgen commands. Domain
packages remain transport-free. Gateway reads immutable local snapshots. Controller
owns provider fetch, pool lifecycle, routing state, extraction, persistence, and worker
distribution. PostgreSQL persists management state and Admin audit/outbox. Redis owns
the rebuildable TTL Proxy activity pool and atomic extraction; Gateway hot paths remain
memory-only.

**Tech Stack:** Go 1.26, `go.yaml.in/yaml/v4`, pgx/v5, go-redis/v9, gRPC/Protobuf,
Prometheus, PostgreSQL, Redis, Docker Compose, Kubernetes.

---

## File Structure

```text
cmd/
  proxy-gateway/main.go
  proxy-controller/main.go
  proxy-checker/main.go
  proxy-loadgen/main.go
internal/
  config/{config.go,load.go,validate.go}
  domain/proxy/{proxy.go,state.go,capacity.go}
  domain/routing/{rule.go,strategy.go,sequential.go}
  domain/upstream/{upstream.go,fetch_result.go,pool.go}
  domain/extraction/{extraction.go,store.go}
  domain/client/client.go
  gateway/{server,dispatch,snapshot,transport}/
  controller/{provider,pool,routing,extraction,health,distribution,operations,runtime,bootstrap}/
  adapters/{memory,postgres,redis,providerapi}/
  platform/{logging,metrics,shutdown}/
api/{openapi,proto}/
configs/
deploy/{compose,kubernetes,haproxy,prometheus,grafana}/
docs/{design,development,configuration,api,operations,testing,adr,requirements}/
diagrams/
examples/
test/{fixtures,integration,e2e,load}/
```

## Task 1: Repository and Build Baseline

**Files:** `go.mod`, `.golangci.yml`, `README.md`, `scripts/verify.ps1`,
`.github/workflows/ci.yml`

- [x] Create Go module `proxy-pool` with Go 1.26.
- [x] Pin YAML v4, pgx/v5, go-redis/v9, gRPC, protobuf, Prometheus, and x/sync.
- [x] Add `scripts/verify.ps1` that runs format check, `go vet`, unit tests, race tests,
  and builds all commands, each test command bounded to 60 seconds.
- [x] Add CI for Windows and Linux with unit/race/build jobs.
- [x] Verify `go mod tidy`, `go test ./...`, and `go build ./cmd/...` succeed.

## Task 2: Strict Configuration

**Files:** `internal/config/config.go`, `load.go`, `validate.go`, corresponding tests,
`configs/default.yaml`, `docs/configuration/reference.md`

- [x] Define versioned types for security, gateway, distribution, admin, metrics,
  storage, routing, upstream/provider/api/proxyAuth/pool/capacity/lifecycle/fetch/check.
- [x] Decode one YAML document with known fields enabled and resolve `${ENV}` plus
  secret file references without logging values.
- [x] Validate listener protection, routing references/order, regexes, strategy fields,
  positive limits, TTL margins, pool/fetch limits, auth modes, and exposure modes.
- [x] Add table tests for every invalid condition in CFG requirements.

## Task 3: Proxy Domain and Capacity

**Files:** `internal/domain/proxy/*.go`, corresponding tests

- [x] Implement Proxy fields, UTC TTL precedence, canonical host/port, and unique key.
- [x] Implement state transitions and reject illegal transitions.
- [x] Implement packed per-Proxy runtime counters with CAS Reserve, Commit, Cancel, Release.
- [x] Prove with 1,000 concurrent goroutines that effective capacity is never exceeded.
- [x] Add race coverage and duplicate-release invariant metrics hook.

当前进度（2026-08-02）：固定 Max 下的每 Proxy 打包 CAS、Cancel/Commit/Release
生命周期、重复终结、错误顺序和同一 Reservation 并发终结已通过领域测试；已退出
Proxy 会停止新预留、保留非零 Drain 计数，并在归零后由后续 Snapshot Apply 回收。
Snapshot 降低 `maxConcurrency` 时会保留既有 Active 工作、禁止新预留，直到运行计数
低于新上限；运行态复用、降容和恢复预留均有回归测试。领域的
`CapacityInvariantObserver` 只发出固定生命周期违规枚举，Gateway 指标
`proxy_pool_gateway_capacity_invariant_violations_total{operation}` 不含 Proxy、请求或
Worker 维度；该测试纳入 Linux CI 的 race 范围，本机因 `CGO_ENABLED=0` 未执行 race。

## Task 4: Routing and Sequential Switching

**Files:** `internal/domain/routing/*.go`, corresponding tests

- [x] Compile first-match host/method/path/header rules into an immutable RuleSet.
- [x] Implement random, round-robin, weighted, least-connections, and sequential.
- [x] Model upstream empty counters separately from per-routing current indexes.
- [x] Implement versioned CAS switch so simultaneous threshold observers advance once.
- [x] Cover four-empty-then-success, five-empty, A-to-B-only, disabled references,
  end behavior, and explicit onUnavailable.

当前进度（2026-08-02）：领域构造器与严格配置已统一 Sequential 至少两个
Upstream、`endBehavior` 默认 `stop`，并覆盖列表末端停止。Provider Stats 现为每次
连续空结果分配单调代次；Controller 的公共 Sequential 协调器只接收有界通知，在独立
循环中读取权威配置、管理快照和 Stats，并以既有 `ExpectedCurrent` CAS 自动切换。
它会跳过禁用 Upstream，支持 `loop`/`stayLast`，同一空结果代次不会在循环后重复切换，
成功后立即广播完整 Snapshot。末端 `stop` 使用独立 `DisableRouting` 公用命令，
以 `ExpectedCurrent` CAS 原子停用 Routing，并在同一权威事务写入审计与 Outbox；
内存与 PostgreSQL Adapter 都运行相同并发、幂等和失败回滚契约。Gateway 已执行
reject/wait/direct，并会在快照刷新后看到停用状态。

## Task 5: Provider Fetch Classification and Scheduling

**Files:** `internal/controller/provider/*.go`, `internal/domain/upstream/*.go`, tests

- [x] Implement Valid, Empty, DuplicateOnly, and Error result classes exactly as the
  traceability matrix defines.
- [x] Implement one coalesced reconcile signal per Upstream using singleflight.
- [x] Enforce requestInterval, maxInFlight, maxSize, maxTotal, timeout, retry,
  exponential backoff, jitter, and Retry-After.
- [x] Define ProviderAdapter and safe TemplateParser ports; add fixture adapters.
- [x] Test that 100 concurrent capacity signals do not fan out 100 Provider calls.

## Task 6: Pool Reconciliation and Ownership

**Files:** `internal/controller/pool/*.go`, `internal/domain/upstream/pool.go`, tests

- [x] Compute Available Slots from eligible Proxy capacity, Active, Reserved, TTL,
  health, ownership, pending expected fetch, and gateway reserve.
- [x] Implement pool.maxSize and fetch.maxTotal as distinct counters.
- [x] Allocate each Proxy to one Worker with epoch/version/expiry ownership.
- [x] Implement revoke -> drain -> ACK -> unowned transition.
- [x] Test Worker crash expiry and prevent simultaneous dual ownership.

## Task 7: Exclusive Extraction

**Files:** `internal/domain/extraction/*.go`, `internal/controller/extraction/*.go`,
`internal/adapters/memory/extraction.go`, tests

- [x] Implement POST extraction command with protocol/region/carrier/upstream filters.
- [x] Enforce minRemainingTTL, maxHealthCheckAge, maxCount, client limits, and
  reserveForGateway.
- [x] Atomically remove selected AVAILABLE entries from the allocatable set and return
  the result; MemoryPool and the production Redis Adapter run the same shared contract.
- [x] Implement partial and allOrNothing without Lease, release, or renewal concepts.
- [x] Run 1,000 concurrent claim attempts and prove every Proxy ID appears at most once.

## Task 8: Immutable Snapshot and Dispatch

**Files:** `internal/gateway/snapshot/*.go`, `internal/gateway/dispatch/*.go`, tests

- [x] Define cluster/worker/epoch/version/checksum snapshot envelopes.
- [x] Build indexes before publication and atomically swap complete snapshots.
- [x] Reject version gaps and wrong epochs; request full resync.
- [x] Implement Dispatch Acquire/Commit/Release over local owned Proxy runtime.
- [x] Benchmark 100k Proxy snapshots and record allocations and latency.

## Task 9: Gateway Transport

**Files:** `internal/gateway/server/*.go`, `internal/gateway/transport/*.go`, tests

- [x] Implement HTTP forward proxy and HTTPS CONNECT through an upstream proxy.
- [x] Add Client auth/access/admission and destination policy checks before routing.
- [x] Implement safe retry commit points and prevent non-idempotent/established tunnel
  replay.
- [x] Use bounded buffers, deadlines, connection pools, and graceful shutdown.
- [x] Add local fake upstream end-to-end tests for success, 407, timeout, cancel, half
  close, retry, and blocked private destinations.

## Task 10: Controller APIs and Persistence Ports

**Files:** `internal/controller/distribution/*.go`, `admin/*.go`,
`internal/adapters/postgres/*.go`, `internal/adapters/redis/*.go`, migrations, tests

- [x] Define the PostgreSQL management seam for ConfigVersion, Upstream/Routing state,
  AdminAudit and leased Outbox; the public contract has no Proxy or extraction detail types.
- [x] Add the six-table PostgreSQL management migration with static data-boundary checks.
- [x] Implement the pgx PostgreSQL Adapter and run the public contract against PostgreSQL 18.
- [x] Implement the Redis TTL activity pool and one atomic extraction operation covering
  candidate eligibility, Gateway reserve, ownership, removal, and short-lived idempotency.
- [x] Implement Redis Worker ownership, drain/ACK, expiry reclaim, inventory and bounded
  sweep primitives with a monotonic global epoch.
- [x] Implement Redis Provider leader, distributed request quota, Client limit and
  automatic Provider inventory rebuild after Redis state loss.
- [x] Implement the Worker heartbeat receiving path and session lifecycle.
- [x] Keep Provider output in Redis TTL activity state and node memory only; keep the
  Gateway request path on immutable local snapshots with no Redis/PostgreSQL calls.
- [x] Expose Distribution extraction/status and Admin status/audit/enable/disable/switch/reload
  HTTP handlers and contracts with credential-level endpoint permissions and
  Distribution credential-level extraction boundaries; enforce matched
  Gateway credential routing boundaries before local dispatch.
- [x] Add Compose-backed Redis 8.2 integration and shared Adapter contract tests.
- [x] Add PostgreSQL management Adapter and Compose-backed integration tests.

当前进度（2026-07-31）：已实现共享 `platform/httpapi`、Distribution
extract/live/ready Handler 与 Admin status/enable/disable/switch/reload Handler；
定向契约测试已覆盖严格 JSON、Body 上限、Request ID、幂等 Header、DTO 映射、
404/405 及业务错误映射。共享 `platform/httpsecurity` 已补齐 Basic/API Key/
Bearer/CIDR、可信代理、Client ID、本地准入和 API 401/Gateway 407 差异，并作为
Admin/Distribution 必需依赖。共享 `platform/httpserver` 与
`controller/runtime` 已完成 Distribution/Admin 独立监听器、首错联动关闭和
有界优雅停机。Admin mutation 已携带认证 Actor/SourceIP；公用 `adminstate`
事务契约、MemoryStore、100 并发 Routing CAS、租约 Outbox 和六表管理 Schema
已完成。pgx Adapter 已在真实 PostgreSQL 18 上运行同一公用契约，并验证
Repeatable Read 快照、`SKIP LOCKED`、原子 ACK、审计/Outbox 故障回滚和数据边界。
Admin `ApplicationService` 已将 mutation、权威管理快照、低基数运行态
聚合与配置重载接到同一公用 seam；严格文件加载、外部密钥 HMAC 管理指纹及
revision 单调配置发布已通过失败路径和确定性并发测试。`cmd/proxy-controller` 与公用 `controller/bootstrap`
已完成配置单次加载、PostgreSQL 连接/迁移、Redis 活动池、状态聚合、
Distribution/Admin 服务构造、错误合并和资源关闭。生产 Provider Supervisor 已按
权威管理状态动态装配 Upstream，并与 HTTP Runtime 通过公用 lifecycle Group 联动
停机；Admin disable 会取消 Runtime，reload 在提交前预检并在发布后替换运行实例。
组合 fixture 已验证隔离 Redis namespace 下的选主、Provider HTTP 调用、模板解析
和活动池写入。Controller Metrics 独立入口现已提供 `/livez`、`/readyz` 与基础
Prometheus 运行时指标，三监听器隔离已通过测试；Checker、Gateway、Drain、
Provider、Extraction 与容量业务指标已接入；进程级密钥安全结构化日志也已落地。双存储 bootstrap 已通过 PostgreSQL 18 + Redis 8.2 组合 fixture，覆盖
迁移、启动配置提交、Readiness、Admin Status 和 Metrics 探针。

WorkerControlPlane 现已接入 Controller 生命周期：Register、ACK 和 Runtime
报告均经 Redis 服务端 TTL 的 session/issued-snapshot/ACK 栅栏校验；mTLS SPIFFE
身份、消息/流限制和有界停机已实现。`WatchSnapshots` 会发送当前 epoch 的基础完整
Snapshot 并保持连接；Gateway 已具备 Register/Watch/ACK/Runtime 会话协调组件与
`proxy-gateway` 进程装配。
按 Worker 的可下发 ownership 索引已进入 Redis 原子脚本，并可构建已归属 Proxy、
去重凭据材料和 Routing 的完整 payload。Snapshot 签发与 session 匹配在同一 Redis Lua 操作中完成，
重注册会清除旧引用，避免迟到 Stream 覆盖新 session。Worker 服务端会在最近完整
Snapshot 的 `valid_until` 到达时结束流；公用 `SessionSupervisor` 已为 Gateway 调用方
提供可恢复错误的有界指数退避重连，并在参数/认证/协议错误时停止。Gateway Routing
payload 已按配置顺序和 Admin revision/current 状态发布并覆盖 checksum；Gateway 已将其与
Proxy 原子编译为同版本 View，动态 Router 只匹配该未过期 View。派发器的五种上游选择已
接入该 View，并在容量耗尽时在同版本候选中回退；`onUnavailable` 的 reject、wait 与 direct
已接入 Gateway；`proxy-gateway` 已装配本地 HTTP/Metrics 监听、快照就绪探针和
控制面重连与快照凭据分发，以及有界 Outcome 上报、序列确认和精确重试。

已新增公用 `domain/activitypool` 契约及并发安全内存参考实现，Provider
Reconciler 通过 `UpsertFetched` 写入带供应商 TTL 和分配安全余量的批次；已覆盖
`usableUntil` 向 Worker Snapshot 的传播与 Gateway 本地截止过滤、
重复刷新、过期淘汰、独占提取、短期幂等及 Worker ownership 互斥。生产 Redis
Adapter 已通过真实 Redis 8.2 运行同一套公用契约；原子 Lua 覆盖提取、所有权和
有界清理。新增低基数 StateInventory Hash，五类写脚本在同一原子边界维护状态
计数，读取不扫描 Proxy 明细；过期清理积压或负计数时 fail-closed。Redis
Sentinel/故障转移验证与代表性多节点压测仍待实施。

Provider 分布式协调已新增公用 `Coordinator.RunLeader` / `LeaderSession` seam 与
独立 `redisprovider` Adapter。真实 Redis 8.2 已验证同 Upstream 双实例互斥、
generation + epoch fence、全局 requestInterval、全局 maxInFlight Permit、TTL
回收及 Redis 状态丢失后的新 generation 自动重建；Redis 异常期间不发放请求。
补池配置新增必填 `refill` 双水位和 `fetch.estimatedIPsPerCall`，Pool Reconciler
已实现迟滞与 pending 槽位折算，FetchBudget 仅在无 pending 时同步 Redis 权威
Managed。Gateway 已增加打包原子 Active/Reserved 读取与完整稀疏运行态快照；
公用 `workerruntime` session/report/read seam 同时提供并发安全内存参考实现和
生产 Redis Adapter。Redis 以服务端时间、Worker session、已 ACK snapshot/epoch、
单调 report sequence 和报告 TTL 原子隔离旧实例，并由 `pool.InventoryReader` 汇总
权威 Managed/Available Slots；真实 Redis 8.2 已覆盖空报告、幂等重放、倒序、
冲突、超前 epoch、过期、单 Upstream 扫描隔离和预算耗尽的 fail-closed 行为。
Redis Provider Permit 现已把 requestInterval、maxInFlight 与 maxTotal 放在同一
原子边界，按 expected 预留、实际合法数量结算，并支持失败保守计费、换主后结算和
过期回收。响应型代理凭据使用独立、有界 lease，在 Redis Upsert、候选截断或解析
失败后按版本释放；Provider Empty/Error 低基数计数已接入 Admin Status，配置删除
时回收历史统计容量。Redis inventory 扫描上限固定覆盖配置允许的最大池，支持小池
启动后动态扩容。Supervisor 以 PostgreSQL 权威 HMAC 指纹和 revision 栅栏协调
多副本 reload：管理库瞬断沿用 last-known 状态，本地共享源落后时停止旧 Provider，
源匹配并预检后自动替换，迟到旧 revision 不覆盖新配置。
Distribution 现通过公用 `admission.Admitter` 接入独立 `redisadmission` Adapter；
全局和单 Client 分钟额度使用 Redis 服务端时间并在单个 Lua 原子边界内检查、递增，
Controller 多副本共享同一计数。Client 身份只以 SHA-256 摘要进入 Redis，窗口切换
原子回收历史字段；Redis 异常 fail-closed 并返回 503，真实额度耗尽返回 429。
Gateway 请求热路径仍只使用本地准入，不增加 Redis/PostgreSQL 调用。
WorkerControlPlane gRPC 接收端、session 签发/心跳、Snapshot ACK 账本、基础
Snapshot 流、Gateway 会话客户端与快照凭据分发已完成；权威 Proxy/Routing 发布、
Outcome 上报已完成为有界队列、批次序列/摘要栅栏和确认重试；健康 BASIC/EGRESS/TARGET
执行链已完成；持续 UNHEALTHY 的未分配 Proxy 可由后台有界回收，拥有 Worker
所有权的项会通过健康/归属条件栅栏自动发起 Drain，并在排除快照 ACK 与运行态归零后回收。

## Task 11: Checker and Health Reducer

**Files:** `internal/controller/health/*.go`, `cmd/proxy-checker/main.go`, tests

- [x] Schedule global and route health with jitter and bounded maxInFlight.
- [x] Implement FETCHED -> CHECKING -> AVAILABLE and SUSPECT/UNHEALTHY transitions.
- [x] Ensure target failures affect only the target profile.
- [x] Add fixture target server and deterministic clock/scheduler tests.

当前进度（2026-07-31）：已新增不依赖 gRPC 的 `domain/health` Observation、全局/Target
Profile Reducer 与任务摘要幂等语义；BASIC/EGRESS 结果经活动池窄接口在 Memory 和 Redis
Lua 同一原子边界归并，覆盖首次 CHECKING 失败进入 UNHEALTHY、AVAILABLE/SUSPECT 的
阈值降级、精确重放、冲突拒绝与成功恢复。TARGET Profile 也已在 Memory 和 Redis 中独立
归并，以哈希键保存并随代理 TTL 过期，绝不写入 Proxy 全局状态或选择索引。调度器、
Checker Observation 上报 RPC 已复用既有控制面监听接入 Controller：SPIFFE `checker` 身份、
每批上限、配置阈值解析、代理归属查询与 Reducer 均已闭环。`StreamCheckTasks` 现已实现为
有界 pull，并通过通用任务 broker 契约完成能力协商、同 Checker 并发窗口、租约到期回收、
领取者加不可预测 lease token 的栅栏和完成后重放；任务凭据仅由认证流在执行期下发。
Redis 共享 due-index/租约持久化、每 Upstream 的跨副本 in-flight 限制和生产 broker 已完成；
`proxy-checker` 独立进程、固定大小 worker-pool、任务期重试/微批上报和 HTTP/HTTPS/SOCKS5
BASIC/EGRESS 探测器已完成并有测试。Redis 已按固定 EGRESS due-index 保存任务执行 URL，
与 BASIC 独立引用并通过上游共享并发限制；配置化监督器以有界轮转组调度每个 `check.urls`。
EGRESS 的出口身份响应解析支持固定上限的纯文本和常见 JSON IP 字段。TARGET 已以
`routing.check.targets` 作为配置入口，限定每个 Routing 的 URL 数和每个 Upstream 的 Profile
总数；Redis 为 `(routing_name, target_url, proxy_id)` 维护独立 due-index，完成、租约回收和
代理过期均沿用同一原子任务边界。BASIC、EGRESS 和 TARGET 分组轮转并共享 Upstream in-flight
上限，因此本任务已完成。持续 UNHEALTHY 回收使用每 Upstream 的
`check.unhealthyRemoveAfter`：全局 Reducer 保存首次进入 UNHEALTHY 的时间，Redis 以
有序索引有界扫描；没有 Worker ownership 的项沿用统一清理边界删除，已拥有 Worker 的
项通过包含 assignment epoch 与首次异常时间的候选，由条件式 Drain 原子重校验后进入
Ticket/排除快照/Runtime 零计数闭环。

补充进度（2026-08-02）：BASIC 调度已改为配置驱动监督器。它每轮读取已发布快照并复用
有界派发逻辑，所以 reload 后已启用上游的策略变更、停用，以及新启用上游都无需重启
Controller 即可生效。Redis 任务存储现已扩展 BASIC/EGRESS/TARGET 的独立有界索引；路由目标
Profile 在启用 Routing 与 Upstream 的组合上才进入调度。

补充进度（2026-08-02）：配置停用已进入 ownership Drain 编排。Controller 以静态配置
与 Admin 管理态的交集生成完整、版本化的 Upstream 策略视图；Redis 只从对应 `owned` 索引
有界读取仍可用的已归属 Proxy。候选携带 Upstream revision，`ownership.lua` 在复用既有
Ticket/排除 Snapshot/Runtime 零计数闭环前，原子复核策略仍为停用、Proxy source、Worker、
assignment epoch 和未过期租约。重启用后的策略 revision 会使旧候选返回无操作。Provider
Supervisor 也改为同时服从静态配置与管理态，消除两条启停消费链的不一致。

补充进度（2026-08-02）：已新增 `proxy-loadgen` HTTP 与 CONNECT 长连接场景。固定请求数
和固定时长两种模式均通过固定 worker 数与有界派发通道执行，可选 QPS 限速；报告使用固定大小
延迟直方图，输出状态分类、CONNECT 建立数、Extract 校验数、吞吐和 Go 内存/GC 快照。CONNECT
以原始 TCP 握手连接 HTTP Gateway，建连成功后按 `hold` 保持，且不透明读取隧道内容。`extract`
场景会自动生成独立 Request/Idempotency 标识，校验返回数量与单响应 ID 唯一性，且不记录地址或
凭据。报告可选按最大错误率和 p99 延迟门禁；违反阈值时仍完整写出 JSON 证据并以
专用退出码失败。故障注入以及代表性集群报告仍未实现。

## Task 12: Machine-readable Contracts

**Files:** `api/openapi/proxy-pool.yaml`, `api/proto/controlplane/v1/controlplane.proto`,
`docs/api/*.md`

- [x] Specify Distribution/Admin REST schemas, status codes, authentication, examples,
  and idempotency behavior.
- [x] Specify Worker register, snapshot, delta, ACK, report, heartbeat, ownership drain,
  and resync messages.
- [x] Validate OpenAPI contracts and compile protobuf descriptors/generated-code drift in CI.

当前进度（2026-07-29）：Distribution/Admin OpenAPI 已由 Go 测试在双平台 CI
校验本地 `$ref` 闭合、operationId 唯一、响应存在及 security scheme 引用；
`scripts/verify-proto.ps1` 已可复现编译包含 imports/source info 的 descriptor，
并使用 SHA-256 固定的 `protoc` 35.0 安装器在 CI 完整验证 descriptor 与生成代码漂移；
OpenAPI 结构契约由 Go 测试在双平台 CI 执行；`scripts/verify-openapi.ps1` 固定
`@redocly/cli@2.25.4`，按 OpenAPI 3.1 最小规则集验证两份文档并将 Tag 描述作为错误。

## Task 13: Deployment and Observability

**Files:** `deploy/**`, `internal/platform/**`, `docs/operations/**`

- [x] Add Compose for local Controller/Gateway/Checker/PostgreSQL/Redis/Prometheus/
  Grafana/HAProxy.
- [x] Add Kubernetes Deployments, Services, PDBs, HPA, NetworkPolicy, Secrets examples,
  probes, resource limits, topology spread, and graceful termination.
- [ ] Add the production Kubernetes mTLS identity overlay, certificate rotation and unique
  elastic Worker identity wiring.
- [x] Add low-cardinality Prometheus metrics and structured secret-safe logs.
- [x] Document backup, recovery, rollout, rollback, capacity, kernel, file descriptor,
  NAT/conntrack, and incident runbooks.

当前进度（2026-08-02）：已接入 Checker 任务流指标
`proxy_pool_checker_tasks_dispatched_total{level}` 与
`proxy_pool_checker_observations_total{level,result}`；标签值仅允许固定的
`BASIC`、`EGRESS`、`TARGET` 级别及 `accepted`、`rejected` 结果。Gateway 还暴露
`proxy_pool_gateway_outcomes_total{stage,result}` 和
`proxy_pool_gateway_outcome_queue_dropped_total`，其中阶段和结果均为固定枚举。Provider、
拉取已暴露 `proxy_pool_controller_provider_fetch_results_total{class}`、
`proxy_pool_controller_provider_valid_candidates_total` 与
`proxy_pool_controller_provider_new_proxies_total`；`class` 仅允许 `valid`、`empty`、
`duplicate_only`、`error`。Extraction 还暴露固定 `result` 的请求次数、请求数与响应
交付数，幂等重放按响应交付统计。容量指标由既有 Provider 库存对账周期聚合，不进入
Gateway 热路径。`platform/logging` 以 JSON `slog` 输出进程级致命错误，字段、URL
用户信息、查询 Secret 和错误对象均经过脱敏，且不在请求热路径逐条写日志。
CI 另有独立 Deployment job，在占位凭据下渲染 Compose，并使用 `kubectl kustomize`
渲染 Kubernetes base 和单副本 development mTLS Overlay；该 job 同时展开测试 Compose、
校验本地 Compose 配置。它不启动容器、不访问真实存储或密钥。
独立 Image job 会构建 Linux 多阶段应用镜像，覆盖 Dockerfile、Linux 交叉编译与运行时层；
容器服务编排和业务运行链路仍需在具备镜像网络的环境中验证。
development Overlay 已挂载独立 Controller/Gateway/Checker TLS Secret、启用 Checker 和最小
网络策略，并从固定开发证书派生 `gateway-a`、`checker-a` 身份且 HPA 锁为单副本，不能代替生产证书轮换。
控制面 TLS 叶证书和信任根已在每次新握手时从文件加载，短暂文件不一致时保留最后一次有效材料；
生产 Overlay 仍必须由工作负载身份系统提供每副本唯一 SPIFFE URI 和弹性身份注入。

## Task 14: Documentation, Examples, and Diagrams

**Files:** `docs/**`, `examples/**`, `diagrams/**`

- [x] Complete README navigation, design document, developer guide, configuration
  reference, API guide, deployment guide, security model, testing guide, and roadmap.
- [x] Provide at least 20 validated configuration examples.
- [x] Provide at least 30 Mermaid architecture, flow, sequence, state, and failure diagrams.
- [x] Generate `proxy-pool-docs-v1.0.zip` from versioned documentation assets.

`scripts/package-docs.ps1` 会将 README、`docs/`、图表、OpenAPI、Proto 契约与部署手册复制至临时目录，
生成包含 Git revision、文件大小和 SHA-256 的 `manifest.json`，再以临时 ZIP 原子替换目标。
默认输出为被 Git 忽略的 `dist/proxy-pool-docs-v1.0.zip`；Go 回归测试实际执行脚本并校验归档内容。

## Task 15: Completion Audit

- [x] Map every requirement ID to code, test, contract, document, or verified runtime evidence.
- [ ] Run `gofmt`, `go vet`, unit tests, race tests, builds, contract validation, and
  documentation link/example validation.
- [x] Run bounded local performance benchmarks; label 100k QPS as unverified until a
  representative cluster load run exists.

`scripts/benchmark-gateway.ps1` 固定使用 `-count=1 -benchtime=1x` 分别执行 100k 索引调度、
Round Robin 和 Snapshot Apply 基准，并将 Git revision 与原始输出保存到 `dist/`。该基线不包含
网络、TLS、上游 RTT、真实连接或多 Worker 协调，不能替代代表性集群报告。
- [x] Confirm no TODO/TBD/placeholders, secrets, unbounded queues, high-cardinality metric
  labels, extraction Lease APIs, or conflicting maxSize semantics remain.

`docs` 回归测试扫描生产源与部署清单中的未完成标记和私钥材料；OpenAPI 测试拒绝
Extract Lease/Release/Renew 路径，Gateway Outcome Queue 构造器和配置校验拒绝无界容量，
指标/部署契约拒绝高基数标签，`pool.maxSize` 与累计 `fetch.maxTotal` 的边界由配置和
活动池契约测试共同锁定。示例、CI fixture 与环境变量引用不是未实现占位或生产 Secret。