# Proxy Pool Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use
> `superpowers:subagent-driven-development` or `superpowers:executing-plans`.
> Every step is tracked with checkbox syntax and must preserve the requirement IDs in
> `docs/requirements/traceability.md`.

**Goal:** Build a production-oriented Go repository whose domain behavior, interfaces,
configuration, contracts, documentation, and deployment layout implement the final
semantics in `对话内容.md`.

**Architecture:** Separate Gateway, Controller, Checker, and Loadgen commands. Domain
packages remain transport-free. Gateway reads immutable local snapshots. Controller
owns provider fetch, pool lifecycle, routing state, extraction, persistence, and worker
distribution. PostgreSQL persists management state and Admin audit/outbox. Redis owns
the rebuildable TTL Proxy activity pool and atomic extraction; Gateway hot paths remain
memory-only.

**Tech Stack:** Go 1.26, `go.yaml.in/yaml/v4`, pgx/v5, go-redis/v9, gRPC/Protobuf,
Prometheus, PostgreSQL, Redis, Docker Compose, Kubernetes.

---

## File Structure

```text
cmd/
  proxy-gateway/main.go
  proxy-controller/main.go
  proxy-checker/main.go
  proxy-loadgen/main.go
internal/
  config/{config.go,load.go,validate.go}
  domain/proxy/{proxy.go,state.go,capacity.go}
  domain/routing/{rule.go,strategy.go,sequential.go}
  domain/upstream/{upstream.go,fetch_result.go,pool.go}
  domain/extraction/{extraction.go,store.go}
  domain/client/client.go
  gateway/{server,dispatch,snapshot,transport}/
  controller/{provider,pool,routing,extraction,health,distribution,operations,runtime,bootstrap}/
  adapters/{memory,postgres,redis,providerapi}/
  platform/{logging,metrics,shutdown}/
api/{openapi,proto}/
configs/
deploy/{compose,kubernetes,haproxy,prometheus,grafana}/
docs/{design,development,configuration,api,operations,testing,adr,requirements}/
diagrams/
examples/
test/{fixtures,integration,e2e,load}/
```

## Task 1: Repository and Build Baseline

**Files:** `go.mod`, `.golangci.yml`, `README.md`, `scripts/verify.ps1`,
`.github/workflows/ci.yml`

- [ ] Create module `github.com/proxy-pool/proxy-pool` with Go 1.26.
- [ ] Pin YAML v4, pgx/v5, go-redis/v9, gRPC, protobuf, Prometheus, and x/sync.
- [x] Add `scripts/verify.ps1` that runs format check, `go vet`, unit tests, race tests,
  and builds all commands, each test command bounded to 60 seconds.
- [x] Add CI for Windows and Linux with unit/race/build jobs.
- [ ] Verify `go mod tidy`, `go test ./...`, and `go build ./cmd/...` succeed.

## Task 2: Strict Configuration

**Files:** `internal/config/config.go`, `load.go`, `validate.go`, corresponding tests,
`configs/default.yaml`, `docs/configuration/reference.md`

- [x] Define versioned types for security, gateway, distribution, admin, metrics,
  storage, routing, upstream/provider/api/proxyAuth/pool/capacity/lifecycle/fetch/check.
- [x] Decode one YAML document with known fields enabled and resolve `${ENV}` plus
  secret file references without logging values.
- [x] Validate listener protection, routing references/order, regexes, strategy fields,
  positive limits, TTL margins, pool/fetch limits, auth modes, and exposure modes.
- [x] Add table tests for every invalid condition in CFG requirements.

## Task 3: Proxy Domain and Capacity

**Files:** `internal/domain/proxy/*.go`, corresponding tests

- [x] Implement Proxy fields, UTC TTL precedence, canonical host/port, and unique key.
- [x] Implement state transitions and reject illegal transitions.
- [ ] Implement sharded runtime counters with CAS Reserve, Commit, Cancel, Release.
- [x] Prove with 1,000 concurrent goroutines that effective capacity is never exceeded.
- [ ] Add race coverage and duplicate-release invariant metrics hook.

当前进度（2026-07-29）：固定 Max 下的每 Proxy 打包 CAS、Cancel/Commit/Release
生命周期、重复终结、错误顺序和同一 Reservation 并发终结已通过领域测试；动态
降容契约、低基数不变量指标、Linux race 证据及短 TTL runtime 排空回收待完成。

## Task 4: Routing and Sequential Switching

**Files:** `internal/domain/routing/*.go`, corresponding tests

- [x] Compile first-match host/method/path/header rules into an immutable RuleSet.
- [x] Implement random, round-robin, weighted, least-connections, and sequential.
- [x] Model upstream empty counters separately from per-routing current indexes.
- [x] Implement versioned CAS switch so simultaneous threshold observers advance once.
- [ ] Cover four-empty-then-success, five-empty, A-to-B-only, disabled references,
  end behavior, and explicit onUnavailable.

当前进度（2026-07-29）：领域构造器与严格配置已统一 Sequential 至少两个
Upstream、`endBehavior` 默认 `stop`，并覆盖列表末端停止；disabled candidate、
跨实例恢复和 `onUnavailable` 运行链仍待完成。

## Task 5: Provider Fetch Classification and Scheduling

**Files:** `internal/controller/provider/*.go`, `internal/domain/upstream/*.go`, tests

- [x] Implement Valid, Empty, DuplicateOnly, and Error result classes exactly as the
  traceability matrix defines.
- [x] Implement one coalesced reconcile signal per Upstream using singleflight.
- [x] Enforce requestInterval, maxInFlight, maxSize, maxTotal, timeout, retry,
  exponential backoff, jitter, and Retry-After.
- [x] Define ProviderAdapter and safe TemplateParser ports; add fixture adapters.
- [x] Test that 100 concurrent capacity signals do not fan out 100 Provider calls.

## Task 6: Pool Reconciliation and Ownership

**Files:** `internal/controller/pool/*.go`, `internal/domain/upstream/pool.go`, tests

- [ ] Compute Available Slots from eligible Proxy capacity, Active, Reserved, TTL,
  health, ownership, pending expected fetch, and gateway reserve.
- [x] Implement pool.maxSize and fetch.maxTotal as distinct counters.
- [x] Allocate each Proxy to one Worker with epoch/version/expiry ownership.
- [x] Implement revoke -> drain -> ACK -> unowned transition.
- [x] Test Worker crash expiry and prevent simultaneous dual ownership.

## Task 7: Exclusive Extraction

**Files:** `internal/domain/extraction/*.go`, `internal/controller/extraction/*.go`,
`internal/adapters/memory/extraction.go`, tests

- [x] Implement POST extraction command with protocol/region/carrier/upstream filters.
- [x] Enforce minRemainingTTL, maxHealthCheckAge, maxCount, client limits, and
  reserveForGateway.
- [x] Atomically remove selected AVAILABLE entries from the allocatable set and return
  the result; MemoryPool and the production Redis Adapter run the same shared contract.
- [x] Implement partial and allOrNothing without Lease, release, or renewal concepts.
- [x] Run 1,000 concurrent claim attempts and prove every Proxy ID appears at most once.

## Task 8: Immutable Snapshot and Dispatch

**Files:** `internal/gateway/snapshot/*.go`, `internal/gateway/dispatch/*.go`, tests

- [x] Define cluster/worker/epoch/version/checksum snapshot envelopes.
- [x] Build indexes before publication and atomically swap complete snapshots.
- [x] Reject version gaps and wrong epochs; request full resync.
- [x] Implement Dispatch Acquire/Commit/Release over local owned Proxy runtime.
- [x] Benchmark 100k Proxy snapshots and record allocations and latency.

## Task 9: Gateway Transport

**Files:** `internal/gateway/server/*.go`, `internal/gateway/transport/*.go`, tests

- [x] Implement HTTP forward proxy and HTTPS CONNECT through an upstream proxy.
- [x] Add Client auth/access/admission and destination policy checks before routing.
- [x] Implement safe retry commit points and prevent non-idempotent/established tunnel
  replay.
- [x] Use bounded buffers, deadlines, connection pools, and graceful shutdown.
- [x] Add local fake upstream end-to-end tests for success, 407, timeout, cancel, half
  close, retry, and blocked private destinations.

## Task 10: Controller APIs and Persistence Ports

**Files:** `internal/controller/distribution/*.go`, `admin/*.go`,
`internal/adapters/postgres/*.go`, `internal/adapters/redis/*.go`, migrations, tests

- [x] Define the PostgreSQL management seam for ConfigVersion, Upstream/Routing state,
  AdminAudit and leased Outbox; the public contract has no Proxy or extraction detail types.
- [x] Add the six-table PostgreSQL management migration with static data-boundary checks.
- [x] Implement the pgx PostgreSQL Adapter and run the public contract against PostgreSQL 18.
- [x] Implement the Redis TTL activity pool and one atomic extraction operation covering
  candidate eligibility, Gateway reserve, ownership, removal, and short-lived idempotency.
- [x] Implement Redis Worker ownership, drain/ACK, expiry reclaim, inventory and bounded
  sweep primitives with a monotonic global epoch.
- [ ] Implement Redis Provider leader, distributed rate, Client limit and Worker
  heartbeat; wire automatic Provider inventory rebuild after Redis loss.
- [x] Keep Provider output in Redis TTL activity state and node memory only; keep the
  Gateway request path on immutable local snapshots with no Redis/PostgreSQL calls.
- [x] Expose Distribution extraction/status and Admin status/enable/disable/switch/reload
  HTTP handlers and contracts.
- [x] Add Compose-backed Redis 8.2 integration and shared Adapter contract tests.
- [x] Add PostgreSQL management Adapter and Compose-backed integration tests.

当前进度（2026-07-30）：已实现共享 `platform/httpapi`、Distribution
extract/live/ready Handler 与 Admin status/enable/disable/switch/reload Handler；
定向契约测试已覆盖严格 JSON、Body 上限、Request ID、幂等 Header、DTO 映射、
404/405 及业务错误映射。共享 `platform/httpsecurity` 已补齐 Basic/API Key/
Bearer/CIDR、可信代理、Client ID、本地准入和 API 401/Gateway 407 差异，并作为
Admin/Distribution 必需依赖。共享 `platform/httpserver` 与
`controller/runtime` 已完成 Distribution/Admin 独立监听器、首错联动关闭和
有界优雅停机。Admin mutation 已携带认证 Actor/SourceIP；公用 `adminstate`
事务契约、MemoryStore、100 并发 Routing CAS、租约 Outbox 和六表管理 Schema
已完成。pgx Adapter 已在真实 PostgreSQL 18 上运行同一公用契约，并验证
Repeatable Read 快照、`SKIP LOCKED`、原子 ACK、审计/Outbox 故障回滚和数据边界。
Admin `ApplicationService` 已将 mutation、权威管理快照、低基数运行态
聚合与配置重载接到同一公用 seam；严格文件加载、脱敏管理摘要及原子配置发布
已通过失败路径和并发测试。`cmd/proxy-controller` 与公用 `controller/bootstrap`
已完成配置单次加载、PostgreSQL 连接/迁移、Redis 活动池、状态聚合、
Distribution/Admin 服务构造、错误合并和资源关闭；Provider 调度及完整 HTTP
进程端到端测试仍待实现。Controller Metrics 独立入口现已提供 `/livez`、
`/readyz` 与基础 Prometheus 运行时指标，三监听器隔离已通过测试；业务指标仍待
实现。双存储 bootstrap 已通过 PostgreSQL 18 + Redis 8.2 组合 fixture，覆盖
迁移、启动配置提交、Readiness、Admin Status 和 Metrics 探针。

已新增公用 `domain/activitypool` 契约及并发安全内存参考实现，Provider
Reconciler 通过 `UpsertFetched` 写入带供应商 TTL 和分配安全余量的批次；已覆盖
`usableUntil` 向 Worker Snapshot 的传播与 Gateway 本地截止过滤、
重复刷新、过期淘汰、独占提取、短期幂等及 Worker ownership 互斥。生产 Redis
Adapter 已通过真实 Redis 8.2 运行同一套公用契约；原子 Lua 覆盖提取、所有权和
有界清理。新增低基数 StateInventory Hash，五类写脚本在同一原子边界维护状态
计数，读取不扫描 Proxy 明细；过期清理积压或负计数时 fail-closed。Redis
Sentinel/故障转移验证与代表性多节点压测仍待实施。

## Task 11: Checker and Health Reducer

**Files:** `internal/controller/health/*.go`, `cmd/proxy-checker/main.go`, tests

- [ ] Schedule global and route health with jitter and bounded maxInFlight.
- [ ] Implement FETCHED -> CHECKING -> AVAILABLE and SUSPECT/UNHEALTHY transitions.
- [ ] Ensure target failures affect only the target profile.
- [ ] Add fixture target server and deterministic clock/scheduler tests.

## Task 12: Machine-readable Contracts

**Files:** `api/openapi/proxy-pool.yaml`, `api/proto/controlplane/v1/controlplane.proto`,
`docs/api/*.md`

- [x] Specify Distribution/Admin REST schemas, status codes, authentication, examples,
  and idempotency behavior.
- [x] Specify Worker register, snapshot, delta, ACK, report, heartbeat, ownership drain,
  and resync messages.
- [ ] Validate OpenAPI and compile protobuf descriptors in CI.

当前进度（2026-07-29）：Distribution/Admin OpenAPI 已由 Go 测试在双平台 CI
校验本地 `$ref` 闭合、operationId 唯一、响应存在及 security scheme 引用；
`scripts/verify-proto.ps1` 已可复现编译包含 imports/source info 的 descriptor，
并在本地存在 `protoc` 时进入完整验证；完整 OAS 工具验证与 CI 强制安装/执行
`protoc` 仍待完成。

## Task 13: Deployment and Observability

**Files:** `deploy/**`, `internal/platform/**`, `docs/operations/**`

- [ ] Add Compose for local Controller/Gateway/Checker/PostgreSQL/Redis/Prometheus/
  Grafana/HAProxy.
- [ ] Add Kubernetes Deployments, Services, PDBs, HPA, NetworkPolicy, Secrets examples,
  probes, resource limits, topology spread, and graceful termination.
- [ ] Add low-cardinality Prometheus metrics and structured secret-safe logs.
- [ ] Document backup, recovery, rollout, rollback, capacity, kernel, file descriptor,
  NAT/conntrack, and incident runbooks.

## Task 14: Documentation, Examples, and Diagrams

**Files:** `docs/**`, `examples/**`, `diagrams/**`

- [x] Complete README navigation, design document, developer guide, configuration
  reference, API guide, deployment guide, security model, testing guide, and roadmap.
- [x] Provide at least 20 validated configuration examples.
- [x] Provide at least 30 Mermaid architecture, flow, sequence, state, and failure diagrams.
- [ ] Generate `proxy-pool-docs-v1.0.zip` from versioned documentation assets.

## Task 15: Completion Audit

- [ ] Map every requirement ID to code, test, contract, document, or verified runtime evidence.
- [ ] Run `gofmt`, `go vet`, unit tests, race tests, builds, contract validation, and
  documentation link/example validation.
- [ ] Run bounded local performance benchmarks; label 100k QPS as unverified until a
  representative cluster load run exists.
- [ ] Confirm no TODO/TBD/placeholders, secrets, unbounded queues, high-cardinality metric
  labels, extraction Lease APIs, or conflicting maxSize semantics remain.