youfak/proxy-pool
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
b24d71fb90
proxy-pool/docs/development/implementation-plan.md
youfak b24d71fb90 feat: expose capacity metrics
2026-08-02 14:21:18 +08:00

24 KiB
Raw Blame History

Proxy Pool Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development or superpowers:executing-plans. Every step is tracked with checkbox syntax and must preserve the requirement IDs in docs/requirements/traceability.md.

Goal: Build a production-oriented Go repository whose domain behavior, interfaces, configuration, contracts, documentation, and deployment layout implement the final semantics in 对话内容.md.

Architecture: Separate Gateway, Controller, Checker, and Loadgen commands. Domain packages remain transport-free. Gateway reads immutable local snapshots. Controller owns provider fetch, pool lifecycle, routing state, extraction, persistence, and worker distribution. PostgreSQL persists management state and Admin audit/outbox. Redis owns the rebuildable TTL Proxy activity pool and atomic extraction; Gateway hot paths remain memory-only.

Tech Stack: Go 1.26, go.yaml.in/yaml/v4, pgx/v5, go-redis/v9, gRPC/Protobuf, Prometheus, PostgreSQL, Redis, Docker Compose, Kubernetes.

File Structure

cmd/
  proxy-gateway/main.go
  proxy-controller/main.go
  proxy-checker/main.go
  proxy-loadgen/main.go
internal/
  config/{config.go,load.go,validate.go}
  domain/proxy/{proxy.go,state.go,capacity.go}
  domain/routing/{rule.go,strategy.go,sequential.go}
  domain/upstream/{upstream.go,fetch_result.go,pool.go}
  domain/extraction/{extraction.go,store.go}
  domain/client/client.go
  gateway/{server,dispatch,snapshot,transport}/
  controller/{provider,pool,routing,extraction,health,distribution,operations,runtime,bootstrap}/
  adapters/{memory,postgres,redis,providerapi}/
  platform/{logging,metrics,shutdown}/
api/{openapi,proto}/
configs/
deploy/{compose,kubernetes,haproxy,prometheus,grafana}/
docs/{design,development,configuration,api,operations,testing,adr,requirements}/
diagrams/
examples/
test/{fixtures,integration,e2e,load}/

Task 1: Repository and Build Baseline

Files: go.mod, .golangci.yml, README.md, scripts/verify.ps1, .github/workflows/ci.yml

  • Create module github.com/proxy-pool/proxy-pool with Go 1.26.
  • Pin YAML v4, pgx/v5, go-redis/v9, gRPC, protobuf, Prometheus, and x/sync.
  • Add scripts/verify.ps1 that runs format check, go vet, unit tests, race tests, and builds all commands, each test command bounded to 60 seconds.
  • Add CI for Windows and Linux with unit/race/build jobs.
  • Verify go mod tidy, go test ./..., and go build ./cmd/... succeed.

Task 2: Strict Configuration

Files: internal/config/config.go, load.go, validate.go, corresponding tests, configs/default.yaml, docs/configuration/reference.md

  • Define versioned types for security, gateway, distribution, admin, metrics, storage, routing, upstream/provider/api/proxyAuth/pool/capacity/lifecycle/fetch/check.
  • Decode one YAML document with known fields enabled and resolve ${ENV} plus secret file references without logging values.
  • Validate listener protection, routing references/order, regexes, strategy fields, positive limits, TTL margins, pool/fetch limits, auth modes, and exposure modes.
  • Add table tests for every invalid condition in CFG requirements.

Task 3: Proxy Domain and Capacity

Files: internal/domain/proxy/*.go, corresponding tests

  • Implement Proxy fields, UTC TTL precedence, canonical host/port, and unique key.
  • Implement state transitions and reject illegal transitions.
  • Implement sharded runtime counters with CAS Reserve, Commit, Cancel, Release.
  • Prove with 1,000 concurrent goroutines that effective capacity is never exceeded.
  • Add race coverage and duplicate-release invariant metrics hook.

当前进度（2026-07-29）：固定 Max 下的每 Proxy 打包 CAS、Cancel/Commit/Release 生命周期、重复终结、错误顺序和同一 Reservation 并发终结已通过领域测试；已退出 Proxy 会停止新预留、保留非零 Drain 计数，并在归零后由后续 Snapshot Apply 回收。 动态降容契约、低基数不变量指标和 Linux race 证据仍待完成。

Task 4: Routing and Sequential Switching

Files: internal/domain/routing/*.go, corresponding tests

  • Compile first-match host/method/path/header rules into an immutable RuleSet.
  • Implement random, round-robin, weighted, least-connections, and sequential.
  • Model upstream empty counters separately from per-routing current indexes.
  • Implement versioned CAS switch so simultaneous threshold observers advance once.
  • Cover four-empty-then-success, five-empty, A-to-B-only, disabled references, end behavior, and explicit onUnavailable.

当前进度（2026-07-29）：领域构造器与严格配置已统一 Sequential 至少两个 Upstream、endBehavior 默认 stop，并覆盖列表末端停止；disabled candidate、 跨实例恢复和 onUnavailable 运行链仍待完成。

Task 5: Provider Fetch Classification and Scheduling

Files: internal/controller/provider/*.go, internal/domain/upstream/*.go, tests

  • Implement Valid, Empty, DuplicateOnly, and Error result classes exactly as the traceability matrix defines.
  • Implement one coalesced reconcile signal per Upstream using singleflight.
  • Enforce requestInterval, maxInFlight, maxSize, maxTotal, timeout, retry, exponential backoff, jitter, and Retry-After.
  • Define ProviderAdapter and safe TemplateParser ports; add fixture adapters.
  • Test that 100 concurrent capacity signals do not fan out 100 Provider calls.

Task 6: Pool Reconciliation and Ownership

Files: internal/controller/pool/*.go, internal/domain/upstream/pool.go, tests

  • Compute Available Slots from eligible Proxy capacity, Active, Reserved, TTL, health, ownership, pending expected fetch, and gateway reserve.
  • Implement pool.maxSize and fetch.maxTotal as distinct counters.
  • Allocate each Proxy to one Worker with epoch/version/expiry ownership.
  • Implement revoke -> drain -> ACK -> unowned transition.
  • Test Worker crash expiry and prevent simultaneous dual ownership.

Task 7: Exclusive Extraction

Files: internal/domain/extraction/*.go, internal/controller/extraction/*.go, internal/adapters/memory/extraction.go, tests

  • Implement POST extraction command with protocol/region/carrier/upstream filters.
  • Enforce minRemainingTTL, maxHealthCheckAge, maxCount, client limits, and reserveForGateway.
  • Atomically remove selected AVAILABLE entries from the allocatable set and return the result; MemoryPool and the production Redis Adapter run the same shared contract.
  • Implement partial and allOrNothing without Lease, release, or renewal concepts.
  • Run 1,000 concurrent claim attempts and prove every Proxy ID appears at most once.

Task 8: Immutable Snapshot and Dispatch

Files: internal/gateway/snapshot/*.go, internal/gateway/dispatch/*.go, tests

  • Define cluster/worker/epoch/version/checksum snapshot envelopes.
  • Build indexes before publication and atomically swap complete snapshots.
  • Reject version gaps and wrong epochs; request full resync.
  • Implement Dispatch Acquire/Commit/Release over local owned Proxy runtime.
  • Benchmark 100k Proxy snapshots and record allocations and latency.

Task 9: Gateway Transport

Files: internal/gateway/server/*.go, internal/gateway/transport/*.go, tests

  • Implement HTTP forward proxy and HTTPS CONNECT through an upstream proxy.
  • Add Client auth/access/admission and destination policy checks before routing.
  • Implement safe retry commit points and prevent non-idempotent/established tunnel replay.
  • Use bounded buffers, deadlines, connection pools, and graceful shutdown.
  • Add local fake upstream end-to-end tests for success, 407, timeout, cancel, half close, retry, and blocked private destinations.

Task 10: Controller APIs and Persistence Ports

Files: internal/controller/distribution/*.go, admin/*.go, internal/adapters/postgres/*.go, internal/adapters/redis/*.go, migrations, tests

  • Define the PostgreSQL management seam for ConfigVersion, Upstream/Routing state, AdminAudit and leased Outbox; the public contract has no Proxy or extraction detail types.
  • Add the six-table PostgreSQL management migration with static data-boundary checks.
  • Implement the pgx PostgreSQL Adapter and run the public contract against PostgreSQL 18.
  • Implement the Redis TTL activity pool and one atomic extraction operation covering candidate eligibility, Gateway reserve, ownership, removal, and short-lived idempotency.
  • Implement Redis Worker ownership, drain/ACK, expiry reclaim, inventory and bounded sweep primitives with a monotonic global epoch.
  • Implement Redis Provider leader, distributed request quota, Client limit and automatic Provider inventory rebuild after Redis state loss.
  • Implement the Worker heartbeat receiving path and session lifecycle.
  • Keep Provider output in Redis TTL activity state and node memory only; keep the Gateway request path on immutable local snapshots with no Redis/PostgreSQL calls.
  • Expose Distribution extraction/status and Admin status/enable/disable/switch/reload HTTP handlers and contracts.
  • Add Compose-backed Redis 8.2 integration and shared Adapter contract tests.
  • Add PostgreSQL management Adapter and Compose-backed integration tests.

当前进度（2026-07-31）：已实现共享 platform/httpapi、Distribution extract/live/ready Handler 与 Admin status/enable/disable/switch/reload Handler； 定向契约测试已覆盖严格 JSON、Body 上限、Request ID、幂等 Header、DTO 映射、 404/405 及业务错误映射。共享 platform/httpsecurity 已补齐 Basic/API Key/ Bearer/CIDR、可信代理、Client ID、本地准入和 API 401/Gateway 407 差异，并作为 Admin/Distribution 必需依赖。共享 platform/httpserver 与 controller/runtime 已完成 Distribution/Admin 独立监听器、首错联动关闭和 有界优雅停机。Admin mutation 已携带认证 Actor/SourceIP；公用 adminstate 事务契约、MemoryStore、100 并发 Routing CAS、租约 Outbox 和六表管理 Schema 已完成。pgx Adapter 已在真实 PostgreSQL 18 上运行同一公用契约，并验证 Repeatable Read 快照、SKIP LOCKED、原子 ACK、审计/Outbox 故障回滚和数据边界。 Admin ApplicationService 已将 mutation、权威管理快照、低基数运行态 聚合与配置重载接到同一公用 seam；严格文件加载、外部密钥 HMAC 管理指纹及 revision 单调配置发布已通过失败路径和确定性并发测试。cmd/proxy-controller 与公用 controller/bootstrap 已完成配置单次加载、PostgreSQL 连接/迁移、Redis 活动池、状态聚合、 Distribution/Admin 服务构造、错误合并和资源关闭。生产 Provider Supervisor 已按 权威管理状态动态装配 Upstream，并与 HTTP Runtime 通过公用 lifecycle Group 联动 停机；Admin disable 会取消 Runtime，reload 在提交前预检并在发布后替换运行实例。 组合 fixture 已验证隔离 Redis namespace 下的选主、Provider HTTP 调用、模板解析 和活动池写入。Controller Metrics 独立入口现已提供 /livez、/readyz 与基础 Prometheus 运行时指标，三监听器隔离已通过测试；Checker、Gateway、Drain、 Provider、Extraction 与容量业务指标已接入，密钥安全的结构化日志仍待实现。双存储 bootstrap 已通过 PostgreSQL 18 + Redis 8.2 组合 fixture，覆盖 迁移、启动配置提交、Readiness、Admin Status 和 Metrics 探针。

WorkerControlPlane 现已接入 Controller 生命周期：Register、ACK 和 Runtime 报告均经 Redis 服务端 TTL 的 session/issued-snapshot/ACK 栅栏校验；mTLS SPIFFE 身份、消息/流限制和有界停机已实现。WatchSnapshots 会发送当前 epoch 的基础完整 Snapshot 并保持连接；Gateway 已具备 Register/Watch/ACK/Runtime 会话协调组件与 proxy-gateway 进程装配。 按 Worker 的可下发 ownership 索引已进入 Redis 原子脚本，并可构建已归属 Proxy、 去重凭据材料和 Routing 的完整 payload。Snapshot 签发与 session 匹配在同一 Redis Lua 操作中完成， 重注册会清除旧引用，避免迟到 Stream 覆盖新 session。Worker 服务端会在最近完整 Snapshot 的 valid_until 到达时结束流；公用 SessionSupervisor 已为 Gateway 调用方 提供可恢复错误的有界指数退避重连，并在参数/认证/协议错误时停止。Gateway Routing payload 已按配置顺序和 Admin revision/current 状态发布并覆盖 checksum；Gateway 已将其与 Proxy 原子编译为同版本 View，动态 Router 只匹配该未过期 View。派发器的五种上游选择已 接入该 View，并在容量耗尽时在同版本候选中回退；onUnavailable 的 reject、wait 与 direct 已接入 Gateway；proxy-gateway 已装配本地 HTTP/Metrics 监听、快照就绪探针和 控制面重连与快照凭据分发，以及有界 Outcome 上报、序列确认和精确重试。

已新增公用 domain/activitypool 契约及并发安全内存参考实现，Provider Reconciler 通过 UpsertFetched 写入带供应商 TTL 和分配安全余量的批次；已覆盖 usableUntil 向 Worker Snapshot 的传播与 Gateway 本地截止过滤、 重复刷新、过期淘汰、独占提取、短期幂等及 Worker ownership 互斥。生产 Redis Adapter 已通过真实 Redis 8.2 运行同一套公用契约；原子 Lua 覆盖提取、所有权和 有界清理。新增低基数 StateInventory Hash，五类写脚本在同一原子边界维护状态 计数，读取不扫描 Proxy 明细；过期清理积压或负计数时 fail-closed。Redis Sentinel/故障转移验证与代表性多节点压测仍待实施。

Provider 分布式协调已新增公用 Coordinator.RunLeader / LeaderSession seam 与 独立 redisprovider Adapter。真实 Redis 8.2 已验证同 Upstream 双实例互斥、 generation + epoch fence、全局 requestInterval、全局 maxInFlight Permit、TTL 回收及 Redis 状态丢失后的新 generation 自动重建；Redis 异常期间不发放请求。 补池配置新增必填 refill 双水位和 fetch.estimatedIPsPerCall，Pool Reconciler 已实现迟滞与 pending 槽位折算，FetchBudget 仅在无 pending 时同步 Redis 权威 Managed。Gateway 已增加打包原子 Active/Reserved 读取与完整稀疏运行态快照； 公用 workerruntime session/report/read seam 同时提供并发安全内存参考实现和 生产 Redis Adapter。Redis 以服务端时间、Worker session、已 ACK snapshot/epoch、 单调 report sequence 和报告 TTL 原子隔离旧实例，并由 pool.InventoryReader 汇总 权威 Managed/Available Slots；真实 Redis 8.2 已覆盖空报告、幂等重放、倒序、 冲突、超前 epoch、过期、单 Upstream 扫描隔离和预算耗尽的 fail-closed 行为。 Redis Provider Permit 现已把 requestInterval、maxInFlight 与 maxTotal 放在同一 原子边界，按 expected 预留、实际合法数量结算，并支持失败保守计费、换主后结算和 过期回收。响应型代理凭据使用独立、有界 lease，在 Redis Upsert、候选截断或解析 失败后按版本释放；Provider Empty/Error 低基数计数已接入 Admin Status，配置删除 时回收历史统计容量。Redis inventory 扫描上限固定覆盖配置允许的最大池，支持小池 启动后动态扩容。Supervisor 以 PostgreSQL 权威 HMAC 指纹和 revision 栅栏协调 多副本 reload：管理库瞬断沿用 last-known 状态，本地共享源落后时停止旧 Provider， 源匹配并预检后自动替换，迟到旧 revision 不覆盖新配置。 Distribution 现通过公用 admission.Admitter 接入独立 redisadmission Adapter； 全局和单 Client 分钟额度使用 Redis 服务端时间并在单个 Lua 原子边界内检查、递增， Controller 多副本共享同一计数。Client 身份只以 SHA-256 摘要进入 Redis，窗口切换 原子回收历史字段；Redis 异常 fail-closed 并返回 503，真实额度耗尽返回 429。 Gateway 请求热路径仍只使用本地准入，不增加 Redis/PostgreSQL 调用。 WorkerControlPlane gRPC 接收端、session 签发/心跳、Snapshot ACK 账本、基础 Snapshot 流、Gateway 会话客户端与快照凭据分发已完成；权威 Proxy/Routing 发布、 Outcome 上报已完成为有界队列、批次序列/摘要栅栏和确认重试；健康 BASIC/EGRESS/TARGET 执行链已完成；持续 UNHEALTHY 的未分配 Proxy 可由后台有界回收，拥有 Worker 所有权的项会通过健康/归属条件栅栏自动发起 Drain，并在排除快照 ACK 与运行态归零后回收。

Task 11: Checker and Health Reducer

Files: internal/controller/health/*.go, cmd/proxy-checker/main.go, tests

  • Schedule global and route health with jitter and bounded maxInFlight.
  • Implement FETCHED -> CHECKING -> AVAILABLE and SUSPECT/UNHEALTHY transitions.
  • Ensure target failures affect only the target profile.
  • Add fixture target server and deterministic clock/scheduler tests.

当前进度（2026-07-31）：已新增不依赖 gRPC 的 domain/health Observation、全局/Target Profile Reducer 与任务摘要幂等语义；BASIC/EGRESS 结果经活动池窄接口在 Memory 和 Redis Lua 同一原子边界归并，覆盖首次 CHECKING 失败进入 UNHEALTHY、AVAILABLE/SUSPECT 的 阈值降级、精确重放、冲突拒绝与成功恢复。TARGET Profile 也已在 Memory 和 Redis 中独立 归并，以哈希键保存并随代理 TTL 过期，绝不写入 Proxy 全局状态或选择索引。调度器、 Checker Observation 上报 RPC 已复用既有控制面监听接入 Controller：SPIFFE checker 身份、 每批上限、配置阈值解析、代理归属查询与 Reducer 均已闭环。StreamCheckTasks 现已实现为 有界 pull，并通过通用任务 broker 契约完成能力协商、同 Checker 并发窗口、租约到期回收、 领取者加不可预测 lease token 的栅栏和完成后重放；任务凭据仅由认证流在执行期下发。 Redis 共享 due-index/租约持久化、每 Upstream 的跨副本 in-flight 限制和生产 broker 已完成； proxy-checker 独立进程、固定大小 worker-pool、任务期重试/微批上报和 HTTP/HTTPS/SOCKS5 BASIC/EGRESS 探测器已完成并有测试。Redis 已按固定 EGRESS due-index 保存任务执行 URL， 与 BASIC 独立引用并通过上游共享并发限制；配置化监督器以有界轮转组调度每个 check.urls。 EGRESS 的出口身份响应解析支持固定上限的纯文本和常见 JSON IP 字段。TARGET 已以 routing.check.targets 作为配置入口，限定每个 Routing 的 URL 数和每个 Upstream 的 Profile 总数；Redis 为 (routing_name, target_url, proxy_id) 维护独立 due-index，完成、租约回收和 代理过期均沿用同一原子任务边界。BASIC、EGRESS 和 TARGET 分组轮转并共享 Upstream in-flight 上限，因此本任务已完成。持续 UNHEALTHY 回收使用每 Upstream 的 check.unhealthyRemoveAfter：全局 Reducer 保存首次进入 UNHEALTHY 的时间，Redis 以 有序索引有界扫描；没有 Worker ownership 的项沿用统一清理边界删除，已拥有 Worker 的 项通过包含 assignment epoch 与首次异常时间的候选，由条件式 Drain 原子重校验后进入 Ticket/排除快照/Runtime 零计数闭环。

补充进度（2026-08-02）：BASIC 调度已改为配置驱动监督器。它每轮读取已发布快照并复用 有界派发逻辑，所以 reload 后已启用上游的策略变更、停用，以及新启用上游都无需重启 Controller 即可生效。Redis 任务存储现已扩展 BASIC/EGRESS/TARGET 的独立有界索引；路由目标 Profile 在启用 Routing 与 Upstream 的组合上才进入调度。

补充进度（2026-08-02）：配置停用已进入 ownership Drain 编排。Controller 以静态配置 与 Admin 管理态的交集生成完整、版本化的 Upstream 策略视图；Redis 只从对应 owned 索引 有界读取仍可用的已归属 Proxy。候选携带 Upstream revision，ownership.lua 在复用既有 Ticket/排除 Snapshot/Runtime 零计数闭环前，原子复核策略仍为停用、Proxy source、Worker、 assignment epoch 和未过期租约。重启用后的策略 revision 会使旧候选返回无操作。Provider Supervisor 也改为同时服从静态配置与管理态，消除两条启停消费链的不一致。

补充进度（2026-08-02）：已新增 proxy-loadgen HTTP 与 CONNECT 长连接场景。固定请求数 和固定时长两种模式均通过固定 worker 数与有界派发通道执行，可选 QPS 限速；报告使用固定大小 延迟直方图，输出状态分类、CONNECT 建立数、Extract 校验数、吞吐和 Go 内存/GC 快照。CONNECT 以原始 TCP 握手连接 HTTP Gateway，建连成功后按 hold 保持，且不透明读取隧道内容。extract 场景会自动生成独立 Request/Idempotency 标识，校验返回数量与单响应 ID 唯一性，且不记录地址或 凭据。故障注入以及代表性集群报告仍未实现。

Task 12: Machine-readable Contracts

Files: api/openapi/proxy-pool.yaml, api/proto/controlplane/v1/controlplane.proto, docs/api/*.md

  • Specify Distribution/Admin REST schemas, status codes, authentication, examples, and idempotency behavior.
  • Specify Worker register, snapshot, delta, ACK, report, heartbeat, ownership drain, and resync messages.
  • Validate OpenAPI and compile protobuf descriptors in CI.

当前进度（2026-07-29）：Distribution/Admin OpenAPI 已由 Go 测试在双平台 CI 校验本地 $ref 闭合、operationId 唯一、响应存在及 security scheme 引用； scripts/verify-proto.ps1 已可复现编译包含 imports/source info 的 descriptor， 并在本地存在 protoc 时进入完整验证；完整 OAS 工具验证与 CI 强制安装/执行 protoc 仍待完成。

Task 13: Deployment and Observability

Files: deploy/**, internal/platform/**, docs/operations/**

  • Add Compose for local Controller/Gateway/Checker/PostgreSQL/Redis/Prometheus/ Grafana/HAProxy.
  • Add Kubernetes Deployments, Services, PDBs, HPA, NetworkPolicy, Secrets examples, probes, resource limits, topology spread, and graceful termination.
  • Add low-cardinality Prometheus metrics and structured secret-safe logs.
  • Document backup, recovery, rollout, rollback, capacity, kernel, file descriptor, NAT/conntrack, and incident runbooks.

当前进度（2026-08-02）：已接入 Checker 任务流指标 proxy_pool_checker_tasks_dispatched_total{level} 与 proxy_pool_checker_observations_total{level,result}；标签值仅允许固定的 BASIC、EGRESS、TARGET 级别及 accepted、rejected 结果。Gateway 还暴露 proxy_pool_gateway_outcomes_total{stage,result} 和 proxy_pool_gateway_outcome_queue_dropped_total，其中阶段和结果均为固定枚举。Provider、 拉取已暴露 proxy_pool_controller_provider_fetch_results_total{class}、 proxy_pool_controller_provider_valid_candidates_total 与 proxy_pool_controller_provider_new_proxies_total；class 仅允许 valid、empty、 duplicate_only、error。Extraction 还暴露固定 result 的请求次数、请求数与响应 交付数，幂等重放按响应交付统计。容量指标由既有 Provider 库存对账周期聚合，不进入 Gateway 热路径；仅密钥安全的结构化日志仍待补齐。

Task 14: Documentation, Examples, and Diagrams

Files: docs/**, examples/**, diagrams/**

  • Complete README navigation, design document, developer guide, configuration reference, API guide, deployment guide, security model, testing guide, and roadmap.
  • Provide at least 20 validated configuration examples.
  • Provide at least 30 Mermaid architecture, flow, sequence, state, and failure diagrams.
  • Generate proxy-pool-docs-v1.0.zip from versioned documentation assets.

Task 15: Completion Audit

  • Map every requirement ID to code, test, contract, document, or verified runtime evidence.
  • Run gofmt, go vet, unit tests, race tests, builds, contract validation, and documentation link/example validation.
  • Run bounded local performance benchmarks; label 100k QPS as unverified until a representative cluster load run exists.
  • Confirm no TODO/TBD/placeholders, secrets, unbounded queues, high-cardinality metric labels, extraction Lease APIs, or conflicting maxSize semantics remain.