217 lines
10 KiB
Markdown
217 lines
10 KiB
Markdown
# Proxy Pool Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use
|
|
> `superpowers:subagent-driven-development` or `superpowers:executing-plans`.
|
|
> Every step is tracked with checkbox syntax and must preserve the requirement IDs in
|
|
> `docs/requirements/traceability.md`.
|
|
|
|
**Goal:** Build a production-oriented Go repository whose domain behavior, interfaces,
|
|
configuration, contracts, documentation, and deployment layout implement the final
|
|
semantics in `对话内容.md`.
|
|
|
|
**Architecture:** Separate Gateway, Controller, Checker, and Loadgen commands. Domain
|
|
packages remain transport-free. Gateway reads immutable local snapshots. Controller
|
|
owns provider fetch, pool lifecycle, routing state, extraction, persistence, and worker
|
|
distribution. PostgreSQL is authoritative; Redis stores rebuildable coordination state.
|
|
|
|
**Tech Stack:** Go 1.26, `go.yaml.in/yaml/v4`, pgx/v5, go-redis/v9, gRPC/Protobuf,
|
|
Prometheus, PostgreSQL, Redis, Docker Compose, Kubernetes.
|
|
|
|
---
|
|
|
|
## File Structure
|
|
|
|
```text
|
|
cmd/
|
|
proxy-gateway/main.go
|
|
proxy-controller/main.go
|
|
proxy-checker/main.go
|
|
proxy-loadgen/main.go
|
|
internal/
|
|
config/{config.go,load.go,validate.go}
|
|
domain/proxy/{proxy.go,state.go,capacity.go}
|
|
domain/routing/{rule.go,strategy.go,sequential.go}
|
|
domain/upstream/{upstream.go,fetch_result.go,pool.go}
|
|
domain/extraction/{extraction.go,store.go}
|
|
domain/client/client.go
|
|
gateway/{server,dispatch,snapshot,transport}/
|
|
controller/{provider,pool,routing,extraction,health,distribution}/
|
|
adapters/{memory,postgres,redis,providerapi}/
|
|
platform/{logging,metrics,shutdown}/
|
|
api/{openapi,proto}/
|
|
configs/
|
|
deploy/{compose,kubernetes,haproxy,prometheus,grafana}/
|
|
docs/{design,development,configuration,api,operations,testing,adr,requirements}/
|
|
diagrams/
|
|
examples/
|
|
test/{fixtures,integration,e2e,load}/
|
|
```
|
|
|
|
## Task 1: Repository and Build Baseline
|
|
|
|
**Files:** `go.mod`, `.golangci.yml`, `README.md`, `scripts/verify.ps1`,
|
|
`.github/workflows/ci.yml`
|
|
|
|
- [ ] Create module `github.com/proxy-pool/proxy-pool` with Go 1.26.
|
|
- [ ] Pin YAML v4, pgx/v5, go-redis/v9, gRPC, protobuf, Prometheus, and x/sync.
|
|
- [ ] Add `scripts/verify.ps1` that runs format check, `go vet`, unit tests, race tests,
|
|
and builds all commands, each test command bounded to 60 seconds.
|
|
- [ ] Add CI for Windows and Linux with unit/race/build jobs.
|
|
- [ ] Verify `go mod tidy`, `go test ./...`, and `go build ./cmd/...` succeed.
|
|
|
|
## Task 2: Strict Configuration
|
|
|
|
**Files:** `internal/config/config.go`, `load.go`, `validate.go`, corresponding tests,
|
|
`configs/default.yaml`, `docs/configuration/reference.md`
|
|
|
|
- [x] Define versioned types for security, gateway, distribution, admin, metrics,
|
|
storage, routing, upstream/provider/api/proxyAuth/pool/capacity/lifecycle/fetch/check.
|
|
- [x] Decode one YAML document with known fields enabled and resolve `${ENV}` plus
|
|
secret file references without logging values.
|
|
- [x] Validate listener protection, routing references/order, regexes, strategy fields,
|
|
positive limits, TTL margins, pool/fetch limits, auth modes, and exposure modes.
|
|
- [x] Add table tests for every invalid condition in CFG requirements.
|
|
|
|
## Task 3: Proxy Domain and Capacity
|
|
|
|
**Files:** `internal/domain/proxy/*.go`, corresponding tests
|
|
|
|
- [x] Implement Proxy fields, UTC TTL precedence, canonical host/port, and unique key.
|
|
- [x] Implement state transitions and reject illegal transitions.
|
|
- [ ] Implement sharded runtime counters with CAS Reserve, Commit, Cancel, Release.
|
|
- [x] Prove with 1,000 concurrent goroutines that effective capacity is never exceeded.
|
|
- [ ] Add race coverage and duplicate-release invariant metrics hook.
|
|
|
|
## Task 4: Routing and Sequential Switching
|
|
|
|
**Files:** `internal/domain/routing/*.go`, corresponding tests
|
|
|
|
- [x] Compile first-match host/method/path/header rules into an immutable RuleSet.
|
|
- [x] Implement random, round-robin, weighted, least-connections, and sequential.
|
|
- [x] Model upstream empty counters separately from per-routing current indexes.
|
|
- [x] Implement versioned CAS switch so simultaneous threshold observers advance once.
|
|
- [ ] Cover four-empty-then-success, five-empty, A-to-B-only, disabled references,
|
|
end behavior, and explicit onUnavailable.
|
|
|
|
## Task 5: Provider Fetch Classification and Scheduling
|
|
|
|
**Files:** `internal/controller/provider/*.go`, `internal/domain/upstream/*.go`, tests
|
|
|
|
- [x] Implement Valid, Empty, DuplicateOnly, and Error result classes exactly as the
|
|
traceability matrix defines.
|
|
- [x] Implement one coalesced reconcile signal per Upstream using singleflight.
|
|
- [x] Enforce requestInterval, maxInFlight, maxSize, maxTotal, timeout, retry,
|
|
exponential backoff, jitter, and Retry-After.
|
|
- [ ] Define ProviderAdapter and safe TemplateParser ports; add fixture adapters.
|
|
- [x] Test that 100 concurrent capacity signals do not fan out 100 Provider calls.
|
|
|
|
## Task 6: Pool Reconciliation and Ownership
|
|
|
|
**Files:** `internal/controller/pool/*.go`, `internal/domain/upstream/pool.go`, tests
|
|
|
|
- [ ] Compute Available Slots from eligible Proxy capacity, Active, Reserved, TTL,
|
|
health, ownership, pending expected fetch, and gateway reserve.
|
|
- [x] Implement pool.maxSize and fetch.maxTotal as distinct counters.
|
|
- [x] Allocate each Proxy to one Worker with epoch/version/expiry ownership.
|
|
- [x] Implement revoke -> drain -> ACK -> unowned transition.
|
|
- [x] Test Worker crash expiry and prevent simultaneous dual ownership.
|
|
|
|
## Task 7: Exclusive Extraction
|
|
|
|
**Files:** `internal/domain/extraction/*.go`, `internal/controller/extraction/*.go`,
|
|
`internal/adapters/memory/extraction.go`, tests
|
|
|
|
- [x] Implement POST extraction command with protocol/region/carrier/upstream filters.
|
|
- [x] Enforce minRemainingTTL, maxHealthCheckAge, maxCount, client limits, and
|
|
reserveForGateway.
|
|
- [x] Atomically transition AVAILABLE to EXTRACTED and append audit records.
|
|
- [x] Implement partial and allOrNothing without Lease, release, or renewal concepts.
|
|
- [x] Run 1,000 concurrent claim attempts and prove every Proxy ID appears at most once.
|
|
|
|
## Task 8: Immutable Snapshot and Dispatch
|
|
|
|
**Files:** `internal/gateway/snapshot/*.go`, `internal/gateway/dispatch/*.go`, tests
|
|
|
|
- [x] Define cluster/worker/epoch/version/checksum snapshot envelopes.
|
|
- [ ] Build indexes in the background and atomically swap complete snapshots.
|
|
- [x] Reject version gaps and wrong epochs; request full resync.
|
|
- [x] Implement Dispatch Acquire/Commit/Release over local owned Proxy runtime.
|
|
- [x] Benchmark 100k Proxy snapshots and record allocations and latency.
|
|
|
|
## Task 9: Gateway Transport
|
|
|
|
**Files:** `internal/gateway/server/*.go`, `internal/gateway/transport/*.go`, tests
|
|
|
|
- [ ] Implement HTTP forward proxy and HTTPS CONNECT through an upstream proxy.
|
|
- [ ] Add Client auth/access/admission and destination policy checks before routing.
|
|
- [ ] Implement safe retry commit points and prevent non-idempotent/established tunnel
|
|
replay.
|
|
- [ ] Use bounded buffers, deadlines, connection pools, and graceful shutdown.
|
|
- [ ] Add local fake upstream end-to-end tests for success, 407, timeout, cancel, half
|
|
close, retry, and blocked private destinations.
|
|
|
|
## Task 10: Controller APIs and Persistence Ports
|
|
|
|
**Files:** `internal/controller/distribution/*.go`, `admin/*.go`,
|
|
`internal/adapters/postgres/*.go`, `internal/adapters/redis/*.go`, migrations, tests
|
|
|
|
- [ ] Define repository ports for Proxy, RoutingRuntime, Ownership, ExtractionRecord,
|
|
Client, ConfigVersion, and Outbox.
|
|
- [ ] Implement PostgreSQL extraction with one transaction and `FOR UPDATE SKIP LOCKED`.
|
|
- [ ] Implement Redis coordination for Provider leader, distributed rate, Client limit,
|
|
and Worker heartbeat; keep all state rebuildable.
|
|
- [ ] Expose Distribution extraction/status and Admin status/enable/disable/switch/reload.
|
|
- [ ] Add integration tests using Compose-backed PostgreSQL/Redis.
|
|
|
|
## Task 11: Checker and Health Reducer
|
|
|
|
**Files:** `internal/controller/health/*.go`, `cmd/proxy-checker/main.go`, tests
|
|
|
|
- [ ] Schedule global and route health with jitter and bounded maxInFlight.
|
|
- [ ] Implement FETCHED -> CHECKING -> AVAILABLE and SUSPECT/UNHEALTHY transitions.
|
|
- [ ] Ensure target failures affect only the target profile.
|
|
- [ ] Add fixture target server and deterministic clock/scheduler tests.
|
|
|
|
## Task 12: Machine-readable Contracts
|
|
|
|
**Files:** `api/openapi/proxy-pool.yaml`, `api/proto/controlplane/v1/controlplane.proto`,
|
|
`docs/api/*.md`
|
|
|
|
- [ ] Specify Distribution/Admin REST schemas, status codes, authentication, examples,
|
|
and idempotency behavior.
|
|
- [ ] Specify Worker register, snapshot, delta, ACK, report, heartbeat, ownership drain,
|
|
and resync messages.
|
|
- [ ] Validate OpenAPI and compile protobuf descriptors in CI.
|
|
|
|
## Task 13: Deployment and Observability
|
|
|
|
**Files:** `deploy/**`, `internal/platform/**`, `docs/operations/**`
|
|
|
|
- [ ] Add Compose for local Controller/Gateway/Checker/PostgreSQL/Redis/Prometheus/
|
|
Grafana/HAProxy.
|
|
- [ ] Add Kubernetes Deployments, Services, PDBs, HPA, NetworkPolicy, Secrets examples,
|
|
probes, resource limits, topology spread, and graceful termination.
|
|
- [ ] Add low-cardinality Prometheus metrics and structured secret-safe logs.
|
|
- [ ] Document backup, recovery, rollout, rollback, capacity, kernel, file descriptor,
|
|
NAT/conntrack, and incident runbooks.
|
|
|
|
## Task 14: Documentation, Examples, and Diagrams
|
|
|
|
**Files:** `docs/**`, `examples/**`, `diagrams/**`
|
|
|
|
- [ ] Complete README navigation, design document, developer guide, configuration
|
|
reference, API guide, deployment guide, security model, testing guide, and roadmap.
|
|
- [ ] Provide at least 20 validated configuration examples.
|
|
- [ ] Provide at least 30 Mermaid architecture, flow, sequence, state, and failure diagrams.
|
|
- [ ] Generate `proxy-pool-docs-v1.0.zip` from versioned documentation assets.
|
|
|
|
## Task 15: Completion Audit
|
|
|
|
- [ ] Map every requirement ID to code, test, contract, document, or verified runtime evidence.
|
|
- [ ] Run `gofmt`, `go vet`, unit tests, race tests, builds, contract validation, and
|
|
documentation link/example validation.
|
|
- [ ] Run bounded local performance benchmarks; label 100k QPS as unverified until a
|
|
representative cluster load run exists.
|
|
- [ ] Confirm no TODO/TBD/placeholders, secrets, unbounded queues, high-cardinality metric
|
|
labels, extraction Lease APIs, or conflicting maxSize semantics remain.
|