docs: define proxy pool architecture from full conversation
This commit is contained in:
youfak
commit
dab16fda12
25
.gitignore
vendored
Normal file
25
.gitignore
vendored
Normal file
@ -0,0 +1,25 @@
|
||||
# Local planning and browser artifacts
|
||||
.playwright-mcp/
|
||||
temp/
|
||||
|
||||
# Build and test outputs
|
||||
bin/
|
||||
dist/
|
||||
coverage/
|
||||
*.out
|
||||
*.test
|
||||
*.prof
|
||||
|
||||
# Local configuration and secrets
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
configs/local.yaml
|
||||
*.pem
|
||||
*.key
|
||||
|
||||
# Editors and operating systems
|
||||
.idea/
|
||||
.vscode/
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
54
CONTEXT.md
Normal file
54
CONTEXT.md
Normal file
@ -0,0 +1,54 @@
|
||||
# Proxy Pool 统一领域语言
|
||||
|
||||
## 核心实体
|
||||
|
||||
- **Client**：使用 Gateway 或 Distribution API 的调用主体。认证关闭时，
|
||||
由可信来源 IP 形成匿名 Client。
|
||||
- **Routing**：一组有序匹配规则和选择策略，决定某类请求使用哪些 Upstream。
|
||||
- **Upstream**：一个供应商配置及其聚合代理池。Upstream 运行时全局共享，
|
||||
不随 Routing 重复创建。
|
||||
- **Provider**：Upstream 背后的外部代理供应商及其获取接口能力。
|
||||
- **Proxy**：从 Provider 获取并标准化后的代理资源，不等同于简单 IP。
|
||||
- **Worker**：承载 Gateway 流量的数据面节点，仅使用本地快照选路。
|
||||
- **Controller**：集中管理 Provider 获取、代理生命周期、Routing 状态、
|
||||
所有权和快照发布的控制面节点。
|
||||
- **Checker**：执行基础、出口和目标级健康检查的可扩缩执行节点。
|
||||
- **Extraction**：Distribution API 对一个或一组 Proxy 的一次性独占发放。
|
||||
- **Extraction Record**：Extraction 的审计事实，不承担租约或释放语义。
|
||||
|
||||
## 状态与计数
|
||||
|
||||
- **Proxy State**：`FETCHED`、`CHECKING`、`AVAILABLE`、`SUSPECT`、
|
||||
`DRAINING`、`UNHEALTHY`、`EXTRACTED`、`EXPIRED`、`REMOVED`。
|
||||
- **Active Concurrency**：已建立并正在使用 Proxy 的 Gateway 并发。
|
||||
- **Reserved Concurrency**：已选中、正在建连但尚未转为 Active 的并发。
|
||||
- **Available Slots**：所有可分配 Proxy 的有效并发上限减去 Active 与
|
||||
Reserved 后的总和。
|
||||
- **Consecutive Empty Fetch**：Provider 调用成功且解析成功，但解析后没有
|
||||
任何合法代理的连续次数。
|
||||
- **Fetch Error Count**：超时、DNS、HTTP、认证、解析或模板执行错误次数。
|
||||
- **Current Upstream**：Sequential Routing 当前指向的 Upstream 索引。
|
||||
|
||||
## 行为术语
|
||||
|
||||
- **Gateway Allocation**：Worker 原子预留本地所有 Proxy 容量，建连成功后
|
||||
转为 Active，结束后释放。
|
||||
- **Exclusive Extraction**：控制面原子把可提取 Proxy 从 `AVAILABLE` 改为
|
||||
`EXTRACTED`，成功后该 Proxy 永不再次被系统分配。
|
||||
- **Drain**：停止新分配，等待 Reserved 与 Active 归零后转换状态或撤销所有权。
|
||||
- **Empty Fetch**：不是错误、不是重复，而是有效 Provider 响应中没有任何
|
||||
合法 Proxy 候选。
|
||||
- **Switch**：Routing 使用 CAS 从当前 Upstream 前进到下一个；不会销毁旧
|
||||
Upstream 已有 Proxy。
|
||||
- **Snapshot**：Controller 发布给 Worker 的不可变、版本化 Routing、Proxy
|
||||
所有权和策略视图。
|
||||
|
||||
## 配置语义
|
||||
|
||||
- **pool.maxSize**：当前系统维护且尚未被提取的 Proxy 硬上限。
|
||||
- **fetch.maxTotal**：可选的计费周期累计获取上限，和 pool.maxSize 无关。
|
||||
- **allocationSafetyMargin**：距离过期不足此时间时停止新分配。
|
||||
- **reserveForGateway**：共享池中不能被 Distribution 提取的最低可用数量。
|
||||
- **fulfillment.partial**：尽量返回，允许少于请求数量。
|
||||
- **fulfillment.allOrNothing**：不足时一个也不提取。
|
||||
|
||||
458
docs/design/architecture.md
Normal file
458
docs/design/architecture.md
Normal file
@ -0,0 +1,458 @@
|
||||
# Proxy Pool 总体架构设计
|
||||
|
||||
## 1. 范围
|
||||
|
||||
Proxy Pool 是多供应商代理聚合平台，同时提供：
|
||||
|
||||
- **Gateway**：接受 HTTP/HTTPS CONNECT 请求，选择上游代理并代转发。
|
||||
- **Distribution API**：一次性独占发放真实代理地址，发放后不再管理其使用。
|
||||
- **Admin API**：查询状态、启停 Upstream、切换 Routing、触发重载。
|
||||
- **Control Plane**：Provider 获取、健康、生命周期、容量、切换、持久化与分发。
|
||||
|
||||
首要容量目标是集群峰值 100,000 QPS。该数字是设计目标，必须通过后续
|
||||
容量测试证明，不能由文档直接宣称实现。
|
||||
|
||||
## 2. 选择的方案
|
||||
|
||||
采用“独立数据面 + 集中控制面 + 独立健康执行器”：
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
C[Clients] --> LB[HAProxy / Envoy / LVS]
|
||||
LB --> G1[Gateway Worker]
|
||||
LB --> GN[Gateway Worker N]
|
||||
G1 --> PX[Owned Proxies]
|
||||
GN --> PX
|
||||
|
||||
E[Extract Clients] --> D[Distribution API]
|
||||
A[Operators] --> ADM[Admin API]
|
||||
|
||||
D --> CP[Controller]
|
||||
ADM --> CP
|
||||
CP --> PG[(PostgreSQL)]
|
||||
CP --> R[(Redis)]
|
||||
CP --> F[Provider APIs]
|
||||
CP --> CK[Checker Workers]
|
||||
CP --> G1
|
||||
CP --> GN
|
||||
```
|
||||
|
||||
### 2.1 为什么不用单体
|
||||
|
||||
Provider、数据库和健康探测是不可控 I/O。把它们和 Gateway 放在一个进程，
|
||||
会让供应商抖动、配置重载和数据库故障直接污染 100k QPS 热路径。
|
||||
|
||||
### 2.2 为什么不全面微服务化
|
||||
|
||||
Provider、Pool、Routing、Distribution 在首版需要共享事务和一致性规则。
|
||||
先保持 Controller 模块化单体，避免提前引入事件顺序和分布式事务；Checker
|
||||
因负载特征不同独立扩缩，Gateway 因热路径独立部署。
|
||||
|
||||
## 3. 进程与职责
|
||||
|
||||
### 3.1 proxy-gateway
|
||||
|
||||
- Client 认证、CIDR 访问控制和租户限流。
|
||||
- HTTP 正向代理与 HTTPS CONNECT。
|
||||
- 首条命中 Routing、Upstream 选择和 Proxy least-connections 选择。
|
||||
- 本地原子容量预留、建连、Active 计数和结果上报。
|
||||
- 本地不可变 Snapshot；热路径无数据库/Redis/Provider API。
|
||||
- SSRF 与 DNS Rebinding 防护。
|
||||
|
||||
### 3.2 proxy-controller
|
||||
|
||||
- 配置加载、校验、热更新和版本管理。
|
||||
- Upstream/Provider fetch Leader、独立限流、singleflight 和退避。
|
||||
- Proxy 解析、去重、TTL、状态机、pool.maxSize 与 fetch.maxTotal。
|
||||
- Routing Sequential 当前选择和原子切换。
|
||||
- Worker 所有权/容量切片、Snapshot 发布、ACK 和重同步。
|
||||
- Distribution 与 Admin HTTP 接口。
|
||||
- PostgreSQL 权威状态和 Redis 短期协调。
|
||||
|
||||
### 3.3 proxy-checker
|
||||
|
||||
- 消费检查任务，执行基础连接、出口和目标级探测。
|
||||
- 使用 jitter、maxInFlight、超时和分级复检。
|
||||
- 只上报 Observation，不直接修改最终状态。
|
||||
- Controller 的确定性 reducer 根据 Observation 更新状态。
|
||||
|
||||
### 3.4 proxy-loadgen
|
||||
|
||||
- 分别生成 HTTP QPS、CONNECT 活跃连接、建连速率和 Extract 并发。
|
||||
- 输出环境、场景、延迟、错误、CPU、RSS、句柄和网络结果。
|
||||
|
||||
## 4. 模块边界
|
||||
|
||||
```text
|
||||
domain/
|
||||
proxy Proxy、状态机、TTL、唯一键、容量
|
||||
routing 规则、匹配、策略与 Sequential 状态
|
||||
upstream Provider 能力、Fetch 分类与 Pool 需求
|
||||
extraction 独占提取命令、结果和审计事实
|
||||
client 认证主体、权限与限制
|
||||
|
||||
gateway/
|
||||
ingress HTTP/CONNECT 接入
|
||||
dispatch Routing、Proxy 选择、容量预留、重试资格
|
||||
transport 上游连接复用、握手、隧道和失败阶段
|
||||
snapshot 版本校验、后台构建和原子切换
|
||||
|
||||
controller/
|
||||
provider Fetch 计划、限流、singleflight、错误分类
|
||||
pool Pool reconcile、生命周期和所有权
|
||||
health 任务计划和 Observation reducer
|
||||
routing Sequential 原子切换和恢复
|
||||
extraction 批量原子提取与审计
|
||||
distribution Worker 注册、Snapshot/Delta 与结果上报
|
||||
|
||||
adapters/
|
||||
postgres, redis, grpc, http, provider_template
|
||||
```
|
||||
|
||||
领域模块不导入 HTTP、gRPC、SQL、Redis 或模板引擎。Adapter 依赖领域接口，
|
||||
进程装配只发生在 `cmd`。
|
||||
|
||||
## 5. Proxy 模型
|
||||
|
||||
```go
|
||||
type Proxy struct {
|
||||
ID ProxyID
|
||||
Scheme Scheme
|
||||
Host string
|
||||
Port uint16
|
||||
Username string
|
||||
CredentialVersion string
|
||||
SecretRef string
|
||||
SourceUpstream UpstreamID
|
||||
CreatedAt time.Time
|
||||
ExpiresAt *time.Time
|
||||
LastCheckedAt *time.Time
|
||||
LastSuccessAt *time.Time
|
||||
Latency time.Duration
|
||||
MaxConcurrency int64
|
||||
State ProxyState
|
||||
Tags map[string]string
|
||||
}
|
||||
```
|
||||
|
||||
密码不参与日志可见唯一键；`CredentialVersion` 区分同一用户名的凭据轮换。
|
||||
运行态 `active` 与 `reserved` 存在 Worker 本地、按 Proxy ID 分片，不写入
|
||||
不可变 Snapshot。
|
||||
|
||||
## 6. Proxy 状态机
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> FETCHED
|
||||
FETCHED --> CHECKING
|
||||
CHECKING --> AVAILABLE: basic check passed
|
||||
CHECKING --> UNHEALTHY: check exhausted
|
||||
AVAILABLE --> SUSPECT: first meaningful failure
|
||||
SUSPECT --> AVAILABLE: recheck passed
|
||||
SUSPECT --> UNHEALTHY: consecutive failures
|
||||
AVAILABLE --> DRAINING: TTL margin / disable / ownership revoke
|
||||
DRAINING --> EXPIRED: no active or reserved capacity
|
||||
AVAILABLE --> EXTRACTED: atomic exclusive extraction
|
||||
UNHEALTHY --> REMOVED
|
||||
EXTRACTED --> EXPIRED: expires
|
||||
EXPIRED --> REMOVED
|
||||
```
|
||||
|
||||
`ACTIVE` 与 `BUSY` 不是持久状态，而是容量计数。一个 AVAILABLE Proxy 可以
|
||||
同时承载多个 Gateway 请求，直到有效并发上限。
|
||||
|
||||
## 7. Gateway 热路径
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Client
|
||||
participant G as Gateway
|
||||
participant D as Dispatch
|
||||
participant T as Transport
|
||||
participant P as Upstream Proxy
|
||||
|
||||
C->>G: HTTP / CONNECT
|
||||
G->>G: auth + access + admission
|
||||
G->>D: RouteRequest
|
||||
D->>D: first-match route + local candidate filter
|
||||
D->>D: CAS reserved +1
|
||||
D-->>G: Allocation
|
||||
G->>T: execute allocation
|
||||
T->>P: dial / proxy handshake
|
||||
alt connected
|
||||
T->>D: reserved -1, active +1
|
||||
T-->>C: response / tunnel
|
||||
T->>D: active -1 + outcome
|
||||
else failed before commit
|
||||
T->>D: reserved -1 + failure
|
||||
G->>D: optional safe retry with exclusion
|
||||
end
|
||||
```
|
||||
|
||||
### 7.1 Dispatch 接口
|
||||
|
||||
```go
|
||||
type Dispatcher interface {
|
||||
Acquire(context.Context, RouteRequest) (*Allocation, error)
|
||||
Commit(*Allocation) error
|
||||
Release(*Allocation, Outcome)
|
||||
}
|
||||
```
|
||||
|
||||
不变量：
|
||||
|
||||
- `Acquire` 只读本地 Snapshot 与本地分片运行态。
|
||||
- `Allocation` 创建前必须 CAS 预留成功。
|
||||
- Commit 将 Reserved 恰好一次转换为 Active。
|
||||
- 未 Commit 的失败释放 Reserved；已 Commit 的结束释放 Active。
|
||||
- Release 重复调用安全但产生错误指标。
|
||||
|
||||
### 7.2 重试提交点
|
||||
|
||||
- HTTP：响应头写给 Client 前可按方法与失败阶段重试。
|
||||
- CONNECT：上游 CONNECT 成功且向 Client 写 200 后不可重试。
|
||||
- GET/HEAD 默认最多尝试 2 个不同 Proxy。
|
||||
- POST/PUT/PATCH/DELETE 默认不自动重试。
|
||||
|
||||
## 8. Provider Fetch
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
S[Capacity signal] --> SF{singleflight running?}
|
||||
SF -->|yes| W[Coalesce signal]
|
||||
SF -->|no| R[Read current demand]
|
||||
R --> L[Acquire Provider leader]
|
||||
L --> C{Under maxSize/maxTotal?}
|
||||
C -->|no| X[Stop]
|
||||
C -->|yes| I[Wait requestInterval]
|
||||
I --> M[Acquire maxInFlight]
|
||||
M --> API[Call Provider API]
|
||||
API --> CL{Classify}
|
||||
CL -->|error| BO[Backoff + fetchErrorCount]
|
||||
CL -->|empty| EC[emptyCount++]
|
||||
CL -->|valid| DD[Deduplicate + reset empty]
|
||||
DD --> HC[Create FETCHED and schedule check]
|
||||
```
|
||||
|
||||
### 8.1 Empty、Duplicate 与 Error
|
||||
|
||||
- **Empty**：HTTP/认证成功、模板执行成功，解析后合法 Proxy 数为 0。
|
||||
- **Duplicate-only**：合法 Proxy 数大于 0，但去重后新增为 0；不计 Empty，
|
||||
重置 Empty 连续计数并增加 duplicate 指标。
|
||||
- **Error**：超时、DNS、非预期 HTTP、认证、响应超限、模板或解析异常；
|
||||
Empty 计数保持不变，增加 Error 并进入退避。
|
||||
- **Success**：至少一个合法候选；Empty 计数归零。是否最终入池由去重、
|
||||
maxSize、TTL 和健康结果决定。
|
||||
|
||||
## 9. Sequential 切换
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant F as Provider Controller
|
||||
participant U as Upstream Runtime
|
||||
participant R as Routing Runtime
|
||||
participant DB as PostgreSQL
|
||||
|
||||
F->>U: EmptyFetch
|
||||
U->>U: consecutiveEmpty++
|
||||
alt below threshold
|
||||
U-->>F: continue current
|
||||
else threshold reached
|
||||
U->>DB: persist depleted generation
|
||||
DB-->>R: notify affected routings
|
||||
R->>DB: CAS current index A -> B
|
||||
DB-->>R: switched once
|
||||
R->>R: old A proxies drain naturally
|
||||
end
|
||||
```
|
||||
|
||||
Upstream 的 Empty 事实全局共享；每条 Routing 独立 CAS 当前索引。多个并发
|
||||
协程只能有一个成功从 A 切到 B，其他协程读取新版本，不会再切到 C。
|
||||
|
||||
## 10. Exclusive Extraction
|
||||
|
||||
### 10.1 核心流程
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Extract Client
|
||||
participant API as Distribution API
|
||||
participant CP as Controller
|
||||
participant W as Gateway Worker
|
||||
participant DB as PostgreSQL
|
||||
|
||||
C->>API: POST extract(count, filters)
|
||||
API->>API: auth + access + limits
|
||||
API->>CP: Extract command
|
||||
CP->>DB: select eligible unowned proxies
|
||||
alt candidate owned by Worker
|
||||
CP->>W: revoke and drain
|
||||
W-->>CP: active=0, reserved=0, ownership released
|
||||
end
|
||||
CP->>DB: transaction AVAILABLE -> EXTRACTED
|
||||
DB->>DB: insert extraction records
|
||||
DB-->>CP: committed rows
|
||||
CP-->>API: proxy + expiry
|
||||
API-->>C: requested/returned/proxies
|
||||
```
|
||||
|
||||
### 10.2 原子批量提取
|
||||
|
||||
PostgreSQL Adapter 使用事务和 `FOR UPDATE SKIP LOCKED` 选择满足以下条件的
|
||||
行，并在同一事务更新状态与写审计记录：
|
||||
|
||||
- State 为 AVAILABLE。
|
||||
- 未分配 Worker 所有权，Reserved/Active 为 0。
|
||||
- 剩余 TTL 不低于 `minRemainingTTL`。
|
||||
- 健康检查时间不早于 `maxHealthCheckAge`。
|
||||
- 提取后仍保留 `reserveForGateway`。
|
||||
- 符合 protocol、region、carrier 与 allowedUpstreams。
|
||||
|
||||
`partial` 提交实际可得数量；`allOrNothing` 在锁定数量不足时回滚。
|
||||
|
||||
### 10.3 与 Gateway 共池
|
||||
|
||||
Controller 优先维护两类库存：Worker-owned Gateway 容量和 unowned Extract
|
||||
库存。Distribution 只直接提取 unowned Proxy。需要从 Gateway 回收时，先
|
||||
发布 DRAINING、等待 Worker ACK 和容量归零，再清除所有权并提取。这样避免
|
||||
Snapshot 传播延迟造成同一 Proxy 同时被 Gateway 新分配和 API 发放。
|
||||
|
||||
## 11. Cluster 所有权
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
CP[Controller] -->|capacity lease| W1[Worker 1]
|
||||
CP -->|capacity lease| W2[Worker 2]
|
||||
P1[Proxy set A] --> W1
|
||||
P2[Proxy set B] --> W2
|
||||
U[Unowned extract reserve] --> CP
|
||||
```
|
||||
|
||||
- 同一个 Proxy 同一时刻只归一个 Worker 所有。
|
||||
- Worker 仅对自身 Proxy 做本地原子计数。
|
||||
- 所有权租约有 epoch 与过期时间；Worker 失联后先等待租约过期，再分配给
|
||||
其他 Worker，避免双主。
|
||||
- Snapshot 包含 Worker 专属 Proxy 集、Routing、策略和版本校验和。
|
||||
- Worker 断开控制面后在 `maxStaleAge` 内使用最后快照；超限停止接收新流量，
|
||||
已有隧道排空。
|
||||
|
||||
## 12. 存储与一致性
|
||||
|
||||
### 12.1 PostgreSQL
|
||||
|
||||
权威保存：配置版本、Upstream、Proxy 生命周期、Routing 当前选择、Worker
|
||||
所有权、Extraction Record、Client、审计与 outbox。
|
||||
|
||||
### 12.2 Redis
|
||||
|
||||
保存可重建短期状态：Provider Leader 租约、分布式限流、singleflight 信号、
|
||||
短期 Client 限流和 Worker 心跳。Redis 不保存唯一权威业务状态。
|
||||
|
||||
### 12.3 Outbox
|
||||
|
||||
任何需要发布 Snapshot/事件的 PostgreSQL 状态更新同时写 outbox。发布成功
|
||||
后标记；Worker 使用 epoch/version 幂等应用，缺口触发完整 Snapshot。
|
||||
|
||||
## 13. 健康模型
|
||||
|
||||
Health Observation 包含 Proxy、检查层级、Routing/目标组、阶段、结果、延迟、
|
||||
出口信息与时间。Reducer 规则：
|
||||
|
||||
- 新 Proxy 必须通过基础检查才可 AVAILABLE。
|
||||
- 第一次有意义失败进入 SUSPECT，不立即删除。
|
||||
- 连续失败达到配置阈值进入 UNHEALTHY。
|
||||
- SUSPECT 复检成功恢复 AVAILABLE。
|
||||
- 目标级失败只影响对应 Routing/Target Profile，不直接全局删除。
|
||||
- 任务按稳定哈希分散，并对 interval 加 jitter。
|
||||
|
||||
## 14. 配置热更新
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
F[Read file] --> P[Parse]
|
||||
P --> V[Validate references and regex]
|
||||
V --> B[Build immutable config]
|
||||
B --> D[Diff old/new]
|
||||
D --> S[Atomic swap]
|
||||
S --> G[Drain removed tasks/resources]
|
||||
```
|
||||
|
||||
热更新规则：
|
||||
|
||||
- 删除/禁用 Upstream：停止 Fetch 和新分配，已有 Proxy/连接 Drain。
|
||||
- 减小 maxSize：不强杀连接，停止补池并按策略自然缩容。
|
||||
- Routing 立即对新请求生效；旧请求持有旧 Snapshot 完成。
|
||||
- 修改 API 地址或凭据版本会重建 Provider Adapter，但不会把错误计成 Empty。
|
||||
- 新配置任何校验失败时，保留旧版本并报告完整错误。
|
||||
|
||||
## 15. 安全
|
||||
|
||||
- Gateway、Distribution、Admin 认证相互独立。
|
||||
- `auth.mode` 支持 none、usernamePassword、apiKey、ipWhitelist 与组合 any。
|
||||
- `access.allowCIDRs` 独立于认证；代理头只在来源属于 trustedProxies 时接受。
|
||||
- 严格模式下，非回环监听且 auth=none、allowCIDRs 为空时启动失败。
|
||||
- 目的地址解析前后都拒绝 loopback、private、link-local、metadata 和配置禁区。
|
||||
- Secret 使用环境变量或文件引用；日志统一 Secret 类型脱敏。
|
||||
- Provider 模板只能使用白名单纯函数，禁止文件、网络和环境变量读取。
|
||||
- Distribution 返回真实凭据属于 Raw 弱控制模式，文档必须明确其风险。
|
||||
|
||||
## 16. 可观测性
|
||||
|
||||
关键指标：
|
||||
|
||||
- Gateway 请求、连接、建连、字节、延迟、重试和拒绝。
|
||||
- Routing 当前 Upstream、切换、无可用和策略选择。
|
||||
- Upstream Proxy 状态、Available Slots、Active、Reserved、Pending Fetch。
|
||||
- Provider Fetch 成功、Empty、Duplicate、Error、429、退避和耗时。
|
||||
- Extraction requested、returned、insufficient、atomic conflict 和审计失败。
|
||||
- Snapshot epoch/version、陈旧时长、应用耗时和重同步。
|
||||
|
||||
Prometheus 标签不包含 Proxy IP、Client ID、session、完整 URL 或 request ID。
|
||||
这些信息进入受控、脱敏、可采样日志。
|
||||
|
||||
## 17. 性能与容量
|
||||
|
||||
### 17.1 集群容量公式
|
||||
|
||||
```text
|
||||
worker_replicas =
|
||||
ceil(peak_qps / (tested_worker_qps * target_utilization))
|
||||
+ largest_failure_domain_replicas
|
||||
```
|
||||
|
||||
默认 `peak_qps=100000`、`target_utilization<=0.60`。单 Worker 能力必须在相同
|
||||
CPU、内存、网络、Go 版本、配置和上游响应模型下测得。
|
||||
|
||||
### 17.2 热路径预算
|
||||
|
||||
- Dispatch 无 I/O、无全局锁，100k Proxy Snapshot 下 p99 小于 100 微秒。
|
||||
- 所有队列、buffer、重试和日志均有界。
|
||||
- Listener、Client、Routing、Worker 和 Proxy 均有独立准入限制。
|
||||
- 过载在路由/建连前快速拒绝，不允许请求堆积耗尽内存。
|
||||
- 结果上报批量、异步、有界，不反压请求热路径。
|
||||
|
||||
### 17.3 独立场景
|
||||
|
||||
必须分别测试：10k 稳态 QPS、100k 峰值 QPS、活跃 CONNECT、建连速率、
|
||||
大 Snapshot 更新、Provider 故障、Controller 断线和 Worker 故障域丢失。
|
||||
|
||||
## 18. 故障行为
|
||||
|
||||
| 故障 | 行为 |
|
||||
|---|---|
|
||||
| Provider 超时/500 | 计 Error、退避；不计 Empty，不影响已有 Proxy |
|
||||
| Provider 合法空响应 | Empty++；达到阈值触发相关 Routing 原子切换 |
|
||||
| Redis 不可用 | Gateway 不受影响；Controller 使用本地退化并停止高风险写操作 |
|
||||
| PostgreSQL 不可用 | Gateway 使用最后快照；停止 Extract 和权威状态变更 |
|
||||
| Controller 断线 | Worker 在 maxStaleAge 内继续；超限拒绝新流量并排空 |
|
||||
| Worker 崩溃 | 所有权租约过期后重新分配；期间不双重所有 |
|
||||
| Checker 积压 | 降低普通复检频率，优先新 Proxy 与 SUSPECT，不无限排队 |
|
||||
| Snapshot 缺版本 | 丢弃增量并请求完整 Snapshot |
|
||||
| 所有 Upstream 不可用 | 执行显式 onUnavailable，默认 reject |
|
||||
|
||||
## 19. 交付边界
|
||||
|
||||
项目架构必须包含四个命令、领域模块、Gateway/Controller/Checker 模块、
|
||||
存储/协议 Adapter、OpenAPI/Proto、配置样例、Compose/Kubernetes、监控、
|
||||
迁移、测试 fixture、负载场景和开发文档。目录存在但没有契约或测试不算完成。
|
||||
|
||||
217
docs/development/implementation-plan.md
Normal file
217
docs/development/implementation-plan.md
Normal file
@ -0,0 +1,217 @@
|
||||
# Proxy Pool Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use
|
||||
> `superpowers:subagent-driven-development` or `superpowers:executing-plans`.
|
||||
> Every step is tracked with checkbox syntax and must preserve the requirement IDs in
|
||||
> `docs/requirements/traceability.md`.
|
||||
|
||||
**Goal:** Build a production-oriented Go repository whose domain behavior, interfaces,
|
||||
configuration, contracts, documentation, and deployment layout implement the final
|
||||
semantics in `对话内容.md`.
|
||||
|
||||
**Architecture:** Separate Gateway, Controller, Checker, and Loadgen commands. Domain
|
||||
packages remain transport-free. Gateway reads immutable local snapshots. Controller
|
||||
owns provider fetch, pool lifecycle, routing state, extraction, persistence, and worker
|
||||
distribution. PostgreSQL is authoritative; Redis stores rebuildable coordination state.
|
||||
|
||||
**Tech Stack:** Go 1.26, `go.yaml.in/yaml/v4`, pgx/v5, go-redis/v9, gRPC/Protobuf,
|
||||
Prometheus, PostgreSQL, Redis, Docker Compose, Kubernetes.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```text
|
||||
cmd/
|
||||
proxy-gateway/main.go
|
||||
proxy-controller/main.go
|
||||
proxy-checker/main.go
|
||||
proxy-loadgen/main.go
|
||||
internal/
|
||||
config/{config.go,load.go,validate.go}
|
||||
domain/proxy/{proxy.go,state.go,capacity.go}
|
||||
domain/routing/{rule.go,strategy.go,sequential.go}
|
||||
domain/upstream/{upstream.go,fetch_result.go,pool.go}
|
||||
domain/extraction/{extraction.go,store.go}
|
||||
domain/client/client.go
|
||||
gateway/{server,dispatch,snapshot,transport}/
|
||||
controller/{provider,pool,routing,extraction,health,distribution}/
|
||||
adapters/{memory,postgres,redis,providerapi}/
|
||||
platform/{logging,metrics,shutdown}/
|
||||
api/{openapi,proto}/
|
||||
configs/
|
||||
deploy/{compose,kubernetes,haproxy,prometheus,grafana}/
|
||||
docs/{design,development,configuration,api,operations,testing,adr,requirements}/
|
||||
diagrams/
|
||||
examples/
|
||||
test/{fixtures,integration,e2e,load}/
|
||||
```
|
||||
|
||||
## Task 1: Repository and Build Baseline
|
||||
|
||||
**Files:** `go.mod`, `.golangci.yml`, `README.md`, `scripts/verify.ps1`,
|
||||
`.github/workflows/ci.yml`
|
||||
|
||||
- [ ] Create module `github.com/proxy-pool/proxy-pool` with Go 1.26.
|
||||
- [ ] Pin YAML v4, pgx/v5, go-redis/v9, gRPC, protobuf, Prometheus, and x/sync.
|
||||
- [ ] Add `scripts/verify.ps1` that runs format check, `go vet`, unit tests, race tests,
|
||||
and builds all commands, each test command bounded to 60 seconds.
|
||||
- [ ] Add CI for Windows and Linux with unit/race/build jobs.
|
||||
- [ ] Verify `go mod tidy`, `go test ./...`, and `go build ./cmd/...` succeed.
|
||||
|
||||
## Task 2: Strict Configuration
|
||||
|
||||
**Files:** `internal/config/config.go`, `load.go`, `validate.go`, corresponding tests,
|
||||
`configs/default.yaml`, `docs/configuration/reference.md`
|
||||
|
||||
- [ ] Define versioned types for security, gateway, distribution, admin, metrics,
|
||||
storage, routing, upstream/provider/api/proxyAuth/pool/capacity/lifecycle/fetch/check.
|
||||
- [ ] Decode one YAML document with known fields enabled and resolve `${ENV}` plus
|
||||
secret file references without logging values.
|
||||
- [ ] Validate listener protection, routing references/order, regexes, strategy fields,
|
||||
positive limits, TTL margins, pool/fetch limits, auth modes, and exposure modes.
|
||||
- [ ] Add table tests for every invalid condition in CFG requirements.
|
||||
|
||||
## Task 3: Proxy Domain and Capacity
|
||||
|
||||
**Files:** `internal/domain/proxy/*.go`, corresponding tests
|
||||
|
||||
- [ ] Implement Proxy fields, UTC TTL precedence, canonical host/port, and unique key.
|
||||
- [ ] Implement state transitions and reject illegal transitions.
|
||||
- [ ] Implement sharded runtime counters with CAS Reserve, Commit, Cancel, Release.
|
||||
- [ ] Prove with 1,000 concurrent goroutines that effective capacity is never exceeded.
|
||||
- [ ] Add race coverage and duplicate-release invariant metrics hook.
|
||||
|
||||
## Task 4: Routing and Sequential Switching
|
||||
|
||||
**Files:** `internal/domain/routing/*.go`, corresponding tests
|
||||
|
||||
- [ ] Compile first-match host/method/path/header rules into an immutable RuleSet.
|
||||
- [ ] Implement random, round-robin, weighted, least-connections, and sequential.
|
||||
- [ ] Model upstream empty counters separately from per-routing current indexes.
|
||||
- [ ] Implement versioned CAS switch so simultaneous threshold observers advance once.
|
||||
- [ ] Cover four-empty-then-success, five-empty, A-to-B-only, disabled references,
|
||||
end behavior, and explicit onUnavailable.
|
||||
|
||||
## Task 5: Provider Fetch Classification and Scheduling
|
||||
|
||||
**Files:** `internal/controller/provider/*.go`, `internal/domain/upstream/*.go`, tests
|
||||
|
||||
- [ ] Implement Valid, Empty, DuplicateOnly, and Error result classes exactly as the
|
||||
traceability matrix defines.
|
||||
- [ ] Implement one coalesced reconcile signal per Upstream using singleflight.
|
||||
- [ ] Enforce requestInterval, maxInFlight, maxSize, maxTotal, timeout, retry,
|
||||
exponential backoff, jitter, and Retry-After.
|
||||
- [ ] Define ProviderAdapter and safe TemplateParser ports; add fixture adapters.
|
||||
- [ ] Test that 100 concurrent capacity signals do not fan out 100 Provider calls.
|
||||
|
||||
## Task 6: Pool Reconciliation and Ownership
|
||||
|
||||
**Files:** `internal/controller/pool/*.go`, `internal/domain/upstream/pool.go`, tests
|
||||
|
||||
- [ ] Compute Available Slots from eligible Proxy capacity, Active, Reserved, TTL,
|
||||
health, ownership, pending expected fetch, and gateway reserve.
|
||||
- [ ] Implement pool.maxSize and fetch.maxTotal as distinct counters.
|
||||
- [ ] Allocate each Proxy to one Worker with epoch/version/expiry ownership.
|
||||
- [ ] Implement revoke -> drain -> ACK -> unowned transition.
|
||||
- [ ] Test Worker crash expiry and prevent simultaneous dual ownership.
|
||||
|
||||
## Task 7: Exclusive Extraction
|
||||
|
||||
**Files:** `internal/domain/extraction/*.go`, `internal/controller/extraction/*.go`,
|
||||
`internal/adapters/memory/extraction.go`, tests
|
||||
|
||||
- [ ] Implement POST extraction command with protocol/region/carrier/upstream filters.
|
||||
- [ ] Enforce minRemainingTTL, maxHealthCheckAge, maxCount, client limits, and
|
||||
reserveForGateway.
|
||||
- [ ] Atomically transition AVAILABLE to EXTRACTED and append audit records.
|
||||
- [ ] Implement partial and allOrNothing without Lease, release, or renewal concepts.
|
||||
- [ ] Run 1,000 concurrent claim attempts and prove every Proxy ID appears at most once.
|
||||
|
||||
## Task 8: Immutable Snapshot and Dispatch
|
||||
|
||||
**Files:** `internal/gateway/snapshot/*.go`, `internal/gateway/dispatch/*.go`, tests
|
||||
|
||||
- [ ] Define cluster/worker/epoch/version/checksum snapshot envelopes.
|
||||
- [ ] Build indexes in the background and atomically swap complete snapshots.
|
||||
- [ ] Reject version gaps and wrong epochs; request full resync.
|
||||
- [ ] Implement Dispatch Acquire/Commit/Release over local owned Proxy runtime.
|
||||
- [ ] Benchmark 100k Proxy snapshots and record allocations and latency.
|
||||
|
||||
## Task 9: Gateway Transport
|
||||
|
||||
**Files:** `internal/gateway/server/*.go`, `internal/gateway/transport/*.go`, tests
|
||||
|
||||
- [ ] Implement HTTP forward proxy and HTTPS CONNECT through an upstream proxy.
|
||||
- [ ] Add Client auth/access/admission and destination policy checks before routing.
|
||||
- [ ] Implement safe retry commit points and prevent non-idempotent/established tunnel
|
||||
replay.
|
||||
- [ ] Use bounded buffers, deadlines, connection pools, and graceful shutdown.
|
||||
- [ ] Add local fake upstream end-to-end tests for success, 407, timeout, cancel, half
|
||||
close, retry, and blocked private destinations.
|
||||
|
||||
## Task 10: Controller APIs and Persistence Ports
|
||||
|
||||
**Files:** `internal/controller/distribution/*.go`, `admin/*.go`,
|
||||
`internal/adapters/postgres/*.go`, `internal/adapters/redis/*.go`, migrations, tests
|
||||
|
||||
- [ ] Define repository ports for Proxy, RoutingRuntime, Ownership, ExtractionRecord,
|
||||
Client, ConfigVersion, and Outbox.
|
||||
- [ ] Implement PostgreSQL extraction with one transaction and `FOR UPDATE SKIP LOCKED`.
|
||||
- [ ] Implement Redis coordination for Provider leader, distributed rate, Client limit,
|
||||
and Worker heartbeat; keep all state rebuildable.
|
||||
- [ ] Expose Distribution extraction/status and Admin status/enable/disable/switch/reload.
|
||||
- [ ] Add integration tests using Compose-backed PostgreSQL/Redis.
|
||||
|
||||
## Task 11: Checker and Health Reducer
|
||||
|
||||
**Files:** `internal/controller/health/*.go`, `cmd/proxy-checker/main.go`, tests
|
||||
|
||||
- [ ] Schedule global and route health with jitter and bounded maxInFlight.
|
||||
- [ ] Implement FETCHED -> CHECKING -> AVAILABLE and SUSPECT/UNHEALTHY transitions.
|
||||
- [ ] Ensure target failures affect only the target profile.
|
||||
- [ ] Add fixture target server and deterministic clock/scheduler tests.
|
||||
|
||||
## Task 12: Machine-readable Contracts
|
||||
|
||||
**Files:** `api/openapi/proxy-pool.yaml`, `api/proto/controlplane/v1/controlplane.proto`,
|
||||
`docs/api/*.md`
|
||||
|
||||
- [ ] Specify Distribution/Admin REST schemas, status codes, authentication, examples,
|
||||
and idempotency behavior.
|
||||
- [ ] Specify Worker register, snapshot, delta, ACK, report, heartbeat, ownership drain,
|
||||
and resync messages.
|
||||
- [ ] Validate OpenAPI and compile protobuf descriptors in CI.
|
||||
|
||||
## Task 13: Deployment and Observability
|
||||
|
||||
**Files:** `deploy/**`, `internal/platform/**`, `docs/operations/**`
|
||||
|
||||
- [ ] Add Compose for local Controller/Gateway/Checker/PostgreSQL/Redis/Prometheus/
|
||||
Grafana/HAProxy.
|
||||
- [ ] Add Kubernetes Deployments, Services, PDBs, HPA, NetworkPolicy, Secrets examples,
|
||||
probes, resource limits, topology spread, and graceful termination.
|
||||
- [ ] Add low-cardinality Prometheus metrics and structured secret-safe logs.
|
||||
- [ ] Document backup, recovery, rollout, rollback, capacity, kernel, file descriptor,
|
||||
NAT/conntrack, and incident runbooks.
|
||||
|
||||
## Task 14: Documentation, Examples, and Diagrams
|
||||
|
||||
**Files:** `docs/**`, `examples/**`, `diagrams/**`
|
||||
|
||||
- [ ] Complete README navigation, design document, developer guide, configuration
|
||||
reference, API guide, deployment guide, security model, testing guide, and roadmap.
|
||||
- [ ] Provide at least 20 validated configuration examples.
|
||||
- [ ] Provide at least 30 Mermaid architecture, flow, sequence, state, and failure diagrams.
|
||||
- [ ] Generate `proxy-pool-docs-v1.0.zip` from versioned documentation assets.
|
||||
|
||||
## Task 15: Completion Audit
|
||||
|
||||
- [ ] Map every requirement ID to code, test, contract, document, or verified runtime evidence.
|
||||
- [ ] Run `gofmt`, `go vet`, unit tests, race tests, builds, contract validation, and
|
||||
documentation link/example validation.
|
||||
- [ ] Run bounded local performance benchmarks; label 100k QPS as unverified until a
|
||||
representative cluster load run exists.
|
||||
- [ ] Confirm no TODO/TBD/placeholders, secrets, unbounded queues, high-cardinality metric
|
||||
labels, extraction Lease APIs, or conflicting maxSize semantics remain.
|
||||
|
||||
90
docs/requirements/traceability.md
Normal file
90
docs/requirements/traceability.md
Normal file
@ -0,0 +1,90 @@
|
||||
# 需求追踪矩阵
|
||||
|
||||
本文将 `对话内容.md` 的演进讨论压缩为最终可验收需求。后出现的明确修订
|
||||
覆盖早期方案，尤其是 Distribution 的 Lease 设计。
|
||||
|
||||
## 架构
|
||||
|
||||
| ID | 最终需求 | 来源 | 验证证据 |
|
||||
|---|---|---|---|
|
||||
| ARCH-001 | 数据面 Worker 与控制面 Controller 分离 | 1-70 | 进程结构、架构图、构建产物 |
|
||||
| ARCH-002 | 热路径只做认证、本地路由和网络转发 | 1-70, 380-430 | 依赖规则、测试、性能剖析 |
|
||||
| ARCH-003 | Gateway、Distribution、Admin、Metrics 独立入口 | 8904-8958 | 配置、监听装配、端口测试 |
|
||||
| ARCH-004 | Controller 集中 Provider 获取与切换 | 1403-1580 | Leader、singleflight、集成测试 |
|
||||
| ARCH-005 | 100k QPS 峰值使用多 Worker 集群 | 当前会话 | 容量公式、负载场景、部署清单 |
|
||||
|
||||
## Routing 与 Upstream
|
||||
|
||||
| ID | 最终需求 | 来源 | 验证证据 |
|
||||
|---|---|---|---|
|
||||
| ROUTE-001 | Routing 自上而下匹配，首条命中停止 | 3534-3798, 5825-6467 | 路由单测 |
|
||||
| ROUTE-002 | Routing 与 Upstream 生命周期解耦 | 3534-3798 | 包依赖与配置模型 |
|
||||
| ROUTE-003 | 支持 sequential、random、roundRobin、weighted、leastConnections | 5825-6467 | 策略契约测试 |
|
||||
| ROUTE-004 | Sequential 连续空结果达到阈值后原子切换一次 | 5295-5824, 6520-6617 | 并发切换测试 |
|
||||
| ROUTE-005 | 空计数属于 Upstream，当前选择属于 Routing | 8442-8529 | 状态模型与多 Routing 测试 |
|
||||
| ROUTE-006 | 旧 Upstream 已有 Proxy 继续耗尽，不因切换直接丢弃 | 6618-6641 | Drain 测试 |
|
||||
| ROUTE-007 | 无可用 Upstream 时显式 reject、wait 或 direct，默认 reject | 5075-5294, 6743-6760 | 配置默认值与端到端测试 |
|
||||
|
||||
## Provider 与补池
|
||||
|
||||
| ID | 最终需求 | 来源 | 验证证据 |
|
||||
|---|---|---|---|
|
||||
| FETCH-001 | 每个 Provider 有独立 requestInterval、maxInFlight、timeout 和 retry | 968-2394 | Fetcher 单测 |
|
||||
| FETCH-002 | 大量缺池信号合并为 singleflight/容量 1 通知 | 2067-2136, 8808-8849 | 100 并发请求测试 |
|
||||
| FETCH-003 | 错误使用指数退避和抖动，429 尊重 Retry-After | 1601-1831, 8808-8856 | 时钟驱动测试 |
|
||||
| FETCH-004 | Provider 获取由单逻辑 Leader 执行 | 1403-1580 | 多实例锁测试 |
|
||||
| FETCH-005 | Empty 与 Error 分开；只有合法候选为零时 Empty++ | 8442-8529 | 分类表驱动测试 |
|
||||
| FETCH-006 | 重复候选不当作 Empty，记录独立指标 | 8442-8480 | 去重测试 |
|
||||
| FETCH-007 | 模板限制响应大小、执行时间、函数集和外部访问 | 8808-8856 | 安全测试 |
|
||||
| FETCH-008 | pool.maxSize 与 fetch.maxTotal 语义分离 | 9190-9280 | 配置校验与计数测试 |
|
||||
|
||||
## Proxy 生命周期与容量
|
||||
|
||||
| ID | 最终需求 | 来源 | 验证证据 |
|
||||
|---|---|---|---|
|
||||
| PROXY-001 | Proxy 保存协议、地址、凭据引用、来源、TTL、健康、容量和标签 | 71-105, 8605-8678 | Domain 类型与序列化测试 |
|
||||
| PROXY-002 | 唯一键包含 scheme、host、port、username、credentialVersion | 6655-6727, 8605-8678 | 去重单测 |
|
||||
| PROXY-003 | TTL 来源优先级明确并统一 UTC | 681-747, 8655-8678 | TTL 表驱动测试 |
|
||||
| CAP-001 | Gateway 分配使用 Reserved -> Active 原子转换 | 1203-1467, 8530-8597 | 高并发竞态测试 |
|
||||
| CAP-002 | 补池依据 Available Slots，不只看 Proxy 数量 | 1203-1402, 8530-8597 | 容量单测 |
|
||||
| CAP-003 | pool.maxSize 包括 FETCHED/CHECKING/AVAILABLE/SUSPECT/DRAINING 与 pending expected | 3001-3533, 6642-6680 | 并发 fetch 上限测试 |
|
||||
| CAP-004 | TTL safety margin 内禁止新分配 | 173-220, 6728-6741 | 时钟测试 |
|
||||
| CAP-005 | 多 Worker 不在热路径访问 Redis 计数 | 1403-1467 | 依赖审计与压测 |
|
||||
|
||||
## Gateway
|
||||
|
||||
| ID | 最终需求 | 来源 | 验证证据 |
|
||||
|---|---|---|---|
|
||||
| GW-001 | 支持 HTTP 与 HTTPS CONNECT，SOCKS5 保留扩展接口 | 1-70 | 协议端到端测试 |
|
||||
| GW-002 | GET/HEAD 可配置安全重试，非幂等方法默认不重试 | 2600-2654, 8737-8807 | Retry 表驱动测试 |
|
||||
| GW-003 | CONNECT 建立后不得透明重放 | 221-300 | 隧道故障测试 |
|
||||
| GW-004 | Client 认证可关闭，但访问控制、身份识别和限流独立 | 8112-8441 | 配置矩阵测试 |
|
||||
| GW-005 | 防私网、回环、链路本地、元数据地址和 DNS Rebinding | 8904-8931 | 目的地址策略测试 |
|
||||
|
||||
## Distribution
|
||||
|
||||
| ID | 最终需求 | 来源 | 验证证据 |
|
||||
|---|---|---|---|
|
||||
| DIST-001 | API 提取固定为一次性独占发放，不使用 Lease | 9083-9404 | Domain 状态机与 API 测试 |
|
||||
| DIST-002 | AVAILABLE -> EXTRACTED 必须原子完成后才能返回 | 9083-9189 | 并发提取测试 |
|
||||
| DIST-003 | 支持 partial 与 allOrNothing，默认 partial | 9190-9215 | API 契约测试 |
|
||||
| DIST-004 | 保存审计记录，不提供释放接口 | 9216-9252 | Repository 测试与 OpenAPI |
|
||||
| DIST-005 | 返回 expiresAt 与 remainingTtlSeconds | 9334-9360 | 响应测试 |
|
||||
| DIST-006 | 提取前校验 minRemainingTTL 与 maxHealthCheckAge | 9334-9369 | 过滤测试 |
|
||||
| DIST-007 | reserveForGateway 防止 Extract 清空共享池 | 9281-9333 | 共享池测试 |
|
||||
| DIST-008 | 提取认证可关闭，关闭后仍有来源识别与全局限制 | 8112-8441 | 安全配置测试 |
|
||||
|
||||
## 健康、安全、运维与测试
|
||||
|
||||
| ID | 最终需求 | 来源 | 验证证据 |
|
||||
|---|---|---|---|
|
||||
| HEALTH-001 | 全局健康与 Routing/目标健康分离 | 221-270, 8679-8708 | 健康 reducer 测试 |
|
||||
| HEALTH-002 | 健康调度有 jitter、maxInFlight 和分级频率 | 8679-8736 | 调度测试 |
|
||||
| HEALTH-003 | 失败分级 SUSPECT -> UNHEALTHY -> REMOVE | 8679-8736 | 状态机测试 |
|
||||
| SEC-001 | API 认证与 Proxy 认证分离，Secret 统一脱敏 | 7528-8111, 8904-8945 | 配置类型与日志测试 |
|
||||
| SEC-002 | 非回环监听无保护时严格模式启动失败 | 8112-8441 | 配置校验测试 |
|
||||
| OPS-001 | 配置校验后构建不可变快照并原子替换 | 8959-8999 | 热更新并发测试 |
|
||||
| OPS-002 | 优雅停机停止新请求/Fetch，等待现有流量后超时关闭 | 8981-9000 | 进程测试 |
|
||||
| OBS-001 | 指标禁止 Proxy IP、session、Client、完整 URL 高基数标签 | 9001-9029 | 指标描述符测试 |
|
||||
| TEST-001 | 覆盖对话中列出的 11 个关键并发与故障场景 | 9030-9082 | CI 测试清单 |
|
||||
|
||||
85
findings.md
Normal file
85
findings.md
Normal file
@ -0,0 +1,85 @@
|
||||
# 对话内容提取结果
|
||||
|
||||
## 权威来源
|
||||
|
||||
- 文件：`对话内容.md`
|
||||
- 大小：167,284 字节
|
||||
- 行数：9,404 行
|
||||
- 读取时间：2026-07-28
|
||||
|
||||
## 最终产品边界
|
||||
|
||||
系统同时提供两种入口：
|
||||
|
||||
1. Gateway：系统选择上游代理并代转发 HTTP、HTTPS CONNECT，预留
|
||||
SOCKS5 扩展。
|
||||
2. Distribution API：一次性、独占地返回真实上游代理。代理成功提取后
|
||||
立即从可分配池移除，不需要租约、释放接口或使用量追踪。
|
||||
|
||||
系统管理多个 Upstream Provider，并使用配置化 Routing 决定 Gateway 或
|
||||
Extract 请求使用哪些 Upstream。
|
||||
|
||||
## 后出现并覆盖早期建议的决策
|
||||
|
||||
- Distribution API 的 Lease/Release/Shared Allocation 方案被最终的
|
||||
Exclusive Extraction 取代。
|
||||
- `pool.maxSize` 表示当前系统维护的未提取代理硬上限；累计供应商提取额度
|
||||
使用独立的 `fetch.maxTotal`，不能复用同一个字段。
|
||||
- `switchAfterEmptyFetch` 只统计“上游请求成功、解析成功，但解析后没有任何
|
||||
合法代理”的结果；超时、HTTP 错误、认证错误、DNS 错误和模板错误只计
|
||||
`fetchErrorCount`。全重复结果不当作空结果，单独记录。
|
||||
- `consecutiveEmptyFetch` 属于 Upstream；Sequential 当前选择属于 Routing。
|
||||
某 Upstream 达到阈值时，引用它的 Routing 原子切换；已有代理继续耗尽。
|
||||
- Extract API 默认部分满足 `partial`；也支持 `allOrNothing`。
|
||||
- Extract API 从 `AVAILABLE` 原子转换到 `EXTRACTED` 后才返回，保证同一代理
|
||||
永不发放两次。
|
||||
|
||||
## 核心不变量
|
||||
|
||||
- 请求热路径不得调用 Provider API，也不得查询全量 Redis/PostgreSQL 后排序。
|
||||
- 代理分配必须原子预留容量，防止并发超卖。
|
||||
- 代理唯一键为 `scheme + host + port + username + credentialVersion`；日志
|
||||
和指标不得暴露密码。
|
||||
- TTL 优先级为响应 `expiresAt`、响应 `ttl`、配置固定 TTL、不过期；内部
|
||||
时间统一 UTC。
|
||||
- 健康检查至少区分全局健康和 Routing/目标健康，并使用抖动和并发上限。
|
||||
- GET/HEAD 可按配置安全重试；非幂等方法默认不自动重试；CONNECT 建立后
|
||||
不透明重放。
|
||||
- 默认不直连；所有 Upstream 不可用时必须显式选择 reject、wait 或 direct。
|
||||
- Gateway、Distribution、Admin、Metrics 使用独立监听和认证/访问控制。
|
||||
- 非回环监听且无认证、无 CIDR 保护时，严格模式必须拒绝启动。
|
||||
|
||||
## 集群与性能
|
||||
|
||||
- 用户补充：高峰可能达到 100,000 请求/秒。
|
||||
- 数据面采用多 Worker，本地不可变代理快照和本地容量计数。
|
||||
- 同一代理必须由单个 Worker 所有，或由控制面下发容量切片；禁止每请求
|
||||
访问 Redis 做全局并发计数。
|
||||
- 控制面集中 Provider 获取、独立限流、singleflight、Leader 选举、状态
|
||||
持久化和快照分发。
|
||||
- 副本数必须由单 Worker 实测能力、目标利用率和故障域余量计算。
|
||||
|
||||
## 配置模型
|
||||
|
||||
顶层包含：`version`、`defaults`、`security`、`gateway`、`distribution`、
|
||||
`admin`、`metrics`、`storage`、`routing`、`upstreams`。
|
||||
|
||||
每个 Upstream 包含：`enabled`、`exposure`、`provider`、`api`、`proxyAuth`、
|
||||
`pool`、`capacity`、`lifecycle`、`fetch`、`check`。
|
||||
|
||||
Routing 自上而下匹配，首条命中停止；支持 Gateway 与 Extract 两种 purpose，
|
||||
策略至少包括 sequential、random、roundRobin、weighted、leastConnections。
|
||||
|
||||
## 必测场景
|
||||
|
||||
- 连续 4 次空后成功不得切换；连续 5 次空只切换一次。
|
||||
- 100 个并发缺池请求只触发有限次 Provider fetch。
|
||||
- 并发切换不能从 A 一次跳到 C。
|
||||
- 并发 fetch 不得突破 pool.maxSize 或 fetch.maxTotal。
|
||||
- TTL safety margin 内不得分配。
|
||||
- Gateway 与 Extract 共享池时不得容量超卖或重复提取。
|
||||
- 配置原子热更新期间请求不中断。
|
||||
- Provider 超时不得计入 Empty Fetch。
|
||||
- 重复代理不得重复入池，也不得触发空结果切换。
|
||||
- 所有 Upstream 不可用时按显式策略执行。
|
||||
|
||||
11
progress.md
Normal file
11
progress.md
Normal file
@ -0,0 +1,11 @@
|
||||
# 项目进度
|
||||
|
||||
## 2026-07-28
|
||||
|
||||
- 用户删除了先前基于不完整网页内容生成的设计文件。
|
||||
- 已重新读取当前 `对话内容.md` 全部 9,404 行。
|
||||
- 已按主题定位配置定稿、实施方案、Distribution API、认证、安全、并发、
|
||||
故障语义和最终 Exclusive Extraction 修订。
|
||||
- 已建立新的任务计划与事实记录，旧网页摘要不再作为需求证据。
|
||||
- 尚未创建实现代码。
|
||||
|
||||
45
task_plan.md
Normal file
45
task_plan.md
Normal file
@ -0,0 +1,45 @@
|
||||
# Proxy Pool 项目交付计划
|
||||
|
||||
## 目标
|
||||
|
||||
以 `对话内容.md` 的 9404 行完整讨论为唯一需求基线，交付设计文案、
|
||||
开发文档和可编译、可测试、可扩展的 Go 项目架构。
|
||||
|
||||
## 需求覆盖原则
|
||||
|
||||
- 同一主题存在多轮修订时，以文件中最后出现的明确结论为准。
|
||||
- 用户补充的集群峰值为 100,000 QPS，覆盖文档早期 10,000 并发示例。
|
||||
- Extract API 固定采用一次性独占发放；早期 Lease/Release 设计作废。
|
||||
- 不保留空实现、TBD、TODO 或没有接口契约的占位目录。
|
||||
|
||||
## 阶段
|
||||
|
||||
1. [已完成] 完整读取对话并识别覆盖关系
|
||||
2. [进行中] 建立需求追踪矩阵与统一领域模型
|
||||
3. [待开始] 编写总体设计、详细设计和 ADR
|
||||
4. [待开始] 编写开发、配置、API、测试和运维文档
|
||||
5. [待开始] 搭建 Go 模块、命令、核心包、契约和部署目录
|
||||
6. [待开始] 实现核心状态机、路由、容量、提取与配置校验
|
||||
7. [待开始] 执行单元测试、竞态检查、静态检查和构建
|
||||
8. [待开始] 按需求矩阵逐项审计并打包交付
|
||||
|
||||
## 串并行关系
|
||||
|
||||
```text
|
||||
全文解析 ─> 需求矩阵 ─> 统一设计 ─┬─> 设计/开发文档 ─┐
|
||||
├─> API/配置契约 ───┼─> 集成验证
|
||||
└─> Go 项目架构 ────┘
|
||||
```
|
||||
|
||||
## 测试约束
|
||||
|
||||
- 单条后台测试命令最长运行 60 秒。
|
||||
- 核心并发代码必须执行 `go test -race`。
|
||||
- 100k QPS 只能由可复现的集群压测结果证明，架构文档不得把设计容量
|
||||
写成已验证性能。
|
||||
|
||||
## 已知环境限制
|
||||
|
||||
- Docker CLI 已安装，但 Linux daemon 状态需在集成验证前再次确认。
|
||||
- 当前仓库尚无提交；`对话内容.md` 和 `.gitignore` 为现有文件。
|
||||
|
||||
Loading…
Reference in New Issue
Block a user