diff --git a/CONTEXT.md b/CONTEXT.md index 7d24dc6..6c880e5 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -8,13 +8,21 @@ - **Upstream**：一个供应商配置及其聚合代理池。Upstream 运行时全局共享， 不随 Routing 重复创建。 - **Provider**：Upstream 背后的外部代理供应商及其获取接口能力。 -- **Proxy**：从 Provider 获取并标准化后的代理资源，不等同于简单 IP。 +- **Proxy**：从 Provider 获取并标准化后的短效代理资源，不等同于简单 IP。 + Proxy 明细不进入 PostgreSQL，只存在于带 TTL 的 Redis 活动池和持有其快照的 + 节点内存；丢失后由 Provider 重新获取。 - **Worker**：承载 Gateway 流量的数据面节点，仅使用本地快照选路。 - **Controller**：集中管理 Provider 获取、代理生命周期、Routing 状态、 所有权和快照发布的控制面节点。 - **Checker**：执行基础、出口和目标级健康检查的可扩缩执行节点。 - **Extraction**：Distribution API 对一个或一组 Proxy 的一次性独占发放。 -- **Extraction Record**：Extraction 的审计事实，不承担租约或释放语义。 +- **Activity Pool**：Redis 中带 TTL 的短效 Proxy 运行时集合，是分配、所有权 + 和独占提取的原子操作位置，可在丢失后由 Provider 重建。 +- **Extraction Result**：Redis 中短期保存的幂等响应，不承担租约、释放或 + 长期审计语义。 +- **Persistent Management State**：PostgreSQL 中的配置版本、Upstream/Routing + 管理状态、Admin 审计与 Outbox，以及可选聚合指标；不包含 Proxy 明细或逐次 + 提取记录。 ## 状态与计数 @@ -33,8 +41,10 @@ - **Gateway Allocation**：Worker 原子预留本地所有 Proxy 容量，建连成功后 转为 Active，结束后释放。 -- **Exclusive Extraction**：控制面原子把可提取 Proxy 从 `AVAILABLE` 改为 - `EXTRACTED`，成功后该 Proxy 永不再次被系统分配。 +- **Exclusive Extraction**：控制面通过 Redis 原子操作把可提取 Proxy 从 + `AVAILABLE` 改为不可再次分配，并在有限 TTL 内保存幂等结果；成功后该 + Proxy 在当前 TTL 生命周期内不再被系统分配。Redis 整体丢失后的 Provider + 重建属于新的活动池代次，不承诺延续已丢失代次的排他状态。 - **Drain**：停止新分配，等待 Reserved 与 Active 归零后转换状态或撤销所有权。 - **Empty Fetch**：不是错误、不是重复，而是有效 Provider 响应中没有任何 合法 Proxy 候选。 @@ -51,4 +61,3 @@ - **reserveForGateway**：共享池中不能被 Distribution 提取的最低可用数量。 - **fulfillment.partial**：尽量返回，允许少于请求数量。 - **fulfillment.allOrNothing**：不足时一个也不提取。 - diff --git a/api/openapi/proxy-pool.yaml b/api/openapi/proxy-pool.yaml index 3cd66ee..9b6b490 100644 --- a/api/openapi/proxy-pool.yaml +++ b/api/openapi/proxy-pool.yaml @@ -4,9 +4,10 @@ info: version: 1.0.0 description: | Distribution API performs one-time exclusive extraction. A successful - operation atomically transitions every returned proxy from AVAILABLE to - EXTRACTED. Extracted proxies are never allocated again and there is no - release, renew, or lease API. + Redis operation atomically removes or marks every returned proxy as + extracted in the TTL activity pool. Proxy details and per-extraction + records are not persisted in PostgreSQL. There is no release, renew, or + lease API. servers: - url: http://127.0.0.1:8081 description: Distribution API @@ -20,13 +21,14 @@ paths: operationId: extractProxies summary: 一次性独占提取代理 description: | - 服务端先完成筛选、行锁定、AVAILABLE -> EXTRACTED 状态更新和审计记录 - 写入，再返回代理。相同代理不会返回给两个成功请求。 + 服务端通过单个 Redis 原子操作完成筛选、TTL/所有权复核、Gateway 预留、 + 从可分配池移除所选条目和短期幂等结果写入，再返回代理。相同代理不会 + 返回给两个成功请求，提取路径不访问 PostgreSQL。 `partial` 允许实际返回数量小于请求数量；`allOrNothing` 数量不足时不 提取任何代理并返回 409。未传 `fulfillment` 时使用服务端配置，默认 - 为 `partial`。`Idempotency-Key` 可避免客户端因响应丢失重试而再次消耗 - 库存。 + 为 `partial`。`Idempotency-Key` 在 Redis 的有界 TTL 窗口内避免客户端因 + 响应丢失重试而再次消耗库存。 security: - ApiKeyAuth: [] - BasicAuth: [] @@ -57,7 +59,7 @@ paths: allowedUpstreams: [provider-a, provider-b] responses: '200': - description: 提取事务已提交；返回的代理已永久退出可分配池 + description: Redis 原子提取已提交；返回的代理已退出当前可分配池 headers: X-Request-ID: $ref: '#/components/headers/RequestID' @@ -117,7 +119,7 @@ paths: tags: [Health] operationId: getReadiness summary: Distribution 就绪探针 - description: PostgreSQL 不可用或权威状态不可写时返回 503。 + description: Redis TTL 活动池不可用或无法保证原子提取时返回 503。 security: [] responses: '200': @@ -375,7 +377,7 @@ components: schema: $ref: '#/components/schemas/Problem' ServiceUnavailable: - description: 权威存储不可用或服务正在排空 + description: Redis TTL 活动池不可用、原子提取不可执行或服务正在排空 headers: X-Request-ID: $ref: '#/components/headers/RequestID' diff --git a/api/proto/controlplane/v1/controlplane.proto b/api/proto/controlplane/v1/controlplane.proto index e9be4bf..f3804d8 100644 --- a/api/proto/controlplane/v1/controlplane.proto +++ b/api/proto/controlplane/v1/controlplane.proto @@ -2,7 +2,7 @@ syntax = "proto3"; package proxy_pool.controlplane.v1; -option go_package = "github.com/proxy-pool/proxy-pool/gen/controlplane/v1;controlplanev1"; +option go_package = "proxy-pool/gen/controlplane/v1;controlplanev1"; import "google/protobuf/duration.proto"; import "google/protobuf/empty.proto"; @@ -125,6 +125,7 @@ message OwnedProxy { uint32 max_concurrency = 10; map tags = 11; uint64 ownership_epoch = 12; + google.protobuf.Timestamp usable_until = 13; } enum ProxyProtocol { diff --git a/configs/proxy-pool.yaml b/configs/proxy-pool.yaml index c8d9600..19ff874 100644 --- a/configs/proxy-pool.yaml +++ b/configs/proxy-pool.yaml @@ -72,6 +72,7 @@ distribution: minRemainingTTL: 30s maxHealthCheckAge: 15s reserveForGateway: 100 + idempotencyTTL: 5m admin: enabled: true diff --git a/deploy/config/local.yaml b/deploy/config/local.yaml index d83b855..307f0f6 100644 --- a/deploy/config/local.yaml +++ b/deploy/config/local.yaml @@ -46,6 +46,7 @@ distribution: minRemainingTTL: 30s maxHealthCheckAge: 30s reserveForGateway: 1000 + idempotencyTTL: 5m admin: enabled: true diff --git a/deploy/kubernetes/base/configmap.yaml b/deploy/kubernetes/base/configmap.yaml index d017e3f..52f1329 100644 --- a/deploy/kubernetes/base/configmap.yaml +++ b/deploy/kubernetes/base/configmap.yaml @@ -50,6 +50,7 @@ data: minRemainingTTL: 30s maxHealthCheckAge: 30s reserveForGateway: 5000 + idempotencyTTL: 5m admin: enabled: true listen: 0.0.0.0:8082 diff --git a/deploy/tools/configcheck/main.go b/deploy/tools/configcheck/main.go index 0d310cf..4601770 100644 --- a/deploy/tools/configcheck/main.go +++ b/deploy/tools/configcheck/main.go @@ -5,7 +5,7 @@ import ( "fmt" "os" - "github.com/proxy-pool/proxy-pool/internal/config" + "proxy-pool/internal/config" ) func main() { diff --git a/diagrams/README.md b/diagrams/README.md index ee21838..3ac36e4 100644 --- a/diagrams/README.md +++ b/diagrams/README.md @@ -245,16 +245,16 @@ flowchart TB ```mermaid sequenceDiagram - participant DB as PostgreSQL + participant R as Redis Activity Pool participant C as Controller participant W as Worker - DB->>C: outbox revision 42 + R->>C: activity or ownership revision 42 C->>C: build worker snapshot C->>W: epoch 8 version 42 checksum W->>W: validate and build indexes W->>W: atomic swap W-->>C: ACK epoch 8 version 42 - C->>DB: mark outbox delivered + C->>R: record worker ACK with TTL ``` ## 17 Snapshot 缺口恢复 @@ -316,12 +316,10 @@ flowchart LR sequenceDiagram participant C as Client participant API as Distribution - participant DB as PostgreSQL + participant R as Redis Activity Pool C->>API: count 10 fulfillment partial - API->>DB: lock eligible rows - DB-->>API: 6 rows - API->>DB: update 6 to EXTRACTED and audit - DB-->>API: commit + API->>R: atomic extract with filters and reserve + R-->>API: mark 6 EXTRACTED and return them API-->>C: requested 10 returned 6 ``` @@ -331,11 +329,10 @@ sequenceDiagram sequenceDiagram participant C as Client participant API as Distribution - participant DB as PostgreSQL + participant R as Redis Activity Pool C->>API: count 10 fulfillment allOrNothing - API->>DB: lock eligible rows - DB-->>API: only 6 rows - API->>DB: rollback entire transaction + API->>R: atomic extract with filters and reserve + R-->>API: insufficient inventory and no mutation API-->>C: insufficient inventory and returned 0 ``` @@ -345,13 +342,12 @@ sequenceDiagram sequenceDiagram participant C as Controller participant W as Gateway Worker - participant DB as PostgreSQL + participant R as Redis Activity Pool C->>W: mark Proxy DRAINING at epoch 13 W->>W: stop new allocations W-->>C: ACK active 0 reserved 0 - C->>DB: clear worker ownership - C->>DB: AVAILABLE to EXTRACTED plus audit - DB-->>C: committed exclusive result + C->>R: atomically clear ownership and extract + R-->>C: committed exclusive result ``` ## 24 reserveForGateway 不变量 @@ -367,18 +363,16 @@ flowchart TD Result -->|allOrNothing insufficient| Rollback[Return zero] ``` -## 25 PostgreSQL 独占事务 +## 25 Redis 独占原子操作 ```mermaid flowchart TD - Begin[BEGIN] --> Select[SELECT eligible FOR UPDATE SKIP LOCKED] + Begin[Lua script or Redis Function] --> Select[Filter eligible TTL entries] Select --> Enough{Quantity satisfies mode?} - Enough -->|no allOrNothing| Rollback[ROLLBACK] - Enough -->|yes or partial| Update[UPDATE AVAILABLE to EXTRACTED] - Update --> Audit[INSERT extraction records] - Audit --> Outbox[INSERT outbox] - Outbox --> Commit[COMMIT] - Commit --> Return[Return proxies with expiry] + Enough -->|no allOrNothing| Noop[Return zero without mutation] + Enough -->|yes or partial| Update[Mark selected entries EXTRACTED] + Update --> Idempotency[Optionally cache bounded idempotency result] + Idempotency --> Return[Return proxies with expiry] ``` ## 26 并发提取互斥 @@ -386,17 +380,16 @@ flowchart TD ```mermaid sequenceDiagram participant A as Extract request A - participant DB as PostgreSQL + participant R as Redis Activity Pool participant B as Extract request B - A->>DB: lock rows 1 to 10 - B->>DB: skip locked rows 1 to 10 - B->>DB: lock rows 11 to 20 - A->>DB: commit EXTRACTED 1 to 10 - B->>DB: commit EXTRACTED 11 to 20 + A->>R: execute atomic extraction + R-->>A: commit EXTRACTED 1 to 10 + B->>R: execute atomic extraction + R-->>B: commit EXTRACTED 11 to 20 Note over A,B: no Proxy returned twice ``` -## 27 Outbox 一致性 +## 27 管理面 Outbox 一致性 ```mermaid sequenceDiagram @@ -404,7 +397,7 @@ sequenceDiagram participant DB as PostgreSQL participant P as Publisher participant W as Worker - C->>DB: state change plus outbox + C->>DB: management state change plus outbox DB-->>C: atomic commit P->>DB: read undelivered event P->>W: publish versioned event @@ -533,4 +526,3 @@ flowchart TD Failure --> Verify[Verify SLO and all invariants] Verify --> Evidence[Archive raw metrics config and image digest] ``` - diff --git a/docs/adr/README.md b/docs/adr/README.md index 9df4793..e01f9d6 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -12,9 +12,9 @@ Gateway 只依赖本地不可变 Snapshot；Provider、数据库、配置重载 **状态：** 接受并覆盖早期 Lease 方案。 -成功响应前在一个事务中执行 `AVAILABLE -> EXTRACTED`。系统保存审计事实， -但不提供 release、renew 或使用跟踪。这样契合“拿走真实代理后平台不再管理” -的最终产品语义，并消除重复发放。 +成功响应前，通过 Redis 原子操作完成候选选择、`AVAILABLE -> EXTRACTED` 和 +短期幂等结果写入；不提供 release、renew 或使用跟踪。这样契合“拿走真实 +代理后平台不再管理”的最终产品语义，并消除并发重复发放。 ## ADR-003：单 Worker 所有权 @@ -24,11 +24,14 @@ Gateway 只依赖本地不可变 Snapshot；Provider、数据库、配置重载 Distribution 只提取无所有权 Proxy；回收时执行 drain/ACK/归零/解除所有权。 该选择避免每请求访问 Redis 做全局并发计数。 -## ADR-004：PostgreSQL 权威、Redis 可重建 +## ADR-004：PostgreSQL 保存控制面事实，Redis 承载短效活动池 **状态：** 接受。 -Proxy 状态、Extraction、配置版本和 Outbox 由 PostgreSQL 持久化。Redis 只 -承担 Leader、短期速率与心跳等协调；Redis 丢失后可从权威状态恢复，避免 -双写状态成为不可判定的事实源。 +Proxy 明细不进入 PostgreSQL。短效 Proxy 及其状态、所有权和过期时间只存在于 +带 TTL 的 Redis 活动池及节点内存；Redis 同时承担独占提取、短期幂等结果、 +Provider Leader、分布式速率和 Worker 心跳。Redis 丢失时活动池作废，由 +Provider 重新获取并重建，不从 PostgreSQL 恢复原 Proxy。 +PostgreSQL 只持久化配置版本、Upstream/Routing 管理状态、Admin 审计与 Outbox， +以及可选的无 Proxy 明细聚合指标。两类存储不双写 Proxy，也不建立跨存储事务。 diff --git a/docs/api/control-plane.md b/docs/api/control-plane.md index c02d5f3..3a98c4f 100644 --- a/docs/api/control-plane.md +++ b/docs/api/control-plane.md @@ -10,7 +10,7 @@ Proto 源文件位于 `api/proto/controlplane/v1/controlplane.proto`，包含两 - `CheckerControlPlane`：健康检查任务流和 Observation 批量上报。 该协议不承载 Client 的独占提取，也没有 extraction lease/release。Proxy 的 -`AVAILABLE -> EXTRACTED` 只在 Controller 权威事务中完成。 +`AVAILABLE -> EXTRACTED` 只在 Controller 调用的 Redis 原子操作中完成。 ## 2. Worker 会话 @@ -40,9 +40,14 @@ sequenceDiagram - 单调 `version`、`ownership_epoch`、生成时间和有效期。 - 对该 Worker 可见的有序 Routing。 -- 仅归该 Worker 所有的 Proxy 与每个 Proxy 的容量。 +- 仅归该 Worker 所有的 Proxy、每个 Proxy 的容量、硬过期时间 `expires_at` + 和停止新分配的 `usable_until`。 - 内容 `checksum`。 +`usable_until = expires_at - allocationSafetyMargin`。Worker 必须以 +`usable_until` 作为最后可分配时刻；达到该时间后即使尚未到 `expires_at`， +也不得再为新请求选择该 Proxy。 + Delta 声明 `base_version`。Worker 只有在本地版本恰好等于 base 且 checksum 验证成功时才能应用；否则丢弃 Delta 并请求完整 Snapshot。构建在后台完成， 热路径只读取一次原子指针。 @@ -57,11 +62,11 @@ Delta 声明 `base_version`。Worker 只有在本地版本恰好等于 base 且 1. 新 Snapshot 标记或移除该 Proxy，使 Worker 停止新预留。 2. Worker 上报 `draining=true` 以及 Active/Reserved。 3. 两个计数都归零后 Controller 清除所有权。 -4. 无所有权 Proxy 才能进入 Distribution 提取事务。 +4. 无所有权 Proxy 才能进入 Distribution 的 Redis 原子提取操作。 Worker 崩溃时必须等待所有权 epoch/有效期失效后再转移，避免双主。Proto 中 `ReportRuntimeResponse.revoke_proxy_ids` 是加速 Drain 的控制信号，不绕过 -Snapshot 版本和权威持久化。 +Snapshot 版本和 Redis 中的权威所有权状态。 ## 5. Outcome 上报 @@ -73,8 +78,9 @@ Outcome 按 Worker 单调 `sequence` 批量上报。Controller 返回已接受 - `RESPONSE_HEADERS`：目标响应头前失败。 - `TUNNEL`：隧道建立后结束或失败。 -Outcome 是 Observation，不直接让 Worker 修改 PostgreSQL 状态。异步上报队列 -必须有界；队列满时丢弃低价值样本并计指标，不能反压 Gateway 热路径。 +Outcome 是 Observation，不直接让 Worker 修改 Redis 活动池状态，也不创建 +PostgreSQL Proxy 明细。异步上报队列必须有界；队列满时丢弃低价值样本并计 +指标，不能反压 Gateway 热路径。 ## 6. Checker 任务 @@ -85,7 +91,8 @@ Checker 注册自身最大并发与支持层级，Controller 发送有 deadline - `TARGET`：针对 Routing/目标组的可达性。 Checker 只返回 `HealthObservation`。Controller reducer 按 Proxy、检查层级和 -Routing 决定 AVAILABLE、SUSPECT 或 UNHEALTHY，避免多个 Checker 并发写状态。 +Routing 决定 AVAILABLE、SUSPECT 或 UNHEALTHY，并更新 Redis 活动池，避免 +多个 Checker 并发写状态。 ## 7. 兼容与演进 diff --git a/docs/api/distribution.md b/docs/api/distribution.md index 10f4d3b..18e83ce 100644 --- a/docs/api/distribution.md +++ b/docs/api/distribution.md @@ -5,18 +5,17 @@ Distribution API 只提供一次性独占提取： ```text -筛选 AVAILABLE - -> 锁定候选 - -> 校验 Gateway 预留、TTL、健康与过滤条件 - -> 原子 AVAILABLE -> EXTRACTED - -> 写 Extraction Record - -> 提交事务 +读取 Redis TTL 活动池中的 AVAILABLE 候选 + -> 校验 Gateway 预留、TTL、健康、所有权与过滤条件 + -> Redis 原子移出可分配池并写入短期幂等结果 -> 返回真实代理地址 ``` -事务提交前不得把地址写给 Client。返回成功后，该 Proxy 不再参与 Gateway、 -再次提取或可用库存统计。系统不跟踪 Client 是否使用、使用并发或何时停止， -也不提供 release、renew、status 或租约端点。 +Redis 原子操作成功前不得把地址写给 Client。返回成功后，该 Proxy 在当前 TTL +生命周期中不再参与 Gateway、再次提取或可用库存统计。系统不跟踪 Client 是否 +使用、使用并发或何时停止，也不提供 release、renew、status 或租约端点。短效 +Proxy 明细只存在于 Redis TTL 活动池和节点内存；PostgreSQL 不保存 Proxy 明细 +或逐次提取记录。 HTTP 契约源文件：`api/openapi/proxy-pool.yaml`。 @@ -105,7 +104,7 @@ Request ID，不记录地址或凭据。 ### 4.1 partial -锁定的符合条件数量少于 `count` 时，提交实际数量： +符合条件数量少于 `count` 时，原子提交实际数量： ```text requested=10, eligible=6, reserve=0 -> returned=6 @@ -126,13 +125,14 @@ requested=10, eligible=6, reserve=0 -> returned=6 } ``` -冲突响应后，之前被该请求临时锁定的 Proxy 仍为 AVAILABLE。 +冲突响应后，候选 Proxy 仍在 Redis TTL 活动池中保持 AVAILABLE。 ## 5. 资格过滤与 Gateway 预留 候选必须同时满足： -1. 权威状态是 AVAILABLE。 +1. Redis TTL 活动池条目仍为 AVAILABLE，且当前时间早于按供应商安全余量计算的 + `usableUntil`。 2. 没有 Worker 所有权，或已完成 Drain 且 Active/Reserved 均为零。 3. 剩余 TTL 不低于 `minRemainingTTL`。 4. 最近健康检查不早于 `maxHealthCheckAge`。 @@ -140,29 +140,33 @@ requested=10, eligible=6, reserve=0 -> returned=6 6. 提取后符合条件的共享库存不低于 `reserveForGateway`。 Worker-owned Proxy 不得直接提取。Controller 需要先发布 DRAINING，等待 Worker -确认没有 Active/Reserved，再清除所有权并进入提取事务。快照延迟时仍禁止 -Gateway 与 Client 同时获得同一 Proxy。 +确认没有 Active/Reserved，再清除 Redis 中的所有权并进入原子提取。快照延迟时 +仍禁止 Gateway 与 Client 同时获得同一 Proxy。 -## 6. 并发与事务 +## 6. 并发与原子性 -PostgreSQL Adapter 应在一个事务内使用 `FOR UPDATE SKIP LOCKED` 获取候选， -更新状态并写审计记录。所有状态更新必须包含 `state = 'AVAILABLE'` 前置条件。 +Redis Adapter 应通过单个 Lua 脚本、Redis Function 或等价的原子原语完成候选 +筛选、TTL/健康/所有权复核、Gateway 预留计算、满足模式判断、从可分配池移除 +所选条目，并写入短期幂等结果。提取过程中不访问 PostgreSQL。 关键不变量： - 两个并发成功响应的 Proxy ID 集合交集为空。 -- 状态更新或审计写入任一步失败，整个批次不返回。 -- `allOrNothing` 不足时零行变为 EXTRACTED。 -- PostgreSQL 不可写时返回 503，不以内存结果冒充成功。 +- 原子操作失败时整个批次不返回，也不得留下部分移除结果。 +- `allOrNothing` 不足时零个条目退出活动池。 +- Redis 活动池不可用时返回 503，不以内存副本冒充成功。 +- PostgreSQL 不可用不阻断提取；需要 PostgreSQL 的 Admin 管理写入单独降级。 ## 7. 幂等 提取是消耗库存的写操作。客户端收到超时后盲目重试可能再次提取一批不同 Proxy，因此自动重试应提供稳定的 `Idempotency-Key`。 -服务端幂等记录至少包含：Client ID、Key、请求体摘要、提交结果和过期时间。 -同一 Client、同一 Key、相同摘要返回首次结果；摘要不同返回 409。幂等记录和 -Extraction Record 必须与状态更新处在相同事务边界或由同一权威恢复流程保证。 +服务端在 Redis 中保存有界 TTL 的幂等记录，至少包含 Client ID、Key、请求体 +摘要、提交结果和过期时间。同一 Client、同一 Key、相同摘要返回首次结果； +摘要不同返回 409。幂等结果必须和活动池状态变更处于同一个 Redis 原子操作。 +TTL 到期或 Redis 数据丢失后不再保证旧 Key 去重，系统不回退到 PostgreSQL +保存逐次提取事实。 ## 8. 认证、识别与限制 @@ -186,28 +190,30 @@ Extraction Record 必须与状态更新处在相同事务边界或由同一权 - `415`：请求体不是 `application/json`。 - `422`：数量、枚举或过滤组合违反业务约束。 - `429`：全局或 Client 速率限制，响应 `Retry-After`。 -- `503`：PostgreSQL 不可写、服务排空或权威状态不可用。 +- `503`：Redis 活动池不可用、原子提取不可执行或服务正在排空。 - `500`：未分类的内部错误；响应不包含底层错误文本。 错误响应不得包含 Provider Secret、Proxy 凭据、SQL 或内部拓扑。 -## 10. 审计记录 +## 10. 短期运行记录与数据最小化 -每个被提交的 Proxy 对应一条 Extraction Record： +Redis 幂等结果只在配置的 TTL 窗口内保留重放响应所需的数据： ```text -proxyId, clientId, sourceIP, requestId, upstream, extractedAt, expiresAt +clientId, idempotencyKey, requestDigest, response, expiresAt ``` -无认证时 `clientId` 使用可信代理链解析后的规范化来源 IP 稳定标识，并保留 -`sourceIP`。记录只用于审计、排错和计费事实，不承担资源归还语义。Proxy -到期后可以清理运行记录，但 Extraction Record 按审计保留策略归档。 +无认证时 `clientId` 使用可信代理链解析后的规范化来源 IP 稳定标识。应用日志 +仅记录 Request ID、数量、结果码和过滤摘要，不记录 Proxy 地址或凭据。需要长期 +分析时只向 PostgreSQL 写入不含 Proxy 明细的可选聚合指标；系统不创建或归档 +逐个 Proxy 的 Extraction Record。 ## 11. 运行时实现边界 `distribution.Handler` 是薄 HTTP Adapter，只负责严格解码、Header/DTO 校验、 身份结果注入、错误映射和健康探针。独占提取、TTL、Gateway 预留及幂等事务 -继续由 `extraction.Service` 和持久化 Store 承担。 +继续由 `extraction.Service` 和 Redis 活动池 Adapter 承担。当前内存 Store 仅作为 +原子行为的参考实现与测试替身，不是生产持久化层。 请求体解码、Request ID 与 Problem JSON 统一复用 `platform/httpapi`。认证、 可信代理、来源控制、Client ID 和准入限流由必需的 `IdentityResolver` 注入， diff --git a/docs/configuration/reference.md b/docs/configuration/reference.md index aa60450..13c0b5a 100644 --- a/docs/configuration/reference.md +++ b/docs/configuration/reference.md @@ -145,20 +145,25 @@ distribution: minRemainingTTL: 30s maxHealthCheckAge: 15s reserveForGateway: 5 + idempotencyTTL: 5m ``` Extraction 是固定的一次性独占行为，**没有** `mode`、`leaseDuration`、 -`release` 或 `renew` 配置。事务提交后执行 `AVAILABLE -> EXTRACTED`，该 Proxy -不再由 Gateway 或 Distribution 分配。 +`release` 或 `renew` 配置。Redis 在一个原子操作内选择候选、执行 +`AVAILABLE -> EXTRACTED` 并短期保存幂等结果；成功后该 Proxy 不再由 Gateway +或 Distribution 在当前 TTL 生命周期中分配。 - `fulfillment`：`partial` 或 `allOrNothing`，默认语义为 `partial`。 - `maxCountPerRequest`：单次请求硬上限。 - `minRemainingTTL`：剩余寿命低于此值时不参与提取。 - `maxHealthCheckAge`：最近检查早于此窗口时不参与提取。 - `reserveForGateway`：提取后必须留给 Gateway 的最低符合条件库存数量。 +- `idempotencyTTL`：Redis 幂等结果的最长保留时间；省略时为 5 分钟，实际 + 保留时间不会超过本次返回代理中最早的 `expiresAt`。 -`partial` 会提交实际可得数量；`allOrNothing` 数量不足时事务回滚，一个也不 -提取。认证关闭时仍应使用 `sourceIP` 识别匿名 Client 并执行全局/来源限流。 +`partial` 会原子提取实际可得数量；`allOrNothing` 数量不足时不改变任何候选， +一个也不提取。认证关闭时仍应使用 `sourceIP` 识别匿名 Client 并执行全局/ +来源限流。 `clientIdentification.mode` 支持： @@ -326,9 +331,14 @@ storage: redisURL: "${REDIS_URL}" ``` -PostgreSQL 是 Proxy 生命周期、Routing 选择、Worker 所有权和 Extraction Record -的权威存储。Redis 只承载可重建的短期协调状态，不能成为独占提取的唯一事实 -来源。Metrics 标签禁止 Proxy IP、Client ID、Session、完整 URL 和 Request ID。 +Proxy 明细不写入 PostgreSQL。Redis 是短效 Proxy 活动池的运行时事实源，使用 +TTL 保存 Proxy 状态、所有权和过期时间，并原子执行独占提取及短期幂等结果 +读写；Redis 丢失后由 Provider 重新补池，不从 PostgreSQL 恢复原 Proxy。 + +PostgreSQL 只保存配置版本、Upstream/Routing 管理状态、Admin 审计与 Outbox， +以及可选的无 Proxy 明细聚合指标。Distribution 提取不依赖 PostgreSQL，因而 +PostgreSQL 故障本身不应使 Redis 中可完成的 Extract 返回 `503`。Metrics 标签 +禁止 Proxy IP、Client ID、Session、完整 URL 和 Request ID。 ## 9. 启动前校验清单 diff --git a/docs/design/architecture.md b/docs/design/architecture.md index c71bac2..39a174c 100644 --- a/docs/design/architecture.md +++ b/docs/design/architecture.md @@ -67,7 +67,7 @@ Provider、Pool、Routing、Distribution 在首版需要共享事务和一致性 - Routing Sequential 当前选择和原子切换。 - Worker 所有权/容量切片、Snapshot 发布、ACK 和重同步。 - Distribution 与 Admin HTTP 接口。 -- PostgreSQL 权威状态和 Redis 短期协调。 +- PostgreSQL 管理面持久化和 Redis TTL 活动池。 ### 3.3 proxy-checker @@ -88,7 +88,8 @@ domain/ proxy Proxy、状态机、TTL、唯一键、容量 routing 规则、匹配、策略与 Sequential 状态 upstream Provider 能力、Fetch 分类与 Pool 需求 - extraction 独占提取命令、结果和审计事实 + activitypool 短效 Proxy、TTL、所有权和独占消费 + extraction 独占提取命令与结果 client 认证主体、权限与限制 gateway/ @@ -102,7 +103,7 @@ controller/ pool Pool reconcile、生命周期和所有权 health 任务计划和 Observation reducer routing Sequential 原子切换和恢复 - extraction 批量原子提取与审计 + extraction Redis 活动池批量原子提取与短期幂等 distribution Worker 注册、Snapshot/Delta 与结果上报 adapters/ @@ -126,6 +127,7 @@ type Proxy struct { SourceUpstream UpstreamID CreatedAt time.Time ExpiresAt *time.Time + UsableUntil *time.Time LastCheckedAt *time.Time LastSuccessAt *time.Time Latency time.Duration @@ -136,6 +138,8 @@ type Proxy struct { ``` 密码不参与日志可见唯一键；`CredentialVersion` 区分同一用户名的凭据轮换。 +唯一键不包含 Upstream：多个供应商返回同一唯一键时，当前生命周期的首个来源 +保持归属，其他来源只计重复且不得覆盖 TTL；原来源硬过期淘汰后可重新归属。 运行态 `active` 与 `reserved` 存在 Worker 本地、按 Proxy ID 分片，不写入 不可变 Snapshot。 @@ -279,27 +283,26 @@ sequenceDiagram participant API as Distribution API participant CP as Controller participant W as Gateway Worker - participant DB as PostgreSQL + participant R as Redis Activity Pool C->>API: POST extract(count, filters) API->>API: auth + access + limits API->>CP: Extract command - CP->>DB: select eligible unowned proxies + CP->>R: atomic select eligible unowned proxies alt candidate owned by Worker CP->>W: revoke and drain W-->>CP: active=0, reserved=0, ownership released end - CP->>DB: transaction AVAILABLE -> EXTRACTED - DB->>DB: insert extraction records - DB-->>CP: committed rows + CP->>R: AVAILABLE -> EXTRACTED + bounded idempotency result + R-->>CP: consumed proxies CP-->>API: proxy + expiry API-->>C: requested/returned/proxies ``` ### 10.2 原子批量提取 -PostgreSQL Adapter 使用事务和 `FOR UPDATE SKIP LOCKED` 选择满足以下条件的 -行，并在同一事务更新状态与写审计记录： +Redis Adapter 使用单个 Lua 脚本或等价原子命令，选择满足以下条件的活动池 +条目，并同时完成状态迁移和带 TTL 的幂等结果写入： - State 为 AVAILABLE。 - 未分配 Worker 所有权，Reserved/Active 为 0。 @@ -308,7 +311,8 @@ PostgreSQL Adapter 使用事务和 `FOR UPDATE SKIP LOCKED` 选择满足以下 - 提取后仍保留 `reserveForGateway`。 - 符合 protocol、region、carrier 与 allowedUpstreams。 -`partial` 提交实际可得数量；`allOrNothing` 在锁定数量不足时回滚。 +`partial` 原子消费实际可得数量；`allOrNothing` 数量不足时不改变任何条目。 +Proxy 地址、凭据和逐次提取明细不写 PostgreSQL。 ### 10.3 与 Gateway 共池 @@ -332,7 +336,9 @@ flowchart LR - Worker 仅对自身 Proxy 做本地原子计数。 - 所有权租约有 epoch 与过期时间；Worker 失联后先等待租约过期，再分配给 其他 Worker，避免双主。 -- Snapshot 包含 Worker 专属 Proxy 集、Routing、策略和版本校验和。 +- Snapshot 包含 Worker 专属 Proxy 集、Routing、策略、`ExpiresAt`、 + `UsableUntil` 和版本校验和。Worker 在 `UsableUntil` 到达后立即停止新分配， + 不等待供应商硬过期时间 `ExpiresAt`。 - Worker 断开控制面后在 `maxStaleAge` 内使用最后快照；超限停止接收新流量， 已有隧道排空。 @@ -340,18 +346,22 @@ flowchart LR ### 12.1 PostgreSQL -权威保存：配置版本、Upstream、Proxy 生命周期、Routing 当前选择、Worker -所有权、Extraction Record、Client、审计与 outbox。 +持久化配置版本、Upstream/Routing 管理状态、Admin 审计和管理事件 outbox， +以及可选的不含 Proxy 明细的聚合指标。PostgreSQL 不保存 Proxy 地址、凭据、 +生命周期、Worker 所有权或逐次提取记录。 ### 12.2 Redis -保存可重建短期状态：Provider Leader 租约、分布式限流、singleflight 信号、 -短期 Client 限流和 Worker 心跳。Redis 不保存唯一权威业务状态。 +保存带 TTL 的短效 Proxy 活动池、生命周期、Worker 所有权、独占提取状态和 +短期幂等结果，并承载 Provider Leader、分布式限流、singleflight 信号、 +短期 Client 限流和 Worker 心跳。Redis 丢失后由 Provider 重新获取代理重建， +不会从 PostgreSQL 恢复旧 Proxy。 ### 12.3 Outbox -任何需要发布 Snapshot/事件的 PostgreSQL 状态更新同时写 outbox。发布成功 -后标记；Worker 使用 epoch/version 幂等应用，缺口触发完整 Snapshot。 +管理状态变更在 PostgreSQL 事务内同时写 outbox。活动池变更通过 Redis 事件流 +通知 Controller 构建 Snapshot，不跨 PostgreSQL/Redis 双写 Proxy；Worker 使用 +epoch/version 幂等应用，缺口时从当前活动池构建完整 Snapshot。 ## 13. 健康模型 @@ -404,7 +414,7 @@ flowchart LR - Routing 当前 Upstream、切换、无可用和策略选择。 - Upstream Proxy 状态、Available Slots、Active、Reserved、Pending Fetch。 - Provider Fetch 成功、Empty、Duplicate、Error、429、退避和耗时。 -- Extraction requested、returned、insufficient、atomic conflict 和审计失败。 +- Extraction requested、returned、insufficient、atomic conflict 和幂等缓存失败。 - Snapshot epoch/version、陈旧时长、应用耗时和重同步。 Prometheus 标签不包含 Proxy IP、Client ID、session、完整 URL 或 request ID。 @@ -443,8 +453,8 @@ CPU、内存、网络、Go 版本、配置和上游响应模型下测得。 |---|---| | Provider 超时/500 | 计 Error、退避；不计 Empty，不影响已有 Proxy | | Provider 合法空响应 | Empty++；达到阈值触发相关 Routing 原子切换 | -| Redis 不可用 | Gateway 不受影响；Controller 使用本地退化并停止高风险写操作 | -| PostgreSQL 不可用 | Gateway 使用最后快照；停止 Extract 和权威状态变更 | +| Redis 不可用 | Gateway 暂用未过期快照；停止 Fetch 入池、Extract 和所有权变更 | +| PostgreSQL 不可用 | Gateway 与 Redis Extract 不受影响；停止管理状态变更和 Admin 审计 | | Controller 断线 | Worker 在 maxStaleAge 内继续；超限拒绝新流量并排空 | | Worker 崩溃 | 所有权租约过期后重新分配；期间不双重所有 | | Checker 积压 | 降低普通复检频率，优先新 Proxy 与 SUSPECT，不无限排队 | diff --git a/docs/design/product-design.md b/docs/design/product-design.md index d89ad64..bf7863b 100644 --- a/docs/design/product-design.md +++ b/docs/design/product-design.md @@ -16,7 +16,7 @@ Proxy Pool 把多个供应商的动态代理统一成一个可运营的资源系 - **业务调用方**：通过稳定入口使用代理，不感知供应商差异。 - **代理直提调用方**：按协议、区域、运营商或 Upstream 条件独占提取。 - **平台管理员**：管理 Provider、Routing、容量、健康和故障切换。 -- **SRE**：依据低基数指标、审计记录和运行手册进行容量与故障管理。 +- **SRE**：依据低基数指标、管理面审计和运行手册进行容量与故障管理。 ## 3. 核心价值 @@ -30,11 +30,11 @@ Provider Adapter 负责请求格式、认证和响应解析。标准化后，Rou Gateway Worker 只读取本地不可变快照并维护本地容量计数。供应商延迟、 数据库抖动和控制面重载不会成为每请求依赖。 -### 3.3 明确且可审计的资源语义 +### 3.3 明确的资源语义 -系统区分“短时使用容量”和“一次性独占提取”。每次 Extraction 保存请求、 -调用方、来源、Upstream、提取时间和过期时间审计事实，但不追踪提取后的 -实际使用，也不存在归还接口。 +系统区分“短时使用容量”和“一次性独占提取”。Extraction 只在 Redis 中短期 +保存有界幂等结果，不向 PostgreSQL 写逐个 Proxy 的提取事实；系统不追踪 +提取后的实际使用，也不存在归还接口。 ## 4. 主要流程 @@ -51,7 +51,8 @@ Gateway Worker 只读取本地不可变快照并维护本地容量计数。供 1. 调用方提交数量、过滤条件和 fulfillment。 2. Controller 校验调用方限额、TTL、健康新鲜度及 Gateway 保留量。 -3. 在一个数据库事务中锁定候选并执行 `AVAILABLE -> EXTRACTED`。 +3. 在一个 Redis Lua 脚本、Redis Function 或等价原子操作中筛选候选并执行 + `AVAILABLE -> EXTRACTED`。 4. `partial` 尽量返回；`allOrNothing` 数量不足时零提取。 5. 响应返回代理 URL、`expiresAt` 和 `remainingTtlSeconds`。 @@ -88,6 +89,6 @@ required_workers = ceil(peak_qps / (measured_worker_qps * target_utilization)) ## 7. 首版范围 首版包含 HTTP 正向代理、HTTPS CONNECT、REST Distribution/Admin、Provider -适配、健康与生命周期、Sequential 等路由策略、PostgreSQL 权威状态、Redis -可重建协调和集群快照。SOCKS5、跨地域主动主动和高级成本优化保留扩展边界。 - +适配、健康与生命周期、Sequential 等路由策略、PostgreSQL 管理面持久化、 +Redis TTL 活动池和集群快照。SOCKS5、跨地域主动主动和高级成本优化保留扩展 +边界。 diff --git a/docs/design/project-structure.md b/docs/design/project-structure.md index a2b1275..e80c569 100644 --- a/docs/design/project-structure.md +++ b/docs/design/project-structure.md @@ -37,8 +37,8 @@ proxy-pool/ ### proxy-controller Controller 是首版模块化单体。Provider、Pool、Routing 和 Extraction 共享 -事务边界和状态演进，避免过早拆成分布式事务。对外端口定义在领域或控制器 -模块，具体 PostgreSQL/Redis/HTTP 实现在 `adapters`。 +运行时策略和状态演进，但不建立 PostgreSQL/Redis 跨存储事务。对外端口定义 +在领域或控制器模块，具体 PostgreSQL/Redis/HTTP 实现在 `adapters`。 ### proxy-checker @@ -71,20 +71,28 @@ platform -------------------------> standard library / observability SDK ## 4. 数据所有权 -- PostgreSQL：Proxy 生命周期、Extraction 审计、配置版本和 Outbox 的权威源。 -- Redis：Leader、分布式速率、Worker 心跳等可丢失且可重建状态。 +- PostgreSQL：配置版本、Upstream/Routing 管理状态、Admin 审计与 Outbox， + 以及可选的无 Proxy 明细聚合指标。 +- Redis：带 TTL 的短效 Proxy 活动池及其状态、所有权和过期时间；同时保存 + Provider Leader、分布式速率、Worker 心跳和短期提取幂等结果。 - Worker：仅拥有分配给自己的 Proxy 本地容量计数和不可变快照。 - Controller：拥有 Provider 调度、Routing 运行态与 Worker 所有权编排。 - Checker：不拥有 Proxy 状态，只拥有执行中的检查任务。 +Proxy 明细不进入 PostgreSQL。Redis 活动池丢失后由 Provider 重新获取并重建， +节点内存中的旧快照随版本或有效期失效，不能把 PostgreSQL 当成恢复来源。 + ## 5. 一致性边界 -- Extraction 使用 PostgreSQL 单事务和行锁跳过锁定候选，提交后才响应。 +- Extraction 在 Redis 内原子完成候选筛选、排他状态迁移和短期幂等结果写入， + 成功后才响应；该路径不访问 PostgreSQL。 - Worker 所有权采用 `worker + epoch + version + expiry`，同一 Proxy 至多归属 一个 Worker。 - 从 Worker 回收 Proxy 时先 Drain，等待 ACK 且 Active/Reserved 为零，再 解除所有权；只有无所有权 Proxy 能被 Distribution 提取。 - Snapshot 为整代不可变对象，通过校验和和严格版本序列原子替换。 +- PostgreSQL 故障会影响管理状态变更和 Admin 审计，但不应阻断 Redis 中可完成 + 的独占提取；Redis 故障则使活动池暂不可用，并触发 Provider 重建。 ## 6. 扩展规则 diff --git a/docs/development/implementation-plan.md b/docs/development/implementation-plan.md index 0809bfc..5ce4e30 100644 --- a/docs/development/implementation-plan.md +++ b/docs/development/implementation-plan.md @@ -12,7 +12,9 @@ semantics in `对话内容.md`. **Architecture:** Separate Gateway, Controller, Checker, and Loadgen commands. Domain packages remain transport-free. Gateway reads immutable local snapshots. Controller owns provider fetch, pool lifecycle, routing state, extraction, persistence, and worker -distribution. PostgreSQL is authoritative; Redis stores rebuildable coordination state. +distribution. PostgreSQL persists management state and Admin audit/outbox. Redis owns +the rebuildable TTL Proxy activity pool and atomic extraction; Gateway hot paths remain +memory-only. **Tech Stack:** Go 1.26, `go.yaml.in/yaml/v4`, pgx/v5, go-redis/v9, gRPC/Protobuf, Prometheus, PostgreSQL, Redis, Docker Compose, Kubernetes. @@ -124,7 +126,8 @@ test/{fixtures,integration,e2e,load}/ - [x] Implement POST extraction command with protocol/region/carrier/upstream filters. - [x] Enforce minRemainingTTL, maxHealthCheckAge, maxCount, client limits, and reserveForGateway. -- [x] Atomically transition AVAILABLE to EXTRACTED and append audit records. +- [x] Atomically remove selected AVAILABLE entries from the allocatable set and return + the result; the current memory Store models the production Redis atomic boundary. - [x] Implement partial and allOrNothing without Lease, release, or renewal concepts. - [x] Run 1,000 concurrent claim attempts and prove every Proxy ID appears at most once. @@ -155,11 +158,15 @@ test/{fixtures,integration,e2e,load}/ **Files:** `internal/controller/distribution/*.go`, `admin/*.go`, `internal/adapters/postgres/*.go`, `internal/adapters/redis/*.go`, migrations, tests -- [ ] Define repository ports for Proxy, RoutingRuntime, Ownership, ExtractionRecord, - Client, ConfigVersion, and Outbox. -- [ ] Implement PostgreSQL extraction with one transaction and `FOR UPDATE SKIP LOCKED`. -- [ ] Implement Redis coordination for Provider leader, distributed rate, Client limit, - and Worker heartbeat; keep all state rebuildable. +- [ ] Define PostgreSQL ports for ConfigVersion, Upstream/Routing management state, + AdminAudit, Outbox, and optional aggregate metrics; never persist Proxy details or + per-extraction records. +- [ ] Implement the Redis TTL activity pool and one atomic extraction operation covering + candidate eligibility, Gateway reserve, ownership, removal, and short-lived idempotency. +- [ ] Implement Redis Provider leader, distributed rate, Client limit, and Worker + heartbeat/ownership; rebuild short-lived Proxy inventory from Providers after loss. +- [ ] Keep Provider output in Redis TTL activity state and node memory only; keep the + Gateway request path on immutable local snapshots with no Redis/PostgreSQL calls. - [ ] Expose Distribution extraction/status and Admin status/enable/disable/switch/reload. - [ ] Add integration tests using Compose-backed PostgreSQL/Redis. @@ -170,8 +177,14 @@ extract/live/ready Handler 与 Admin status/enable/disable/switch/reload Handler Bearer/CIDR、可信代理、Client ID、本地准入和 API 401/Gateway 407 差异，并作为 Admin/Distribution 必需依赖。共享 `platform/httpserver` 与 `controller/runtime` 已完成 Distribution/Admin 独立监听器、首错联动关闭和 -有界优雅停机；端点正式勾选仍等待 PostgreSQL/Redis Adapter、命令入口与 -Compose 集成测试。 +有界优雅停机；端点正式勾选仍等待 Redis 活动池/原子提取 Adapter、PostgreSQL +管理面 Adapter、命令入口与 Compose 集成测试。 + +已新增公用 `domain/activitypool` 契约及并发安全内存参考实现，Provider +Reconciler 通过 `UpsertFetched` 写入带供应商 TTL 和分配安全余量的批次；已覆盖 +`usableUntil` 向 Worker Snapshot 的传播与 Gateway 本地截止过滤、 +重复刷新、过期淘汰、独占提取、短期幂等及 Worker ownership 互斥。生产 Redis +Lua/Function Adapter 和多节点集成测试仍待实现。 ## Task 11: Checker and Health Reducer diff --git a/docs/operations/production-readiness.md b/docs/operations/production-readiness.md index 6c61677..699060c 100644 --- a/docs/operations/production-readiness.md +++ b/docs/operations/production-readiness.md @@ -8,8 +8,9 @@ - [ ] Sequential 并发 Empty 只切换一次，旧 Upstream Proxy 自然耗尽。 - [ ] `pool.maxSize` 与 `fetch.maxTotal` 分别按当前库存和累计获取计数。 - [ ] Extract 只有 `AVAILABLE -> EXTRACTED`，OpenAPI 不存在 release/renew。 -- [ ] Extract 状态更新和审计记录位于同一数据库事务。 -- [ ] `partial` 和 `allOrNothing` 均通过并发事务测试。 +- [ ] Extract 状态更新和短期幂等结果位于同一个 Redis 原子操作。 +- [ ] PostgreSQL 中不存在 Proxy 明细或逐次提取记录。 +- [ ] `partial` 和 `allOrNothing` 均通过 Redis 原子并发测试。 - [ ] `reserveForGateway` 在所有提取路径上统一执行。 ## 安全 @@ -25,7 +26,8 @@ ## 可用性 -- [ ] PostgreSQL 和 Redis 跨可用区，有监控、备份和恢复演练证据。 +- [ ] PostgreSQL 跨可用区，有监控、备份和恢复演练证据。 +- [ ] Redis 跨可用区并受监控，完成丢失后由 Provider 重建活动池的演练。 - [ ] Gateway、Controller、Checker 均跨主机/可用区分散。 - [ ] PDB、优雅终止与最大连接时长的组合经过驱逐测试。 - [ ] Controller 断线时 Gateway 在 `maxStaleAge` 内继续，超限拒绝新请求。 @@ -58,4 +60,3 @@ - [ ] 100k QPS 验证报告包含环境、命令、版本、场景、原始指标和结论。 任一关键一致性、安全或恢复项未完成时，不标记生产就绪。 - diff --git a/docs/operations/runbook.md b/docs/operations/runbook.md index c1f9b58..db4fba1 100644 --- a/docs/operations/runbook.md +++ b/docs/operations/runbook.md @@ -3,12 +3,15 @@ ## 1. 运行边界 - Gateway 是数据面，正常请求热路径不访问 PostgreSQL、Redis 或 Provider。 -- Controller 是权威控制面，负责 Fetch、生命周期、所有权、Snapshot、Extract - 和审计；多个副本只有一个 Provider 逻辑 Leader。 +- Controller 编排 Fetch、生命周期、所有权、Snapshot 与 Extract；多个副本只有 + 一个 Provider 逻辑 Leader。短效 Proxy 明细只存在于 Redis TTL 活动池和节点 + 内存，可由 Provider 重建。 +- PostgreSQL 只持久化配置版本、Upstream/Routing 管理状态、Admin 审计与 + outbox，以及可选聚合指标；不保存 Proxy 明细或逐次提取记录。 - Checker 执行有界健康探测，只上报 Observation，最终状态由 Controller reducer 决定。 -- Extract 是一次性独占发放。提交后状态为 `EXTRACTED`，没有 Lease、续租或 - Release 接口。 +- Extract 是一次性独占发放。Redis 在单次原子操作中校验并从可分配活动池移除 + 候选，同时写入带 TTL 的幂等结果；没有 Lease、续租或 Release 接口。 - `reserveForGateway` 是共享池硬约束，Extract 不得把 Gateway 库存清空。 - 集群峰值 100,000 QPS 是设计目标，只有完成本文容量验收后才能作为已验证 能力对外承诺。 @@ -88,7 +91,8 @@ kubectl -n proxy-pool rollout status deployment/proxy-gateway --timeout=10m liveness 失败。 - `/readyz`：进程可以接收新工作。Gateway 只有在持有未超过 `maxStaleAge` 的完整 Snapshot 且仍有准入能力时才 Ready。 -- Controller 只有在配置有效、存储可用、迁移兼容且控制接口已监听时才 Ready。 +- Controller 按能力判定就绪：Distribution/Fetch 依赖 Redis 活动池，Admin 持久化 + 写依赖 PostgreSQL 与兼容迁移。PostgreSQL 故障不得单独使 Extract 返回 503。 - Checker 在任务消费与结果上报通道可用时 Ready。 - `/metrics`：独立于业务入口，NetworkPolicy 仅允许监控命名空间访问。 @@ -140,9 +144,9 @@ HPA 使用 CPU/内存作为保护性信号；生产环境建议通过 Prometheus 3. Available Slots、Reserved、Active、各 Proxy 状态数量。 4. Provider Success、Empty、Duplicate-only、Error、429 与退避。 5. Checker 队列、SUSPECT 数、检查延迟和目标级失败。 -6. Extract requested/returned、insufficient、冲突和审计写入。 -7. PostgreSQL 连接、锁等待、事务失败、WAL 与备份。 -8. Redis 延迟、内存、主从状态和 Leader 租约抖动。 +6. Extract requested/returned、insufficient、冲突和幂等命中。 +7. PostgreSQL 连接、管理事务失败、outbox backlog、WAL 与备份。 +8. Redis 延迟、内存、TTL 淘汰、原子操作错误、主从状态和 Leader 租约抖动。 Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、request ID。 需要逐请求调查时使用受控、脱敏且采样的结构化日志。 @@ -160,8 +164,8 @@ Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、requ 1. 停止接收新的 Extract/Admin 写请求。 2. 停止发起 Fetch，释放 Provider Leader 租约。 -3. 完成已进入数据库事务的 Extract 或回滚。 -4. 刷新 outbox、审计和 Worker ACK，再关闭连接池。 +3. 等待已提交 Redis 原子操作返回；未确认请求的客户端必须使用相同幂等键重试。 +4. 刷新 Admin outbox、审计和 Worker ACK，再关闭 PostgreSQL/Redis 连接池。 ### Checker @@ -184,16 +188,23 @@ Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、requ ### 7.2 PostgreSQL 不可用 1. Gateway 继续使用最后有效 Snapshot。 -2. Controller 将 Distribution 和权威写操作置为不可用，避免返回未提交代理。 -3. Provider Fetch 停止写入；已有流量不受影响。 -4. 恢复后核对迁移、事务回滚、outbox backlog 与 Extract 审计连续性。 +2. Redis 健康且运行配置有效时，Distribution 继续执行原子 Extract，Provider + 继续刷新 TTL 活动池。 +3. 拒绝配置版本、Upstream/Routing 管理状态和其他需要 Admin 审计/outbox 的写入； + 不得把 Proxy 明细临时落入 PostgreSQL。 +4. 恢复后核对迁移、管理事务回滚、Admin 审计与 outbox backlog；不存在 Proxy + 明细或逐次提取记录恢复步骤。 ### 7.3 Redis 不可用 -1. Gateway 不受影响。 -2. Controller 停止需要分布式互斥的高风险工作，防止多个 Fetch Leader。 -3. 本地限流只作为临时降级，不能声称满足全局额度。 -4. 恢复后确认 Leader 唯一、租约 epoch 单调和重复 Fetch 去重。 +1. Gateway 继续读取节点内最后有效 Snapshot，直到 `maxStaleAge`；请求热路径不 + 回查 PostgreSQL 或 Provider。 +2. Distribution 立即失败关闭并返回 503，禁止本地内存提取或 PostgreSQL 兜底。 +3. Controller 停止活动池写入和需要分布式互斥的工作，防止多个 Fetch Leader； + 本地限流不能声称满足全局额度。 +4. 恢复后确认 Leader 唯一和租约 epoch 单调，由 Provider 重新 Fetch 并构建 TTL + 活动池，再恢复 Distribution。Redis 整体丢失会终止原活动池代次的排他状态和 + 短期幂等窗口；高可用、持久化、监控和告警必须明确并降低该风险。 ### 7.4 Provider 故障 @@ -213,11 +224,11 @@ Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、requ ### 7.6 Extract 库存不足 -1. `partial` 返回实际数量；`allOrNothing` 不足时整批回滚。 +1. `partial` 返回实际数量；`allOrNothing` 不足时不改变 Redis 活动池。 2. 检查 TTL、health age、filter、Worker ownership 和 `reserveForGateway`。 3. 不得降低 `reserveForGateway` 到导致 Gateway 容量告警的水平。 4. 回收 Worker-owned Proxy 必须先 DRAINING、等待 active/reserved 为零、清除 - ownership，再执行 `AVAILABLE -> EXTRACTED` 事务。 + ownership，再由 Redis 原子操作将其从可分配活动池移除并写入短期幂等结果。 5. 已提取代理没有 Release；客户端归还请求只记录为无效调用，不恢复库存。 ### 7.7 Checker 积压 @@ -229,14 +240,17 @@ Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、requ ## 8. 备份与恢复 -- PostgreSQL：每日全量、连续 WAL/PITR，至少每季度做恢复演练。 -- Redis：仅保存可重建协调状态；不得把 Redis 备份当权威业务备份。 +- PostgreSQL：每日全量、连续 WAL/PITR，保护配置版本、Upstream/Routing 管理 + 状态、Admin 审计与 outbox；至少每季度做恢复演练。 +- Redis：保存可由 Provider 重建的 TTL 活动池、所有权/Leader 协调和短期幂等 + 结果；使用高可用与持久化降低窗口丢失风险，但不把它当长期业务档案。 - 配置：版本化保存校验通过的不可变 Revision 与校验和。 - Secret：由密钥平台版本化，日志和备份中不得出现明文。 -恢复顺序：PostgreSQL -> Redis -> Controller -> Checker -> Gateway。恢复后验证 -Proxy 状态、Extraction Record、Worker ownership epoch、outbox 和配置 Revision -单调一致，再开放 Gateway 与 Distribution。 +管理面恢复顺序：PostgreSQL -> Controller/Admin；代理运行面恢复顺序：Redis -> +Provider 重建活动池 -> Controller/Checker -> 发布新 Snapshot。验证 Worker +ownership epoch、活动池 TTL、短期幂等窗口、outbox 和配置 Revision 一致后，再 +开放相应能力；Gateway 在旧 Snapshot 未超过 `maxStaleAge` 时无需等待数据库恢复。 ## 9. Secret 轮换 diff --git a/docs/requirements/completion-audit.md b/docs/requirements/completion-audit.md index a8e8a7c..d0d16c5 100644 --- a/docs/requirements/completion-audit.md +++ b/docs/requirements/completion-audit.md @@ -11,15 +11,14 @@ - 产品设计、总体架构、项目结构、四项 ADR。 - 开发、配置、Distribution/Admin API、控制面协议、安全、测试、运维文档。 - 20 个配置场景和 35 张 Mermaid 架构/流程/状态/故障图。 -- 版本化文档包 `proxy-pool-docs-v1.0.zip`，包含 50 个条目。 ### 机器契约 - Distribution OpenAPI：一次性独占提取、partial/allOrNothing、幂等键、 - TTL/健康过滤结果与标准错误。 + Redis TTL 活动池原子语义、TTL/健康过滤结果与标准错误。 - Admin OpenAPI：状态、Upstream 启停、Routing 切换和配置重载。 -- Protobuf：Worker 注册、全量/增量 Snapshot、ACK、运行态/结果上报、Checker - 任务与 Observation。 +- Protobuf：Worker 注册、全量/增量 Snapshot、`usable_until`、ACK、运行态/ + 结果上报、Checker 任务与 Observation。 ### 核心参考实现 @@ -29,8 +28,8 @@ 原子计数；1,000 goroutine 不超卖测试。 - `ROUTE-001 / ROUTE-004`：首条命中规则与 Concurrent Sequential 单次切换。 - `FETCH-005 / FETCH-006`：Valid、Empty、DuplicateOnly、Error 分类。 -- `DIST-001..003 / DIST-006..007`：内存事务模型验证独占提取、满足模式、TTL、 - 健康时效与 Gateway 保留量；1,000 并发不重复。 +- `DIST-001..003 / DIST-006..007`：内存活动池参考实现验证独占提取、满足模式、 + TTL、健康时效与 Gateway 保留量；1,000 并发不重复。 - `OPS-001`：完整 Snapshot 目标、epoch/version、校验和验证及原子替换。 - `CAP-001 / GW 热路径边界`：本地 Dispatch 条件过滤与原子容量预留。 - `GW-*`：HTTP 正向代理、HTTPS CONNECT、双向 Tunnel、超时、重试、保护链与 @@ -65,8 +64,12 @@ CI 已配置 Linux race job。Docker/Kubernetes 仅完成静态验证，没有 1. `cmd/proxy-gateway/controller/checker/loadgen` 进程装配。 2. Gateway 进程装配、生产连接池调优与代表性流量压测。 3. Provider 分布式 singleflight/Leader、长期凭据回收和累计额度执行器。 -4. PostgreSQL repository、Extraction 行锁事务、Outbox 和迁移。 -5. Redis Leader、速率限制、心跳与可重建协调适配器。 +4. PostgreSQL 配置版本、Upstream/Routing 管理状态、Admin 审计/Outbox + repository 和迁移；可选聚合指标不包含 Proxy 明细。 +5. Redis TTL 活动池、原子独占提取、短期幂等结果、Leader、速率限制、心跳与 + 所有权适配器；活动池可由 Provider 重建。接线时必须验证进入 + `allocationSafetyMargin` 的不可分配条目不会长期占用补池额度，同时不得 + 突破 `pool.maxSize` 硬上限。 6. Worker ownership drain/ACK/过期回收和网络快照流。 7. Checker 调度、探测器和健康 reducer。 8. Admin/Distribution 细粒度授权、分布式限流和审计查询。 diff --git a/docs/security/security-model.md b/docs/security/security-model.md index f89ee91..f06a5e3 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -5,7 +5,8 @@ - Gateway、Distribution、Admin 和 Metrics 为四个独立监听边界。 - Provider API 和上游 Proxy 属于外部不可信网络。 - Worker 与 Controller 通道必须进行双向身份校验并绑定 cluster/worker。 -- PostgreSQL 保存权威状态；Redis 数据默认按可重建缓存与协调信息处理。 +- PostgreSQL 保存管理面持久状态；Redis 保存可重建的短效 Proxy 活动池与协调 + 信息，Proxy 地址和凭据不写入 PostgreSQL。 ## 2. 入口控制 diff --git a/docs/testing/failure-injection.md b/docs/testing/failure-injection.md index 09439d8..ba0c713 100644 --- a/docs/testing/failure-injection.md +++ b/docs/testing/failure-injection.md @@ -4,7 +4,9 @@ - 先在隔离环境建立 15 分钟稳定基线，再注入单一故障。 - 每次只改变一个变量，记录开始/恢复时间与所有指标。 -- 任何数据库写故障后都核对 Proxy 状态、审计和 outbox，而非只看 HTTP 状态码。 +- PostgreSQL 写故障后核对配置版本、Upstream/Routing 管理状态、Admin 审计和 + outbox；Redis 写故障后核对活动池、独占提取和短期幂等键，不能只看 HTTP + 状态码。 - 故障恢复后至少观察两个健康检查周期和一个 Snapshot 完整发布周期。 ## 场景 @@ -22,15 +24,18 @@ duplicate-only 不触发 Sequential 切换。 - 断开 60 秒、连接池耗尽、锁等待、事务提交失败、只读切换。 -预期：Gateway 继续使用最后 Snapshot；Extract 不返回未提交记录；恢复后 outbox -补发且不重复应用。 +预期：Gateway 继续使用最后 Snapshot；Redis 健康时 Fetch、活动池更新和 Extract +不被 PostgreSQL 故障阻断；依赖持久化的配置/Admin 写操作失败关闭。恢复后 +outbox 补发且不重复应用，PostgreSQL 中没有 Proxy 明细或逐次提取记录需要恢复。 ### Redis -- 断开、延迟 2 秒、Leader key 丢失、主从切换。 +- 断开、延迟 2 秒、Leader key 丢失、活动池与幂等键丢失、主从切换。 -预期：Gateway 无影响；Provider Fetch 不出现多个有效 Leader；全局限流明确降级； -恢复后 epoch 单调。 +预期：Gateway 在 Snapshot 新鲜度窗口内无影响；Distribution 失败关闭并返回 503， +不得改写 PostgreSQL 兜底；Provider Fetch 不出现多个有效 Leader；全局限流明确 +降级。恢复后 epoch 单调，由 Provider 重新拉取并构建 TTL 活动池；已经丢失的短期 +幂等窗口和旧活动池排他状态不伪装成仍然有效，重建结果视为新的活动池代次。 ### Controller @@ -55,10 +60,12 @@ Fetch；Delta 缺口触发完整 Snapshot。 ### Distribution -- 100 个并发请求争用相同 Proxy；数据库在事务提交时断开；Gateway 同时满载。 +- 100 个并发请求争用相同 Proxy；Redis 在原子操作执行或主从切换时断开；Gateway + 同时满载。 -预期：每个 Proxy 最多返回一次；提交失败不返回代理；allOrNothing 整批回滚； -池中始终剩余 `reserveForGateway`；没有 Release 恢复路径。 +预期：同一 Redis 活动池代次内每个 Proxy 最多返回一次；该代次仍保留时，未确认 +成功的客户端使用同一幂等键重试并得到已缓存结果或明确失败，不出现部分可见状态； +allOrNothing 整批不变；池中始终剩余 `reserveForGateway`；没有 Release 恢复路径。 ### 配置与 Secret @@ -66,4 +73,3 @@ Fetch；Delta 缺口触发完整 Snapshot。 预期：新 Revision 整体拒绝，旧不可变 Snapshot 继续；认证错误计 Error 而不是 Empty；日志不出现 Secret。 - diff --git a/docs/testing/test-strategy.md b/docs/testing/test-strategy.md index 50f138b..ef6daf8 100644 --- a/docs/testing/test-strategy.md +++ b/docs/testing/test-strategy.md @@ -5,6 +5,8 @@ - 先证明领域不变量，再证明 Adapter 契约，最后证明跨进程行为。 - 并发测试必须在 race detector 下运行，不能只依赖单线程示例。 - 时间、随机、网络、Provider 和存储均通过可替换接口或 fixture 控制。 +- 生产 Adapter 必须证明数据边界：Proxy 明细只进入 Redis TTL 活动池和节点 + 内存，PostgreSQL 不得出现 Proxy 明细或逐次提取记录。 - 100,000 QPS 是待验证的集群目标，不以架构图、副本数或短时峰值替代证据。 - 性能通过与正确性通过相互独立；高 QPS 下出现超卖、重复 Extract 或状态 回退时，结果一律失败。 @@ -24,9 +26,11 @@ ### 契约测试 - Provider 响应模板的大小、超时、函数白名单和解析边界。 -- PostgreSQL `FOR UPDATE SKIP LOCKED` 并发批量提取。 -- Outbox 状态更新、发布与幂等重放。 -- Redis Leader 租约、限流和失联恢复。 +- Redis Lua/Function 对候选筛选、库存预留、独占移除和短期幂等结果执行单次 + 原子操作；并发与主从切换下不得部分提交。 +- PostgreSQL 配置版本、Upstream/Routing 管理状态、Admin 审计与 outbox 的事务 + 更新及幂等重放；测试库断言不包含 Proxy 明细或逐次提取记录。 +- Redis TTL 活动池、Leader 租约、限流、短期幂等窗口和失联恢复。 - Snapshot/Delta/ACK/Report 的版本与校验和兼容性。 - OpenAPI 错误模型、认证矩阵、批量 fulfillment。 @@ -34,7 +38,8 @@ - HTTP 正向代理成功、上游连接前失败和安全重试。 - HTTPS CONNECT 建立后不透明重放。 -- Controller Fetch -> Check -> AVAILABLE -> Worker Snapshot -> Gateway 转发。 +- Controller Fetch -> Redis TTL 活动池 -> Check -> AVAILABLE -> Worker Snapshot -> + Gateway 转发；Gateway 请求线程始终只读节点内存。 - Distribution 原子提取后 Gateway 不再分配同一 Proxy。 - 配置热更新失败保留旧 Revision，成功后新请求使用新 Snapshot。 @@ -51,9 +56,10 @@ 重置 Empty；Success 重置 Empty。 6. **Sequential 竞态**：达到阈值时多协程只能将 A 切到 B 一次，不能越过 B。 7. **Drain**：切换或禁用 Upstream 后停止新分配，已有连接完成后才回收。 -8. **Extract 竞态**：多个请求并发提取同一候选集合，每个 Proxy 最多返回一次。 -9. **Extract 批量语义**：partial 提交实际数量；allOrNothing 不足时状态和审计 - 全回滚；始终保留 `reserveForGateway`。 +8. **Extract 竞态**：多个请求并发提取同一候选集合，同一 Redis 活动池代次内 + 每个 Proxy 最多返回一次。 +9. **Extract 批量语义**：partial 提交实际数量；allOrNothing 不足时 Redis + 活动池和幂等结果均不变；始终保留 `reserveForGateway`。 10. **Worker ownership**：回收过程严格经过 DRAINING、ACK、active/reserved=0、 unowned，旧 Snapshot 不可再分配。 11. **控制面故障**：Redis、PostgreSQL、Controller、Checker 与 Provider 分别 @@ -70,8 +76,9 @@ go vet ./... go build ./... ``` -需要 PostgreSQL/Redis 的测试使用独立数据库和短生命周期容器，不复用开发数据。 -测试结束后验证没有残留 Worker ownership、Leader 租约或未提交 Extraction Record。 +需要 PostgreSQL/Redis 的测试使用独立实例和短生命周期容器，不复用开发数据。 +测试结束后验证没有残留 Worker ownership、Leader 租约、活动池条目或幂等键， +并检查 PostgreSQL 中不存在 Proxy 明细和逐次提取记录。 ## 5. 负载模型 @@ -109,7 +116,7 @@ go build ./... 业务 SLO 由产品最终确认，但至少满足以下工程门槛： -- 无容量超卖、负计数、重复 Extract、审计缺失或状态非法回退。 +- 无容量超卖、负计数、幂等窗口内重复 Extract、Admin 审计缺失或状态非法回退。 - 100k 峰值期间无进程 OOM、FD 耗尽、无界队列或全局锁热点。 - Gateway 热路径在 PostgreSQL、Redis、Provider 失效时不发起同步访问。 - p99 Dispatch 小于 100 微秒的设计预算需要在 100k Proxy Snapshot 下单独证明。 @@ -125,6 +132,6 @@ go build ./... 配置：Snapshot 规模、Proxy 容量、路由、重试、日志级别 场景：协议、连接复用、响应体、持续时间、升压曲线、故障注入 结果：QPS、建连速率、active、p50/p95/p99、错误、CPU、RSS、GC、FD、网络 -不变量：capacity、ownership、extraction、audit、reserve 检查结果 +不变量：capacity、ownership、extraction、idempotency、admin audit、reserve 检查结果 结论：通过/失败，以及适用边界 ``` diff --git a/examples/config/10-round-robin-routing.yaml b/examples/config/10-round-robin-routing.yaml index 7eb9e45..727dbe1 100644 --- a/examples/config/10-round-robin-routing.yaml +++ b/examples/config/10-round-robin-routing.yaml @@ -19,6 +19,7 @@ upstreams: proxyAuth: {type: response} pool: {maxSize: 1000} capacity: {maxConcurrencyPerProxy: 10} + lifecycle: {ttl: 2m, allocationSafetyMargin: 15s} fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1} provider-b: enabled: true @@ -28,4 +29,5 @@ upstreams: proxyAuth: {type: response} pool: {maxSize: 1000} capacity: {maxConcurrencyPerProxy: 10} + lifecycle: {ttl: 2m, allocationSafetyMargin: 15s} fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1} diff --git a/examples/config/11-random-routing.yaml b/examples/config/11-random-routing.yaml index 2777134..27ff8d9 100644 --- a/examples/config/11-random-routing.yaml +++ b/examples/config/11-random-routing.yaml @@ -19,6 +19,7 @@ upstreams: proxyAuth: {type: response} pool: {maxSize: 1000} capacity: {maxConcurrencyPerProxy: 10} + lifecycle: {ttl: 2m, allocationSafetyMargin: 15s} fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1} provider-b: enabled: true @@ -28,4 +29,5 @@ upstreams: proxyAuth: {type: response} pool: {maxSize: 1000} capacity: {maxConcurrencyPerProxy: 10} + lifecycle: {ttl: 2m, allocationSafetyMargin: 15s} fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1} diff --git a/examples/config/examples_test.go b/examples/config/examples_test.go index e7c3c0a..20068dd 100644 --- a/examples/config/examples_test.go +++ b/examples/config/examples_test.go @@ -6,7 +6,7 @@ import ( "strings" "testing" - projectconfig "github.com/proxy-pool/proxy-pool/internal/config" + projectconfig "proxy-pool/internal/config" ) func TestAllExamplesLoadStrictly(t *testing.T) { diff --git a/findings.md b/findings.md index 2f7b9c2..25afdc1 100644 --- a/findings.md +++ b/findings.md @@ -40,8 +40,8 @@ Extract 请求使用哪些 Upstream。 - 代理分配必须原子预留容量，防止并发超卖。 - 代理唯一键为 `scheme + host + port + username + credentialVersion`；日志 和指标不得暴露密码。 -- TTL 优先级为响应 `expiresAt`、响应 `ttl`、配置固定 TTL、不过期；内部 - 时间统一 UTC。 +- TTL 优先级为响应 `expiresAt`、响应 `ttl`、配置固定 TTL；活动池中的每个 + Proxy 必须有明确过期时间，内部时间统一 UTC。 - 健康检查至少区分全局健康和 Routing/目标健康，并使用抖动和并发上限。 - GET/HEAD 可按配置安全重试；非幂等方法默认不自动重试；CONNECT 建立后 不透明重放。 @@ -55,8 +55,8 @@ Extract 请求使用哪些 Upstream。 - 数据面采用多 Worker，本地不可变代理快照和本地容量计数。 - 同一代理必须由单个 Worker 所有，或由控制面下发容量切片；禁止每请求 访问 Redis 做全局并发计数。 -- 控制面集中 Provider 获取、独立限流、singleflight、Leader 选举、状态 - 持久化和快照分发。 +- 控制面集中 Provider 获取、独立限流、singleflight、Leader 选举和快照分发。 + Proxy 明细只进入 Redis TTL 活动池，不写 PostgreSQL。 - 副本数必须由单 Worker 实测能力、目标利用率和故障域余量计算。 ## 配置模型 @@ -82,4 +82,3 @@ Routing 自上而下匹配，首条命中停止；支持 Gateway 与 Extract 两 - Provider 超时不得计入 Empty Fetch。 - 重复代理不得重复入池，也不得触发空结果切换。 - 所有 Upstream 不可用时按显式策略执行。 - diff --git a/go.mod b/go.mod index 68ad726..766da3e 100644 --- a/go.mod +++ b/go.mod @@ -1,4 +1,4 @@ -module github.com/proxy-pool/proxy-pool +module proxy-pool go 1.26.0 diff --git a/internal/adapters/providerapi/credential_chain_test.go b/internal/adapters/providerapi/credential_chain_test.go index 75c9fa2..af2f36d 100644 --- a/internal/adapters/providerapi/credential_chain_test.go +++ b/internal/adapters/providerapi/credential_chain_test.go @@ -5,10 +5,10 @@ import ( "testing" "time" - "github.com/proxy-pool/proxy-pool/internal/adapters/providerapi" - "github.com/proxy-pool/proxy-pool/internal/config" - "github.com/proxy-pool/proxy-pool/internal/gateway/transport" - "github.com/proxy-pool/proxy-pool/internal/platform/credentials" + "proxy-pool/internal/adapters/providerapi" + "proxy-pool/internal/config" + "proxy-pool/internal/gateway/transport" + "proxy-pool/internal/platform/credentials" ) func TestParsedCredentialsResolveInGatewayTransport(t *testing.T) { diff --git a/internal/adapters/providerapi/http_adapter.go b/internal/adapters/providerapi/http_adapter.go index dfbccdf..eeeb5ec 100644 --- a/internal/adapters/providerapi/http_adapter.go +++ b/internal/adapters/providerapi/http_adapter.go @@ -12,8 +12,8 @@ import ( "strings" "time" - "github.com/proxy-pool/proxy-pool/internal/config" - controllerProvider "github.com/proxy-pool/proxy-pool/internal/controller/provider" + "proxy-pool/internal/config" + controllerProvider "proxy-pool/internal/controller/provider" ) const defaultRetryAfterMax = 30 * time.Second diff --git a/internal/adapters/providerapi/http_adapter_test.go b/internal/adapters/providerapi/http_adapter_test.go index 4483b61..9d0a2c3 100644 --- a/internal/adapters/providerapi/http_adapter_test.go +++ b/internal/adapters/providerapi/http_adapter_test.go @@ -12,8 +12,8 @@ import ( "testing" "time" - "github.com/proxy-pool/proxy-pool/internal/config" - controllerProvider "github.com/proxy-pool/proxy-pool/internal/controller/provider" + "proxy-pool/internal/config" + controllerProvider "proxy-pool/internal/controller/provider" ) var _ controllerProvider.ProviderAdapter = (*HTTPAdapter)(nil) diff --git a/internal/adapters/providerapi/template_complexity_test.go b/internal/adapters/providerapi/template_complexity_test.go index 8014527..7c1384f 100644 --- a/internal/adapters/providerapi/template_complexity_test.go +++ b/internal/adapters/providerapi/template_complexity_test.go @@ -4,7 +4,7 @@ import ( "errors" "testing" - "github.com/proxy-pool/proxy-pool/internal/config" + "proxy-pool/internal/config" ) func TestTemplateParserRejectsNestedRanges(t *testing.T) { diff --git a/internal/adapters/providerapi/template_parser.go b/internal/adapters/providerapi/template_parser.go index 173171a..5dcb4db 100644 --- a/internal/adapters/providerapi/template_parser.go +++ b/internal/adapters/providerapi/template_parser.go @@ -11,9 +11,9 @@ import ( "text/template" "time" - "github.com/proxy-pool/proxy-pool/internal/config" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" - "github.com/proxy-pool/proxy-pool/internal/platform/credentials" + "proxy-pool/internal/config" + proxyDomain "proxy-pool/internal/domain/proxy" + "proxy-pool/internal/platform/credentials" ) const ( diff --git a/internal/adapters/providerapi/template_parser_test.go b/internal/adapters/providerapi/template_parser_test.go index 52409d3..e6bd536 100644 --- a/internal/adapters/providerapi/template_parser_test.go +++ b/internal/adapters/providerapi/template_parser_test.go @@ -10,10 +10,10 @@ import ( "testing" "time" - "github.com/proxy-pool/proxy-pool/internal/config" - controllerProvider "github.com/proxy-pool/proxy-pool/internal/controller/provider" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" - "github.com/proxy-pool/proxy-pool/internal/platform/credentials" + "proxy-pool/internal/config" + controllerProvider "proxy-pool/internal/controller/provider" + proxyDomain "proxy-pool/internal/domain/proxy" + "proxy-pool/internal/platform/credentials" ) var _ controllerProvider.Parser = (*TemplateParser)(nil) diff --git a/internal/config/config.go b/internal/config/config.go index d5d2e47..b04c7cb 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -113,6 +113,7 @@ type Extraction struct { MinRemainingTTL Duration `yaml:"minRemainingTTL"` MaxHealthCheckAge Duration `yaml:"maxHealthCheckAge"` ReserveForGateway int `yaml:"reserveForGateway"` + IdempotencyTTL Duration `yaml:"idempotencyTTL"` } type Metrics struct { diff --git a/internal/config/config_test.go b/internal/config/config_test.go index 936aa87..7a3dfb4 100644 --- a/internal/config/config_test.go +++ b/internal/config/config_test.go @@ -6,6 +6,7 @@ import ( "path/filepath" "strings" "testing" + "time" "go.yaml.in/yaml/v4" ) @@ -352,6 +353,22 @@ func TestValidateRejectsInvalidConfigurationMatrix(t *testing.T) { }, want: "reserveForGateway", }, + { + name: "negative extraction idempotency ttl", + mutate: func(cfg *Config) { + cfg.Distribution.Extraction.IdempotencyTTL = Duration(-time.Second) + }, + want: "idempotencyTTL", + }, + { + name: "enabled upstream without ttl", + mutate: func(cfg *Config) { + upstream := cfg.Upstreams["provider-a"] + upstream.Lifecycle.TTL = 0 + cfg.Upstreams["provider-a"] = upstream + }, + want: "lifecycle.ttl", + }, { name: "no enabled upstream", mutate: func(cfg *Config) { diff --git a/internal/config/validate.go b/internal/config/validate.go index a5e22d7..a80ae0d 100644 --- a/internal/config/validate.go +++ b/internal/config/validate.go @@ -81,6 +81,9 @@ func Validate(cfg *Config) error { if err := requireNonNegative("distribution.reserveForGateway", cfg.Distribution.Extraction.ReserveForGateway); err != nil { return err } + if err := requireNonNegative("distribution.idempotencyTTL", cfg.Distribution.Extraction.IdempotencyTTL); err != nil { + return err + } } return nil } @@ -312,7 +315,7 @@ func validateUpstream(name string, upstream Upstream) error { if err := requirePositive(scope+" capacity.maxConcurrencyPerProxy", upstream.Capacity.MaxConcurrencyPerProxy); err != nil { return err } - if err := requireNonNegative(scope+" lifecycle.ttl", upstream.Lifecycle.TTL); err != nil { + if err := requirePositive(scope+" lifecycle.ttl", upstream.Lifecycle.TTL); err != nil { return err } if err := requireNonNegative(scope+" lifecycle.allocationSafetyMargin", upstream.Lifecycle.AllocationSafetyMargin); err != nil { diff --git a/internal/controller/admin/handler.go b/internal/controller/admin/handler.go index ed5fe44..e8b16d1 100644 --- a/internal/controller/admin/handler.go +++ b/internal/controller/admin/handler.go @@ -7,8 +7,8 @@ import ( "net/http" "strings" - "github.com/proxy-pool/proxy-pool/internal/platform/httpapi" - "github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity" + "proxy-pool/internal/platform/httpapi" + "proxy-pool/internal/platform/httpsecurity" ) const ( diff --git a/internal/controller/admin/handler_test.go b/internal/controller/admin/handler_test.go index cbef09f..660b4a0 100644 --- a/internal/controller/admin/handler_test.go +++ b/internal/controller/admin/handler_test.go @@ -9,8 +9,8 @@ import ( "strings" "testing" - "github.com/proxy-pool/proxy-pool/internal/platform/httpapi" - "github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity" + "proxy-pool/internal/platform/httpapi" + "proxy-pool/internal/platform/httpsecurity" ) func TestHandlerReturnsStatusWithoutSensitiveDetails(t *testing.T) { diff --git a/internal/controller/distribution/handler.go b/internal/controller/distribution/handler.go index d690b16..471e1d5 100644 --- a/internal/controller/distribution/handler.go +++ b/internal/controller/distribution/handler.go @@ -7,10 +7,10 @@ import ( "strings" "time" - controllerExtraction "github.com/proxy-pool/proxy-pool/internal/controller/extraction" - domainExtraction "github.com/proxy-pool/proxy-pool/internal/domain/extraction" - "github.com/proxy-pool/proxy-pool/internal/platform/httpapi" - "github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity" + controllerExtraction "proxy-pool/internal/controller/extraction" + domainExtraction "proxy-pool/internal/domain/extraction" + "proxy-pool/internal/platform/httpapi" + "proxy-pool/internal/platform/httpsecurity" ) const ( diff --git a/internal/controller/distribution/handler_test.go b/internal/controller/distribution/handler_test.go index 0d027d3..f82b0d7 100644 --- a/internal/controller/distribution/handler_test.go +++ b/internal/controller/distribution/handler_test.go @@ -11,10 +11,10 @@ import ( "testing" "time" - controllerExtraction "github.com/proxy-pool/proxy-pool/internal/controller/extraction" - domainExtraction "github.com/proxy-pool/proxy-pool/internal/domain/extraction" - "github.com/proxy-pool/proxy-pool/internal/platform/httpapi" - "github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity" + controllerExtraction "proxy-pool/internal/controller/extraction" + domainExtraction "proxy-pool/internal/domain/extraction" + "proxy-pool/internal/platform/httpapi" + "proxy-pool/internal/platform/httpsecurity" ) func TestNewHandlerValidatesDependenciesAndBodyLimit(t *testing.T) { diff --git a/internal/controller/extraction/service.go b/internal/controller/extraction/service.go index a762b28..0df7cb9 100644 --- a/internal/controller/extraction/service.go +++ b/internal/controller/extraction/service.go @@ -6,7 +6,7 @@ import ( "fmt" "time" - domain "github.com/proxy-pool/proxy-pool/internal/domain/extraction" + domain "proxy-pool/internal/domain/extraction" ) var ( @@ -24,6 +24,7 @@ type Policy struct { MinRemainingTTL time.Duration MaxHealthCheckAge time.Duration ReserveForGateway int + IdempotencyTTL time.Duration } type Filters struct { @@ -85,7 +86,7 @@ func NewService(store domain.Store, policy Policy, admission Admission, now func return nil, fmt.Errorf("%w: store is required", ErrInvalidServicePolicy) } if policy.MaxCountPerRequest <= 0 || policy.MinRemainingTTL < 0 || - policy.MaxHealthCheckAge < 0 || policy.ReserveForGateway < 0 { + policy.MaxHealthCheckAge < 0 || policy.ReserveForGateway < 0 || policy.IdempotencyTTL < 0 { return nil, ErrInvalidServicePolicy } if policy.DefaultFulfillment != domain.Partial && policy.DefaultFulfillment != domain.AllOrNothing { @@ -119,13 +120,14 @@ func (s *Service) Extract(ctx context.Context, request Request) (Response, error now := s.now().UTC() clientID := request.ClientID if clientID == "" { - clientID = "anonymous" + clientID = request.SourceIP } result, err := s.store.Extract(ctx, domain.Command{ RequestID: request.RequestID, ClientID: clientID, SourceIP: request.SourceIP, IdempotencyKey: request.IdempotencyKey, + IdempotencyTTL: s.policy.IdempotencyTTL, Requested: request.Count, Fulfillment: fulfillment, Now: now, diff --git a/internal/controller/extraction/service_test.go b/internal/controller/extraction/service_test.go index 16c1fb2..4995473 100644 --- a/internal/controller/extraction/service_test.go +++ b/internal/controller/extraction/service_test.go @@ -6,7 +6,9 @@ import ( "testing" "time" - domain "github.com/proxy-pool/proxy-pool/internal/domain/extraction" + "proxy-pool/internal/domain/activitypool" + domain "proxy-pool/internal/domain/extraction" + proxyDomain "proxy-pool/internal/domain/proxy" ) func TestServiceAppliesPolicyAndBuildsResponse(t *testing.T) { @@ -30,6 +32,7 @@ func TestServiceAppliesPolicyAndBuildsResponse(t *testing.T) { MinRemainingTTL: 30 * time.Second, MaxHealthCheckAge: 15 * time.Second, ReserveForGateway: 5, + IdempotencyTTL: 2 * time.Minute, }, allowAllAdmission{}, func() time.Time { return now }) if err != nil { t.Fatalf("NewService(): %v", err) @@ -53,7 +56,10 @@ func TestServiceAppliesPolicyAndBuildsResponse(t *testing.T) { t.Fatalf("store command policy = %+v", store.command) } if store.command.RequestID != "req-1" || store.command.ClientID != "client-1" { - t.Fatalf("store command audit context = %+v", store.command) + t.Fatalf("store command request context = %+v", store.command) + } + if store.command.IdempotencyTTL != 2*time.Minute { + t.Fatalf("store command idempotency TTL = %s, want 2m", store.command.IdempotencyTTL) } if response.RequestID != "req-1" || response.Requested != 2 || response.Returned != 1 { t.Fatalf("response = %+v", response) @@ -116,15 +122,45 @@ func TestServiceAppliesAdmissionBeforeStoreUsingStableIdentity(t *testing.T) { } } +func TestServiceUsesSourceIdentityForEphemeralIdempotency(t *testing.T) { + store := &recordingStore{} + service, err := NewService(store, Policy{ + MaxCountPerRequest: 1, + DefaultFulfillment: domain.Partial, + }, allowAllAdmission{}, time.Now) + if err != nil { + t.Fatalf("NewService(): %v", err) + } + + _, err = service.Extract(context.Background(), Request{ + RequestID: "req-1", + SourceIP: "192.0.2.30", + Count: 1, + }) + if err != nil { + t.Fatalf("Extract(): %v", err) + } + if store.command.ClientID != "192.0.2.30" { + t.Fatalf("store command ClientID = %q, want source IP", store.command.ClientID) + } +} + func TestServiceIdempotentReplayKeepsOriginalExtractionTime(t *testing.T) { firstTime := time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC) secondTime := firstTime.Add(time.Minute) - store := domain.NewMemoryStore([]domain.Candidate{{ - ID: "p1", - State: domain.Available, - ExpiresAt: firstTime.Add(2 * time.Minute), - LastCheckedAt: firstTime, - }}) + store := activitypool.NewMemoryPool() + checkedAt := firstTime + upserted, err := store.UpsertFetched(context.Background(), "provider-a", activitypool.FetchedBatch{ + ObservedAt: firstTime, + ConfiguredTTL: 2 * time.Minute, + Proxies: []proxyDomain.Proxy{{ + ID: "p1", Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8080, + State: proxyDomain.StateAvailable, LastCheckedAt: &checkedAt, + }}, + }) + if err != nil || upserted.Inserted != 1 { + t.Fatalf("UpsertFetched() = %+v, %v", upserted, err) + } times := []time.Time{firstTime, secondTime} service, err := NewService(store, Policy{ MaxCountPerRequest: 1, diff --git a/internal/controller/pool/fetch_budget.go b/internal/controller/pool/fetch_budget.go index 466b84d..7626594 100644 --- a/internal/controller/pool/fetch_budget.go +++ b/internal/controller/pool/fetch_budget.go @@ -5,7 +5,7 @@ import ( "fmt" "sync" - "github.com/proxy-pool/proxy-pool/internal/domain/upstream" + "proxy-pool/internal/domain/upstream" ) var ( diff --git a/internal/controller/pool/ownership.go b/internal/controller/pool/ownership.go index 499db51..1dbd04d 100644 --- a/internal/controller/pool/ownership.go +++ b/internal/controller/pool/ownership.go @@ -3,7 +3,7 @@ package pool import ( "time" - ownershipDomain "github.com/proxy-pool/proxy-pool/internal/domain/ownership" + ownershipDomain "proxy-pool/internal/domain/ownership" ) var ( diff --git a/internal/controller/pool/ownership_test.go b/internal/controller/pool/ownership_test.go index d93fcbc..560ca9e 100644 --- a/internal/controller/pool/ownership_test.go +++ b/internal/controller/pool/ownership_test.go @@ -9,7 +9,9 @@ import ( "testing" "time" - extractionDomain "github.com/proxy-pool/proxy-pool/internal/domain/extraction" + "proxy-pool/internal/domain/activitypool" + extractionDomain "proxy-pool/internal/domain/extraction" + proxyDomain "proxy-pool/internal/domain/proxy" ) func TestOwnershipManagerPreventsDualAssignment(t *testing.T) { @@ -64,9 +66,7 @@ func TestOwnershipManagerRenewsOnlyCurrentAssignment(t *testing.T) { func TestSharedRepositoryMakesOwnershipAndExtractionMutuallyExclusive(t *testing.T) { for iteration := range 100 { now := time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC) - store := extractionDomain.NewMemoryStore([]extractionDomain.Candidate{{ - ID: "proxy-1", State: extractionDomain.Available, - }}) + store := newTestActivityPool(t, now, "proxy-1") manager, err := NewOwnershipManager(store) if err != nil { t.Fatalf("iteration %d NewOwnershipManager(): %v", iteration, err) @@ -159,15 +159,29 @@ func TestOwnershipManagerExpiresCrashedWorkerAssignment(t *testing.T) { func newTestOwnershipManager(t *testing.T, proxyIDs ...string) *OwnershipManager { t.Helper() - candidates := make([]extractionDomain.Candidate, 0, len(proxyIDs)) - for _, proxyID := range proxyIDs { - candidates = append(candidates, extractionDomain.Candidate{ - ID: proxyID, State: extractionDomain.Available, - }) - } - manager, err := NewOwnershipManager(extractionDomain.NewMemoryStore(candidates)) + now := time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC) + manager, err := NewOwnershipManager(newTestActivityPool(t, now, proxyIDs...)) if err != nil { t.Fatalf("NewOwnershipManager(): %v", err) } return manager } + +func newTestActivityPool(t *testing.T, now time.Time, proxyIDs ...string) *activitypool.MemoryPool { + t.Helper() + proxies := make([]proxyDomain.Proxy, 0, len(proxyIDs)) + for index, proxyID := range proxyIDs { + proxies = append(proxies, proxyDomain.Proxy{ + ID: proxyID, Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.1", + Port: uint16(8000 + index), State: proxyDomain.StateAvailable, + }) + } + store := activitypool.NewMemoryPool() + result, err := store.UpsertFetched(context.Background(), "provider-a", activitypool.FetchedBatch{ + ObservedAt: now, ConfiguredTTL: 10 * time.Minute, Proxies: proxies, + }) + if err != nil || result.Inserted != len(proxyIDs) { + t.Fatalf("UpsertFetched() = %+v, %v", result, err) + } + return store +} diff --git a/internal/controller/pool/reconciler.go b/internal/controller/pool/reconciler.go index 3ab72a5..7108bba 100644 --- a/internal/controller/pool/reconciler.go +++ b/internal/controller/pool/reconciler.go @@ -4,7 +4,7 @@ import ( "errors" "time" - "github.com/proxy-pool/proxy-pool/internal/domain/upstream" + "proxy-pool/internal/domain/upstream" ) var ErrInvalidReconcilePolicy = errors.New("invalid pool reconcile policy") diff --git a/internal/controller/pool/reconciler_test.go b/internal/controller/pool/reconciler_test.go index 8da9aab..ca2c6bc 100644 --- a/internal/controller/pool/reconciler_test.go +++ b/internal/controller/pool/reconciler_test.go @@ -4,8 +4,8 @@ import ( "testing" "time" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" - "github.com/proxy-pool/proxy-pool/internal/domain/upstream" + proxyDomain "proxy-pool/internal/domain/proxy" + "proxy-pool/internal/domain/upstream" ) func TestPoolReconcilerSignalsOnlyWhenSlotsAreLowAndWholeFetchFits(t *testing.T) { diff --git a/internal/controller/provider/ports.go b/internal/controller/provider/ports.go index 0c88eb3..c88d936 100644 --- a/internal/controller/provider/ports.go +++ b/internal/controller/provider/ports.go @@ -4,8 +4,9 @@ import ( "context" "time" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" - "github.com/proxy-pool/proxy-pool/internal/domain/upstream" + "proxy-pool/internal/domain/activitypool" + proxyDomain "proxy-pool/internal/domain/proxy" + "proxy-pool/internal/domain/upstream" ) type FetchResponse struct { @@ -31,13 +32,6 @@ type Parser interface { Parse(context.Context, []byte) ([]proxyDomain.Proxy, error) } -// CandidateSink owns deduplication, is concurrency-safe, and returns the new -// count. On error it must not retain candidates; a successful count must be -// between zero and len(candidates). -type CandidateSink interface { - Add(context.Context, string, []proxyDomain.Proxy) (int, error) -} - type Result struct { UpstreamID string Class upstream.FetchClass @@ -53,9 +47,9 @@ type ResultRecorder interface { } type Ports struct { - Adapter ProviderAdapter - Parser Parser - Candidates CandidateSink - Results ResultRecorder - Capacity upstream.FetchCapacity + Adapter ProviderAdapter + Parser Parser + Activity activitypool.Upserter + Results ResultRecorder + Capacity upstream.FetchCapacity } diff --git a/internal/controller/provider/reconciler.go b/internal/controller/provider/reconciler.go index b3ee354..1514cd2 100644 --- a/internal/controller/provider/reconciler.go +++ b/internal/controller/provider/reconciler.go @@ -8,17 +8,20 @@ import ( "sync" "time" - "github.com/proxy-pool/proxy-pool/internal/domain/upstream" - "github.com/proxy-pool/proxy-pool/internal/platform/coalesce" + "proxy-pool/internal/domain/activitypool" + "proxy-pool/internal/domain/upstream" + "proxy-pool/internal/platform/coalesce" ) type Config struct { - UpstreamID string - RequestInterval time.Duration - Timeout time.Duration - MaxAttempts int - MaxInFlight int - Retry RetryConfig + UpstreamID string + RequestInterval time.Duration + Timeout time.Duration + MaxAttempts int + MaxInFlight int + TTL time.Duration + AllocationSafetyMargin time.Duration + Retry RetryConfig } type RetryConfig struct { @@ -62,6 +65,10 @@ func NewReconciler(config Config, ports Ports, runtimes ...Runtime) (*Reconciler if config.RequestInterval < 0 || config.Timeout <= 0 || config.MaxAttempts <= 0 || config.MaxInFlight <= 0 { return nil, fmt.Errorf("new provider reconciler: fetch limits must be positive") } + if config.TTL < 0 || config.AllocationSafetyMargin < 0 || + (config.TTL > 0 && config.AllocationSafetyMargin >= config.TTL) { + return nil, fmt.Errorf("new provider reconciler: lifecycle settings are invalid") + } if config.Retry.Initial < 0 || config.Retry.Max < 0 || config.Retry.Jitter < 0 || config.Retry.Jitter > 100 { return nil, fmt.Errorf("new provider reconciler: retry settings are invalid") } @@ -71,7 +78,7 @@ func NewReconciler(config Config, ports Ports, runtimes ...Runtime) (*Reconciler if config.Retry.Max > 0 && config.Retry.Initial > config.Retry.Max { return nil, fmt.Errorf("new provider reconciler: retry initial delay exceeds maximum") } - if ports.Adapter == nil || ports.Parser == nil || ports.Candidates == nil || + if ports.Adapter == nil || ports.Parser == nil || ports.Activity == nil || ports.Results == nil || ports.Capacity == nil { return nil, fmt.Errorf("new provider reconciler: all ports are required") } @@ -202,7 +209,14 @@ func (r *Reconciler) fetchAttempt(ctx context.Context, attempt int) (FetchRespon if expected := permit.Expected(); expected < len(retained) { retained = retained[:expected] } - newCount, candidateErr = r.ports.Candidates.Add(callCtx, r.config.UpstreamID, retained) + upserted, err := r.ports.Activity.UpsertFetched(callCtx, r.config.UpstreamID, activitypool.FetchedBatch{ + ObservedAt: r.runtime.Clock.Now().UTC(), + ConfiguredTTL: r.config.TTL, + AllocationSafetyMargin: r.config.AllocationSafetyMargin, + Proxies: retained, + }) + candidateErr = err + newCount = upserted.Inserted } } if callErr == nil && parseErr == nil && candidateErr == nil { diff --git a/internal/controller/provider/reconciler_test.go b/internal/controller/provider/reconciler_test.go index 9314b43..b6dc41f 100644 --- a/internal/controller/provider/reconciler_test.go +++ b/internal/controller/provider/reconciler_test.go @@ -8,10 +8,47 @@ import ( "testing" "time" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" - "github.com/proxy-pool/proxy-pool/internal/domain/upstream" + "proxy-pool/internal/domain/activitypool" + proxyDomain "proxy-pool/internal/domain/proxy" + "proxy-pool/internal/domain/upstream" ) +func TestReconcilerWritesProviderTTLPolicyToEphemeralPool(t *testing.T) { + now := time.Date(2026, 7, 29, 12, 0, 0, 0, time.UTC) + clock := newFakeClock(now) + results := make(chan Result, 1) + batches := make(chan activitypool.FetchedBatch, 1) + ports := successfulPorts(func() {}, results) + ports.Activity = activitySinkFunc(func(_ context.Context, upstreamID string, batch activitypool.FetchedBatch) (activitypool.UpsertResult, error) { + if upstreamID != "provider-a" { + t.Errorf("upstream ID = %q, want provider-a", upstreamID) + } + batches <- batch + return activitypool.UpsertResult{Accepted: 1, Inserted: 1}, nil + }) + reconciler, err := NewReconciler(Config{ + UpstreamID: "provider-a", + Timeout: time.Second, + MaxAttempts: 1, + MaxInFlight: 1, + TTL: 30 * time.Second, + AllocationSafetyMargin: 3 * time.Second, + }, ports, Runtime{Clock: clock}) + if err != nil { + t.Fatalf("NewReconciler(): %v", err) + } + + result := runSingleReconcile(t, reconciler, results) + if result.ValidCount != 1 || result.NewCount != 1 { + t.Fatalf("result = %+v, want valid=1 new=1", result) + } + batch := <-batches + if !batch.ObservedAt.Equal(now) || batch.ConfiguredTTL != 30*time.Second || + batch.AllocationSafetyMargin != 3*time.Second { + t.Fatalf("activity batch = %+v", batch) + } +} + func TestReconcilerCoalescesConcurrentNotifications(t *testing.T) { var calls atomic.Int64 result := make(chan Result, 1) @@ -28,8 +65,8 @@ func TestReconcilerCoalescesConcurrentNotifications(t *testing.T) { Parser: parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) { return []proxyDomain.Proxy{{ID: "proxy-1"}}, nil }), - Candidates: candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) { - return 1, nil + Activity: activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) { + return activitypool.UpsertResult{Accepted: 1, Inserted: 1}, nil }), Results: resultRecorderFunc(func(got Result) { result <- got }), Capacity: unlimitedFetchCapacity{}, @@ -122,8 +159,8 @@ func TestReconcilerRetriesErrorsWithExponentialBackoffAndJitter(t *testing.T) { } return FetchResponse{Body: []byte("fixture")}, nil }) - ports.Candidates = candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) { - return 0, nil + ports.Activity = activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) { + return activitypool.UpsertResult{Accepted: 1}, nil }) reconciler, err := NewReconciler(Config{ UpstreamID: "provider-a", @@ -196,8 +233,8 @@ func TestReconcilerClassifiesFetchResults(t *testing.T) { Parser: parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) { return make([]proxyDomain.Proxy, tt.candidates), tt.parseErr }), - Candidates: candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) { - return tt.newCount, nil + Activity: activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) { + return activitypool.UpsertResult{Accepted: tt.candidates, Inserted: tt.newCount}, nil }), Results: resultRecorderFunc(func(got Result) { results <- got }), Capacity: unlimitedFetchCapacity{}, @@ -325,9 +362,9 @@ func TestReconcilerAppliesAttemptTimeoutToEveryPort(t *testing.T) { assertDeadline(ctx, "Parser.Parse") return []proxyDomain.Proxy{{ID: "proxy-1"}}, nil }) - ports.Candidates = candidateSinkFunc(func(ctx context.Context, _ string, _ []proxyDomain.Proxy) (int, error) { - assertDeadline(ctx, "CandidateSink.Add") - return 1, nil + ports.Activity = activitySinkFunc(func(ctx context.Context, _ string, _ activitypool.FetchedBatch) (activitypool.UpsertResult, error) { + assertDeadline(ctx, "ActivitySink.UpsertFetched") + return activitypool.UpsertResult{Accepted: 1, Inserted: 1}, nil }) reconciler, err := NewReconciler(Config{ UpstreamID: "provider-a", @@ -493,14 +530,14 @@ func TestReconcilerUsesConfiguredMaxInFlight(t *testing.T) { } } -func TestReconcilerDoesNotRefetchWhenCandidateSinkFails(t *testing.T) { +func TestReconcilerDoesNotRefetchWhenActivitySinkFails(t *testing.T) { clock := newFakeClock(time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC)) sleeper := &fakeSleeper{clock: clock} results := make(chan Result, 3) var calls atomic.Int64 ports := successfulPorts(func() { calls.Add(1) }, results) - ports.Candidates = candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) { - return 0, errors.New("candidate store unavailable") + ports.Activity = activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) { + return activitypool.UpsertResult{}, errors.New("activity pool unavailable") }) reconciler, err := NewReconciler(Config{ UpstreamID: "provider-a", @@ -597,11 +634,11 @@ func TestReconcilerCompletesFetchPermitWithFetchedAndRetainedCounts(t *testing.T ports.Parser = parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) { return []proxyDomain.Proxy{{ID: "p1"}, {ID: "p2"}, {ID: "p3"}}, nil }) - ports.Candidates = candidateSinkFunc(func(_ context.Context, _ string, candidates []proxyDomain.Proxy) (int, error) { - if len(candidates) != 2 { - t.Errorf("CandidateSink candidates = %d, want permit limit 2", len(candidates)) + ports.Activity = activitySinkFunc(func(_ context.Context, _ string, batch activitypool.FetchedBatch) (activitypool.UpsertResult, error) { + if len(batch.Proxies) != 2 { + t.Errorf("activity batch proxies = %d, want permit limit 2", len(batch.Proxies)) } - return 1, nil + return activitypool.UpsertResult{Accepted: 2, Inserted: 1}, nil }) ports.Capacity = fetchCapacityFunc(func(string) (upstream.FetchPermit, bool, error) { return &recordingFetchPermit{expected: 2, completed: completed}, true, nil @@ -720,8 +757,8 @@ func successfulPorts(onFetch func(), results chan<- Result) Ports { Parser: parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) { return []proxyDomain.Proxy{{ID: "proxy-1"}}, nil }), - Candidates: candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) { - return 1, nil + Activity: activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) { + return activitypool.UpsertResult{Accepted: 1, Inserted: 1}, nil }), Results: resultRecorderFunc(func(got Result) { results <- got }), Capacity: unlimitedFetchCapacity{}, @@ -796,10 +833,10 @@ func (f parserFunc) Parse(ctx context.Context, body []byte) ([]proxyDomain.Proxy return f(ctx, body) } -type candidateSinkFunc func(context.Context, string, []proxyDomain.Proxy) (int, error) +type activitySinkFunc func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) -func (f candidateSinkFunc) Add(ctx context.Context, upstreamID string, candidates []proxyDomain.Proxy) (int, error) { - return f(ctx, upstreamID, candidates) +func (f activitySinkFunc) UpsertFetched(ctx context.Context, upstreamID string, batch activitypool.FetchedBatch) (activitypool.UpsertResult, error) { + return f(ctx, upstreamID, batch) } type fetchCapacityFunc func(string) (upstream.FetchPermit, bool, error) diff --git a/internal/controller/runtime/runtime.go b/internal/controller/runtime/runtime.go index 608a975..013eab7 100644 --- a/internal/controller/runtime/runtime.go +++ b/internal/controller/runtime/runtime.go @@ -8,11 +8,11 @@ import ( "net/http" "strings" - "github.com/proxy-pool/proxy-pool/internal/config" - "github.com/proxy-pool/proxy-pool/internal/controller/admin" - "github.com/proxy-pool/proxy-pool/internal/controller/distribution" - "github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity" - "github.com/proxy-pool/proxy-pool/internal/platform/httpserver" + "proxy-pool/internal/config" + "proxy-pool/internal/controller/admin" + "proxy-pool/internal/controller/distribution" + "proxy-pool/internal/platform/httpsecurity" + "proxy-pool/internal/platform/httpserver" ) const ( diff --git a/internal/controller/runtime/runtime_test.go b/internal/controller/runtime/runtime_test.go index 754a14c..79bf936 100644 --- a/internal/controller/runtime/runtime_test.go +++ b/internal/controller/runtime/runtime_test.go @@ -10,10 +10,10 @@ import ( "testing" "time" - "github.com/proxy-pool/proxy-pool/internal/config" - "github.com/proxy-pool/proxy-pool/internal/controller/admin" - controllerExtraction "github.com/proxy-pool/proxy-pool/internal/controller/extraction" - "github.com/proxy-pool/proxy-pool/internal/platform/httpserver" + "proxy-pool/internal/config" + "proxy-pool/internal/controller/admin" + controllerExtraction "proxy-pool/internal/controller/extraction" + "proxy-pool/internal/platform/httpserver" ) func TestRuntimeServesDistributionAndAdminOnIndependentListeners(t *testing.T) { diff --git a/internal/domain/activitypool/pool.go b/internal/domain/activitypool/pool.go new file mode 100644 index 0000000..9ea9cd3 --- /dev/null +++ b/internal/domain/activitypool/pool.go @@ -0,0 +1,590 @@ +package activitypool + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "sort" + "sync" + "time" + + extractionDomain "proxy-pool/internal/domain/extraction" + ownershipDomain "proxy-pool/internal/domain/ownership" + proxyDomain "proxy-pool/internal/domain/proxy" +) + +const defaultIdempotencyTTL = 5 * time.Minute + +var ErrInvalidBatch = errors.New("invalid activity pool batch") + +// FetchedBatch describes one ephemeral provider response. Proxies without a +// usable expiry are dropped because this pool is intentionally rebuildable. +type FetchedBatch struct { + ObservedAt time.Time + ConfiguredTTL time.Duration + AllocationSafetyMargin time.Duration + Proxies []proxyDomain.Proxy +} + +type UpsertResult struct { + Accepted int + Inserted int + Refreshed int + Dropped int +} + +type Upserter interface { + // A proxy unique key has one incumbent upstream for its current lifetime. + // Duplicates from other upstreams must not replace that source lifecycle. + UpsertFetched(context.Context, string, FetchedBatch) (UpsertResult, error) +} + +type Entry struct { + Proxy proxyDomain.Proxy + UsableUntil time.Time + OwnerWorkerID string + State proxyDomain.State +} + +type MemoryPool struct { + mu sync.Mutex + + entries map[string]Entry + keyByID map[string]string + idempotent map[string]idempotencyEntry + ownership map[string]ownershipDomain.Assignment + nextEpoch uint64 +} + +type idempotencyEntry struct { + command extractionDomain.Command + result extractionDomain.Result + expiresAt time.Time +} + +var ( + _ Upserter = (*MemoryPool)(nil) + _ extractionDomain.Store = (*MemoryPool)(nil) + _ ownershipDomain.Repository = (*MemoryPool)(nil) +) + +func NewMemoryPool() *MemoryPool { + return &MemoryPool{ + entries: make(map[string]Entry), + keyByID: make(map[string]string), + idempotent: make(map[string]idempotencyEntry), + ownership: make(map[string]ownershipDomain.Assignment), + } +} + +func (p *MemoryPool) UpsertFetched(ctx context.Context, upstreamID string, batch FetchedBatch) (UpsertResult, error) { + var result UpsertResult + if err := ctx.Err(); err != nil { + return result, err + } + if p == nil || upstreamID == "" || batch.ObservedAt.IsZero() || batch.ConfiguredTTL < 0 || + batch.AllocationSafetyMargin < 0 || + (batch.ConfiguredTTL > 0 && batch.AllocationSafetyMargin >= batch.ConfiguredTTL) { + return result, ErrInvalidBatch + } + for _, candidate := range batch.Proxies { + if candidate.SourceUpstream != "" && candidate.SourceUpstream != upstreamID { + return result, ErrInvalidBatch + } + } + + p.mu.Lock() + defer p.mu.Unlock() + if err := ctx.Err(); err != nil { + return result, err + } + p.purgeExpiredLocked(batch.ObservedAt) + seenIDs := make(map[string]string, len(batch.Proxies)) + for _, candidate := range batch.Proxies { + if !validProxyIdentity(candidate) { + continue + } + key := candidate.UniqueKey() + proxyID := candidate.ID + if proxyID == "" { + proxyID = stableProxyID(key) + } + if existingKey, exists := p.keyByID[proxyID]; exists && existingKey != key { + return UpsertResult{}, ErrInvalidBatch + } + if existingKey, exists := seenIDs[proxyID]; exists && existingKey != key { + return UpsertResult{}, ErrInvalidBatch + } + seenIDs[proxyID] = key + } + + for _, candidate := range batch.Proxies { + if !validProxyIdentity(candidate) { + result.Dropped++ + continue + } + candidate.SourceUpstream = upstreamID + expiresAt := proxyDomain.EffectiveExpiry(batch.ObservedAt, candidate.ExpiresAt, 0, batch.ConfiguredTTL) + if expiresAt == nil { + result.Dropped++ + continue + } + usableUntil := expiresAt.Add(-batch.AllocationSafetyMargin) + if !usableUntil.After(batch.ObservedAt) { + result.Dropped++ + continue + } + candidate.ExpiresAt = expiresAt + candidate.UsableUntil = &usableUntil + if candidate.CreatedAt.IsZero() { + candidate.CreatedAt = batch.ObservedAt.UTC() + } + if candidate.State == "" { + candidate.State = proxyDomain.StateFetched + } + key := candidate.UniqueKey() + result.Accepted++ + + if current, exists := p.entries[key]; exists { + result.Refreshed++ + if current.State == proxyDomain.StateExtracted || current.Proxy.SourceUpstream != upstreamID { + continue + } + candidate.ID = current.Proxy.ID + candidate.CreatedAt = current.Proxy.CreatedAt + candidate.State = current.State + candidate.LastCheckedAt = current.Proxy.LastCheckedAt + candidate.LastSuccessAt = current.Proxy.LastSuccessAt + candidate.Latency = current.Proxy.Latency + p.entries[key] = Entry{ + Proxy: cloneProxy(candidate), + UsableUntil: usableUntil, + OwnerWorkerID: current.OwnerWorkerID, + State: current.State, + } + continue + } + + if candidate.ID == "" { + candidate.ID = stableProxyID(key) + } + p.entries[key] = Entry{ + Proxy: cloneProxy(candidate), + UsableUntil: usableUntil, + State: candidate.State, + } + p.keyByID[candidate.ID] = key + result.Inserted++ + } + return result, nil +} + +func (p *MemoryPool) Snapshot(now time.Time) []Entry { + if p == nil { + return nil + } + p.mu.Lock() + defer p.mu.Unlock() + p.purgeExpiredLocked(now) + entries := make([]Entry, 0, len(p.entries)) + for _, entry := range p.entries { + entry.Proxy = cloneProxy(entry.Proxy) + entries = append(entries, entry) + } + sort.Slice(entries, func(i, j int) bool { return entries[i].Proxy.ID < entries[j].Proxy.ID }) + return entries +} + +func (p *MemoryPool) PurgeExpired(now time.Time) int { + if p == nil { + return 0 + } + p.mu.Lock() + defer p.mu.Unlock() + return p.purgeExpiredLocked(now) +} + +func (p *MemoryPool) Extract(ctx context.Context, command extractionDomain.Command) (extractionDomain.Result, error) { + result := extractionDomain.Result{Requested: command.Requested} + if err := ctx.Err(); err != nil { + return result, err + } + if p == nil || command.Now.IsZero() || command.Requested < 0 || command.ReserveForGateway < 0 || + command.MinRemainingTTL < 0 || command.MaxHealthCheckAge < 0 || command.IdempotencyTTL < 0 || + (command.IdempotencyKey != "" && command.ClientID == "") || + (command.Fulfillment != extractionDomain.Partial && command.Fulfillment != extractionDomain.AllOrNothing) { + return result, extractionDomain.ErrInvalidCommand + } + + p.mu.Lock() + defer p.mu.Unlock() + if err := ctx.Err(); err != nil { + return result, err + } + p.purgeExpiredLocked(command.Now) + + idempotencyKey := command.ClientID + "\x00" + command.IdempotencyKey + if command.IdempotencyKey != "" { + if committed, ok := p.idempotent[idempotencyKey]; ok { + if !sameIdempotentRequest(committed.command, command) { + return result, extractionDomain.ErrIdempotencyConflict + } + return cloneResult(committed.result), nil + } + } + if command.Requested == 0 { + return result, nil + } + + eligible := make([]string, 0, len(p.entries)) + for key, entry := range p.entries { + if eligibleForExtraction(entry, command) { + eligible = append(eligible, key) + } + } + sort.Slice(eligible, func(i, j int) bool { + return p.entries[eligible[i]].UsableUntil.After(p.entries[eligible[j]].UsableUntil) + }) + available := len(eligible) - command.ReserveForGateway + if available < 0 { + available = 0 + } + if command.Fulfillment == extractionDomain.AllOrNothing && available < command.Requested { + return result, extractionDomain.ErrInsufficientProxies + } + count := command.Requested + if count > available { + count = available + } + for _, key := range eligible[:count] { + entry := p.entries[key] + entry.State = proxyDomain.StateExtracted + entry.Proxy.State = proxyDomain.StateExtracted + p.entries[key] = entry + result.Items = append(result.Items, extractionCandidate(entry)) + } + result.Returned = len(result.Items) + if result.Returned > 0 { + result.ExtractedAt = command.Now.UTC() + } + if command.IdempotencyKey != "" { + expiresAt := command.Now.Add(idempotencyTTL(command.IdempotencyTTL)) + for _, item := range result.Items { + if !item.ExpiresAt.IsZero() && item.ExpiresAt.Before(expiresAt) { + expiresAt = item.ExpiresAt + } + } + if expiresAt.After(command.Now) { + p.idempotent[idempotencyKey] = idempotencyEntry{ + command: cloneCommand(command), result: cloneResult(result), expiresAt: expiresAt, + } + } + } + return result, nil +} + +func (p *MemoryPool) Assign(now time.Time, proxyID, workerID string, ttl time.Duration) (ownershipDomain.Assignment, error) { + if p == nil || now.IsZero() || proxyID == "" || workerID == "" || ttl <= 0 { + return ownershipDomain.Assignment{}, ownershipDomain.ErrInvalidOwnership + } + p.mu.Lock() + defer p.mu.Unlock() + p.purgeExpiredLocked(now) + if current, exists := p.ownership[proxyID]; exists { + if current.ExpiresAt.After(now) { + return ownershipDomain.Assignment{}, ownershipDomain.ErrAlreadyOwned + } + if entry, ok := p.entryByIDLocked(proxyID); ok && entry.OwnerWorkerID == current.WorkerID { + entry.OwnerWorkerID = "" + p.setEntryByIDLocked(proxyID, entry) + } + delete(p.ownership, proxyID) + } + entry, ok := p.entryByIDLocked(proxyID) + if !ok || entry.State != proxyDomain.StateAvailable || entry.OwnerWorkerID != "" || !entry.UsableUntil.After(now) { + return ownershipDomain.Assignment{}, ownershipDomain.ErrOwnershipUnavailable + } + p.nextEpoch++ + expiresAt := minTime(now.UTC().Add(ttl), entry.UsableUntil) + assignment := ownershipDomain.Assignment{ + ProxyID: proxyID, WorkerID: workerID, Epoch: p.nextEpoch, Version: 1, ExpiresAt: expiresAt, + } + entry.OwnerWorkerID = workerID + p.setEntryByIDLocked(proxyID, entry) + p.ownership[proxyID] = assignment + return assignment, nil +} + +func (p *MemoryPool) Renew(now time.Time, proxyID, workerID string, epoch uint64, ttl time.Duration) (ownershipDomain.Assignment, error) { + if p == nil || now.IsZero() || ttl <= 0 { + return ownershipDomain.Assignment{}, ownershipDomain.ErrInvalidOwnership + } + p.mu.Lock() + defer p.mu.Unlock() + p.purgeExpiredLocked(now) + assignment, ok := p.ownership[proxyID] + if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch || !assignment.ExpiresAt.After(now) { + return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment + } + entry, ok := p.entryByIDLocked(proxyID) + if !ok { + return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment + } + assignment.ExpiresAt = minTime(now.UTC().Add(ttl), entry.UsableUntil) + assignment.Version++ + p.ownership[proxyID] = assignment + return assignment, nil +} + +func (p *MemoryPool) BeginDrain(proxyID, workerID string, epoch uint64) (ownershipDomain.Assignment, error) { + if p == nil { + return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment + } + p.mu.Lock() + defer p.mu.Unlock() + assignment, ok := p.ownership[proxyID] + if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch { + return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment + } + if !assignment.Draining { + assignment.Draining = true + assignment.Version++ + p.ownership[proxyID] = assignment + } + return assignment, nil +} + +func (p *MemoryPool) AcknowledgeDrain(proxyID, workerID string, epoch uint64, active, reserved int64) error { + if p == nil || active < 0 || reserved < 0 { + return ownershipDomain.ErrInvalidOwnership + } + p.mu.Lock() + defer p.mu.Unlock() + assignment, ok := p.ownership[proxyID] + if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch { + return ownershipDomain.ErrStaleAssignment + } + if !assignment.Draining { + return ownershipDomain.ErrNotDraining + } + if active > 0 || reserved > 0 { + return ownershipDomain.ErrDrainNotReady + } + if entry, exists := p.entryByIDLocked(proxyID); exists && entry.OwnerWorkerID == workerID { + entry.OwnerWorkerID = "" + p.setEntryByIDLocked(proxyID, entry) + } + delete(p.ownership, proxyID) + return nil +} + +func (p *MemoryPool) Get(proxyID string) (ownershipDomain.Assignment, bool) { + if p == nil { + return ownershipDomain.Assignment{}, false + } + p.mu.Lock() + defer p.mu.Unlock() + assignment, ok := p.ownership[proxyID] + return assignment, ok +} + +func (p *MemoryPool) Expire(now time.Time) []ownershipDomain.Assignment { + if p == nil { + return nil + } + p.mu.Lock() + defer p.mu.Unlock() + expired := make([]ownershipDomain.Assignment, 0) + for proxyID, assignment := range p.ownership { + entry, exists := p.entryByIDLocked(proxyID) + if exists && assignment.ExpiresAt.After(now) { + continue + } + if exists && entry.OwnerWorkerID == assignment.WorkerID { + entry.OwnerWorkerID = "" + p.setEntryByIDLocked(proxyID, entry) + } + expired = append(expired, assignment) + delete(p.ownership, proxyID) + } + p.purgeExpiredLocked(now) + sort.Slice(expired, func(i, j int) bool { return expired[i].ProxyID < expired[j].ProxyID }) + return expired +} + +func (p *MemoryPool) purgeExpiredLocked(now time.Time) int { + removed := 0 + for key, entry := range p.entries { + if entry.Proxy.ExpiresAt == nil || entry.Proxy.ExpiresAt.After(now) { + continue + } + delete(p.ownership, entry.Proxy.ID) + delete(p.keyByID, entry.Proxy.ID) + delete(p.entries, key) + removed++ + } + for key, entry := range p.idempotent { + if !entry.expiresAt.After(now) { + delete(p.idempotent, key) + } + } + return removed +} + +func (p *MemoryPool) entryByIDLocked(proxyID string) (Entry, bool) { + key, ok := p.keyByID[proxyID] + if !ok { + return Entry{}, false + } + entry, ok := p.entries[key] + return entry, ok +} + +func (p *MemoryPool) setEntryByIDLocked(proxyID string, entry Entry) { + if key, ok := p.keyByID[proxyID]; ok { + p.entries[key] = entry + } +} + +func eligibleForExtraction(entry Entry, command extractionDomain.Command) bool { + if entry.State != proxyDomain.StateAvailable || entry.OwnerWorkerID != "" || !entry.UsableUntil.After(command.Now) { + return false + } + if entry.Proxy.ExpiresAt != nil && entry.Proxy.ExpiresAt.Sub(command.Now) < command.MinRemainingTTL { + return false + } + if command.MaxHealthCheckAge > 0 { + if entry.Proxy.LastCheckedAt == nil || command.Now.Sub(*entry.Proxy.LastCheckedAt) > command.MaxHealthCheckAge { + return false + } + } + return matches(command.Protocols, string(entry.Proxy.Scheme)) && + matches(command.Regions, entry.Proxy.Tags["region"]) && + matches(command.Carriers, entry.Proxy.Tags["carrier"]) && + matches(command.Upstreams, entry.Proxy.SourceUpstream) +} + +func extractionCandidate(entry Entry) extractionDomain.Candidate { + expiresAt := time.Time{} + if entry.Proxy.ExpiresAt != nil { + expiresAt = *entry.Proxy.ExpiresAt + } + checkedAt := time.Time{} + if entry.Proxy.LastCheckedAt != nil { + checkedAt = *entry.Proxy.LastCheckedAt + } + return extractionDomain.Candidate{ + ID: entry.Proxy.ID, Protocol: string(entry.Proxy.Scheme), Host: entry.Proxy.Host, + Port: entry.Proxy.Port, Username: entry.Proxy.Username, Region: entry.Proxy.Tags["region"], + Carrier: entry.Proxy.Tags["carrier"], Upstream: entry.Proxy.SourceUpstream, + OwnerWorkerID: entry.OwnerWorkerID, State: extractionDomain.Available, + ExpiresAt: expiresAt, LastCheckedAt: checkedAt, + } +} + +func stableProxyID(key string) string { + digest := sha256.Sum256([]byte(key)) + return "px_" + hex.EncodeToString(digest[:12]) +} + +func validProxyIdentity(candidate proxyDomain.Proxy) bool { + if candidate.Host == "" || candidate.Port == 0 { + return false + } + switch candidate.Scheme { + case proxyDomain.SchemeHTTP, proxyDomain.SchemeHTTPS, proxyDomain.SchemeSOCKS5: + return true + default: + return false + } +} + +func cloneProxy(candidate proxyDomain.Proxy) proxyDomain.Proxy { + if candidate.ExpiresAt != nil { + value := *candidate.ExpiresAt + candidate.ExpiresAt = &value + } + if candidate.UsableUntil != nil { + value := *candidate.UsableUntil + candidate.UsableUntil = &value + } + if candidate.LastCheckedAt != nil { + value := *candidate.LastCheckedAt + candidate.LastCheckedAt = &value + } + if candidate.LastSuccessAt != nil { + value := *candidate.LastSuccessAt + candidate.LastSuccessAt = &value + } + if candidate.Tags != nil { + tags := make(map[string]string, len(candidate.Tags)) + for key, value := range candidate.Tags { + tags[key] = value + } + candidate.Tags = tags + } + return candidate +} + +func cloneResult(result extractionDomain.Result) extractionDomain.Result { + result.Items = append([]extractionDomain.Candidate(nil), result.Items...) + return result +} + +func cloneCommand(command extractionDomain.Command) extractionDomain.Command { + command.Protocols = append([]string(nil), command.Protocols...) + command.Regions = append([]string(nil), command.Regions...) + command.Carriers = append([]string(nil), command.Carriers...) + command.Upstreams = append([]string(nil), command.Upstreams...) + return command +} + +func sameIdempotentRequest(left, right extractionDomain.Command) bool { + return left.Requested == right.Requested && left.Fulfillment == right.Fulfillment && + equalSet(left.Protocols, right.Protocols) && equalSet(left.Regions, right.Regions) && + equalSet(left.Carriers, right.Carriers) && equalSet(left.Upstreams, right.Upstreams) +} + +func equalSet(left, right []string) bool { + if len(left) != len(right) { + return false + } + counts := make(map[string]int, len(left)) + for _, value := range left { + counts[value]++ + } + for _, value := range right { + counts[value]-- + if counts[value] < 0 { + return false + } + } + return true +} + +func matches(allowed []string, value string) bool { + if len(allowed) == 0 { + return true + } + for _, candidate := range allowed { + if candidate == value { + return true + } + } + return false +} + +func idempotencyTTL(configured time.Duration) time.Duration { + if configured > 0 { + return configured + } + return defaultIdempotencyTTL +} + +func minTime(left, right time.Time) time.Time { + if left.Before(right) { + return left + } + return right +} diff --git a/internal/domain/activitypool/pool_test.go b/internal/domain/activitypool/pool_test.go new file mode 100644 index 0000000..2d480af --- /dev/null +++ b/internal/domain/activitypool/pool_test.go @@ -0,0 +1,359 @@ +package activitypool + +import ( + "context" + "errors" + "sync" + "testing" + "time" + + extractionDomain "proxy-pool/internal/domain/extraction" + ownershipDomain "proxy-pool/internal/domain/ownership" + proxyDomain "proxy-pool/internal/domain/proxy" +) + +func TestMemoryPoolUpsertAppliesProviderTTLAndRefreshesWithoutGrowth(t *testing.T) { + now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC) + pool := NewMemoryPool() + batch := FetchedBatch{ + ObservedAt: now, + ConfiguredTTL: 30 * time.Second, + AllocationSafetyMargin: 3 * time.Second, + Proxies: []proxyDomain.Proxy{{ + Scheme: proxyDomain.SchemeHTTP, + Host: "192.0.2.10", + Port: 8080, + State: proxyDomain.StateAvailable, + }}, + } + + first, err := pool.UpsertFetched(context.Background(), "provider-a", batch) + if err != nil { + t.Fatalf("UpsertFetched(first): %v", err) + } + if first.Inserted != 1 || first.Refreshed != 0 || first.Dropped != 0 { + t.Fatalf("first result = %+v", first) + } + + entries := pool.Snapshot(now) + if len(entries) != 1 { + t.Fatalf("Snapshot() entries = %d, want 1", len(entries)) + } + if !entries[0].Proxy.ExpiresAt.Equal(now.Add(30 * time.Second)) { + t.Fatalf("ExpiresAt = %v, want %v", entries[0].Proxy.ExpiresAt, now.Add(30*time.Second)) + } + if !entries[0].UsableUntil.Equal(now.Add(27 * time.Second)) { + t.Fatalf("UsableUntil = %v, want %v", entries[0].UsableUntil, now.Add(27*time.Second)) + } + if entries[0].Proxy.UsableUntil == nil || !entries[0].Proxy.UsableUntil.Equal(now.Add(27*time.Second)) { + t.Fatalf("Proxy.UsableUntil = %v, want %v", entries[0].Proxy.UsableUntil, now.Add(27*time.Second)) + } + + batch.ObservedAt = now.Add(10 * time.Second) + second, err := pool.UpsertFetched(context.Background(), "provider-a", batch) + if err != nil { + t.Fatalf("UpsertFetched(refresh): %v", err) + } + if second.Inserted != 0 || second.Refreshed != 1 || len(pool.Snapshot(batch.ObservedAt)) != 1 { + t.Fatalf("refresh result = %+v, entries=%d", second, len(pool.Snapshot(batch.ObservedAt))) + } + if got := pool.Snapshot(batch.ObservedAt)[0].Proxy.ExpiresAt; !got.Equal(now.Add(40 * time.Second)) { + t.Fatalf("refreshed ExpiresAt = %v, want %v", got, now.Add(40*time.Second)) + } +} + +func TestMemoryPoolCrossProviderDuplicateDoesNotReplaceSourceLifecycle(t *testing.T) { + now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC) + pool := NewMemoryPool() + proxy := proxyDomain.Proxy{ + Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8080, + State: proxyDomain.StateAvailable, + } + first, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{ + ObservedAt: now, ConfiguredTTL: 30 * time.Second, + AllocationSafetyMargin: 3 * time.Second, Proxies: []proxyDomain.Proxy{proxy}, + }) + if err != nil || first.Inserted != 1 { + t.Fatalf("UpsertFetched(provider-a) = %+v, %v", first, err) + } + + duplicate, err := pool.UpsertFetched(context.Background(), "provider-b", FetchedBatch{ + ObservedAt: now.Add(time.Second), ConfiguredTTL: 5 * time.Minute, + AllocationSafetyMargin: 10 * time.Second, Proxies: []proxyDomain.Proxy{proxy}, + }) + if err != nil || duplicate.Inserted != 0 || duplicate.Refreshed != 1 { + t.Fatalf("UpsertFetched(provider-b) = %+v, %v", duplicate, err) + } + + entries := pool.Snapshot(now.Add(time.Second)) + if len(entries) != 1 { + t.Fatalf("Snapshot() entries = %d, want 1", len(entries)) + } + entry := entries[0] + if entry.Proxy.SourceUpstream != "provider-a" { + t.Fatalf("SourceUpstream = %q, want provider-a", entry.Proxy.SourceUpstream) + } + if !entry.Proxy.ExpiresAt.Equal(now.Add(30*time.Second)) || + !entry.UsableUntil.Equal(now.Add(27*time.Second)) { + t.Fatalf("lifecycle = expires %v usable %v, want provider-a lifecycle", entry.Proxy.ExpiresAt, entry.UsableUntil) + } + + afterExpiry, err := pool.UpsertFetched(context.Background(), "provider-b", FetchedBatch{ + ObservedAt: now.Add(31 * time.Second), ConfiguredTTL: 5 * time.Minute, + AllocationSafetyMargin: 10 * time.Second, Proxies: []proxyDomain.Proxy{proxy}, + }) + if err != nil || afterExpiry.Inserted != 1 { + t.Fatalf("UpsertFetched(provider-b after expiry) = %+v, %v", afterExpiry, err) + } + replacement := pool.Snapshot(now.Add(31 * time.Second))[0] + if replacement.Proxy.SourceUpstream != "provider-b" { + t.Fatalf("replacement SourceUpstream = %q, want provider-b", replacement.Proxy.SourceUpstream) + } +} + +func TestMemoryPoolRefreshPreservesRuntimeStateAndHealth(t *testing.T) { + now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC) + checkedAt := now.Add(-time.Second) + pool := NewMemoryPool() + first, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{ + ObservedAt: now, ConfiguredTTL: 30 * time.Second, + Proxies: []proxyDomain.Proxy{{ + Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8080, + State: proxyDomain.StateAvailable, LastCheckedAt: &checkedAt, + }}, + }) + if err != nil || first.Inserted != 1 { + t.Fatalf("UpsertFetched(first) = %+v, %v", first, err) + } + initial := pool.Snapshot(now)[0] + + second, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{ + ObservedAt: now.Add(10 * time.Second), ConfiguredTTL: 30 * time.Second, + Proxies: []proxyDomain.Proxy{{ + Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8080, + State: proxyDomain.StateFetched, + }}, + }) + if err != nil || second.Refreshed != 1 { + t.Fatalf("UpsertFetched(refresh) = %+v, %v", second, err) + } + refreshed := pool.Snapshot(now.Add(10 * time.Second))[0] + if refreshed.Proxy.ID != initial.Proxy.ID || refreshed.State != proxyDomain.StateAvailable || + refreshed.Proxy.LastCheckedAt == nil || !refreshed.Proxy.LastCheckedAt.Equal(checkedAt) || + !refreshed.Proxy.CreatedAt.Equal(initial.Proxy.CreatedAt) { + t.Fatalf("refreshed entry lost runtime state: initial=%+v refreshed=%+v", initial, refreshed) + } +} + +func TestMemoryPoolDropsCandidatesWithoutUsableTTL(t *testing.T) { + now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC) + expired := now.Add(-time.Second) + tooShort := now.Add(2 * time.Second) + pool := NewMemoryPool() + + result, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{ + ObservedAt: now, + AllocationSafetyMargin: 3 * time.Second, + Proxies: []proxyDomain.Proxy{ + {Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8001}, + {Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.11", Port: 8002, ExpiresAt: &expired}, + {Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.12", Port: 8003, ExpiresAt: &tooShort}, + }, + }) + if err != nil { + t.Fatalf("UpsertFetched(): %v", err) + } + if result.Dropped != 3 || result.Accepted != 0 || len(pool.Snapshot(now)) != 0 { + t.Fatalf("result = %+v, entries=%d", result, len(pool.Snapshot(now))) + } +} + +func TestMemoryPoolRejectsInvalidBatchAtomically(t *testing.T) { + now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC) + pool := NewMemoryPool() + result, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{ + ObservedAt: now, + ConfiguredTTL: time.Minute, + Proxies: []proxyDomain.Proxy{ + {Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8001}, + {Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.11", Port: 8002, SourceUpstream: "provider-b"}, + }, + }) + if !errors.Is(err, ErrInvalidBatch) { + t.Fatalf("UpsertFetched() error = %v, want ErrInvalidBatch", err) + } + if result != (UpsertResult{}) || len(pool.Snapshot(now)) != 0 { + t.Fatalf("failed batch retained state: result=%+v entries=%+v", result, pool.Snapshot(now)) + } +} + +func TestMemoryPoolNeverExtractsProxyTwiceAndDoesNotWriteAuditRecords(t *testing.T) { + now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC) + pool := poolWithOneProxy(t, now) + + var wait sync.WaitGroup + results := make(chan string, 1000) + for range 1000 { + wait.Add(1) + go func() { + defer wait.Done() + result, err := pool.Extract(context.Background(), extractionDomain.Command{ + Requested: 1, + Fulfillment: extractionDomain.Partial, + Now: now, + MinRemainingTTL: 10 * time.Second, + MaxHealthCheckAge: time.Minute, + }) + if err != nil { + t.Errorf("Extract(): %v", err) + return + } + for _, item := range result.Items { + results <- item.ID + } + }() + } + wait.Wait() + close(results) + + count := 0 + for range results { + count++ + } + if count != 1 { + t.Fatalf("proxy extracted %d times, want exactly once", count) + } + if got := pool.Snapshot(now); len(got) != 1 || got[0].State != proxyDomain.StateExtracted { + t.Fatalf("post-extraction snapshot = %+v", got) + } +} + +func TestMemoryPoolIdempotencyIsBoundedByProxyExpiry(t *testing.T) { + now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC) + pool := poolWithOneProxy(t, now) + command := extractionDomain.Command{ + ClientID: "client-a", + IdempotencyKey: "idem-a", + IdempotencyTTL: 5 * time.Minute, + Requested: 1, + Fulfillment: extractionDomain.Partial, + Now: now, + MinRemainingTTL: 10 * time.Second, + MaxHealthCheckAge: time.Minute, + } + + first, err := pool.Extract(context.Background(), command) + if err != nil || first.Returned != 1 { + t.Fatalf("first Extract() = %+v, %v", first, err) + } + command.Now = now.Add(20 * time.Second) + replayed, err := pool.Extract(context.Background(), command) + if err != nil || replayed.Returned != 1 || replayed.Items[0].ID != first.Items[0].ID { + t.Fatalf("replayed Extract() = %+v, %v", replayed, err) + } + + if purged := pool.PurgeExpired(now.Add(31 * time.Second)); purged != 1 { + t.Fatalf("PurgeExpired() = %d, want 1", purged) + } + command.Now = now.Add(31 * time.Second) + afterExpiry, err := pool.Extract(context.Background(), command) + if err != nil || afterExpiry.Returned != 0 { + t.Fatalf("Extract(after expiry) = %+v, %v", afterExpiry, err) + } +} + +func TestMemoryPoolAllOrNothingDoesNotConsumePartialInventory(t *testing.T) { + now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC) + pool := poolWithOneProxy(t, now) + result, err := pool.Extract(context.Background(), extractionDomain.Command{ + Requested: 2, Fulfillment: extractionDomain.AllOrNothing, Now: now, + }) + if !errors.Is(err, extractionDomain.ErrInsufficientProxies) || result.Returned != 0 { + t.Fatalf("all-or-nothing Extract() = %+v, %v", result, err) + } + partial, err := pool.Extract(context.Background(), extractionDomain.Command{ + Requested: 1, Fulfillment: extractionDomain.Partial, Now: now, + }) + if err != nil || partial.Returned != 1 { + t.Fatalf("partial Extract() after rollback = %+v, %v", partial, err) + } +} + +func TestMemoryPoolAssignRecoversExpiredLeaseWithoutSeparateSweep(t *testing.T) { + now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC) + pool := poolWithOneProxy(t, now) + proxyID := pool.Snapshot(now)[0].Proxy.ID + first, err := pool.Assign(now, proxyID, "worker-a", 5*time.Second) + if err != nil { + t.Fatalf("Assign(first): %v", err) + } + second, err := pool.Assign(now.Add(5*time.Second), proxyID, "worker-b", 5*time.Second) + if err != nil { + t.Fatalf("Assign(after lease expiry): %v", err) + } + if second.Epoch <= first.Epoch || second.WorkerID != "worker-b" { + t.Fatalf("second assignment = %+v, first=%+v", second, first) + } +} + +func TestMemoryPoolMakesOwnershipAndExtractionMutuallyExclusive(t *testing.T) { + now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC) + for iteration := range 100 { + pool := poolWithOneProxy(t, now) + start := make(chan struct{}) + assigned := make(chan bool, 1) + extracted := make(chan bool, 1) + go func() { + <-start + _, err := pool.Assign(now, pool.Snapshot(now)[0].Proxy.ID, "worker-a", time.Minute) + if err != nil && !errors.Is(err, ownershipDomain.ErrOwnershipUnavailable) { + t.Errorf("Assign(): %v", err) + } + assigned <- err == nil + }() + go func() { + <-start + result, err := pool.Extract(context.Background(), extractionDomain.Command{ + ClientID: "client-a", Requested: 1, Fulfillment: extractionDomain.Partial, Now: now, + }) + if err != nil { + t.Errorf("Extract(): %v", err) + } + extracted <- result.Returned == 1 + }() + close(start) + + wins := 0 + if <-assigned { + wins++ + } + if <-extracted { + wins++ + } + if wins != 1 { + t.Fatalf("iteration %d winners = %d, want 1", iteration, wins) + } + } +} + +func poolWithOneProxy(t *testing.T, now time.Time) *MemoryPool { + t.Helper() + pool := NewMemoryPool() + checkedAt := now + result, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{ + ObservedAt: now, + ConfiguredTTL: 30 * time.Second, + AllocationSafetyMargin: 3 * time.Second, + Proxies: []proxyDomain.Proxy{{ + Scheme: proxyDomain.SchemeHTTP, + Host: "192.0.2.10", + Port: 8080, + State: proxyDomain.StateAvailable, + LastCheckedAt: &checkedAt, + }}, + }) + if err != nil || result.Inserted != 1 { + t.Fatalf("UpsertFetched() = %+v, %v", result, err) + } + return pool +} diff --git a/internal/domain/extraction/extraction.go b/internal/domain/extraction/extraction.go index 1aea894..f58caab 100644 --- a/internal/domain/extraction/extraction.go +++ b/internal/domain/extraction/extraction.go @@ -3,11 +3,7 @@ package extraction import ( "context" "errors" - "sort" - "sync" "time" - - ownershipDomain "github.com/proxy-pool/proxy-pool/internal/domain/ownership" ) type Fulfillment string @@ -52,6 +48,7 @@ type Command struct { ClientID string SourceIP string IdempotencyKey string + IdempotencyTTL time.Duration Requested int Fulfillment Fulfillment Now time.Time @@ -64,16 +61,6 @@ type Command struct { Upstreams []string } -type Record struct { - ProxyID string - ClientID string - SourceIP string - RequestID string - Upstream string - ExtractedAt time.Time - ExpiresAt time.Time -} - type Result struct { Requested int Returned int @@ -81,294 +68,8 @@ type Result struct { Items []Candidate } +// Store atomically consumes proxies from a bounded, ephemeral activity pool. +// Implementations must not require durable per-proxy extraction records. type Store interface { Extract(context.Context, Command) (Result, error) } - -type MemoryStore struct { - mu sync.Mutex - candidates map[string]Candidate - records []Record - idempotent map[string]idempotencyEntry - nextEpoch uint64 - ownership map[string]ownershipDomain.Assignment -} - -var _ ownershipDomain.Repository = (*MemoryStore)(nil) - -type idempotencyEntry struct { - command Command - result Result -} - -func NewMemoryStore(candidates []Candidate) *MemoryStore { - items := make(map[string]Candidate, len(candidates)) - for _, candidate := range candidates { - items[candidate.ID] = candidate - } - return &MemoryStore{ - candidates: items, - idempotent: make(map[string]idempotencyEntry), - ownership: make(map[string]ownershipDomain.Assignment), - } -} - -func (s *MemoryStore) Extract(ctx context.Context, command Command) (Result, error) { - result := Result{Requested: command.Requested} - if err := ctx.Err(); err != nil { - return result, err - } - if command.Requested < 0 || command.ReserveForGateway < 0 || - command.MinRemainingTTL < 0 || command.MaxHealthCheckAge < 0 || - (command.Fulfillment != Partial && command.Fulfillment != AllOrNothing) { - return result, ErrInvalidCommand - } - s.mu.Lock() - defer s.mu.Unlock() - if err := ctx.Err(); err != nil { - return result, err - } - - idempotencyKey := command.ClientID + "\x00" + command.IdempotencyKey - if command.IdempotencyKey != "" { - if committed, ok := s.idempotent[idempotencyKey]; ok { - if !sameIdempotentRequest(committed.command, command) { - return result, ErrIdempotencyConflict - } - return cloneResult(committed.result), nil - } - } - if command.Requested <= 0 { - return result, nil - } - eligible := make([]Candidate, 0, len(s.candidates)) - for _, candidate := range s.candidates { - if eligibleForExtraction(candidate, command) { - eligible = append(eligible, candidate) - } - } - sort.Slice(eligible, func(i, j int) bool { - return eligible[i].ExpiresAt.After(eligible[j].ExpiresAt) - }) - available := len(eligible) - command.ReserveForGateway - if available < 0 { - available = 0 - } - if command.Fulfillment == AllOrNothing && available < command.Requested { - return result, ErrInsufficientProxies - } - count := command.Requested - if count > available { - count = available - } - for i := 0; i < count; i++ { - candidate := eligible[i] - candidate.State = Extracted - s.candidates[candidate.ID] = candidate - result.Items = append(result.Items, candidate) - s.records = append(s.records, Record{ - ProxyID: candidate.ID, - ClientID: command.ClientID, - SourceIP: command.SourceIP, - RequestID: command.RequestID, - Upstream: candidate.Upstream, - ExtractedAt: command.Now, - ExpiresAt: candidate.ExpiresAt, - }) - } - result.Returned = len(result.Items) - if result.Returned > 0 { - result.ExtractedAt = command.Now - } - if command.IdempotencyKey != "" { - s.idempotent[idempotencyKey] = idempotencyEntry{ - command: cloneCommand(command), - result: cloneResult(result), - } - } - return result, nil -} - -func (s *MemoryStore) Assign(now time.Time, proxyID, workerID string, ttl time.Duration) (ownershipDomain.Assignment, error) { - if proxyID == "" || workerID == "" || ttl <= 0 { - return ownershipDomain.Assignment{}, ownershipDomain.ErrInvalidOwnership - } - s.mu.Lock() - defer s.mu.Unlock() - candidate, ok := s.candidates[proxyID] - if current, exists := s.ownership[proxyID]; exists && current.ExpiresAt.After(now) { - return ownershipDomain.Assignment{}, ownershipDomain.ErrAlreadyOwned - } else if exists { - if candidate.OwnerWorkerID == current.WorkerID { - candidate.OwnerWorkerID = "" - } - delete(s.ownership, proxyID) - } - if !ok || candidate.State != Available || candidate.OwnerWorkerID != "" { - return ownershipDomain.Assignment{}, ownershipDomain.ErrOwnershipUnavailable - } - s.nextEpoch++ - assignment := ownershipDomain.Assignment{ - ProxyID: proxyID, WorkerID: workerID, Epoch: s.nextEpoch, Version: 1, - ExpiresAt: now.UTC().Add(ttl), - } - candidate.OwnerWorkerID = workerID - s.candidates[proxyID] = candidate - s.ownership[proxyID] = assignment - return assignment, nil -} - -func (s *MemoryStore) Renew(now time.Time, proxyID, workerID string, epoch uint64, ttl time.Duration) (ownershipDomain.Assignment, error) { - if ttl <= 0 { - return ownershipDomain.Assignment{}, ownershipDomain.ErrInvalidOwnership - } - s.mu.Lock() - defer s.mu.Unlock() - assignment, ok := s.ownership[proxyID] - if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch || !assignment.ExpiresAt.After(now) { - return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment - } - assignment.ExpiresAt = now.UTC().Add(ttl) - assignment.Version++ - s.ownership[proxyID] = assignment - return assignment, nil -} - -func (s *MemoryStore) BeginDrain(proxyID, workerID string, epoch uint64) (ownershipDomain.Assignment, error) { - s.mu.Lock() - defer s.mu.Unlock() - assignment, ok := s.ownership[proxyID] - if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch { - return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment - } - if assignment.Draining { - return assignment, nil - } - assignment.Draining = true - assignment.Version++ - s.ownership[proxyID] = assignment - return assignment, nil -} - -func (s *MemoryStore) AcknowledgeDrain(proxyID, workerID string, epoch uint64, active, reserved int64) error { - if active < 0 || reserved < 0 { - return ownershipDomain.ErrInvalidOwnership - } - s.mu.Lock() - defer s.mu.Unlock() - assignment, ok := s.ownership[proxyID] - if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch { - return ownershipDomain.ErrStaleAssignment - } - if !assignment.Draining { - return ownershipDomain.ErrNotDraining - } - if active > 0 || reserved > 0 { - return ownershipDomain.ErrDrainNotReady - } - if candidate, exists := s.candidates[proxyID]; exists && candidate.OwnerWorkerID == workerID { - candidate.OwnerWorkerID = "" - s.candidates[proxyID] = candidate - } - delete(s.ownership, proxyID) - return nil -} - -func (s *MemoryStore) Get(proxyID string) (ownershipDomain.Assignment, bool) { - s.mu.Lock() - defer s.mu.Unlock() - assignment, ok := s.ownership[proxyID] - return assignment, ok -} - -func (s *MemoryStore) Expire(now time.Time) []ownershipDomain.Assignment { - s.mu.Lock() - defer s.mu.Unlock() - expired := make([]ownershipDomain.Assignment, 0) - for proxyID, assignment := range s.ownership { - if assignment.ExpiresAt.After(now) { - continue - } - if candidate, ok := s.candidates[proxyID]; ok && candidate.OwnerWorkerID == assignment.WorkerID { - candidate.OwnerWorkerID = "" - s.candidates[proxyID] = candidate - } - expired = append(expired, assignment) - delete(s.ownership, proxyID) - } - sort.Slice(expired, func(i, j int) bool { return expired[i].ProxyID < expired[j].ProxyID }) - return expired -} - -func (s *MemoryStore) Records() []Record { - s.mu.Lock() - defer s.mu.Unlock() - return append([]Record(nil), s.records...) -} - -func cloneResult(result Result) Result { - result.Items = append([]Candidate(nil), result.Items...) - return result -} - -func cloneCommand(command Command) Command { - command.Protocols = append([]string(nil), command.Protocols...) - command.Regions = append([]string(nil), command.Regions...) - command.Carriers = append([]string(nil), command.Carriers...) - command.Upstreams = append([]string(nil), command.Upstreams...) - return command -} - -func sameIdempotentRequest(left, right Command) bool { - return left.Requested == right.Requested && - left.Fulfillment == right.Fulfillment && - equalSet(left.Protocols, right.Protocols) && - equalSet(left.Regions, right.Regions) && - equalSet(left.Carriers, right.Carriers) && - equalSet(left.Upstreams, right.Upstreams) -} - -func equalSet(left, right []string) bool { - if len(left) != len(right) { - return false - } - counts := make(map[string]int, len(left)) - for _, value := range left { - counts[value]++ - } - for _, value := range right { - counts[value]-- - if counts[value] < 0 { - return false - } - } - return true -} - -func eligibleForExtraction(candidate Candidate, command Command) bool { - if candidate.State != Available || candidate.OwnerWorkerID != "" { - return false - } - if !candidate.ExpiresAt.IsZero() && candidate.ExpiresAt.Sub(command.Now) < command.MinRemainingTTL { - return false - } - if command.MaxHealthCheckAge > 0 && command.Now.Sub(candidate.LastCheckedAt) > command.MaxHealthCheckAge { - return false - } - return matches(command.Protocols, candidate.Protocol) && - matches(command.Regions, candidate.Region) && - matches(command.Carriers, candidate.Carrier) && - matches(command.Upstreams, candidate.Upstream) -} - -func matches(allowed []string, value string) bool { - if len(allowed) == 0 { - return true - } - for _, candidate := range allowed { - if candidate == value { - return true - } - } - return false -} diff --git a/internal/domain/extraction/extraction_test.go b/internal/domain/extraction/extraction_test.go index 27cfde1..ef5f0d4 100644 --- a/internal/domain/extraction/extraction_test.go +++ b/internal/domain/extraction/extraction_test.go @@ -1,246 +1,12 @@ package extraction -import ( - "context" - "errors" - "sync" - "testing" - "time" -) +import "testing" -func TestMemoryStoreNeverExtractsProxyTwice(t *testing.T) { - now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC) - store := NewMemoryStore([]Candidate{ - {ID: "p1", State: Available, ExpiresAt: now.Add(time.Minute), LastCheckedAt: now}, - }) - - var wg sync.WaitGroup - results := make(chan string, 1000) - for range 1000 { - wg.Add(1) - go func() { - defer wg.Done() - result, err := store.Extract(context.Background(), Command{ - Requested: 1, - Fulfillment: Partial, - Now: now, - MinRemainingTTL: 30 * time.Second, - MaxHealthCheckAge: 10 * time.Second, - }) - if err != nil { - t.Errorf("Extract(): %v", err) - return - } - for _, item := range result.Items { - results <- item.ID - } - }() +func TestWireValuesRemainStable(t *testing.T) { + if Partial != "partial" || AllOrNothing != "allOrNothing" { + t.Fatalf("fulfillment values = %q, %q", Partial, AllOrNothing) } - wg.Wait() - close(results) - - count := 0 - for id := range results { - if id != "p1" { - t.Fatalf("unexpected proxy %q", id) - } - count++ - } - if count != 1 { - t.Fatalf("proxy extracted %d times, want exactly once", count) - } -} - -func TestAllOrNothingDoesNotConsumePartialInventory(t *testing.T) { - now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC) - store := NewMemoryStore([]Candidate{ - {ID: "p1", State: Available, ExpiresAt: now.Add(time.Minute), LastCheckedAt: now}, - }) - - result, err := store.Extract(context.Background(), Command{ - Requested: 2, - Fulfillment: AllOrNothing, - Now: now, - MinRemainingTTL: 30 * time.Second, - MaxHealthCheckAge: 10 * time.Second, - }) - if err != ErrInsufficientProxies { - t.Fatalf("Extract() error = %v, want ErrInsufficientProxies", err) - } - if len(result.Items) != 0 { - t.Fatalf("Extract() returned %d items, want 0", len(result.Items)) - } - - partial, err := store.Extract(context.Background(), Command{ - Requested: 1, - Fulfillment: Partial, - Now: now, - MinRemainingTTL: 30 * time.Second, - MaxHealthCheckAge: 10 * time.Second, - }) - if err != nil || len(partial.Items) != 1 { - t.Fatalf("inventory was consumed by failed all-or-nothing: result=%+v err=%v", partial, err) - } -} - -func TestMemoryStoreCommitsAuditWithExtraction(t *testing.T) { - now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC) - store := NewMemoryStore([]Candidate{{ - ID: "p1", - Upstream: "provider-a", - State: Available, - ExpiresAt: now.Add(time.Minute), - LastCheckedAt: now, - }}) - - _, err := store.Extract(context.Background(), Command{ - RequestID: "req-1", - ClientID: "client-1", - SourceIP: "192.0.2.30", - Requested: 1, - Fulfillment: Partial, - Now: now, - MinRemainingTTL: 30 * time.Second, - MaxHealthCheckAge: 10 * time.Second, - }) - if err != nil { - t.Fatalf("Extract(): %v", err) - } - - records := store.Records() - if len(records) != 1 { - t.Fatalf("audit records = %d, want 1", len(records)) - } - record := records[0] - if record.ProxyID != "p1" || record.ClientID != "client-1" || record.RequestID != "req-1" { - t.Fatalf("audit record = %+v", record) - } - if !record.ExtractedAt.Equal(now) || !record.ExpiresAt.Equal(now.Add(time.Minute)) { - t.Fatalf("audit timestamps = %+v", record) - } -} - -func TestMemoryStoreReplaysCommittedIdempotentResult(t *testing.T) { - now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC) - store := NewMemoryStore([]Candidate{{ - ID: "p1", - State: Available, - ExpiresAt: now.Add(time.Minute), - LastCheckedAt: now, - }}) - command := Command{ - RequestID: "req-1", - ClientID: "client-1", - IdempotencyKey: "idem-12345678", - Requested: 1, - Fulfillment: Partial, - Now: now, - MinRemainingTTL: 30 * time.Second, - MaxHealthCheckAge: 10 * time.Second, - } - - first, err := store.Extract(context.Background(), command) - if err != nil { - t.Fatalf("first Extract(): %v", err) - } - command.RequestID = "req-2" - second, err := store.Extract(context.Background(), command) - if err != nil { - t.Fatalf("second Extract(): %v", err) - } - - if len(first.Items) != 1 || len(second.Items) != 1 || second.Items[0].ID != first.Items[0].ID { - t.Fatalf("idempotent results: first=%+v second=%+v", first, second) - } - if got := len(store.Records()); got != 1 { - t.Fatalf("audit records = %d, want 1", got) - } -} - -func TestMemoryStoreRejectsIdempotencyKeyReuseWithDifferentRequest(t *testing.T) { - now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC) - store := NewMemoryStore([]Candidate{{ - ID: "p1", - State: Available, - ExpiresAt: now.Add(time.Minute), - LastCheckedAt: now, - }}) - command := Command{ - RequestID: "req-1", - ClientID: "client-1", - IdempotencyKey: "idem-12345678", - Requested: 1, - Fulfillment: Partial, - Now: now, - MinRemainingTTL: 30 * time.Second, - MaxHealthCheckAge: 10 * time.Second, - } - if _, err := store.Extract(context.Background(), command); err != nil { - t.Fatalf("first Extract(): %v", err) - } - - command.Requested = 2 - if _, err := store.Extract(context.Background(), command); err != ErrIdempotencyConflict { - t.Fatalf("second Extract() error = %v, want ErrIdempotencyConflict", err) - } -} - -func TestMemoryStoreExtractsOnlyUnownedProxy(t *testing.T) { - now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC) - store := NewMemoryStore([]Candidate{ - {ID: "owned", OwnerWorkerID: "worker-1", State: Available, ExpiresAt: now.Add(time.Minute), LastCheckedAt: now}, - {ID: "unowned", State: Available, ExpiresAt: now.Add(time.Minute), LastCheckedAt: now}, - }) - - result, err := store.Extract(context.Background(), Command{ - RequestID: "req-1", - ClientID: "client-1", - Requested: 2, - Fulfillment: Partial, - Now: now, - MinRemainingTTL: 30 * time.Second, - MaxHealthCheckAge: 10 * time.Second, - }) - if err != nil { - t.Fatalf("Extract(): %v", err) - } - if len(result.Items) != 1 || result.Items[0].ID != "unowned" { - t.Fatalf("extracted items = %+v, want only unowned", result.Items) - } -} - -func TestMemoryStoreValidatesCommandAndHonorsCancellation(t *testing.T) { - store := NewMemoryStore([]Candidate{{ID: "p1", State: Available}}) - if _, err := store.Extract(context.Background(), Command{ - Requested: 1, Fulfillment: Partial, ReserveForGateway: -1, - }); !errors.Is(err, ErrInvalidCommand) { - t.Fatalf("Extract(negative reserve) error = %v, want ErrInvalidCommand", err) - } - - ctx, cancel := context.WithCancel(context.Background()) - cancel() - if _, err := store.Extract(ctx, Command{Requested: 1, Fulfillment: Partial}); !errors.Is(err, context.Canceled) { - t.Fatalf("Extract(canceled) error = %v, want context.Canceled", err) - } - result, err := store.Extract(context.Background(), Command{Requested: 1, Fulfillment: Partial}) - if err != nil || result.Returned != 1 { - t.Fatalf("candidate changed after canceled request: result=%+v err=%v", result, err) - } -} - -func TestMemoryStoreClonesIdempotencyCommandFilters(t *testing.T) { - store := NewMemoryStore([]Candidate{{ID: "p1", Protocol: "http", State: Available}}) - protocols := []string{"http"} - command := Command{ - ClientID: "client-1", IdempotencyKey: "idem-1", Requested: 1, - Fulfillment: Partial, Protocols: protocols, - } - if _, err := store.Extract(context.Background(), command); err != nil { - t.Fatalf("first Extract(): %v", err) - } - protocols[0] = "socks5" - command.Protocols = []string{"http"} - if _, err := store.Extract(context.Background(), command); err != nil { - t.Fatalf("idempotent replay after caller mutation: %v", err) + if Available != "AVAILABLE" || Extracted != "EXTRACTED" { + t.Fatalf("state values = %q, %q", Available, Extracted) } } diff --git a/internal/domain/proxy/proxy.go b/internal/domain/proxy/proxy.go index 63d8454..4383890 100644 --- a/internal/domain/proxy/proxy.go +++ b/internal/domain/proxy/proxy.go @@ -27,6 +27,7 @@ type Proxy struct { SourceUpstream string CreatedAt time.Time ExpiresAt *time.Time + UsableUntil *time.Time LastCheckedAt *time.Time LastSuccessAt *time.Time Latency time.Duration diff --git a/internal/domain/upstream/pool.go b/internal/domain/upstream/pool.go index f1bb65c..af7aeca 100644 --- a/internal/domain/upstream/pool.go +++ b/internal/domain/upstream/pool.go @@ -3,7 +3,7 @@ package upstream import ( "time" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" + proxyDomain "proxy-pool/internal/domain/proxy" ) type ProxyCapacity struct { diff --git a/internal/domain/upstream/pool_test.go b/internal/domain/upstream/pool_test.go index d4875fc..b21f52b 100644 --- a/internal/domain/upstream/pool_test.go +++ b/internal/domain/upstream/pool_test.go @@ -4,7 +4,7 @@ import ( "testing" "time" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" + proxyDomain "proxy-pool/internal/domain/proxy" ) func TestInventoryAvailableSlotsUsesOnlyAllocatableCapacity(t *testing.T) { diff --git a/internal/gateway/dispatch/dispatcher.go b/internal/gateway/dispatch/dispatcher.go index 0332cd6..f4852ff 100644 --- a/internal/gateway/dispatch/dispatcher.go +++ b/internal/gateway/dispatch/dispatcher.go @@ -5,8 +5,8 @@ import ( "sync/atomic" "time" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" - "github.com/proxy-pool/proxy-pool/internal/gateway/snapshot" + proxyDomain "proxy-pool/internal/domain/proxy" + "proxy-pool/internal/gateway/snapshot" ) var ErrNoCandidate = errors.New("no local proxy candidate is available") diff --git a/internal/gateway/dispatch/dispatcher_test.go b/internal/gateway/dispatch/dispatcher_test.go index 09bf079..aa12cef 100644 --- a/internal/gateway/dispatch/dispatcher_test.go +++ b/internal/gateway/dispatch/dispatcher_test.go @@ -8,8 +8,8 @@ import ( "testing" "time" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" - "github.com/proxy-pool/proxy-pool/internal/gateway/snapshot" + proxyDomain "proxy-pool/internal/domain/proxy" + "proxy-pool/internal/gateway/snapshot" ) func TestAcquireFiltersAndReservesLocalCapacity(t *testing.T) { diff --git a/internal/gateway/server/bootstrap.go b/internal/gateway/server/bootstrap.go index 23c426c..589c988 100644 --- a/internal/gateway/server/bootstrap.go +++ b/internal/gateway/server/bootstrap.go @@ -4,10 +4,10 @@ import ( "fmt" "time" - "github.com/proxy-pool/proxy-pool/internal/config" - "github.com/proxy-pool/proxy-pool/internal/gateway/policy" - platformAdmission "github.com/proxy-pool/proxy-pool/internal/platform/admission" - "github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity" + "proxy-pool/internal/config" + "proxy-pool/internal/gateway/policy" + platformAdmission "proxy-pool/internal/platform/admission" + "proxy-pool/internal/platform/httpsecurity" ) type Protection struct { diff --git a/internal/gateway/server/bootstrap_test.go b/internal/gateway/server/bootstrap_test.go index 03dc786..ec51bb2 100644 --- a/internal/gateway/server/bootstrap_test.go +++ b/internal/gateway/server/bootstrap_test.go @@ -7,9 +7,9 @@ import ( "net/http/httptest" "testing" - "github.com/proxy-pool/proxy-pool/internal/config" - "github.com/proxy-pool/proxy-pool/internal/gateway/policy" - "github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity" + "proxy-pool/internal/config" + "proxy-pool/internal/gateway/policy" + "proxy-pool/internal/platform/httpsecurity" ) func TestBuildProtectionFromListenerConfig(t *testing.T) { diff --git a/internal/gateway/server/e2e_test.go b/internal/gateway/server/e2e_test.go index 252de3d..e02900b 100644 --- a/internal/gateway/server/e2e_test.go +++ b/internal/gateway/server/e2e_test.go @@ -16,11 +16,11 @@ import ( "testing" "time" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" - "github.com/proxy-pool/proxy-pool/internal/gateway/dispatch" - "github.com/proxy-pool/proxy-pool/internal/gateway/policy" - "github.com/proxy-pool/proxy-pool/internal/gateway/snapshot" - transportDomain "github.com/proxy-pool/proxy-pool/internal/gateway/transport" + proxyDomain "proxy-pool/internal/domain/proxy" + "proxy-pool/internal/gateway/dispatch" + "proxy-pool/internal/gateway/policy" + "proxy-pool/internal/gateway/snapshot" + transportDomain "proxy-pool/internal/gateway/transport" ) func TestHTTPProxyEndToEndRetriesDialFailure(t *testing.T) { diff --git a/internal/gateway/server/handler.go b/internal/gateway/server/handler.go index 438d46d..a49dd00 100644 --- a/internal/gateway/server/handler.go +++ b/internal/gateway/server/handler.go @@ -14,11 +14,11 @@ import ( "sync/atomic" "time" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" - "github.com/proxy-pool/proxy-pool/internal/gateway/dispatch" - "github.com/proxy-pool/proxy-pool/internal/gateway/policy" - transportDomain "github.com/proxy-pool/proxy-pool/internal/gateway/transport" - "github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity" + proxyDomain "proxy-pool/internal/domain/proxy" + "proxy-pool/internal/gateway/dispatch" + "proxy-pool/internal/gateway/policy" + transportDomain "proxy-pool/internal/gateway/transport" + "proxy-pool/internal/platform/httpsecurity" ) type Config struct { diff --git a/internal/gateway/server/handler_test.go b/internal/gateway/server/handler_test.go index 1af9a48..09ee653 100644 --- a/internal/gateway/server/handler_test.go +++ b/internal/gateway/server/handler_test.go @@ -15,11 +15,11 @@ import ( "testing" "time" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" - "github.com/proxy-pool/proxy-pool/internal/gateway/dispatch" - "github.com/proxy-pool/proxy-pool/internal/gateway/policy" - "github.com/proxy-pool/proxy-pool/internal/gateway/snapshot" - transportDomain "github.com/proxy-pool/proxy-pool/internal/gateway/transport" + proxyDomain "proxy-pool/internal/domain/proxy" + "proxy-pool/internal/gateway/dispatch" + "proxy-pool/internal/gateway/policy" + "proxy-pool/internal/gateway/snapshot" + transportDomain "proxy-pool/internal/gateway/transport" ) func TestHandlerRunsProtectionAndTargetPolicyBeforeRouting(t *testing.T) { diff --git a/internal/gateway/server/protection.go b/internal/gateway/server/protection.go index b688935..63a4d5f 100644 --- a/internal/gateway/server/protection.go +++ b/internal/gateway/server/protection.go @@ -6,8 +6,8 @@ import ( "fmt" "net/http" - "github.com/proxy-pool/proxy-pool/internal/gateway/policy" - "github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity" + "proxy-pool/internal/gateway/policy" + "proxy-pool/internal/platform/httpsecurity" ) type HTTPError struct { diff --git a/internal/gateway/server/routing.go b/internal/gateway/server/routing.go index 17718e8..3c543b3 100644 --- a/internal/gateway/server/routing.go +++ b/internal/gateway/server/routing.go @@ -7,8 +7,8 @@ import ( "net/http" "strings" - "github.com/proxy-pool/proxy-pool/internal/domain/routing" - "github.com/proxy-pool/proxy-pool/internal/gateway/dispatch" + "proxy-pool/internal/domain/routing" + "proxy-pool/internal/gateway/dispatch" ) var ( diff --git a/internal/gateway/server/routing_test.go b/internal/gateway/server/routing_test.go index dd12f99..74e6727 100644 --- a/internal/gateway/server/routing_test.go +++ b/internal/gateway/server/routing_test.go @@ -7,7 +7,7 @@ import ( "reflect" "testing" - "github.com/proxy-pool/proxy-pool/internal/domain/routing" + "proxy-pool/internal/domain/routing" ) func TestRulesRouterReturnsMatchedUpstreams(t *testing.T) { diff --git a/internal/gateway/snapshot/store.go b/internal/gateway/snapshot/store.go index 546e42e..6055a19 100644 --- a/internal/gateway/snapshot/store.go +++ b/internal/gateway/snapshot/store.go @@ -11,7 +11,7 @@ import ( "sync/atomic" "time" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" + proxyDomain "proxy-pool/internal/domain/proxy" ) var ( @@ -270,6 +270,9 @@ func matchesQuery(candidate proxyDomain.Proxy, query Query) bool { if _, excluded := query.Exclude[candidate.ID]; excluded { return false } + if candidate.UsableUntil != nil && !candidate.UsableUntil.After(query.Now) { + return false + } if candidate.ExpiresAt != nil && !candidate.ExpiresAt.After(query.Now.Add(query.SafetyMargin)) { return false } diff --git a/internal/gateway/snapshot/store_test.go b/internal/gateway/snapshot/store_test.go index 74cce14..2b6461a 100644 --- a/internal/gateway/snapshot/store_test.go +++ b/internal/gateway/snapshot/store_test.go @@ -7,7 +7,7 @@ import ( "testing" "time" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" + proxyDomain "proxy-pool/internal/domain/proxy" ) func TestStoreAppliesCompleteSnapshotsInOrder(t *testing.T) { @@ -119,6 +119,34 @@ func TestViewSelectFiltersBySchemeUpstreamTagAndExclude(t *testing.T) { } } +func TestViewSelectRejectsProxyAfterProviderUsableDeadline(t *testing.T) { + now := time.Date(2026, 7, 29, 12, 0, 0, 0, time.UTC) + expiresAt := now.Add(30 * time.Second) + usableUntil := now.Add(27 * time.Second) + proxy := proxyDomain.Proxy{ + ID: "short-lived", Scheme: proxyDomain.SchemeHTTP, State: proxyDomain.StateAvailable, + MaxConcurrency: 1, ExpiresAt: &expiresAt, UsableUntil: &usableUntil, + } + store := NewStore("cluster-a", "worker-a") + envelope := Envelope{ + ClusterID: "cluster-a", WorkerID: "worker-a", Epoch: 1, Version: 1, Full: true, + Proxies: []proxyDomain.Proxy{proxy}, + } + envelope.Checksum = Checksum(envelope.Proxies) + if err := store.Apply(envelope); err != nil { + t.Fatalf("Apply(): %v", err) + } + + before := store.Current().Select(Query{Now: now.Add(26 * time.Second)}) + if _, ok := before.EntryAt(0); !ok { + t.Fatal("EntryAt(before usable deadline) = false, want true") + } + after := store.Current().Select(Query{Now: now.Add(27 * time.Second)}) + if _, ok := after.EntryAt(0); ok { + t.Fatal("EntryAt(at usable deadline) = true, want false") + } +} + func TestStoreApplyReusesCapacityAcrossVersionsAndEpochs(t *testing.T) { store := NewStore("cluster-a", "worker-a") firstProxies := []proxyDomain.Proxy{{ diff --git a/internal/gateway/transport/credentials.go b/internal/gateway/transport/credentials.go index 1ab1b84..8dc4214 100644 --- a/internal/gateway/transport/credentials.go +++ b/internal/gateway/transport/credentials.go @@ -4,8 +4,8 @@ import ( "context" "errors" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" - platformCredentials "github.com/proxy-pool/proxy-pool/internal/platform/credentials" + proxyDomain "proxy-pool/internal/domain/proxy" + platformCredentials "proxy-pool/internal/platform/credentials" ) var ( diff --git a/internal/gateway/transport/credentials_test.go b/internal/gateway/transport/credentials_test.go index 500173e..4e36325 100644 --- a/internal/gateway/transport/credentials_test.go +++ b/internal/gateway/transport/credentials_test.go @@ -7,8 +7,8 @@ import ( "strings" "testing" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" - platformCredentials "github.com/proxy-pool/proxy-pool/internal/platform/credentials" + proxyDomain "proxy-pool/internal/domain/proxy" + platformCredentials "proxy-pool/internal/platform/credentials" ) func TestCredentialsFormattingRedactsPassword(t *testing.T) { diff --git a/internal/gateway/transport/transport.go b/internal/gateway/transport/transport.go index 5d3049f..0426e98 100644 --- a/internal/gateway/transport/transport.go +++ b/internal/gateway/transport/transport.go @@ -16,7 +16,7 @@ import ( "sync/atomic" "time" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" + proxyDomain "proxy-pool/internal/domain/proxy" ) const ( diff --git a/internal/gateway/transport/transport_test.go b/internal/gateway/transport/transport_test.go index bc19418..fdfc69a 100644 --- a/internal/gateway/transport/transport_test.go +++ b/internal/gateway/transport/transport_test.go @@ -15,7 +15,7 @@ import ( "testing" "time" - proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy" + proxyDomain "proxy-pool/internal/domain/proxy" ) func TestRoundTripForwardsHTTPViaSelectedProxy(t *testing.T) { diff --git a/internal/platform/httpsecurity/config.go b/internal/platform/httpsecurity/config.go index 3d10d1c..ab9e6b7 100644 --- a/internal/platform/httpsecurity/config.go +++ b/internal/platform/httpsecurity/config.go @@ -4,8 +4,8 @@ import ( "fmt" "time" - "github.com/proxy-pool/proxy-pool/internal/config" - platformAdmission "github.com/proxy-pool/proxy-pool/internal/platform/admission" + "proxy-pool/internal/config" + platformAdmission "proxy-pool/internal/platform/admission" ) func BuildFromListener(listener config.Listener, clientIdentification string, semantics Semantics) (*Protection, error) { diff --git a/internal/platform/httpsecurity/config_test.go b/internal/platform/httpsecurity/config_test.go index 467060b..8739a8d 100644 --- a/internal/platform/httpsecurity/config_test.go +++ b/internal/platform/httpsecurity/config_test.go @@ -6,7 +6,7 @@ import ( "net/http/httptest" "testing" - "github.com/proxy-pool/proxy-pool/internal/config" + "proxy-pool/internal/config" ) func TestNewFromListenerMapsResolvedConfiguration(t *testing.T) { diff --git a/internal/platform/httpsecurity/response.go b/internal/platform/httpsecurity/response.go index 6b6e048..846290d 100644 --- a/internal/platform/httpsecurity/response.go +++ b/internal/platform/httpsecurity/response.go @@ -4,7 +4,7 @@ import ( "errors" "net/http" - "github.com/proxy-pool/proxy-pool/internal/platform/httpapi" + "proxy-pool/internal/platform/httpapi" ) func WriteProblem(writer http.ResponseWriter, requestID string, err error) bool { diff --git a/progress.md b/progress.md index 91c6ec0..4db3730 100644 --- a/progress.md +++ b/progress.md @@ -8,7 +8,14 @@ `httpsecurity`，API 401 与代理 407 语义保持分离。 - 部署配置已使用 `${VAR}` 真实环境变量解析，ConfigMap 与本地配置通过 `LoadResolved` 回归测试。 -- PostgreSQL/Redis Adapter、生产命令入口与代表性 100,000 QPS 集群压测仍待实现。 +- 已新增公用 `activitypool` 契约和并发安全内存参考实现；Provider 按各供应商 + TTL 与安全余量写入，独占提取、短期幂等和 Worker 所有权在同一原子边界内。 +- Proxy 明细不写 PostgreSQL；PostgreSQL 仅保存配置版本、管理状态、Admin + 审计、管理 outbox 和可选聚合指标。 +- `usableUntil` 已进入 Worker Snapshot 契约，Gateway 在供应商硬过期前按安全 + 余量停止新分配。 +- 生产 Redis Adapter、PostgreSQL 管理面 Adapter、生产命令入口与代表性 + 100,000 QPS 集群压测仍待实现。 ## 2026-07-28 @@ -37,8 +44,8 @@ Retry-After、原子 FetchBudget 以及 Pool Reconciler。 - 已强制 Worker 所有权与 Exclusive Extraction 注入同一权威 Repository， 覆盖 100 轮并发竞争、续期、Drain ACK 和 Worker 过期回收。 -- 已实现提取服务层策略映射、幂等审计、来源身份准入和公共 FixedWindow - 全局/Client 限流器。 +- 已实现提取服务层策略映射、Redis 短期幂等语义、来源身份准入和公共 + FixedWindow 全局/Client 限流器；不保存逐代理提取审计记录。 - 已为 100k Proxy 快照建立 scheme/upstream/tag 索引；本机持续基准中 `Acquire` 为 640 ns/op、256 B/op、2 allocs/op。该数据仅证明本地选择热路径， 不代表 100k QPS 集群端到端容量。 diff --git a/task_plan.md b/task_plan.md index 95924a6..b6ec75c 100644 --- a/task_plan.md +++ b/task_plan.md @@ -24,6 +24,8 @@ 7. [已完成] 执行单元测试、静态检查、构建和静态部署/契约验证；本机因 `CGO_ENABLED=0` 且无 C 编译器未运行 race，保留给 Linux CI 8. [已完成] 按需求矩阵逐项审计并生成版本化文档包 +9. [已完成] 将 Proxy 明细、独占提取、短期幂等和 Worker 所有权统一到 + TTL 活动池契约；PostgreSQL 退出代理数据路径 ## 串并行关系 @@ -44,5 +46,5 @@ - Docker Compose 配置与 Kubernetes Kustomize 已完成静态渲染验证；未启动 目标运行拓扑。 -- `cmd/proxy-*`、Provider 调度、PostgreSQL/Redis 适配器、Gateway Transport - 与 Checker 运行时属于后续实施范围，见完成审计。 +- `cmd/proxy-*`、生产 Redis 活动池 Adapter、PostgreSQL 管理面 Adapter 与 + Checker 运行时属于后续实施范围，见完成审计。