feat: add ephemeral proxy activity pool
Some checks are pending
ci / test (ubuntu-latest) (push) Waiting to run
ci / test (windows-latest) (push) Waiting to run
ci / race (push) Waiting to run

This commit is contained in:
youfak 2026-07-29 12:51:18 +08:00
parent ee7fc85031
commit 4de3ffb85f
80 changed files with 1590 additions and 915 deletions

View File

@ -8,13 +8,21 @@
- **Upstream**一个供应商配置及其聚合代理池。Upstream 运行时全局共享,
不随 Routing 重复创建。
- **Provider**Upstream 背后的外部代理供应商及其获取接口能力。
- **Proxy**:从 Provider 获取并标准化后的代理资源,不等同于简单 IP。
- **Proxy**:从 Provider 获取并标准化后的短效代理资源,不等同于简单 IP。
Proxy 明细不进入 PostgreSQL只存在于带 TTL 的 Redis 活动池和持有其快照的
节点内存;丢失后由 Provider 重新获取。
- **Worker**:承载 Gateway 流量的数据面节点,仅使用本地快照选路。
- **Controller**:集中管理 Provider 获取、代理生命周期、Routing 状态、
所有权和快照发布的控制面节点。
- **Checker**:执行基础、出口和目标级健康检查的可扩缩执行节点。
- **Extraction**Distribution API 对一个或一组 Proxy 的一次性独占发放。
- **Extraction Record**Extraction 的审计事实,不承担租约或释放语义。
- **Activity Pool**Redis 中带 TTL 的短效 Proxy 运行时集合,是分配、所有权
和独占提取的原子操作位置,可在丢失后由 Provider 重建。
- **Extraction Result**Redis 中短期保存的幂等响应,不承担租约、释放或
长期审计语义。
- **Persistent Management State**PostgreSQL 中的配置版本、Upstream/Routing
管理状态、Admin 审计与 Outbox以及可选聚合指标不包含 Proxy 明细或逐次
提取记录。
## 状态与计数
@ -33,8 +41,10 @@
- **Gateway Allocation**Worker 原子预留本地所有 Proxy 容量,建连成功后
转为 Active结束后释放。
- **Exclusive Extraction**:控制面原子把可提取 Proxy 从 `AVAILABLE` 改为
`EXTRACTED`,成功后该 Proxy 永不再次被系统分配。
- **Exclusive Extraction**:控制面通过 Redis 原子操作把可提取 Proxy 从
`AVAILABLE` 改为不可再次分配,并在有限 TTL 内保存幂等结果;成功后该
Proxy 在当前 TTL 生命周期内不再被系统分配。Redis 整体丢失后的 Provider
重建属于新的活动池代次,不承诺延续已丢失代次的排他状态。
- **Drain**:停止新分配,等待 Reserved 与 Active 归零后转换状态或撤销所有权。
- **Empty Fetch**:不是错误、不是重复,而是有效 Provider 响应中没有任何
合法 Proxy 候选。
@ -51,4 +61,3 @@
- **reserveForGateway**:共享池中不能被 Distribution 提取的最低可用数量。
- **fulfillment.partial**:尽量返回,允许少于请求数量。
- **fulfillment.allOrNothing**:不足时一个也不提取。

View File

@ -4,9 +4,10 @@ info:
version: 1.0.0
description: |
Distribution API performs one-time exclusive extraction. A successful
operation atomically transitions every returned proxy from AVAILABLE to
EXTRACTED. Extracted proxies are never allocated again and there is no
release, renew, or lease API.
Redis operation atomically removes or marks every returned proxy as
extracted in the TTL activity pool. Proxy details and per-extraction
records are not persisted in PostgreSQL. There is no release, renew, or
lease API.
servers:
- url: http://127.0.0.1:8081
description: Distribution API
@ -20,13 +21,14 @@ paths:
operationId: extractProxies
summary: 一次性独占提取代理
description: |
服务端先完成筛选、行锁定、AVAILABLE -> EXTRACTED 状态更新和审计记录
写入,再返回代理。相同代理不会返回给两个成功请求。
服务端通过单个 Redis 原子操作完成筛选、TTL/所有权复核、Gateway 预留、
从可分配池移除所选条目和短期幂等结果写入,再返回代理。相同代理不会
返回给两个成功请求,提取路径不访问 PostgreSQL。
`partial` 允许实际返回数量小于请求数量;`allOrNothing` 数量不足时不
提取任何代理并返回 409。未传 `fulfillment` 时使用服务端配置,默认
为 `partial`。`Idempotency-Key` 可避免客户端因响应丢失重试而再次消耗
库存。
为 `partial`。`Idempotency-Key` 在 Redis 的有界 TTL 窗口内避免客户端因
响应丢失重试而再次消耗库存。
security:
- ApiKeyAuth: []
- BasicAuth: []
@ -57,7 +59,7 @@ paths:
allowedUpstreams: [provider-a, provider-b]
responses:
'200':
description: 提取事务已提交;返回的代理已永久退出可分配池
description: Redis 原子提取已提交;返回的代理已退出当前可分配池
headers:
X-Request-ID:
$ref: '#/components/headers/RequestID'
@ -117,7 +119,7 @@ paths:
tags: [Health]
operationId: getReadiness
summary: Distribution 就绪探针
description: PostgreSQL 不可用或权威状态不可写时返回 503。
description: Redis TTL 活动池不可用或无法保证原子提取时返回 503。
security: []
responses:
'200':
@ -375,7 +377,7 @@ components:
schema:
$ref: '#/components/schemas/Problem'
ServiceUnavailable:
description: 权威存储不可用或服务正在排空
description: Redis TTL 活动池不可用、原子提取不可执行或服务正在排空
headers:
X-Request-ID:
$ref: '#/components/headers/RequestID'

View File

@ -2,7 +2,7 @@ syntax = "proto3";
package proxy_pool.controlplane.v1;
option go_package = "github.com/proxy-pool/proxy-pool/gen/controlplane/v1;controlplanev1";
option go_package = "proxy-pool/gen/controlplane/v1;controlplanev1";
import "google/protobuf/duration.proto";
import "google/protobuf/empty.proto";
@ -125,6 +125,7 @@ message OwnedProxy {
uint32 max_concurrency = 10;
map<string, string> tags = 11;
uint64 ownership_epoch = 12;
google.protobuf.Timestamp usable_until = 13;
}
enum ProxyProtocol {

View File

@ -72,6 +72,7 @@ distribution:
minRemainingTTL: 30s
maxHealthCheckAge: 15s
reserveForGateway: 100
idempotencyTTL: 5m
admin:
enabled: true

View File

@ -46,6 +46,7 @@ distribution:
minRemainingTTL: 30s
maxHealthCheckAge: 30s
reserveForGateway: 1000
idempotencyTTL: 5m
admin:
enabled: true

View File

@ -50,6 +50,7 @@ data:
minRemainingTTL: 30s
maxHealthCheckAge: 30s
reserveForGateway: 5000
idempotencyTTL: 5m
admin:
enabled: true
listen: 0.0.0.0:8082

View File

@ -5,7 +5,7 @@ import (
"fmt"
"os"
"github.com/proxy-pool/proxy-pool/internal/config"
"proxy-pool/internal/config"
)
func main() {

View File

@ -245,16 +245,16 @@ flowchart TB
```mermaid
sequenceDiagram
participant DB as PostgreSQL
participant R as Redis Activity Pool
participant C as Controller
participant W as Worker
DB->>C: outbox revision 42
R->>C: activity or ownership revision 42
C->>C: build worker snapshot
C->>W: epoch 8 version 42 checksum
W->>W: validate and build indexes
W->>W: atomic swap
W-->>C: ACK epoch 8 version 42
C->>DB: mark outbox delivered
C->>R: record worker ACK with TTL
```
## 17 Snapshot 缺口恢复
@ -316,12 +316,10 @@ flowchart LR
sequenceDiagram
participant C as Client
participant API as Distribution
participant DB as PostgreSQL
participant R as Redis Activity Pool
C->>API: count 10 fulfillment partial
API->>DB: lock eligible rows
DB-->>API: 6 rows
API->>DB: update 6 to EXTRACTED and audit
DB-->>API: commit
API->>R: atomic extract with filters and reserve
R-->>API: mark 6 EXTRACTED and return them
API-->>C: requested 10 returned 6
```
@ -331,11 +329,10 @@ sequenceDiagram
sequenceDiagram
participant C as Client
participant API as Distribution
participant DB as PostgreSQL
participant R as Redis Activity Pool
C->>API: count 10 fulfillment allOrNothing
API->>DB: lock eligible rows
DB-->>API: only 6 rows
API->>DB: rollback entire transaction
API->>R: atomic extract with filters and reserve
R-->>API: insufficient inventory and no mutation
API-->>C: insufficient inventory and returned 0
```
@ -345,13 +342,12 @@ sequenceDiagram
sequenceDiagram
participant C as Controller
participant W as Gateway Worker
participant DB as PostgreSQL
participant R as Redis Activity Pool
C->>W: mark Proxy DRAINING at epoch 13
W->>W: stop new allocations
W-->>C: ACK active 0 reserved 0
C->>DB: clear worker ownership
C->>DB: AVAILABLE to EXTRACTED plus audit
DB-->>C: committed exclusive result
C->>R: atomically clear ownership and extract
R-->>C: committed exclusive result
```
## 24 reserveForGateway 不变量
@ -367,18 +363,16 @@ flowchart TD
Result -->|allOrNothing insufficient| Rollback[Return zero]
```
## 25 PostgreSQL 独占事务
## 25 Redis 独占原子操作
```mermaid
flowchart TD
Begin[BEGIN] --> Select[SELECT eligible FOR UPDATE SKIP LOCKED]
Begin[Lua script or Redis Function] --> Select[Filter eligible TTL entries]
Select --> Enough{Quantity satisfies mode?}
Enough -->|no allOrNothing| Rollback[ROLLBACK]
Enough -->|yes or partial| Update[UPDATE AVAILABLE to EXTRACTED]
Update --> Audit[INSERT extraction records]
Audit --> Outbox[INSERT outbox]
Outbox --> Commit[COMMIT]
Commit --> Return[Return proxies with expiry]
Enough -->|no allOrNothing| Noop[Return zero without mutation]
Enough -->|yes or partial| Update[Mark selected entries EXTRACTED]
Update --> Idempotency[Optionally cache bounded idempotency result]
Idempotency --> Return[Return proxies with expiry]
```
## 26 并发提取互斥
@ -386,17 +380,16 @@ flowchart TD
```mermaid
sequenceDiagram
participant A as Extract request A
participant DB as PostgreSQL
participant R as Redis Activity Pool
participant B as Extract request B
A->>DB: lock rows 1 to 10
B->>DB: skip locked rows 1 to 10
B->>DB: lock rows 11 to 20
A->>DB: commit EXTRACTED 1 to 10
B->>DB: commit EXTRACTED 11 to 20
A->>R: execute atomic extraction
R-->>A: commit EXTRACTED 1 to 10
B->>R: execute atomic extraction
R-->>B: commit EXTRACTED 11 to 20
Note over A,B: no Proxy returned twice
```
## 27 Outbox 一致性
## 27 管理面 Outbox 一致性
```mermaid
sequenceDiagram
@ -404,7 +397,7 @@ sequenceDiagram
participant DB as PostgreSQL
participant P as Publisher
participant W as Worker
C->>DB: state change plus outbox
C->>DB: management state change plus outbox
DB-->>C: atomic commit
P->>DB: read undelivered event
P->>W: publish versioned event
@ -533,4 +526,3 @@ flowchart TD
Failure --> Verify[Verify SLO and all invariants]
Verify --> Evidence[Archive raw metrics config and image digest]
```

View File

@ -12,9 +12,9 @@ Gateway 只依赖本地不可变 SnapshotProvider、数据库、配置重载
**状态:** 接受并覆盖早期 Lease 方案。
成功响应前在一个事务中执行 `AVAILABLE -> EXTRACTED`。系统保存审计事实,
不提供 release、renew 或使用跟踪。这样契合“拿走真实代理后平台不再管理”
的最终产品语义,并消除重复发放。
成功响应前,通过 Redis 原子操作完成候选选择、`AVAILABLE -> EXTRACTED` 和
短期幂等结果写入;不提供 release、renew 或使用跟踪。这样契合“拿走真实
代理后平台不再管理”的最终产品语义,并消除并发重复发放。
## ADR-003单 Worker 所有权
@ -24,11 +24,14 @@ Gateway 只依赖本地不可变 SnapshotProvider、数据库、配置重载
Distribution 只提取无所有权 Proxy回收时执行 drain/ACK/归零/解除所有权。
该选择避免每请求访问 Redis 做全局并发计数。
## ADR-004PostgreSQL 权威、Redis 可重建
## ADR-004PostgreSQL 保存控制面事实Redis 承载短效活动池
**状态:** 接受。
Proxy 状态、Extraction、配置版本和 Outbox 由 PostgreSQL 持久化。Redis 只
承担 Leader、短期速率与心跳等协调Redis 丢失后可从权威状态恢复,避免
双写状态成为不可判定的事实源。
Proxy 明细不进入 PostgreSQL。短效 Proxy 及其状态、所有权和过期时间只存在于
带 TTL 的 Redis 活动池及节点内存Redis 同时承担独占提取、短期幂等结果、
Provider Leader、分布式速率和 Worker 心跳。Redis 丢失时活动池作废,由
Provider 重新获取并重建,不从 PostgreSQL 恢复原 Proxy。
PostgreSQL 只持久化配置版本、Upstream/Routing 管理状态、Admin 审计与 Outbox
以及可选的无 Proxy 明细聚合指标。两类存储不双写 Proxy也不建立跨存储事务。

View File

@ -10,7 +10,7 @@ Proto 源文件位于 `api/proto/controlplane/v1/controlplane.proto`,包含两
- `CheckerControlPlane`:健康检查任务流和 Observation 批量上报。
该协议不承载 Client 的独占提取,也没有 extraction lease/release。Proxy 的
`AVAILABLE -> EXTRACTED` 只在 Controller 权威事务中完成。
`AVAILABLE -> EXTRACTED` 只在 Controller 调用的 Redis 原子操作中完成。
## 2. Worker 会话
@ -40,9 +40,14 @@ sequenceDiagram
- 单调 `version`、`ownership_epoch`、生成时间和有效期。
- 对该 Worker 可见的有序 Routing。
- 仅归该 Worker 所有的 Proxy 与每个 Proxy 的容量。
- 仅归该 Worker 所有的 Proxy、每个 Proxy 的容量、硬过期时间 `expires_at`
和停止新分配的 `usable_until`
- 内容 `checksum`
`usable_until = expires_at - allocationSafetyMargin`。Worker 必须以
`usable_until` 作为最后可分配时刻;达到该时间后即使尚未到 `expires_at`
也不得再为新请求选择该 Proxy。
Delta 声明 `base_version`。Worker 只有在本地版本恰好等于 base 且 checksum
验证成功时才能应用;否则丢弃 Delta 并请求完整 Snapshot。构建在后台完成
热路径只读取一次原子指针。
@ -57,11 +62,11 @@ Delta 声明 `base_version`。Worker 只有在本地版本恰好等于 base 且
1. 新 Snapshot 标记或移除该 Proxy使 Worker 停止新预留。
2. Worker 上报 `draining=true` 以及 Active/Reserved。
3. 两个计数都归零后 Controller 清除所有权。
4. 无所有权 Proxy 才能进入 Distribution 提取事务
4. 无所有权 Proxy 才能进入 Distribution 的 Redis 原子提取操作
Worker 崩溃时必须等待所有权 epoch/有效期失效后再转移避免双主。Proto 中
`ReportRuntimeResponse.revoke_proxy_ids` 是加速 Drain 的控制信号,不绕过
Snapshot 版本和权威持久化
Snapshot 版本和 Redis 中的权威所有权状态
## 5. Outcome 上报
@ -73,8 +78,9 @@ Outcome 按 Worker 单调 `sequence` 批量上报。Controller 返回已接受
- `RESPONSE_HEADERS`:目标响应头前失败。
- `TUNNEL`:隧道建立后结束或失败。
Outcome 是 Observation不直接让 Worker 修改 PostgreSQL 状态。异步上报队列
必须有界;队列满时丢弃低价值样本并计指标,不能反压 Gateway 热路径。
Outcome 是 Observation不直接让 Worker 修改 Redis 活动池状态,也不创建
PostgreSQL Proxy 明细。异步上报队列必须有界;队列满时丢弃低价值样本并计
指标,不能反压 Gateway 热路径。
## 6. Checker 任务
@ -85,7 +91,8 @@ Checker 注册自身最大并发与支持层级Controller 发送有 deadline
- `TARGET`:针对 Routing/目标组的可达性。
Checker 只返回 `HealthObservation`。Controller reducer 按 Proxy、检查层级和
Routing 决定 AVAILABLE、SUSPECT 或 UNHEALTHY避免多个 Checker 并发写状态。
Routing 决定 AVAILABLE、SUSPECT 或 UNHEALTHY并更新 Redis 活动池,避免
多个 Checker 并发写状态。
## 7. 兼容与演进

View File

@ -5,18 +5,17 @@
Distribution API 只提供一次性独占提取:
```text
筛选 AVAILABLE
-> 锁定候选
-> 校验 Gateway 预留、TTL、健康与过滤条件
-> 原子 AVAILABLE -> EXTRACTED
-> 写 Extraction Record
-> 提交事务
读取 Redis TTL 活动池中的 AVAILABLE 候选
-> 校验 Gateway 预留、TTL、健康、所有权与过滤条件
-> Redis 原子移出可分配池并写入短期幂等结果
-> 返回真实代理地址
```
事务提交前不得把地址写给 Client。返回成功后该 Proxy 不再参与 Gateway、
再次提取或可用库存统计。系统不跟踪 Client 是否使用、使用并发或何时停止,
也不提供 release、renew、status 或租约端点。
Redis 原子操作成功前不得把地址写给 Client。返回成功后该 Proxy 在当前 TTL
生命周期中不再参与 Gateway、再次提取或可用库存统计。系统不跟踪 Client 是否
使用、使用并发或何时停止,也不提供 release、renew、status 或租约端点。短效
Proxy 明细只存在于 Redis TTL 活动池和节点内存PostgreSQL 不保存 Proxy 明细
或逐次提取记录。
HTTP 契约源文件:`api/openapi/proxy-pool.yaml`。
@ -105,7 +104,7 @@ Request ID不记录地址或凭据。
### 4.1 partial
锁定的符合条件数量少于 `count` 时,提交实际数量:
符合条件数量少于 `count` 时,原子提交实际数量:
```text
requested=10, eligible=6, reserve=0 -> returned=6
@ -126,13 +125,14 @@ requested=10, eligible=6, reserve=0 -> returned=6
}
```
冲突响应后,之前被该请求临时锁定的 Proxy 仍为 AVAILABLE。
冲突响应后,候选 Proxy 仍在 Redis TTL 活动池中保持 AVAILABLE。
## 5. 资格过滤与 Gateway 预留
候选必须同时满足:
1. 权威状态是 AVAILABLE。
1. Redis TTL 活动池条目仍为 AVAILABLE且当前时间早于按供应商安全余量计算的
`usableUntil`
2. 没有 Worker 所有权,或已完成 Drain 且 Active/Reserved 均为零。
3. 剩余 TTL 不低于 `minRemainingTTL`
4. 最近健康检查不早于 `maxHealthCheckAge`
@ -140,29 +140,33 @@ requested=10, eligible=6, reserve=0 -> returned=6
6. 提取后符合条件的共享库存不低于 `reserveForGateway`
Worker-owned Proxy 不得直接提取。Controller 需要先发布 DRAINING等待 Worker
确认没有 Active/Reserved再清除所有权并进入提取事务。快照延迟时仍禁止
Gateway 与 Client 同时获得同一 Proxy。
确认没有 Active/Reserved再清除 Redis 中的所有权并进入原子提取。快照延迟时
仍禁止 Gateway 与 Client 同时获得同一 Proxy。
## 6. 并发与事务
## 6. 并发与原子性
PostgreSQL Adapter 应在一个事务内使用 `FOR UPDATE SKIP LOCKED` 获取候选,
更新状态并写审计记录。所有状态更新必须包含 `state = 'AVAILABLE'` 前置条件。
Redis Adapter 应通过单个 Lua 脚本、Redis Function 或等价的原子原语完成候选
筛选、TTL/健康/所有权复核、Gateway 预留计算、满足模式判断、从可分配池移除
所选条目,并写入短期幂等结果。提取过程中不访问 PostgreSQL。
关键不变量:
- 两个并发成功响应的 Proxy ID 集合交集为空。
- 状态更新或审计写入任一步失败,整个批次不返回。
- `allOrNothing` 不足时零行变为 EXTRACTED。
- PostgreSQL 不可写时返回 503不以内存结果冒充成功。
- 原子操作失败时整个批次不返回,也不得留下部分移除结果。
- `allOrNothing` 不足时零个条目退出活动池。
- Redis 活动池不可用时返回 503不以内存副本冒充成功。
- PostgreSQL 不可用不阻断提取;需要 PostgreSQL 的 Admin 管理写入单独降级。
## 7. 幂等
提取是消耗库存的写操作。客户端收到超时后盲目重试可能再次提取一批不同
Proxy因此自动重试应提供稳定的 `Idempotency-Key`
服务端幂等记录至少包含Client ID、Key、请求体摘要、提交结果和过期时间。
同一 Client、同一 Key、相同摘要返回首次结果摘要不同返回 409。幂等记录和
Extraction Record 必须与状态更新处在相同事务边界或由同一权威恢复流程保证。
服务端在 Redis 中保存有界 TTL 的幂等记录,至少包含 Client ID、Key、请求体
摘要、提交结果和过期时间。同一 Client、同一 Key、相同摘要返回首次结果
摘要不同返回 409。幂等结果必须和活动池状态变更处于同一个 Redis 原子操作。
TTL 到期或 Redis 数据丢失后不再保证旧 Key 去重,系统不回退到 PostgreSQL
保存逐次提取事实。
## 8. 认证、识别与限制
@ -186,28 +190,30 @@ Extraction Record 必须与状态更新处在相同事务边界或由同一权
- `415`:请求体不是 `application/json`
- `422`:数量、枚举或过滤组合违反业务约束。
- `429`:全局或 Client 速率限制,响应 `Retry-After`
- `503`PostgreSQL 不可写、服务排空或权威状态不可用
- `503`Redis 活动池不可用、原子提取不可执行或服务正在排空
- `500`:未分类的内部错误;响应不包含底层错误文本。
错误响应不得包含 Provider Secret、Proxy 凭据、SQL 或内部拓扑。
## 10. 审计记录
## 10. 短期运行记录与数据最小化
每个被提交的 Proxy 对应一条 Extraction Record
Redis 幂等结果只在配置的 TTL 窗口内保留重放响应所需的数据
```text
proxyId, clientId, sourceIP, requestId, upstream, extractedAt, expiresAt
clientId, idempotencyKey, requestDigest, response, expiresAt
```
无认证时 `clientId` 使用可信代理链解析后的规范化来源 IP 稳定标识,并保留
`sourceIP`。记录只用于审计、排错和计费事实不承担资源归还语义。Proxy
到期后可以清理运行记录,但 Extraction Record 按审计保留策略归档。
无认证时 `clientId` 使用可信代理链解析后的规范化来源 IP 稳定标识。应用日志
仅记录 Request ID、数量、结果码和过滤摘要不记录 Proxy 地址或凭据。需要长期
分析时只向 PostgreSQL 写入不含 Proxy 明细的可选聚合指标;系统不创建或归档
逐个 Proxy 的 Extraction Record。
## 11. 运行时实现边界
`distribution.Handler` 是薄 HTTP Adapter只负责严格解码、Header/DTO 校验、
身份结果注入、错误映射和健康探针。独占提取、TTL、Gateway 预留及幂等事务
继续由 `extraction.Service` 和持久化 Store 承担。
继续由 `extraction.Service` 和 Redis 活动池 Adapter 承担。当前内存 Store 仅作为
原子行为的参考实现与测试替身,不是生产持久化层。
请求体解码、Request ID 与 Problem JSON 统一复用 `platform/httpapi`。认证、
可信代理、来源控制、Client ID 和准入限流由必需的 `IdentityResolver` 注入,

View File

@ -145,20 +145,25 @@ distribution:
minRemainingTTL: 30s
maxHealthCheckAge: 15s
reserveForGateway: 5
idempotencyTTL: 5m
```
Extraction 是固定的一次性独占行为,**没有** `mode`、`leaseDuration`、
`release``renew` 配置。事务提交后执行 `AVAILABLE -> EXTRACTED`,该 Proxy
不再由 Gateway 或 Distribution 分配。
`release``renew` 配置。Redis 在一个原子操作内选择候选、执行
`AVAILABLE -> EXTRACTED` 并短期保存幂等结果;成功后该 Proxy 不再由 Gateway
或 Distribution 在当前 TTL 生命周期中分配。
- `fulfillment``partial` 或 `allOrNothing`,默认语义为 `partial`
- `maxCountPerRequest`:单次请求硬上限。
- `minRemainingTTL`:剩余寿命低于此值时不参与提取。
- `maxHealthCheckAge`:最近检查早于此窗口时不参与提取。
- `reserveForGateway`:提取后必须留给 Gateway 的最低符合条件库存数量。
- `idempotencyTTL`Redis 幂等结果的最长保留时间;省略时为 5 分钟,实际
保留时间不会超过本次返回代理中最早的 `expiresAt`
`partial` 会提交实际可得数量;`allOrNothing` 数量不足时事务回滚,一个也不
提取。认证关闭时仍应使用 `sourceIP` 识别匿名 Client 并执行全局/来源限流。
`partial` 会原子提取实际可得数量;`allOrNothing` 数量不足时不改变任何候选,
一个也不提取。认证关闭时仍应使用 `sourceIP` 识别匿名 Client 并执行全局/
来源限流。
`clientIdentification.mode` 支持:
@ -326,9 +331,14 @@ storage:
redisURL: "${REDIS_URL}"
```
PostgreSQL 是 Proxy 生命周期、Routing 选择、Worker 所有权和 Extraction Record
的权威存储。Redis 只承载可重建的短期协调状态,不能成为独占提取的唯一事实
来源。Metrics 标签禁止 Proxy IP、Client ID、Session、完整 URL 和 Request ID。
Proxy 明细不写入 PostgreSQL。Redis 是短效 Proxy 活动池的运行时事实源,使用
TTL 保存 Proxy 状态、所有权和过期时间,并原子执行独占提取及短期幂等结果
读写Redis 丢失后由 Provider 重新补池,不从 PostgreSQL 恢复原 Proxy。
PostgreSQL 只保存配置版本、Upstream/Routing 管理状态、Admin 审计与 Outbox
以及可选的无 Proxy 明细聚合指标。Distribution 提取不依赖 PostgreSQL因而
PostgreSQL 故障本身不应使 Redis 中可完成的 Extract 返回 `503`。Metrics 标签
禁止 Proxy IP、Client ID、Session、完整 URL 和 Request ID。
## 9. 启动前校验清单

View File

@ -67,7 +67,7 @@ Provider、Pool、Routing、Distribution 在首版需要共享事务和一致性
- Routing Sequential 当前选择和原子切换。
- Worker 所有权/容量切片、Snapshot 发布、ACK 和重同步。
- Distribution 与 Admin HTTP 接口。
- PostgreSQL 权威状态和 Redis 短期协调
- PostgreSQL 管理面持久化和 Redis TTL 活动池
### 3.3 proxy-checker
@ -88,7 +88,8 @@ domain/
proxy Proxy、状态机、TTL、唯一键、容量
routing 规则、匹配、策略与 Sequential 状态
upstream Provider 能力、Fetch 分类与 Pool 需求
extraction 独占提取命令、结果和审计事实
activitypool 短效 Proxy、TTL、所有权和独占消费
extraction 独占提取命令与结果
client 认证主体、权限与限制
gateway/
@ -102,7 +103,7 @@ controller/
pool Pool reconcile、生命周期和所有权
health 任务计划和 Observation reducer
routing Sequential 原子切换和恢复
extraction 批量原子提取与审计
extraction Redis 活动池批量原子提取与短期幂等
distribution Worker 注册、Snapshot/Delta 与结果上报
adapters/
@ -126,6 +127,7 @@ type Proxy struct {
SourceUpstream UpstreamID
CreatedAt time.Time
ExpiresAt *time.Time
UsableUntil *time.Time
LastCheckedAt *time.Time
LastSuccessAt *time.Time
Latency time.Duration
@ -136,6 +138,8 @@ type Proxy struct {
```
密码不参与日志可见唯一键;`CredentialVersion` 区分同一用户名的凭据轮换。
唯一键不包含 Upstream多个供应商返回同一唯一键时当前生命周期的首个来源
保持归属,其他来源只计重复且不得覆盖 TTL原来源硬过期淘汰后可重新归属。
运行态 `active``reserved` 存在 Worker 本地、按 Proxy ID 分片,不写入
不可变 Snapshot。
@ -279,27 +283,26 @@ sequenceDiagram
participant API as Distribution API
participant CP as Controller
participant W as Gateway Worker
participant DB as PostgreSQL
participant R as Redis Activity Pool
C->>API: POST extract(count, filters)
API->>API: auth + access + limits
API->>CP: Extract command
CP->>DB: select eligible unowned proxies
CP->>R: atomic select eligible unowned proxies
alt candidate owned by Worker
CP->>W: revoke and drain
W-->>CP: active=0, reserved=0, ownership released
end
CP->>DB: transaction AVAILABLE -> EXTRACTED
DB->>DB: insert extraction records
DB-->>CP: committed rows
CP->>R: AVAILABLE -> EXTRACTED + bounded idempotency result
R-->>CP: consumed proxies
CP-->>API: proxy + expiry
API-->>C: requested/returned/proxies
```
### 10.2 原子批量提取
PostgreSQL Adapter 使用事务和 `FOR UPDATE SKIP LOCKED` 选择满足以下条件的
行,并在同一事务更新状态与写审计记录
Redis Adapter 使用单个 Lua 脚本或等价原子命令,选择满足以下条件的活动池
条目,并同时完成状态迁移和带 TTL 的幂等结果写入
- State 为 AVAILABLE。
- 未分配 Worker 所有权Reserved/Active 为 0。
@ -308,7 +311,8 @@ PostgreSQL Adapter 使用事务和 `FOR UPDATE SKIP LOCKED` 选择满足以下
- 提取后仍保留 `reserveForGateway`
- 符合 protocol、region、carrier 与 allowedUpstreams。
`partial` 提交实际可得数量;`allOrNothing` 在锁定数量不足时回滚。
`partial` 原子消费实际可得数量;`allOrNothing` 数量不足时不改变任何条目。
Proxy 地址、凭据和逐次提取明细不写 PostgreSQL。
### 10.3 与 Gateway 共池
@ -332,7 +336,9 @@ flowchart LR
- Worker 仅对自身 Proxy 做本地原子计数。
- 所有权租约有 epoch 与过期时间Worker 失联后先等待租约过期,再分配给
其他 Worker避免双主。
- Snapshot 包含 Worker 专属 Proxy 集、Routing、策略和版本校验和。
- Snapshot 包含 Worker 专属 Proxy 集、Routing、策略、`ExpiresAt`、
`UsableUntil` 和版本校验和。Worker 在 `UsableUntil` 到达后立即停止新分配,
不等待供应商硬过期时间 `ExpiresAt`
- Worker 断开控制面后在 `maxStaleAge` 内使用最后快照;超限停止接收新流量,
已有隧道排空。
@ -340,18 +346,22 @@ flowchart LR
### 12.1 PostgreSQL
权威保存配置版本、Upstream、Proxy 生命周期、Routing 当前选择、Worker
所有权、Extraction Record、Client、审计与 outbox。
持久化配置版本、Upstream/Routing 管理状态、Admin 审计和管理事件 outbox
以及可选的不含 Proxy 明细的聚合指标。PostgreSQL 不保存 Proxy 地址、凭据、
生命周期、Worker 所有权或逐次提取记录。
### 12.2 Redis
保存可重建短期状态Provider Leader 租约、分布式限流、singleflight 信号、
短期 Client 限流和 Worker 心跳。Redis 不保存唯一权威业务状态。
保存带 TTL 的短效 Proxy 活动池、生命周期、Worker 所有权、独占提取状态和
短期幂等结果,并承载 Provider Leader、分布式限流、singleflight 信号、
短期 Client 限流和 Worker 心跳。Redis 丢失后由 Provider 重新获取代理重建,
不会从 PostgreSQL 恢复旧 Proxy。
### 12.3 Outbox
任何需要发布 Snapshot/事件的 PostgreSQL 状态更新同时写 outbox。发布成功
后标记Worker 使用 epoch/version 幂等应用,缺口触发完整 Snapshot。
管理状态变更在 PostgreSQL 事务内同时写 outbox。活动池变更通过 Redis 事件流
通知 Controller 构建 Snapshot不跨 PostgreSQL/Redis 双写 ProxyWorker 使用
epoch/version 幂等应用,缺口时从当前活动池构建完整 Snapshot。
## 13. 健康模型
@ -404,7 +414,7 @@ flowchart LR
- Routing 当前 Upstream、切换、无可用和策略选择。
- Upstream Proxy 状态、Available Slots、Active、Reserved、Pending Fetch。
- Provider Fetch 成功、Empty、Duplicate、Error、429、退避和耗时。
- Extraction requested、returned、insufficient、atomic conflict 和审计失败。
- Extraction requested、returned、insufficient、atomic conflict 和幂等缓存失败。
- Snapshot epoch/version、陈旧时长、应用耗时和重同步。
Prometheus 标签不包含 Proxy IP、Client ID、session、完整 URL 或 request ID。
@ -443,8 +453,8 @@ CPU、内存、网络、Go 版本、配置和上游响应模型下测得。
|---|---|
| Provider 超时/500 | 计 Error、退避不计 Empty不影响已有 Proxy |
| Provider 合法空响应 | Empty++;达到阈值触发相关 Routing 原子切换 |
| Redis 不可用 | Gateway 不受影响Controller 使用本地退化并停止高风险写操作 |
| PostgreSQL 不可用 | Gateway 使用最后快照;停止 Extract 和权威状态变更 |
| Redis 不可用 | Gateway 暂用未过期快照;停止 Fetch 入池、Extract 和所有权变更 |
| PostgreSQL 不可用 | Gateway 与 Redis Extract 不受影响;停止管理状态变更和 Admin 审计 |
| Controller 断线 | Worker 在 maxStaleAge 内继续;超限拒绝新流量并排空 |
| Worker 崩溃 | 所有权租约过期后重新分配;期间不双重所有 |
| Checker 积压 | 降低普通复检频率,优先新 Proxy 与 SUSPECT不无限排队 |

View File

@ -16,7 +16,7 @@ Proxy Pool 把多个供应商的动态代理统一成一个可运营的资源系
- **业务调用方**:通过稳定入口使用代理,不感知供应商差异。
- **代理直提调用方**:按协议、区域、运营商或 Upstream 条件独占提取。
- **平台管理员**:管理 Provider、Routing、容量、健康和故障切换。
- **SRE**:依据低基数指标、审计记录和运行手册进行容量与故障管理。
- **SRE**:依据低基数指标、管理面审计和运行手册进行容量与故障管理。
## 3. 核心价值
@ -30,11 +30,11 @@ Provider Adapter 负责请求格式、认证和响应解析。标准化后Rou
Gateway Worker 只读取本地不可变快照并维护本地容量计数。供应商延迟、
数据库抖动和控制面重载不会成为每请求依赖。
### 3.3 明确且可审计的资源语义
### 3.3 明确的资源语义
系统区分“短时使用容量”和“一次性独占提取”。每次 Extraction 保存请求、
调用方、来源、Upstream、提取时间和过期时间审计事实但不追踪提取后的
实际使用,也不存在归还接口。
系统区分“短时使用容量”和“一次性独占提取”。Extraction 只在 Redis 中短期
保存有界幂等结果,不向 PostgreSQL 写逐个 Proxy 的提取事实;系统不追踪
提取后的实际使用,也不存在归还接口。
## 4. 主要流程
@ -51,7 +51,8 @@ Gateway Worker 只读取本地不可变快照并维护本地容量计数。供
1. 调用方提交数量、过滤条件和 fulfillment。
2. Controller 校验调用方限额、TTL、健康新鲜度及 Gateway 保留量。
3. 在一个数据库事务中锁定候选并执行 `AVAILABLE -> EXTRACTED`
3. 在一个 Redis Lua 脚本、Redis Function 或等价原子操作中筛选候选并执行
`AVAILABLE -> EXTRACTED`
4. `partial` 尽量返回;`allOrNothing` 数量不足时零提取。
5. 响应返回代理 URL、`expiresAt` 和 `remainingTtlSeconds`
@ -88,6 +89,6 @@ required_workers = ceil(peak_qps / (measured_worker_qps * target_utilization))
## 7. 首版范围
首版包含 HTTP 正向代理、HTTPS CONNECT、REST Distribution/Admin、Provider
适配、健康与生命周期、Sequential 等路由策略、PostgreSQL 权威状态、Redis
可重建协调和集群快照。SOCKS5、跨地域主动主动和高级成本优化保留扩展边界。
适配、健康与生命周期、Sequential 等路由策略、PostgreSQL 管理面持久化、
Redis TTL 活动池和集群快照。SOCKS5、跨地域主动主动和高级成本优化保留扩展
边界。

View File

@ -37,8 +37,8 @@ proxy-pool/
### proxy-controller
Controller 是首版模块化单体。Provider、Pool、Routing 和 Extraction 共享
事务边界和状态演进,避免过早拆成分布式事务。对外端口定义在领域或控制器
模块,具体 PostgreSQL/Redis/HTTP 实现在 `adapters`
运行时策略和状态演进,但不建立 PostgreSQL/Redis 跨存储事务。对外端口定义
在领域或控制器模块,具体 PostgreSQL/Redis/HTTP 实现在 `adapters`
### proxy-checker
@ -71,20 +71,28 @@ platform -------------------------> standard library / observability SDK
## 4. 数据所有权
- PostgreSQLProxy 生命周期、Extraction 审计、配置版本和 Outbox 的权威源。
- RedisLeader、分布式速率、Worker 心跳等可丢失且可重建状态。
- PostgreSQL配置版本、Upstream/Routing 管理状态、Admin 审计与 Outbox
以及可选的无 Proxy 明细聚合指标。
- Redis带 TTL 的短效 Proxy 活动池及其状态、所有权和过期时间;同时保存
Provider Leader、分布式速率、Worker 心跳和短期提取幂等结果。
- Worker仅拥有分配给自己的 Proxy 本地容量计数和不可变快照。
- Controller拥有 Provider 调度、Routing 运行态与 Worker 所有权编排。
- Checker不拥有 Proxy 状态,只拥有执行中的检查任务。
Proxy 明细不进入 PostgreSQL。Redis 活动池丢失后由 Provider 重新获取并重建,
节点内存中的旧快照随版本或有效期失效,不能把 PostgreSQL 当成恢复来源。
## 5. 一致性边界
- Extraction 使用 PostgreSQL 单事务和行锁跳过锁定候选,提交后才响应。
- Extraction 在 Redis 内原子完成候选筛选、排他状态迁移和短期幂等结果写入,
成功后才响应;该路径不访问 PostgreSQL。
- Worker 所有权采用 `worker + epoch + version + expiry`,同一 Proxy 至多归属
一个 Worker。
- 从 Worker 回收 Proxy 时先 Drain等待 ACK 且 Active/Reserved 为零,再
解除所有权;只有无所有权 Proxy 能被 Distribution 提取。
- Snapshot 为整代不可变对象,通过校验和和严格版本序列原子替换。
- PostgreSQL 故障会影响管理状态变更和 Admin 审计,但不应阻断 Redis 中可完成
的独占提取Redis 故障则使活动池暂不可用,并触发 Provider 重建。
## 6. 扩展规则

View File

@ -12,7 +12,9 @@ semantics in `对话内容.md`.
**Architecture:** Separate Gateway, Controller, Checker, and Loadgen commands. Domain
packages remain transport-free. Gateway reads immutable local snapshots. Controller
owns provider fetch, pool lifecycle, routing state, extraction, persistence, and worker
distribution. PostgreSQL is authoritative; Redis stores rebuildable coordination state.
distribution. PostgreSQL persists management state and Admin audit/outbox. Redis owns
the rebuildable TTL Proxy activity pool and atomic extraction; Gateway hot paths remain
memory-only.
**Tech Stack:** Go 1.26, `go.yaml.in/yaml/v4`, pgx/v5, go-redis/v9, gRPC/Protobuf,
Prometheus, PostgreSQL, Redis, Docker Compose, Kubernetes.
@ -124,7 +126,8 @@ test/{fixtures,integration,e2e,load}/
- [x] Implement POST extraction command with protocol/region/carrier/upstream filters.
- [x] Enforce minRemainingTTL, maxHealthCheckAge, maxCount, client limits, and
reserveForGateway.
- [x] Atomically transition AVAILABLE to EXTRACTED and append audit records.
- [x] Atomically remove selected AVAILABLE entries from the allocatable set and return
the result; the current memory Store models the production Redis atomic boundary.
- [x] Implement partial and allOrNothing without Lease, release, or renewal concepts.
- [x] Run 1,000 concurrent claim attempts and prove every Proxy ID appears at most once.
@ -155,11 +158,15 @@ test/{fixtures,integration,e2e,load}/
**Files:** `internal/controller/distribution/*.go`, `admin/*.go`,
`internal/adapters/postgres/*.go`, `internal/adapters/redis/*.go`, migrations, tests
- [ ] Define repository ports for Proxy, RoutingRuntime, Ownership, ExtractionRecord,
Client, ConfigVersion, and Outbox.
- [ ] Implement PostgreSQL extraction with one transaction and `FOR UPDATE SKIP LOCKED`.
- [ ] Implement Redis coordination for Provider leader, distributed rate, Client limit,
and Worker heartbeat; keep all state rebuildable.
- [ ] Define PostgreSQL ports for ConfigVersion, Upstream/Routing management state,
AdminAudit, Outbox, and optional aggregate metrics; never persist Proxy details or
per-extraction records.
- [ ] Implement the Redis TTL activity pool and one atomic extraction operation covering
candidate eligibility, Gateway reserve, ownership, removal, and short-lived idempotency.
- [ ] Implement Redis Provider leader, distributed rate, Client limit, and Worker
heartbeat/ownership; rebuild short-lived Proxy inventory from Providers after loss.
- [ ] Keep Provider output in Redis TTL activity state and node memory only; keep the
Gateway request path on immutable local snapshots with no Redis/PostgreSQL calls.
- [ ] Expose Distribution extraction/status and Admin status/enable/disable/switch/reload.
- [ ] Add integration tests using Compose-backed PostgreSQL/Redis.
@ -170,8 +177,14 @@ extract/live/ready Handler 与 Admin status/enable/disable/switch/reload Handler
Bearer/CIDR、可信代理、Client ID、本地准入和 API 401/Gateway 407 差异,并作为
Admin/Distribution 必需依赖。共享 `platform/httpserver`
`controller/runtime` 已完成 Distribution/Admin 独立监听器、首错联动关闭和
有界优雅停机;端点正式勾选仍等待 PostgreSQL/Redis Adapter、命令入口与
Compose 集成测试。
有界优雅停机;端点正式勾选仍等待 Redis 活动池/原子提取 Adapter、PostgreSQL
管理面 Adapter、命令入口与 Compose 集成测试。
已新增公用 `domain/activitypool` 契约及并发安全内存参考实现Provider
Reconciler 通过 `UpsertFetched` 写入带供应商 TTL 和分配安全余量的批次;已覆盖
`usableUntil` 向 Worker Snapshot 的传播与 Gateway 本地截止过滤、
重复刷新、过期淘汰、独占提取、短期幂等及 Worker ownership 互斥。生产 Redis
Lua/Function Adapter 和多节点集成测试仍待实现。
## Task 11: Checker and Health Reducer

View File

@ -8,8 +8,9 @@
- [ ] Sequential 并发 Empty 只切换一次,旧 Upstream Proxy 自然耗尽。
- [ ] `pool.maxSize``fetch.maxTotal` 分别按当前库存和累计获取计数。
- [ ] Extract 只有 `AVAILABLE -> EXTRACTED`OpenAPI 不存在 release/renew。
- [ ] Extract 状态更新和审计记录位于同一数据库事务。
- [ ] `partial``allOrNothing` 均通过并发事务测试。
- [ ] Extract 状态更新和短期幂等结果位于同一个 Redis 原子操作。
- [ ] PostgreSQL 中不存在 Proxy 明细或逐次提取记录。
- [ ] `partial``allOrNothing` 均通过 Redis 原子并发测试。
- [ ] `reserveForGateway` 在所有提取路径上统一执行。
## 安全
@ -25,7 +26,8 @@
## 可用性
- [ ] PostgreSQL 和 Redis 跨可用区,有监控、备份和恢复演练证据。
- [ ] PostgreSQL 跨可用区,有监控、备份和恢复演练证据。
- [ ] Redis 跨可用区并受监控,完成丢失后由 Provider 重建活动池的演练。
- [ ] Gateway、Controller、Checker 均跨主机/可用区分散。
- [ ] PDB、优雅终止与最大连接时长的组合经过驱逐测试。
- [ ] Controller 断线时 Gateway 在 `maxStaleAge` 内继续,超限拒绝新请求。
@ -58,4 +60,3 @@
- [ ] 100k QPS 验证报告包含环境、命令、版本、场景、原始指标和结论。
任一关键一致性、安全或恢复项未完成时,不标记生产就绪。

View File

@ -3,12 +3,15 @@
## 1. 运行边界
- Gateway 是数据面,正常请求热路径不访问 PostgreSQL、Redis 或 Provider。
- Controller 是权威控制面,负责 Fetch、生命周期、所有权、Snapshot、Extract
和审计;多个副本只有一个 Provider 逻辑 Leader。
- Controller 编排 Fetch、生命周期、所有权、Snapshot 与 Extract多个副本只有
一个 Provider 逻辑 Leader。短效 Proxy 明细只存在于 Redis TTL 活动池和节点
内存,可由 Provider 重建。
- PostgreSQL 只持久化配置版本、Upstream/Routing 管理状态、Admin 审计与
outbox以及可选聚合指标不保存 Proxy 明细或逐次提取记录。
- Checker 执行有界健康探测,只上报 Observation最终状态由 Controller
reducer 决定。
- Extract 是一次性独占发放。提交后状态为 `EXTRACTED`,没有 Lease、续租或
Release 接口。
- Extract 是一次性独占发放。Redis 在单次原子操作中校验并从可分配活动池移除
候选,同时写入带 TTL 的幂等结果;没有 Lease、续租或 Release 接口。
- `reserveForGateway` 是共享池硬约束Extract 不得把 Gateway 库存清空。
- 集群峰值 100,000 QPS 是设计目标,只有完成本文容量验收后才能作为已验证
能力对外承诺。
@ -88,7 +91,8 @@ kubectl -n proxy-pool rollout status deployment/proxy-gateway --timeout=10m
liveness 失败。
- `/readyz`进程可以接收新工作。Gateway 只有在持有未超过 `maxStaleAge`
的完整 Snapshot 且仍有准入能力时才 Ready。
- Controller 只有在配置有效、存储可用、迁移兼容且控制接口已监听时才 Ready。
- Controller 按能力判定就绪Distribution/Fetch 依赖 Redis 活动池Admin 持久化
写依赖 PostgreSQL 与兼容迁移。PostgreSQL 故障不得单独使 Extract 返回 503。
- Checker 在任务消费与结果上报通道可用时 Ready。
- `/metrics`独立于业务入口NetworkPolicy 仅允许监控命名空间访问。
@ -140,9 +144,9 @@ HPA 使用 CPU/内存作为保护性信号;生产环境建议通过 Prometheus
3. Available Slots、Reserved、Active、各 Proxy 状态数量。
4. Provider Success、Empty、Duplicate-only、Error、429 与退避。
5. Checker 队列、SUSPECT 数、检查延迟和目标级失败。
6. Extract requested/returned、insufficient、冲突和审计写入
7. PostgreSQL 连接、锁等待、事务失败、WAL 与备份。
8. Redis 延迟、内存、主从状态和 Leader 租约抖动。
6. Extract requested/returned、insufficient、冲突和幂等命中
7. PostgreSQL 连接、管理事务失败、outbox backlog、WAL 与备份。
8. Redis 延迟、内存、TTL 淘汰、原子操作错误、主从状态和 Leader 租约抖动。
Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、request ID。
需要逐请求调查时使用受控、脱敏且采样的结构化日志。
@ -160,8 +164,8 @@ Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、requ
1. 停止接收新的 Extract/Admin 写请求。
2. 停止发起 Fetch释放 Provider Leader 租约。
3. 完成已进入数据库事务的 Extract 或回滚
4. 刷新 outbox、审计和 Worker ACK再关闭连接池。
3. 等待已提交 Redis 原子操作返回;未确认请求的客户端必须使用相同幂等键重试
4. 刷新 Admin outbox、审计和 Worker ACK再关闭 PostgreSQL/Redis 连接池。
### Checker
@ -184,16 +188,23 @@ Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、requ
### 7.2 PostgreSQL 不可用
1. Gateway 继续使用最后有效 Snapshot。
2. Controller 将 Distribution 和权威写操作置为不可用,避免返回未提交代理。
3. Provider Fetch 停止写入;已有流量不受影响。
4. 恢复后核对迁移、事务回滚、outbox backlog 与 Extract 审计连续性。
2. Redis 健康且运行配置有效时Distribution 继续执行原子 ExtractProvider
继续刷新 TTL 活动池。
3. 拒绝配置版本、Upstream/Routing 管理状态和其他需要 Admin 审计/outbox 的写入;
不得把 Proxy 明细临时落入 PostgreSQL。
4. 恢复后核对迁移、管理事务回滚、Admin 审计与 outbox backlog不存在 Proxy
明细或逐次提取记录恢复步骤。
### 7.3 Redis 不可用
1. Gateway 不受影响。
2. Controller 停止需要分布式互斥的高风险工作,防止多个 Fetch Leader。
3. 本地限流只作为临时降级,不能声称满足全局额度。
4. 恢复后确认 Leader 唯一、租约 epoch 单调和重复 Fetch 去重。
1. Gateway 继续读取节点内最后有效 Snapshot直到 `maxStaleAge`;请求热路径不
回查 PostgreSQL 或 Provider。
2. Distribution 立即失败关闭并返回 503禁止本地内存提取或 PostgreSQL 兜底。
3. Controller 停止活动池写入和需要分布式互斥的工作,防止多个 Fetch Leader
本地限流不能声称满足全局额度。
4. 恢复后确认 Leader 唯一和租约 epoch 单调,由 Provider 重新 Fetch 并构建 TTL
活动池,再恢复 Distribution。Redis 整体丢失会终止原活动池代次的排他状态和
短期幂等窗口;高可用、持久化、监控和告警必须明确并降低该风险。
### 7.4 Provider 故障
@ -213,11 +224,11 @@ Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、requ
### 7.6 Extract 库存不足
1. `partial` 返回实际数量;`allOrNothing` 不足时整批回滚
1. `partial` 返回实际数量;`allOrNothing` 不足时不改变 Redis 活动池
2. 检查 TTL、health age、filter、Worker ownership 和 `reserveForGateway`
3. 不得降低 `reserveForGateway` 到导致 Gateway 容量告警的水平。
4. 回收 Worker-owned Proxy 必须先 DRAINING、等待 active/reserved 为零、清除
ownership执行 `AVAILABLE -> EXTRACTED` 事务
ownership由 Redis 原子操作将其从可分配活动池移除并写入短期幂等结果
5. 已提取代理没有 Release客户端归还请求只记录为无效调用不恢复库存。
### 7.7 Checker 积压
@ -229,14 +240,17 @@ Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、requ
## 8. 备份与恢复
- PostgreSQL每日全量、连续 WAL/PITR至少每季度做恢复演练。
- Redis仅保存可重建协调状态不得把 Redis 备份当权威业务备份。
- PostgreSQL每日全量、连续 WAL/PITR保护配置版本、Upstream/Routing 管理
状态、Admin 审计与 outbox至少每季度做恢复演练。
- Redis保存可由 Provider 重建的 TTL 活动池、所有权/Leader 协调和短期幂等
结果;使用高可用与持久化降低窗口丢失风险,但不把它当长期业务档案。
- 配置:版本化保存校验通过的不可变 Revision 与校验和。
- Secret由密钥平台版本化日志和备份中不得出现明文。
恢复顺序PostgreSQL -> Redis -> Controller -> Checker -> Gateway。恢复后验证
Proxy 状态、Extraction Record、Worker ownership epoch、outbox 和配置 Revision
单调一致,再开放 Gateway 与 Distribution。
管理面恢复顺序PostgreSQL -> Controller/Admin代理运行面恢复顺序Redis ->
Provider 重建活动池 -> Controller/Checker -> 发布新 Snapshot。验证 Worker
ownership epoch、活动池 TTL、短期幂等窗口、outbox 和配置 Revision 一致后,再
开放相应能力Gateway 在旧 Snapshot 未超过 `maxStaleAge` 时无需等待数据库恢复。
## 9. Secret 轮换

View File

@ -11,15 +11,14 @@
- 产品设计、总体架构、项目结构、四项 ADR。
- 开发、配置、Distribution/Admin API、控制面协议、安全、测试、运维文档。
- 20 个配置场景和 35 张 Mermaid 架构/流程/状态/故障图。
- 版本化文档包 `proxy-pool-docs-v1.0.zip`,包含 50 个条目。
### 机器契约
- Distribution OpenAPI一次性独占提取、partial/allOrNothing、幂等键、
TTL/健康过滤结果与标准错误。
Redis TTL 活动池原子语义、TTL/健康过滤结果与标准错误。
- Admin OpenAPI状态、Upstream 启停、Routing 切换和配置重载。
- ProtobufWorker 注册、全量/增量 Snapshot、ACK、运行态/结果上报、Checker
任务与 Observation。
- ProtobufWorker 注册、全量/增量 Snapshot、`usable_until`、ACK、运行态/
结果上报、Checker 任务与 Observation。
### 核心参考实现
@ -29,8 +28,8 @@
原子计数1,000 goroutine 不超卖测试。
- `ROUTE-001 / ROUTE-004`:首条命中规则与 Concurrent Sequential 单次切换。
- `FETCH-005 / FETCH-006`Valid、Empty、DuplicateOnly、Error 分类。
- `DIST-001..003 / DIST-006..007`:内存事务模型验证独占提取、满足模式、TTL
健康时效与 Gateway 保留量1,000 并发不重复。
- `DIST-001..003 / DIST-006..007`:内存活动池参考实现验证独占提取、满足模式
TTL、健康时效与 Gateway 保留量1,000 并发不重复。
- `OPS-001`:完整 Snapshot 目标、epoch/version、校验和验证及原子替换。
- `CAP-001 / GW 热路径边界`:本地 Dispatch 条件过滤与原子容量预留。
- `GW-*`HTTP 正向代理、HTTPS CONNECT、双向 Tunnel、超时、重试、保护链与
@ -65,8 +64,12 @@ CI 已配置 Linux race job。Docker/Kubernetes 仅完成静态验证,没有
1. `cmd/proxy-gateway/controller/checker/loadgen` 进程装配。
2. Gateway 进程装配、生产连接池调优与代表性流量压测。
3. Provider 分布式 singleflight/Leader、长期凭据回收和累计额度执行器。
4. PostgreSQL repository、Extraction 行锁事务、Outbox 和迁移。
5. Redis Leader、速率限制、心跳与可重建协调适配器。
4. PostgreSQL 配置版本、Upstream/Routing 管理状态、Admin 审计/Outbox
repository 和迁移;可选聚合指标不包含 Proxy 明细。
5. Redis TTL 活动池、原子独占提取、短期幂等结果、Leader、速率限制、心跳与
所有权适配器;活动池可由 Provider 重建。接线时必须验证进入
`allocationSafetyMargin` 的不可分配条目不会长期占用补池额度,同时不得
突破 `pool.maxSize` 硬上限。
6. Worker ownership drain/ACK/过期回收和网络快照流。
7. Checker 调度、探测器和健康 reducer。
8. Admin/Distribution 细粒度授权、分布式限流和审计查询。

View File

@ -5,7 +5,8 @@
- Gateway、Distribution、Admin 和 Metrics 为四个独立监听边界。
- Provider API 和上游 Proxy 属于外部不可信网络。
- Worker 与 Controller 通道必须进行双向身份校验并绑定 cluster/worker。
- PostgreSQL 保存权威状态Redis 数据默认按可重建缓存与协调信息处理。
- PostgreSQL 保存管理面持久状态Redis 保存可重建的短效 Proxy 活动池与协调
信息Proxy 地址和凭据不写入 PostgreSQL。
## 2. 入口控制

View File

@ -4,7 +4,9 @@
- 先在隔离环境建立 15 分钟稳定基线,再注入单一故障。
- 每次只改变一个变量,记录开始/恢复时间与所有指标。
- 任何数据库写故障后都核对 Proxy 状态、审计和 outbox而非只看 HTTP 状态码。
- PostgreSQL 写故障后核对配置版本、Upstream/Routing 管理状态、Admin 审计和
outboxRedis 写故障后核对活动池、独占提取和短期幂等键,不能只看 HTTP
状态码。
- 故障恢复后至少观察两个健康检查周期和一个 Snapshot 完整发布周期。
## 场景
@ -22,15 +24,18 @@ duplicate-only 不触发 Sequential 切换。
- 断开 60 秒、连接池耗尽、锁等待、事务提交失败、只读切换。
预期Gateway 继续使用最后 SnapshotExtract 不返回未提交记录;恢复后 outbox
补发且不重复应用。
预期Gateway 继续使用最后 SnapshotRedis 健康时 Fetch、活动池更新和 Extract
不被 PostgreSQL 故障阻断;依赖持久化的配置/Admin 写操作失败关闭。恢复后
outbox 补发且不重复应用PostgreSQL 中没有 Proxy 明细或逐次提取记录需要恢复。
### Redis
- 断开、延迟 2 秒、Leader key 丢失、主从切换。
- 断开、延迟 2 秒、Leader key 丢失、活动池与幂等键丢失、主从切换。
预期Gateway 无影响Provider Fetch 不出现多个有效 Leader全局限流明确降级
恢复后 epoch 单调。
预期Gateway 在 Snapshot 新鲜度窗口内无影响Distribution 失败关闭并返回 503
不得改写 PostgreSQL 兜底Provider Fetch 不出现多个有效 Leader全局限流明确
降级。恢复后 epoch 单调,由 Provider 重新拉取并构建 TTL 活动池;已经丢失的短期
幂等窗口和旧活动池排他状态不伪装成仍然有效,重建结果视为新的活动池代次。
### Controller
@ -55,10 +60,12 @@ FetchDelta 缺口触发完整 Snapshot。
### Distribution
- 100 个并发请求争用相同 Proxy数据库在事务提交时断开Gateway 同时满载。
- 100 个并发请求争用相同 ProxyRedis 在原子操作执行或主从切换时断开Gateway
同时满载。
预期:每个 Proxy 最多返回一次提交失败不返回代理allOrNothing 整批回滚;
池中始终剩余 `reserveForGateway`;没有 Release 恢复路径。
预期:同一 Redis 活动池代次内每个 Proxy 最多返回一次;该代次仍保留时,未确认
成功的客户端使用同一幂等键重试并得到已缓存结果或明确失败,不出现部分可见状态;
allOrNothing 整批不变;池中始终剩余 `reserveForGateway`;没有 Release 恢复路径。
### 配置与 Secret
@ -66,4 +73,3 @@ FetchDelta 缺口触发完整 Snapshot。
预期:新 Revision 整体拒绝,旧不可变 Snapshot 继续;认证错误计 Error 而不是
Empty日志不出现 Secret。

View File

@ -5,6 +5,8 @@
- 先证明领域不变量,再证明 Adapter 契约,最后证明跨进程行为。
- 并发测试必须在 race detector 下运行,不能只依赖单线程示例。
- 时间、随机、网络、Provider 和存储均通过可替换接口或 fixture 控制。
- 生产 Adapter 必须证明数据边界Proxy 明细只进入 Redis TTL 活动池和节点
内存PostgreSQL 不得出现 Proxy 明细或逐次提取记录。
- 100,000 QPS 是待验证的集群目标,不以架构图、副本数或短时峰值替代证据。
- 性能通过与正确性通过相互独立;高 QPS 下出现超卖、重复 Extract 或状态
回退时,结果一律失败。
@ -24,9 +26,11 @@
### 契约测试
- Provider 响应模板的大小、超时、函数白名单和解析边界。
- PostgreSQL `FOR UPDATE SKIP LOCKED` 并发批量提取。
- Outbox 状态更新、发布与幂等重放。
- Redis Leader 租约、限流和失联恢复。
- Redis Lua/Function 对候选筛选、库存预留、独占移除和短期幂等结果执行单次
原子操作;并发与主从切换下不得部分提交。
- PostgreSQL 配置版本、Upstream/Routing 管理状态、Admin 审计与 outbox 的事务
更新及幂等重放;测试库断言不包含 Proxy 明细或逐次提取记录。
- Redis TTL 活动池、Leader 租约、限流、短期幂等窗口和失联恢复。
- Snapshot/Delta/ACK/Report 的版本与校验和兼容性。
- OpenAPI 错误模型、认证矩阵、批量 fulfillment。
@ -34,7 +38,8 @@
- HTTP 正向代理成功、上游连接前失败和安全重试。
- HTTPS CONNECT 建立后不透明重放。
- Controller Fetch -> Check -> AVAILABLE -> Worker Snapshot -> Gateway 转发。
- Controller Fetch -> Redis TTL 活动池 -> Check -> AVAILABLE -> Worker Snapshot ->
Gateway 转发Gateway 请求线程始终只读节点内存。
- Distribution 原子提取后 Gateway 不再分配同一 Proxy。
- 配置热更新失败保留旧 Revision成功后新请求使用新 Snapshot。
@ -51,9 +56,10 @@
重置 EmptySuccess 重置 Empty。
6. **Sequential 竞态**:达到阈值时多协程只能将 A 切到 B 一次,不能越过 B。
7. **Drain**:切换或禁用 Upstream 后停止新分配,已有连接完成后才回收。
8. **Extract 竞态**:多个请求并发提取同一候选集合,每个 Proxy 最多返回一次。
9. **Extract 批量语义**partial 提交实际数量allOrNothing 不足时状态和审计
全回滚;始终保留 `reserveForGateway`
8. **Extract 竞态**:多个请求并发提取同一候选集合,同一 Redis 活动池代次内
每个 Proxy 最多返回一次。
9. **Extract 批量语义**partial 提交实际数量allOrNothing 不足时 Redis
活动池和幂等结果均不变;始终保留 `reserveForGateway`
10. **Worker ownership**:回收过程严格经过 DRAINING、ACK、active/reserved=0、
unowned旧 Snapshot 不可再分配。
11. **控制面故障**Redis、PostgreSQL、Controller、Checker 与 Provider 分别
@ -70,8 +76,9 @@ go vet ./...
go build ./...
```
需要 PostgreSQL/Redis 的测试使用独立数据库和短生命周期容器,不复用开发数据。
测试结束后验证没有残留 Worker ownership、Leader 租约或未提交 Extraction Record。
需要 PostgreSQL/Redis 的测试使用独立实例和短生命周期容器,不复用开发数据。
测试结束后验证没有残留 Worker ownership、Leader 租约、活动池条目或幂等键,
并检查 PostgreSQL 中不存在 Proxy 明细和逐次提取记录。
## 5. 负载模型
@ -109,7 +116,7 @@ go build ./...
业务 SLO 由产品最终确认,但至少满足以下工程门槛:
- 无容量超卖、负计数、重复 Extract、审计缺失或状态非法回退。
- 无容量超卖、负计数、幂等窗口内重复 Extract、Admin 审计缺失或状态非法回退。
- 100k 峰值期间无进程 OOM、FD 耗尽、无界队列或全局锁热点。
- Gateway 热路径在 PostgreSQL、Redis、Provider 失效时不发起同步访问。
- p99 Dispatch 小于 100 微秒的设计预算需要在 100k Proxy Snapshot 下单独证明。
@ -125,6 +132,6 @@ go build ./...
配置Snapshot 规模、Proxy 容量、路由、重试、日志级别
场景:协议、连接复用、响应体、持续时间、升压曲线、故障注入
结果QPS、建连速率、active、p50/p95/p99、错误、CPU、RSS、GC、FD、网络
不变量capacity、ownership、extraction、audit、reserve 检查结果
不变量capacity、ownership、extraction、idempotency、admin audit、reserve 检查结果
结论:通过/失败,以及适用边界
```

View File

@ -19,6 +19,7 @@ upstreams:
proxyAuth: {type: response}
pool: {maxSize: 1000}
capacity: {maxConcurrencyPerProxy: 10}
lifecycle: {ttl: 2m, allocationSafetyMargin: 15s}
fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1}
provider-b:
enabled: true
@ -28,4 +29,5 @@ upstreams:
proxyAuth: {type: response}
pool: {maxSize: 1000}
capacity: {maxConcurrencyPerProxy: 10}
lifecycle: {ttl: 2m, allocationSafetyMargin: 15s}
fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1}

View File

@ -19,6 +19,7 @@ upstreams:
proxyAuth: {type: response}
pool: {maxSize: 1000}
capacity: {maxConcurrencyPerProxy: 10}
lifecycle: {ttl: 2m, allocationSafetyMargin: 15s}
fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1}
provider-b:
enabled: true
@ -28,4 +29,5 @@ upstreams:
proxyAuth: {type: response}
pool: {maxSize: 1000}
capacity: {maxConcurrencyPerProxy: 10}
lifecycle: {ttl: 2m, allocationSafetyMargin: 15s}
fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1}

View File

@ -6,7 +6,7 @@ import (
"strings"
"testing"
projectconfig "github.com/proxy-pool/proxy-pool/internal/config"
projectconfig "proxy-pool/internal/config"
)
func TestAllExamplesLoadStrictly(t *testing.T) {

View File

@ -40,8 +40,8 @@ Extract 请求使用哪些 Upstream。
- 代理分配必须原子预留容量,防止并发超卖。
- 代理唯一键为 `scheme + host + port + username + credentialVersion`;日志
和指标不得暴露密码。
- TTL 优先级为响应 `expiresAt`、响应 `ttl`、配置固定 TTL、不过期;内部
时间统一 UTC。
- TTL 优先级为响应 `expiresAt`、响应 `ttl`、配置固定 TTL;活动池中的每个
Proxy 必须有明确过期时间,内部时间统一 UTC。
- 健康检查至少区分全局健康和 Routing/目标健康,并使用抖动和并发上限。
- GET/HEAD 可按配置安全重试非幂等方法默认不自动重试CONNECT 建立后
不透明重放。
@ -55,8 +55,8 @@ Extract 请求使用哪些 Upstream。
- 数据面采用多 Worker本地不可变代理快照和本地容量计数。
- 同一代理必须由单个 Worker 所有,或由控制面下发容量切片;禁止每请求
访问 Redis 做全局并发计数。
- 控制面集中 Provider 获取、独立限流、singleflight、Leader 选举、状态
持久化和快照分发
- 控制面集中 Provider 获取、独立限流、singleflight、Leader 选举和快照分发。
Proxy 明细只进入 Redis TTL 活动池,不写 PostgreSQL
- 副本数必须由单 Worker 实测能力、目标利用率和故障域余量计算。
## 配置模型
@ -82,4 +82,3 @@ Routing 自上而下匹配,首条命中停止;支持 Gateway 与 Extract 两
- Provider 超时不得计入 Empty Fetch。
- 重复代理不得重复入池,也不得触发空结果切换。
- 所有 Upstream 不可用时按显式策略执行。

2
go.mod
View File

@ -1,4 +1,4 @@
module github.com/proxy-pool/proxy-pool
module proxy-pool
go 1.26.0

View File

@ -5,10 +5,10 @@ import (
"testing"
"time"
"github.com/proxy-pool/proxy-pool/internal/adapters/providerapi"
"github.com/proxy-pool/proxy-pool/internal/config"
"github.com/proxy-pool/proxy-pool/internal/gateway/transport"
"github.com/proxy-pool/proxy-pool/internal/platform/credentials"
"proxy-pool/internal/adapters/providerapi"
"proxy-pool/internal/config"
"proxy-pool/internal/gateway/transport"
"proxy-pool/internal/platform/credentials"
)
func TestParsedCredentialsResolveInGatewayTransport(t *testing.T) {

View File

@ -12,8 +12,8 @@ import (
"strings"
"time"
"github.com/proxy-pool/proxy-pool/internal/config"
controllerProvider "github.com/proxy-pool/proxy-pool/internal/controller/provider"
"proxy-pool/internal/config"
controllerProvider "proxy-pool/internal/controller/provider"
)
const defaultRetryAfterMax = 30 * time.Second

View File

@ -12,8 +12,8 @@ import (
"testing"
"time"
"github.com/proxy-pool/proxy-pool/internal/config"
controllerProvider "github.com/proxy-pool/proxy-pool/internal/controller/provider"
"proxy-pool/internal/config"
controllerProvider "proxy-pool/internal/controller/provider"
)
var _ controllerProvider.ProviderAdapter = (*HTTPAdapter)(nil)

View File

@ -4,7 +4,7 @@ import (
"errors"
"testing"
"github.com/proxy-pool/proxy-pool/internal/config"
"proxy-pool/internal/config"
)
func TestTemplateParserRejectsNestedRanges(t *testing.T) {

View File

@ -11,9 +11,9 @@ import (
"text/template"
"time"
"github.com/proxy-pool/proxy-pool/internal/config"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
"github.com/proxy-pool/proxy-pool/internal/platform/credentials"
"proxy-pool/internal/config"
proxyDomain "proxy-pool/internal/domain/proxy"
"proxy-pool/internal/platform/credentials"
)
const (

View File

@ -10,10 +10,10 @@ import (
"testing"
"time"
"github.com/proxy-pool/proxy-pool/internal/config"
controllerProvider "github.com/proxy-pool/proxy-pool/internal/controller/provider"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
"github.com/proxy-pool/proxy-pool/internal/platform/credentials"
"proxy-pool/internal/config"
controllerProvider "proxy-pool/internal/controller/provider"
proxyDomain "proxy-pool/internal/domain/proxy"
"proxy-pool/internal/platform/credentials"
)
var _ controllerProvider.Parser = (*TemplateParser)(nil)

View File

@ -113,6 +113,7 @@ type Extraction struct {
MinRemainingTTL Duration `yaml:"minRemainingTTL"`
MaxHealthCheckAge Duration `yaml:"maxHealthCheckAge"`
ReserveForGateway int `yaml:"reserveForGateway"`
IdempotencyTTL Duration `yaml:"idempotencyTTL"`
}
type Metrics struct {

View File

@ -6,6 +6,7 @@ import (
"path/filepath"
"strings"
"testing"
"time"
"go.yaml.in/yaml/v4"
)
@ -352,6 +353,22 @@ func TestValidateRejectsInvalidConfigurationMatrix(t *testing.T) {
},
want: "reserveForGateway",
},
{
name: "negative extraction idempotency ttl",
mutate: func(cfg *Config) {
cfg.Distribution.Extraction.IdempotencyTTL = Duration(-time.Second)
},
want: "idempotencyTTL",
},
{
name: "enabled upstream without ttl",
mutate: func(cfg *Config) {
upstream := cfg.Upstreams["provider-a"]
upstream.Lifecycle.TTL = 0
cfg.Upstreams["provider-a"] = upstream
},
want: "lifecycle.ttl",
},
{
name: "no enabled upstream",
mutate: func(cfg *Config) {

View File

@ -81,6 +81,9 @@ func Validate(cfg *Config) error {
if err := requireNonNegative("distribution.reserveForGateway", cfg.Distribution.Extraction.ReserveForGateway); err != nil {
return err
}
if err := requireNonNegative("distribution.idempotencyTTL", cfg.Distribution.Extraction.IdempotencyTTL); err != nil {
return err
}
}
return nil
}
@ -312,7 +315,7 @@ func validateUpstream(name string, upstream Upstream) error {
if err := requirePositive(scope+" capacity.maxConcurrencyPerProxy", upstream.Capacity.MaxConcurrencyPerProxy); err != nil {
return err
}
if err := requireNonNegative(scope+" lifecycle.ttl", upstream.Lifecycle.TTL); err != nil {
if err := requirePositive(scope+" lifecycle.ttl", upstream.Lifecycle.TTL); err != nil {
return err
}
if err := requireNonNegative(scope+" lifecycle.allocationSafetyMargin", upstream.Lifecycle.AllocationSafetyMargin); err != nil {

View File

@ -7,8 +7,8 @@ import (
"net/http"
"strings"
"github.com/proxy-pool/proxy-pool/internal/platform/httpapi"
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
"proxy-pool/internal/platform/httpapi"
"proxy-pool/internal/platform/httpsecurity"
)
const (

View File

@ -9,8 +9,8 @@ import (
"strings"
"testing"
"github.com/proxy-pool/proxy-pool/internal/platform/httpapi"
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
"proxy-pool/internal/platform/httpapi"
"proxy-pool/internal/platform/httpsecurity"
)
func TestHandlerReturnsStatusWithoutSensitiveDetails(t *testing.T) {

View File

@ -7,10 +7,10 @@ import (
"strings"
"time"
controllerExtraction "github.com/proxy-pool/proxy-pool/internal/controller/extraction"
domainExtraction "github.com/proxy-pool/proxy-pool/internal/domain/extraction"
"github.com/proxy-pool/proxy-pool/internal/platform/httpapi"
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
controllerExtraction "proxy-pool/internal/controller/extraction"
domainExtraction "proxy-pool/internal/domain/extraction"
"proxy-pool/internal/platform/httpapi"
"proxy-pool/internal/platform/httpsecurity"
)
const (

View File

@ -11,10 +11,10 @@ import (
"testing"
"time"
controllerExtraction "github.com/proxy-pool/proxy-pool/internal/controller/extraction"
domainExtraction "github.com/proxy-pool/proxy-pool/internal/domain/extraction"
"github.com/proxy-pool/proxy-pool/internal/platform/httpapi"
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
controllerExtraction "proxy-pool/internal/controller/extraction"
domainExtraction "proxy-pool/internal/domain/extraction"
"proxy-pool/internal/platform/httpapi"
"proxy-pool/internal/platform/httpsecurity"
)
func TestNewHandlerValidatesDependenciesAndBodyLimit(t *testing.T) {

View File

@ -6,7 +6,7 @@ import (
"fmt"
"time"
domain "github.com/proxy-pool/proxy-pool/internal/domain/extraction"
domain "proxy-pool/internal/domain/extraction"
)
var (
@ -24,6 +24,7 @@ type Policy struct {
MinRemainingTTL time.Duration
MaxHealthCheckAge time.Duration
ReserveForGateway int
IdempotencyTTL time.Duration
}
type Filters struct {
@ -85,7 +86,7 @@ func NewService(store domain.Store, policy Policy, admission Admission, now func
return nil, fmt.Errorf("%w: store is required", ErrInvalidServicePolicy)
}
if policy.MaxCountPerRequest <= 0 || policy.MinRemainingTTL < 0 ||
policy.MaxHealthCheckAge < 0 || policy.ReserveForGateway < 0 {
policy.MaxHealthCheckAge < 0 || policy.ReserveForGateway < 0 || policy.IdempotencyTTL < 0 {
return nil, ErrInvalidServicePolicy
}
if policy.DefaultFulfillment != domain.Partial && policy.DefaultFulfillment != domain.AllOrNothing {
@ -119,13 +120,14 @@ func (s *Service) Extract(ctx context.Context, request Request) (Response, error
now := s.now().UTC()
clientID := request.ClientID
if clientID == "" {
clientID = "anonymous"
clientID = request.SourceIP
}
result, err := s.store.Extract(ctx, domain.Command{
RequestID: request.RequestID,
ClientID: clientID,
SourceIP: request.SourceIP,
IdempotencyKey: request.IdempotencyKey,
IdempotencyTTL: s.policy.IdempotencyTTL,
Requested: request.Count,
Fulfillment: fulfillment,
Now: now,

View File

@ -6,7 +6,9 @@ import (
"testing"
"time"
domain "github.com/proxy-pool/proxy-pool/internal/domain/extraction"
"proxy-pool/internal/domain/activitypool"
domain "proxy-pool/internal/domain/extraction"
proxyDomain "proxy-pool/internal/domain/proxy"
)
func TestServiceAppliesPolicyAndBuildsResponse(t *testing.T) {
@ -30,6 +32,7 @@ func TestServiceAppliesPolicyAndBuildsResponse(t *testing.T) {
MinRemainingTTL: 30 * time.Second,
MaxHealthCheckAge: 15 * time.Second,
ReserveForGateway: 5,
IdempotencyTTL: 2 * time.Minute,
}, allowAllAdmission{}, func() time.Time { return now })
if err != nil {
t.Fatalf("NewService(): %v", err)
@ -53,7 +56,10 @@ func TestServiceAppliesPolicyAndBuildsResponse(t *testing.T) {
t.Fatalf("store command policy = %+v", store.command)
}
if store.command.RequestID != "req-1" || store.command.ClientID != "client-1" {
t.Fatalf("store command audit context = %+v", store.command)
t.Fatalf("store command request context = %+v", store.command)
}
if store.command.IdempotencyTTL != 2*time.Minute {
t.Fatalf("store command idempotency TTL = %s, want 2m", store.command.IdempotencyTTL)
}
if response.RequestID != "req-1" || response.Requested != 2 || response.Returned != 1 {
t.Fatalf("response = %+v", response)
@ -116,15 +122,45 @@ func TestServiceAppliesAdmissionBeforeStoreUsingStableIdentity(t *testing.T) {
}
}
func TestServiceUsesSourceIdentityForEphemeralIdempotency(t *testing.T) {
store := &recordingStore{}
service, err := NewService(store, Policy{
MaxCountPerRequest: 1,
DefaultFulfillment: domain.Partial,
}, allowAllAdmission{}, time.Now)
if err != nil {
t.Fatalf("NewService(): %v", err)
}
_, err = service.Extract(context.Background(), Request{
RequestID: "req-1",
SourceIP: "192.0.2.30",
Count: 1,
})
if err != nil {
t.Fatalf("Extract(): %v", err)
}
if store.command.ClientID != "192.0.2.30" {
t.Fatalf("store command ClientID = %q, want source IP", store.command.ClientID)
}
}
func TestServiceIdempotentReplayKeepsOriginalExtractionTime(t *testing.T) {
firstTime := time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC)
secondTime := firstTime.Add(time.Minute)
store := domain.NewMemoryStore([]domain.Candidate{{
ID: "p1",
State: domain.Available,
ExpiresAt: firstTime.Add(2 * time.Minute),
LastCheckedAt: firstTime,
}})
store := activitypool.NewMemoryPool()
checkedAt := firstTime
upserted, err := store.UpsertFetched(context.Background(), "provider-a", activitypool.FetchedBatch{
ObservedAt: firstTime,
ConfiguredTTL: 2 * time.Minute,
Proxies: []proxyDomain.Proxy{{
ID: "p1", Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8080,
State: proxyDomain.StateAvailable, LastCheckedAt: &checkedAt,
}},
})
if err != nil || upserted.Inserted != 1 {
t.Fatalf("UpsertFetched() = %+v, %v", upserted, err)
}
times := []time.Time{firstTime, secondTime}
service, err := NewService(store, Policy{
MaxCountPerRequest: 1,

View File

@ -5,7 +5,7 @@ import (
"fmt"
"sync"
"github.com/proxy-pool/proxy-pool/internal/domain/upstream"
"proxy-pool/internal/domain/upstream"
)
var (

View File

@ -3,7 +3,7 @@ package pool
import (
"time"
ownershipDomain "github.com/proxy-pool/proxy-pool/internal/domain/ownership"
ownershipDomain "proxy-pool/internal/domain/ownership"
)
var (

View File

@ -9,7 +9,9 @@ import (
"testing"
"time"
extractionDomain "github.com/proxy-pool/proxy-pool/internal/domain/extraction"
"proxy-pool/internal/domain/activitypool"
extractionDomain "proxy-pool/internal/domain/extraction"
proxyDomain "proxy-pool/internal/domain/proxy"
)
func TestOwnershipManagerPreventsDualAssignment(t *testing.T) {
@ -64,9 +66,7 @@ func TestOwnershipManagerRenewsOnlyCurrentAssignment(t *testing.T) {
func TestSharedRepositoryMakesOwnershipAndExtractionMutuallyExclusive(t *testing.T) {
for iteration := range 100 {
now := time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC)
store := extractionDomain.NewMemoryStore([]extractionDomain.Candidate{{
ID: "proxy-1", State: extractionDomain.Available,
}})
store := newTestActivityPool(t, now, "proxy-1")
manager, err := NewOwnershipManager(store)
if err != nil {
t.Fatalf("iteration %d NewOwnershipManager(): %v", iteration, err)
@ -159,15 +159,29 @@ func TestOwnershipManagerExpiresCrashedWorkerAssignment(t *testing.T) {
func newTestOwnershipManager(t *testing.T, proxyIDs ...string) *OwnershipManager {
t.Helper()
candidates := make([]extractionDomain.Candidate, 0, len(proxyIDs))
for _, proxyID := range proxyIDs {
candidates = append(candidates, extractionDomain.Candidate{
ID: proxyID, State: extractionDomain.Available,
})
}
manager, err := NewOwnershipManager(extractionDomain.NewMemoryStore(candidates))
now := time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC)
manager, err := NewOwnershipManager(newTestActivityPool(t, now, proxyIDs...))
if err != nil {
t.Fatalf("NewOwnershipManager(): %v", err)
}
return manager
}
func newTestActivityPool(t *testing.T, now time.Time, proxyIDs ...string) *activitypool.MemoryPool {
t.Helper()
proxies := make([]proxyDomain.Proxy, 0, len(proxyIDs))
for index, proxyID := range proxyIDs {
proxies = append(proxies, proxyDomain.Proxy{
ID: proxyID, Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.1",
Port: uint16(8000 + index), State: proxyDomain.StateAvailable,
})
}
store := activitypool.NewMemoryPool()
result, err := store.UpsertFetched(context.Background(), "provider-a", activitypool.FetchedBatch{
ObservedAt: now, ConfiguredTTL: 10 * time.Minute, Proxies: proxies,
})
if err != nil || result.Inserted != len(proxyIDs) {
t.Fatalf("UpsertFetched() = %+v, %v", result, err)
}
return store
}

View File

@ -4,7 +4,7 @@ import (
"errors"
"time"
"github.com/proxy-pool/proxy-pool/internal/domain/upstream"
"proxy-pool/internal/domain/upstream"
)
var ErrInvalidReconcilePolicy = errors.New("invalid pool reconcile policy")

View File

@ -4,8 +4,8 @@ import (
"testing"
"time"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
"github.com/proxy-pool/proxy-pool/internal/domain/upstream"
proxyDomain "proxy-pool/internal/domain/proxy"
"proxy-pool/internal/domain/upstream"
)
func TestPoolReconcilerSignalsOnlyWhenSlotsAreLowAndWholeFetchFits(t *testing.T) {

View File

@ -4,8 +4,9 @@ import (
"context"
"time"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
"github.com/proxy-pool/proxy-pool/internal/domain/upstream"
"proxy-pool/internal/domain/activitypool"
proxyDomain "proxy-pool/internal/domain/proxy"
"proxy-pool/internal/domain/upstream"
)
type FetchResponse struct {
@ -31,13 +32,6 @@ type Parser interface {
Parse(context.Context, []byte) ([]proxyDomain.Proxy, error)
}
// CandidateSink owns deduplication, is concurrency-safe, and returns the new
// count. On error it must not retain candidates; a successful count must be
// between zero and len(candidates).
type CandidateSink interface {
Add(context.Context, string, []proxyDomain.Proxy) (int, error)
}
type Result struct {
UpstreamID string
Class upstream.FetchClass
@ -55,7 +49,7 @@ type ResultRecorder interface {
type Ports struct {
Adapter ProviderAdapter
Parser Parser
Candidates CandidateSink
Activity activitypool.Upserter
Results ResultRecorder
Capacity upstream.FetchCapacity
}

View File

@ -8,8 +8,9 @@ import (
"sync"
"time"
"github.com/proxy-pool/proxy-pool/internal/domain/upstream"
"github.com/proxy-pool/proxy-pool/internal/platform/coalesce"
"proxy-pool/internal/domain/activitypool"
"proxy-pool/internal/domain/upstream"
"proxy-pool/internal/platform/coalesce"
)
type Config struct {
@ -18,6 +19,8 @@ type Config struct {
Timeout time.Duration
MaxAttempts int
MaxInFlight int
TTL time.Duration
AllocationSafetyMargin time.Duration
Retry RetryConfig
}
@ -62,6 +65,10 @@ func NewReconciler(config Config, ports Ports, runtimes ...Runtime) (*Reconciler
if config.RequestInterval < 0 || config.Timeout <= 0 || config.MaxAttempts <= 0 || config.MaxInFlight <= 0 {
return nil, fmt.Errorf("new provider reconciler: fetch limits must be positive")
}
if config.TTL < 0 || config.AllocationSafetyMargin < 0 ||
(config.TTL > 0 && config.AllocationSafetyMargin >= config.TTL) {
return nil, fmt.Errorf("new provider reconciler: lifecycle settings are invalid")
}
if config.Retry.Initial < 0 || config.Retry.Max < 0 || config.Retry.Jitter < 0 || config.Retry.Jitter > 100 {
return nil, fmt.Errorf("new provider reconciler: retry settings are invalid")
}
@ -71,7 +78,7 @@ func NewReconciler(config Config, ports Ports, runtimes ...Runtime) (*Reconciler
if config.Retry.Max > 0 && config.Retry.Initial > config.Retry.Max {
return nil, fmt.Errorf("new provider reconciler: retry initial delay exceeds maximum")
}
if ports.Adapter == nil || ports.Parser == nil || ports.Candidates == nil ||
if ports.Adapter == nil || ports.Parser == nil || ports.Activity == nil ||
ports.Results == nil || ports.Capacity == nil {
return nil, fmt.Errorf("new provider reconciler: all ports are required")
}
@ -202,7 +209,14 @@ func (r *Reconciler) fetchAttempt(ctx context.Context, attempt int) (FetchRespon
if expected := permit.Expected(); expected < len(retained) {
retained = retained[:expected]
}
newCount, candidateErr = r.ports.Candidates.Add(callCtx, r.config.UpstreamID, retained)
upserted, err := r.ports.Activity.UpsertFetched(callCtx, r.config.UpstreamID, activitypool.FetchedBatch{
ObservedAt: r.runtime.Clock.Now().UTC(),
ConfiguredTTL: r.config.TTL,
AllocationSafetyMargin: r.config.AllocationSafetyMargin,
Proxies: retained,
})
candidateErr = err
newCount = upserted.Inserted
}
}
if callErr == nil && parseErr == nil && candidateErr == nil {

View File

@ -8,10 +8,47 @@ import (
"testing"
"time"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
"github.com/proxy-pool/proxy-pool/internal/domain/upstream"
"proxy-pool/internal/domain/activitypool"
proxyDomain "proxy-pool/internal/domain/proxy"
"proxy-pool/internal/domain/upstream"
)
func TestReconcilerWritesProviderTTLPolicyToEphemeralPool(t *testing.T) {
now := time.Date(2026, 7, 29, 12, 0, 0, 0, time.UTC)
clock := newFakeClock(now)
results := make(chan Result, 1)
batches := make(chan activitypool.FetchedBatch, 1)
ports := successfulPorts(func() {}, results)
ports.Activity = activitySinkFunc(func(_ context.Context, upstreamID string, batch activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
if upstreamID != "provider-a" {
t.Errorf("upstream ID = %q, want provider-a", upstreamID)
}
batches <- batch
return activitypool.UpsertResult{Accepted: 1, Inserted: 1}, nil
})
reconciler, err := NewReconciler(Config{
UpstreamID: "provider-a",
Timeout: time.Second,
MaxAttempts: 1,
MaxInFlight: 1,
TTL: 30 * time.Second,
AllocationSafetyMargin: 3 * time.Second,
}, ports, Runtime{Clock: clock})
if err != nil {
t.Fatalf("NewReconciler(): %v", err)
}
result := runSingleReconcile(t, reconciler, results)
if result.ValidCount != 1 || result.NewCount != 1 {
t.Fatalf("result = %+v, want valid=1 new=1", result)
}
batch := <-batches
if !batch.ObservedAt.Equal(now) || batch.ConfiguredTTL != 30*time.Second ||
batch.AllocationSafetyMargin != 3*time.Second {
t.Fatalf("activity batch = %+v", batch)
}
}
func TestReconcilerCoalescesConcurrentNotifications(t *testing.T) {
var calls atomic.Int64
result := make(chan Result, 1)
@ -28,8 +65,8 @@ func TestReconcilerCoalescesConcurrentNotifications(t *testing.T) {
Parser: parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) {
return []proxyDomain.Proxy{{ID: "proxy-1"}}, nil
}),
Candidates: candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) {
return 1, nil
Activity: activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
return activitypool.UpsertResult{Accepted: 1, Inserted: 1}, nil
}),
Results: resultRecorderFunc(func(got Result) { result <- got }),
Capacity: unlimitedFetchCapacity{},
@ -122,8 +159,8 @@ func TestReconcilerRetriesErrorsWithExponentialBackoffAndJitter(t *testing.T) {
}
return FetchResponse{Body: []byte("fixture")}, nil
})
ports.Candidates = candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) {
return 0, nil
ports.Activity = activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
return activitypool.UpsertResult{Accepted: 1}, nil
})
reconciler, err := NewReconciler(Config{
UpstreamID: "provider-a",
@ -196,8 +233,8 @@ func TestReconcilerClassifiesFetchResults(t *testing.T) {
Parser: parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) {
return make([]proxyDomain.Proxy, tt.candidates), tt.parseErr
}),
Candidates: candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) {
return tt.newCount, nil
Activity: activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
return activitypool.UpsertResult{Accepted: tt.candidates, Inserted: tt.newCount}, nil
}),
Results: resultRecorderFunc(func(got Result) { results <- got }),
Capacity: unlimitedFetchCapacity{},
@ -325,9 +362,9 @@ func TestReconcilerAppliesAttemptTimeoutToEveryPort(t *testing.T) {
assertDeadline(ctx, "Parser.Parse")
return []proxyDomain.Proxy{{ID: "proxy-1"}}, nil
})
ports.Candidates = candidateSinkFunc(func(ctx context.Context, _ string, _ []proxyDomain.Proxy) (int, error) {
assertDeadline(ctx, "CandidateSink.Add")
return 1, nil
ports.Activity = activitySinkFunc(func(ctx context.Context, _ string, _ activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
assertDeadline(ctx, "ActivitySink.UpsertFetched")
return activitypool.UpsertResult{Accepted: 1, Inserted: 1}, nil
})
reconciler, err := NewReconciler(Config{
UpstreamID: "provider-a",
@ -493,14 +530,14 @@ func TestReconcilerUsesConfiguredMaxInFlight(t *testing.T) {
}
}
func TestReconcilerDoesNotRefetchWhenCandidateSinkFails(t *testing.T) {
func TestReconcilerDoesNotRefetchWhenActivitySinkFails(t *testing.T) {
clock := newFakeClock(time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC))
sleeper := &fakeSleeper{clock: clock}
results := make(chan Result, 3)
var calls atomic.Int64
ports := successfulPorts(func() { calls.Add(1) }, results)
ports.Candidates = candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) {
return 0, errors.New("candidate store unavailable")
ports.Activity = activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
return activitypool.UpsertResult{}, errors.New("activity pool unavailable")
})
reconciler, err := NewReconciler(Config{
UpstreamID: "provider-a",
@ -597,11 +634,11 @@ func TestReconcilerCompletesFetchPermitWithFetchedAndRetainedCounts(t *testing.T
ports.Parser = parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) {
return []proxyDomain.Proxy{{ID: "p1"}, {ID: "p2"}, {ID: "p3"}}, nil
})
ports.Candidates = candidateSinkFunc(func(_ context.Context, _ string, candidates []proxyDomain.Proxy) (int, error) {
if len(candidates) != 2 {
t.Errorf("CandidateSink candidates = %d, want permit limit 2", len(candidates))
ports.Activity = activitySinkFunc(func(_ context.Context, _ string, batch activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
if len(batch.Proxies) != 2 {
t.Errorf("activity batch proxies = %d, want permit limit 2", len(batch.Proxies))
}
return 1, nil
return activitypool.UpsertResult{Accepted: 2, Inserted: 1}, nil
})
ports.Capacity = fetchCapacityFunc(func(string) (upstream.FetchPermit, bool, error) {
return &recordingFetchPermit{expected: 2, completed: completed}, true, nil
@ -720,8 +757,8 @@ func successfulPorts(onFetch func(), results chan<- Result) Ports {
Parser: parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) {
return []proxyDomain.Proxy{{ID: "proxy-1"}}, nil
}),
Candidates: candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) {
return 1, nil
Activity: activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
return activitypool.UpsertResult{Accepted: 1, Inserted: 1}, nil
}),
Results: resultRecorderFunc(func(got Result) { results <- got }),
Capacity: unlimitedFetchCapacity{},
@ -796,10 +833,10 @@ func (f parserFunc) Parse(ctx context.Context, body []byte) ([]proxyDomain.Proxy
return f(ctx, body)
}
type candidateSinkFunc func(context.Context, string, []proxyDomain.Proxy) (int, error)
type activitySinkFunc func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error)
func (f candidateSinkFunc) Add(ctx context.Context, upstreamID string, candidates []proxyDomain.Proxy) (int, error) {
return f(ctx, upstreamID, candidates)
func (f activitySinkFunc) UpsertFetched(ctx context.Context, upstreamID string, batch activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
return f(ctx, upstreamID, batch)
}
type fetchCapacityFunc func(string) (upstream.FetchPermit, bool, error)

View File

@ -8,11 +8,11 @@ import (
"net/http"
"strings"
"github.com/proxy-pool/proxy-pool/internal/config"
"github.com/proxy-pool/proxy-pool/internal/controller/admin"
"github.com/proxy-pool/proxy-pool/internal/controller/distribution"
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
"github.com/proxy-pool/proxy-pool/internal/platform/httpserver"
"proxy-pool/internal/config"
"proxy-pool/internal/controller/admin"
"proxy-pool/internal/controller/distribution"
"proxy-pool/internal/platform/httpsecurity"
"proxy-pool/internal/platform/httpserver"
)
const (

View File

@ -10,10 +10,10 @@ import (
"testing"
"time"
"github.com/proxy-pool/proxy-pool/internal/config"
"github.com/proxy-pool/proxy-pool/internal/controller/admin"
controllerExtraction "github.com/proxy-pool/proxy-pool/internal/controller/extraction"
"github.com/proxy-pool/proxy-pool/internal/platform/httpserver"
"proxy-pool/internal/config"
"proxy-pool/internal/controller/admin"
controllerExtraction "proxy-pool/internal/controller/extraction"
"proxy-pool/internal/platform/httpserver"
)
func TestRuntimeServesDistributionAndAdminOnIndependentListeners(t *testing.T) {

View File

@ -0,0 +1,590 @@
package activitypool
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"sort"
"sync"
"time"
extractionDomain "proxy-pool/internal/domain/extraction"
ownershipDomain "proxy-pool/internal/domain/ownership"
proxyDomain "proxy-pool/internal/domain/proxy"
)
const defaultIdempotencyTTL = 5 * time.Minute
var ErrInvalidBatch = errors.New("invalid activity pool batch")
// FetchedBatch describes one ephemeral provider response. Proxies without a
// usable expiry are dropped because this pool is intentionally rebuildable.
type FetchedBatch struct {
ObservedAt time.Time
ConfiguredTTL time.Duration
AllocationSafetyMargin time.Duration
Proxies []proxyDomain.Proxy
}
type UpsertResult struct {
Accepted int
Inserted int
Refreshed int
Dropped int
}
type Upserter interface {
// A proxy unique key has one incumbent upstream for its current lifetime.
// Duplicates from other upstreams must not replace that source lifecycle.
UpsertFetched(context.Context, string, FetchedBatch) (UpsertResult, error)
}
type Entry struct {
Proxy proxyDomain.Proxy
UsableUntil time.Time
OwnerWorkerID string
State proxyDomain.State
}
type MemoryPool struct {
mu sync.Mutex
entries map[string]Entry
keyByID map[string]string
idempotent map[string]idempotencyEntry
ownership map[string]ownershipDomain.Assignment
nextEpoch uint64
}
type idempotencyEntry struct {
command extractionDomain.Command
result extractionDomain.Result
expiresAt time.Time
}
var (
_ Upserter = (*MemoryPool)(nil)
_ extractionDomain.Store = (*MemoryPool)(nil)
_ ownershipDomain.Repository = (*MemoryPool)(nil)
)
func NewMemoryPool() *MemoryPool {
return &MemoryPool{
entries: make(map[string]Entry),
keyByID: make(map[string]string),
idempotent: make(map[string]idempotencyEntry),
ownership: make(map[string]ownershipDomain.Assignment),
}
}
func (p *MemoryPool) UpsertFetched(ctx context.Context, upstreamID string, batch FetchedBatch) (UpsertResult, error) {
var result UpsertResult
if err := ctx.Err(); err != nil {
return result, err
}
if p == nil || upstreamID == "" || batch.ObservedAt.IsZero() || batch.ConfiguredTTL < 0 ||
batch.AllocationSafetyMargin < 0 ||
(batch.ConfiguredTTL > 0 && batch.AllocationSafetyMargin >= batch.ConfiguredTTL) {
return result, ErrInvalidBatch
}
for _, candidate := range batch.Proxies {
if candidate.SourceUpstream != "" && candidate.SourceUpstream != upstreamID {
return result, ErrInvalidBatch
}
}
p.mu.Lock()
defer p.mu.Unlock()
if err := ctx.Err(); err != nil {
return result, err
}
p.purgeExpiredLocked(batch.ObservedAt)
seenIDs := make(map[string]string, len(batch.Proxies))
for _, candidate := range batch.Proxies {
if !validProxyIdentity(candidate) {
continue
}
key := candidate.UniqueKey()
proxyID := candidate.ID
if proxyID == "" {
proxyID = stableProxyID(key)
}
if existingKey, exists := p.keyByID[proxyID]; exists && existingKey != key {
return UpsertResult{}, ErrInvalidBatch
}
if existingKey, exists := seenIDs[proxyID]; exists && existingKey != key {
return UpsertResult{}, ErrInvalidBatch
}
seenIDs[proxyID] = key
}
for _, candidate := range batch.Proxies {
if !validProxyIdentity(candidate) {
result.Dropped++
continue
}
candidate.SourceUpstream = upstreamID
expiresAt := proxyDomain.EffectiveExpiry(batch.ObservedAt, candidate.ExpiresAt, 0, batch.ConfiguredTTL)
if expiresAt == nil {
result.Dropped++
continue
}
usableUntil := expiresAt.Add(-batch.AllocationSafetyMargin)
if !usableUntil.After(batch.ObservedAt) {
result.Dropped++
continue
}
candidate.ExpiresAt = expiresAt
candidate.UsableUntil = &usableUntil
if candidate.CreatedAt.IsZero() {
candidate.CreatedAt = batch.ObservedAt.UTC()
}
if candidate.State == "" {
candidate.State = proxyDomain.StateFetched
}
key := candidate.UniqueKey()
result.Accepted++
if current, exists := p.entries[key]; exists {
result.Refreshed++
if current.State == proxyDomain.StateExtracted || current.Proxy.SourceUpstream != upstreamID {
continue
}
candidate.ID = current.Proxy.ID
candidate.CreatedAt = current.Proxy.CreatedAt
candidate.State = current.State
candidate.LastCheckedAt = current.Proxy.LastCheckedAt
candidate.LastSuccessAt = current.Proxy.LastSuccessAt
candidate.Latency = current.Proxy.Latency
p.entries[key] = Entry{
Proxy: cloneProxy(candidate),
UsableUntil: usableUntil,
OwnerWorkerID: current.OwnerWorkerID,
State: current.State,
}
continue
}
if candidate.ID == "" {
candidate.ID = stableProxyID(key)
}
p.entries[key] = Entry{
Proxy: cloneProxy(candidate),
UsableUntil: usableUntil,
State: candidate.State,
}
p.keyByID[candidate.ID] = key
result.Inserted++
}
return result, nil
}
func (p *MemoryPool) Snapshot(now time.Time) []Entry {
if p == nil {
return nil
}
p.mu.Lock()
defer p.mu.Unlock()
p.purgeExpiredLocked(now)
entries := make([]Entry, 0, len(p.entries))
for _, entry := range p.entries {
entry.Proxy = cloneProxy(entry.Proxy)
entries = append(entries, entry)
}
sort.Slice(entries, func(i, j int) bool { return entries[i].Proxy.ID < entries[j].Proxy.ID })
return entries
}
func (p *MemoryPool) PurgeExpired(now time.Time) int {
if p == nil {
return 0
}
p.mu.Lock()
defer p.mu.Unlock()
return p.purgeExpiredLocked(now)
}
func (p *MemoryPool) Extract(ctx context.Context, command extractionDomain.Command) (extractionDomain.Result, error) {
result := extractionDomain.Result{Requested: command.Requested}
if err := ctx.Err(); err != nil {
return result, err
}
if p == nil || command.Now.IsZero() || command.Requested < 0 || command.ReserveForGateway < 0 ||
command.MinRemainingTTL < 0 || command.MaxHealthCheckAge < 0 || command.IdempotencyTTL < 0 ||
(command.IdempotencyKey != "" && command.ClientID == "") ||
(command.Fulfillment != extractionDomain.Partial && command.Fulfillment != extractionDomain.AllOrNothing) {
return result, extractionDomain.ErrInvalidCommand
}
p.mu.Lock()
defer p.mu.Unlock()
if err := ctx.Err(); err != nil {
return result, err
}
p.purgeExpiredLocked(command.Now)
idempotencyKey := command.ClientID + "\x00" + command.IdempotencyKey
if command.IdempotencyKey != "" {
if committed, ok := p.idempotent[idempotencyKey]; ok {
if !sameIdempotentRequest(committed.command, command) {
return result, extractionDomain.ErrIdempotencyConflict
}
return cloneResult(committed.result), nil
}
}
if command.Requested == 0 {
return result, nil
}
eligible := make([]string, 0, len(p.entries))
for key, entry := range p.entries {
if eligibleForExtraction(entry, command) {
eligible = append(eligible, key)
}
}
sort.Slice(eligible, func(i, j int) bool {
return p.entries[eligible[i]].UsableUntil.After(p.entries[eligible[j]].UsableUntil)
})
available := len(eligible) - command.ReserveForGateway
if available < 0 {
available = 0
}
if command.Fulfillment == extractionDomain.AllOrNothing && available < command.Requested {
return result, extractionDomain.ErrInsufficientProxies
}
count := command.Requested
if count > available {
count = available
}
for _, key := range eligible[:count] {
entry := p.entries[key]
entry.State = proxyDomain.StateExtracted
entry.Proxy.State = proxyDomain.StateExtracted
p.entries[key] = entry
result.Items = append(result.Items, extractionCandidate(entry))
}
result.Returned = len(result.Items)
if result.Returned > 0 {
result.ExtractedAt = command.Now.UTC()
}
if command.IdempotencyKey != "" {
expiresAt := command.Now.Add(idempotencyTTL(command.IdempotencyTTL))
for _, item := range result.Items {
if !item.ExpiresAt.IsZero() && item.ExpiresAt.Before(expiresAt) {
expiresAt = item.ExpiresAt
}
}
if expiresAt.After(command.Now) {
p.idempotent[idempotencyKey] = idempotencyEntry{
command: cloneCommand(command), result: cloneResult(result), expiresAt: expiresAt,
}
}
}
return result, nil
}
func (p *MemoryPool) Assign(now time.Time, proxyID, workerID string, ttl time.Duration) (ownershipDomain.Assignment, error) {
if p == nil || now.IsZero() || proxyID == "" || workerID == "" || ttl <= 0 {
return ownershipDomain.Assignment{}, ownershipDomain.ErrInvalidOwnership
}
p.mu.Lock()
defer p.mu.Unlock()
p.purgeExpiredLocked(now)
if current, exists := p.ownership[proxyID]; exists {
if current.ExpiresAt.After(now) {
return ownershipDomain.Assignment{}, ownershipDomain.ErrAlreadyOwned
}
if entry, ok := p.entryByIDLocked(proxyID); ok && entry.OwnerWorkerID == current.WorkerID {
entry.OwnerWorkerID = ""
p.setEntryByIDLocked(proxyID, entry)
}
delete(p.ownership, proxyID)
}
entry, ok := p.entryByIDLocked(proxyID)
if !ok || entry.State != proxyDomain.StateAvailable || entry.OwnerWorkerID != "" || !entry.UsableUntil.After(now) {
return ownershipDomain.Assignment{}, ownershipDomain.ErrOwnershipUnavailable
}
p.nextEpoch++
expiresAt := minTime(now.UTC().Add(ttl), entry.UsableUntil)
assignment := ownershipDomain.Assignment{
ProxyID: proxyID, WorkerID: workerID, Epoch: p.nextEpoch, Version: 1, ExpiresAt: expiresAt,
}
entry.OwnerWorkerID = workerID
p.setEntryByIDLocked(proxyID, entry)
p.ownership[proxyID] = assignment
return assignment, nil
}
func (p *MemoryPool) Renew(now time.Time, proxyID, workerID string, epoch uint64, ttl time.Duration) (ownershipDomain.Assignment, error) {
if p == nil || now.IsZero() || ttl <= 0 {
return ownershipDomain.Assignment{}, ownershipDomain.ErrInvalidOwnership
}
p.mu.Lock()
defer p.mu.Unlock()
p.purgeExpiredLocked(now)
assignment, ok := p.ownership[proxyID]
if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch || !assignment.ExpiresAt.After(now) {
return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment
}
entry, ok := p.entryByIDLocked(proxyID)
if !ok {
return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment
}
assignment.ExpiresAt = minTime(now.UTC().Add(ttl), entry.UsableUntil)
assignment.Version++
p.ownership[proxyID] = assignment
return assignment, nil
}
func (p *MemoryPool) BeginDrain(proxyID, workerID string, epoch uint64) (ownershipDomain.Assignment, error) {
if p == nil {
return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment
}
p.mu.Lock()
defer p.mu.Unlock()
assignment, ok := p.ownership[proxyID]
if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch {
return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment
}
if !assignment.Draining {
assignment.Draining = true
assignment.Version++
p.ownership[proxyID] = assignment
}
return assignment, nil
}
func (p *MemoryPool) AcknowledgeDrain(proxyID, workerID string, epoch uint64, active, reserved int64) error {
if p == nil || active < 0 || reserved < 0 {
return ownershipDomain.ErrInvalidOwnership
}
p.mu.Lock()
defer p.mu.Unlock()
assignment, ok := p.ownership[proxyID]
if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch {
return ownershipDomain.ErrStaleAssignment
}
if !assignment.Draining {
return ownershipDomain.ErrNotDraining
}
if active > 0 || reserved > 0 {
return ownershipDomain.ErrDrainNotReady
}
if entry, exists := p.entryByIDLocked(proxyID); exists && entry.OwnerWorkerID == workerID {
entry.OwnerWorkerID = ""
p.setEntryByIDLocked(proxyID, entry)
}
delete(p.ownership, proxyID)
return nil
}
func (p *MemoryPool) Get(proxyID string) (ownershipDomain.Assignment, bool) {
if p == nil {
return ownershipDomain.Assignment{}, false
}
p.mu.Lock()
defer p.mu.Unlock()
assignment, ok := p.ownership[proxyID]
return assignment, ok
}
func (p *MemoryPool) Expire(now time.Time) []ownershipDomain.Assignment {
if p == nil {
return nil
}
p.mu.Lock()
defer p.mu.Unlock()
expired := make([]ownershipDomain.Assignment, 0)
for proxyID, assignment := range p.ownership {
entry, exists := p.entryByIDLocked(proxyID)
if exists && assignment.ExpiresAt.After(now) {
continue
}
if exists && entry.OwnerWorkerID == assignment.WorkerID {
entry.OwnerWorkerID = ""
p.setEntryByIDLocked(proxyID, entry)
}
expired = append(expired, assignment)
delete(p.ownership, proxyID)
}
p.purgeExpiredLocked(now)
sort.Slice(expired, func(i, j int) bool { return expired[i].ProxyID < expired[j].ProxyID })
return expired
}
func (p *MemoryPool) purgeExpiredLocked(now time.Time) int {
removed := 0
for key, entry := range p.entries {
if entry.Proxy.ExpiresAt == nil || entry.Proxy.ExpiresAt.After(now) {
continue
}
delete(p.ownership, entry.Proxy.ID)
delete(p.keyByID, entry.Proxy.ID)
delete(p.entries, key)
removed++
}
for key, entry := range p.idempotent {
if !entry.expiresAt.After(now) {
delete(p.idempotent, key)
}
}
return removed
}
func (p *MemoryPool) entryByIDLocked(proxyID string) (Entry, bool) {
key, ok := p.keyByID[proxyID]
if !ok {
return Entry{}, false
}
entry, ok := p.entries[key]
return entry, ok
}
func (p *MemoryPool) setEntryByIDLocked(proxyID string, entry Entry) {
if key, ok := p.keyByID[proxyID]; ok {
p.entries[key] = entry
}
}
func eligibleForExtraction(entry Entry, command extractionDomain.Command) bool {
if entry.State != proxyDomain.StateAvailable || entry.OwnerWorkerID != "" || !entry.UsableUntil.After(command.Now) {
return false
}
if entry.Proxy.ExpiresAt != nil && entry.Proxy.ExpiresAt.Sub(command.Now) < command.MinRemainingTTL {
return false
}
if command.MaxHealthCheckAge > 0 {
if entry.Proxy.LastCheckedAt == nil || command.Now.Sub(*entry.Proxy.LastCheckedAt) > command.MaxHealthCheckAge {
return false
}
}
return matches(command.Protocols, string(entry.Proxy.Scheme)) &&
matches(command.Regions, entry.Proxy.Tags["region"]) &&
matches(command.Carriers, entry.Proxy.Tags["carrier"]) &&
matches(command.Upstreams, entry.Proxy.SourceUpstream)
}
func extractionCandidate(entry Entry) extractionDomain.Candidate {
expiresAt := time.Time{}
if entry.Proxy.ExpiresAt != nil {
expiresAt = *entry.Proxy.ExpiresAt
}
checkedAt := time.Time{}
if entry.Proxy.LastCheckedAt != nil {
checkedAt = *entry.Proxy.LastCheckedAt
}
return extractionDomain.Candidate{
ID: entry.Proxy.ID, Protocol: string(entry.Proxy.Scheme), Host: entry.Proxy.Host,
Port: entry.Proxy.Port, Username: entry.Proxy.Username, Region: entry.Proxy.Tags["region"],
Carrier: entry.Proxy.Tags["carrier"], Upstream: entry.Proxy.SourceUpstream,
OwnerWorkerID: entry.OwnerWorkerID, State: extractionDomain.Available,
ExpiresAt: expiresAt, LastCheckedAt: checkedAt,
}
}
func stableProxyID(key string) string {
digest := sha256.Sum256([]byte(key))
return "px_" + hex.EncodeToString(digest[:12])
}
func validProxyIdentity(candidate proxyDomain.Proxy) bool {
if candidate.Host == "" || candidate.Port == 0 {
return false
}
switch candidate.Scheme {
case proxyDomain.SchemeHTTP, proxyDomain.SchemeHTTPS, proxyDomain.SchemeSOCKS5:
return true
default:
return false
}
}
func cloneProxy(candidate proxyDomain.Proxy) proxyDomain.Proxy {
if candidate.ExpiresAt != nil {
value := *candidate.ExpiresAt
candidate.ExpiresAt = &value
}
if candidate.UsableUntil != nil {
value := *candidate.UsableUntil
candidate.UsableUntil = &value
}
if candidate.LastCheckedAt != nil {
value := *candidate.LastCheckedAt
candidate.LastCheckedAt = &value
}
if candidate.LastSuccessAt != nil {
value := *candidate.LastSuccessAt
candidate.LastSuccessAt = &value
}
if candidate.Tags != nil {
tags := make(map[string]string, len(candidate.Tags))
for key, value := range candidate.Tags {
tags[key] = value
}
candidate.Tags = tags
}
return candidate
}
func cloneResult(result extractionDomain.Result) extractionDomain.Result {
result.Items = append([]extractionDomain.Candidate(nil), result.Items...)
return result
}
func cloneCommand(command extractionDomain.Command) extractionDomain.Command {
command.Protocols = append([]string(nil), command.Protocols...)
command.Regions = append([]string(nil), command.Regions...)
command.Carriers = append([]string(nil), command.Carriers...)
command.Upstreams = append([]string(nil), command.Upstreams...)
return command
}
func sameIdempotentRequest(left, right extractionDomain.Command) bool {
return left.Requested == right.Requested && left.Fulfillment == right.Fulfillment &&
equalSet(left.Protocols, right.Protocols) && equalSet(left.Regions, right.Regions) &&
equalSet(left.Carriers, right.Carriers) && equalSet(left.Upstreams, right.Upstreams)
}
func equalSet(left, right []string) bool {
if len(left) != len(right) {
return false
}
counts := make(map[string]int, len(left))
for _, value := range left {
counts[value]++
}
for _, value := range right {
counts[value]--
if counts[value] < 0 {
return false
}
}
return true
}
func matches(allowed []string, value string) bool {
if len(allowed) == 0 {
return true
}
for _, candidate := range allowed {
if candidate == value {
return true
}
}
return false
}
func idempotencyTTL(configured time.Duration) time.Duration {
if configured > 0 {
return configured
}
return defaultIdempotencyTTL
}
func minTime(left, right time.Time) time.Time {
if left.Before(right) {
return left
}
return right
}

View File

@ -0,0 +1,359 @@
package activitypool
import (
"context"
"errors"
"sync"
"testing"
"time"
extractionDomain "proxy-pool/internal/domain/extraction"
ownershipDomain "proxy-pool/internal/domain/ownership"
proxyDomain "proxy-pool/internal/domain/proxy"
)
func TestMemoryPoolUpsertAppliesProviderTTLAndRefreshesWithoutGrowth(t *testing.T) {
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
pool := NewMemoryPool()
batch := FetchedBatch{
ObservedAt: now,
ConfiguredTTL: 30 * time.Second,
AllocationSafetyMargin: 3 * time.Second,
Proxies: []proxyDomain.Proxy{{
Scheme: proxyDomain.SchemeHTTP,
Host: "192.0.2.10",
Port: 8080,
State: proxyDomain.StateAvailable,
}},
}
first, err := pool.UpsertFetched(context.Background(), "provider-a", batch)
if err != nil {
t.Fatalf("UpsertFetched(first): %v", err)
}
if first.Inserted != 1 || first.Refreshed != 0 || first.Dropped != 0 {
t.Fatalf("first result = %+v", first)
}
entries := pool.Snapshot(now)
if len(entries) != 1 {
t.Fatalf("Snapshot() entries = %d, want 1", len(entries))
}
if !entries[0].Proxy.ExpiresAt.Equal(now.Add(30 * time.Second)) {
t.Fatalf("ExpiresAt = %v, want %v", entries[0].Proxy.ExpiresAt, now.Add(30*time.Second))
}
if !entries[0].UsableUntil.Equal(now.Add(27 * time.Second)) {
t.Fatalf("UsableUntil = %v, want %v", entries[0].UsableUntil, now.Add(27*time.Second))
}
if entries[0].Proxy.UsableUntil == nil || !entries[0].Proxy.UsableUntil.Equal(now.Add(27*time.Second)) {
t.Fatalf("Proxy.UsableUntil = %v, want %v", entries[0].Proxy.UsableUntil, now.Add(27*time.Second))
}
batch.ObservedAt = now.Add(10 * time.Second)
second, err := pool.UpsertFetched(context.Background(), "provider-a", batch)
if err != nil {
t.Fatalf("UpsertFetched(refresh): %v", err)
}
if second.Inserted != 0 || second.Refreshed != 1 || len(pool.Snapshot(batch.ObservedAt)) != 1 {
t.Fatalf("refresh result = %+v, entries=%d", second, len(pool.Snapshot(batch.ObservedAt)))
}
if got := pool.Snapshot(batch.ObservedAt)[0].Proxy.ExpiresAt; !got.Equal(now.Add(40 * time.Second)) {
t.Fatalf("refreshed ExpiresAt = %v, want %v", got, now.Add(40*time.Second))
}
}
func TestMemoryPoolCrossProviderDuplicateDoesNotReplaceSourceLifecycle(t *testing.T) {
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
pool := NewMemoryPool()
proxy := proxyDomain.Proxy{
Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8080,
State: proxyDomain.StateAvailable,
}
first, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{
ObservedAt: now, ConfiguredTTL: 30 * time.Second,
AllocationSafetyMargin: 3 * time.Second, Proxies: []proxyDomain.Proxy{proxy},
})
if err != nil || first.Inserted != 1 {
t.Fatalf("UpsertFetched(provider-a) = %+v, %v", first, err)
}
duplicate, err := pool.UpsertFetched(context.Background(), "provider-b", FetchedBatch{
ObservedAt: now.Add(time.Second), ConfiguredTTL: 5 * time.Minute,
AllocationSafetyMargin: 10 * time.Second, Proxies: []proxyDomain.Proxy{proxy},
})
if err != nil || duplicate.Inserted != 0 || duplicate.Refreshed != 1 {
t.Fatalf("UpsertFetched(provider-b) = %+v, %v", duplicate, err)
}
entries := pool.Snapshot(now.Add(time.Second))
if len(entries) != 1 {
t.Fatalf("Snapshot() entries = %d, want 1", len(entries))
}
entry := entries[0]
if entry.Proxy.SourceUpstream != "provider-a" {
t.Fatalf("SourceUpstream = %q, want provider-a", entry.Proxy.SourceUpstream)
}
if !entry.Proxy.ExpiresAt.Equal(now.Add(30*time.Second)) ||
!entry.UsableUntil.Equal(now.Add(27*time.Second)) {
t.Fatalf("lifecycle = expires %v usable %v, want provider-a lifecycle", entry.Proxy.ExpiresAt, entry.UsableUntil)
}
afterExpiry, err := pool.UpsertFetched(context.Background(), "provider-b", FetchedBatch{
ObservedAt: now.Add(31 * time.Second), ConfiguredTTL: 5 * time.Minute,
AllocationSafetyMargin: 10 * time.Second, Proxies: []proxyDomain.Proxy{proxy},
})
if err != nil || afterExpiry.Inserted != 1 {
t.Fatalf("UpsertFetched(provider-b after expiry) = %+v, %v", afterExpiry, err)
}
replacement := pool.Snapshot(now.Add(31 * time.Second))[0]
if replacement.Proxy.SourceUpstream != "provider-b" {
t.Fatalf("replacement SourceUpstream = %q, want provider-b", replacement.Proxy.SourceUpstream)
}
}
func TestMemoryPoolRefreshPreservesRuntimeStateAndHealth(t *testing.T) {
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
checkedAt := now.Add(-time.Second)
pool := NewMemoryPool()
first, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{
ObservedAt: now, ConfiguredTTL: 30 * time.Second,
Proxies: []proxyDomain.Proxy{{
Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8080,
State: proxyDomain.StateAvailable, LastCheckedAt: &checkedAt,
}},
})
if err != nil || first.Inserted != 1 {
t.Fatalf("UpsertFetched(first) = %+v, %v", first, err)
}
initial := pool.Snapshot(now)[0]
second, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{
ObservedAt: now.Add(10 * time.Second), ConfiguredTTL: 30 * time.Second,
Proxies: []proxyDomain.Proxy{{
Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8080,
State: proxyDomain.StateFetched,
}},
})
if err != nil || second.Refreshed != 1 {
t.Fatalf("UpsertFetched(refresh) = %+v, %v", second, err)
}
refreshed := pool.Snapshot(now.Add(10 * time.Second))[0]
if refreshed.Proxy.ID != initial.Proxy.ID || refreshed.State != proxyDomain.StateAvailable ||
refreshed.Proxy.LastCheckedAt == nil || !refreshed.Proxy.LastCheckedAt.Equal(checkedAt) ||
!refreshed.Proxy.CreatedAt.Equal(initial.Proxy.CreatedAt) {
t.Fatalf("refreshed entry lost runtime state: initial=%+v refreshed=%+v", initial, refreshed)
}
}
func TestMemoryPoolDropsCandidatesWithoutUsableTTL(t *testing.T) {
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
expired := now.Add(-time.Second)
tooShort := now.Add(2 * time.Second)
pool := NewMemoryPool()
result, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{
ObservedAt: now,
AllocationSafetyMargin: 3 * time.Second,
Proxies: []proxyDomain.Proxy{
{Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8001},
{Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.11", Port: 8002, ExpiresAt: &expired},
{Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.12", Port: 8003, ExpiresAt: &tooShort},
},
})
if err != nil {
t.Fatalf("UpsertFetched(): %v", err)
}
if result.Dropped != 3 || result.Accepted != 0 || len(pool.Snapshot(now)) != 0 {
t.Fatalf("result = %+v, entries=%d", result, len(pool.Snapshot(now)))
}
}
func TestMemoryPoolRejectsInvalidBatchAtomically(t *testing.T) {
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
pool := NewMemoryPool()
result, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{
ObservedAt: now,
ConfiguredTTL: time.Minute,
Proxies: []proxyDomain.Proxy{
{Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8001},
{Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.11", Port: 8002, SourceUpstream: "provider-b"},
},
})
if !errors.Is(err, ErrInvalidBatch) {
t.Fatalf("UpsertFetched() error = %v, want ErrInvalidBatch", err)
}
if result != (UpsertResult{}) || len(pool.Snapshot(now)) != 0 {
t.Fatalf("failed batch retained state: result=%+v entries=%+v", result, pool.Snapshot(now))
}
}
func TestMemoryPoolNeverExtractsProxyTwiceAndDoesNotWriteAuditRecords(t *testing.T) {
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
pool := poolWithOneProxy(t, now)
var wait sync.WaitGroup
results := make(chan string, 1000)
for range 1000 {
wait.Add(1)
go func() {
defer wait.Done()
result, err := pool.Extract(context.Background(), extractionDomain.Command{
Requested: 1,
Fulfillment: extractionDomain.Partial,
Now: now,
MinRemainingTTL: 10 * time.Second,
MaxHealthCheckAge: time.Minute,
})
if err != nil {
t.Errorf("Extract(): %v", err)
return
}
for _, item := range result.Items {
results <- item.ID
}
}()
}
wait.Wait()
close(results)
count := 0
for range results {
count++
}
if count != 1 {
t.Fatalf("proxy extracted %d times, want exactly once", count)
}
if got := pool.Snapshot(now); len(got) != 1 || got[0].State != proxyDomain.StateExtracted {
t.Fatalf("post-extraction snapshot = %+v", got)
}
}
func TestMemoryPoolIdempotencyIsBoundedByProxyExpiry(t *testing.T) {
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
pool := poolWithOneProxy(t, now)
command := extractionDomain.Command{
ClientID: "client-a",
IdempotencyKey: "idem-a",
IdempotencyTTL: 5 * time.Minute,
Requested: 1,
Fulfillment: extractionDomain.Partial,
Now: now,
MinRemainingTTL: 10 * time.Second,
MaxHealthCheckAge: time.Minute,
}
first, err := pool.Extract(context.Background(), command)
if err != nil || first.Returned != 1 {
t.Fatalf("first Extract() = %+v, %v", first, err)
}
command.Now = now.Add(20 * time.Second)
replayed, err := pool.Extract(context.Background(), command)
if err != nil || replayed.Returned != 1 || replayed.Items[0].ID != first.Items[0].ID {
t.Fatalf("replayed Extract() = %+v, %v", replayed, err)
}
if purged := pool.PurgeExpired(now.Add(31 * time.Second)); purged != 1 {
t.Fatalf("PurgeExpired() = %d, want 1", purged)
}
command.Now = now.Add(31 * time.Second)
afterExpiry, err := pool.Extract(context.Background(), command)
if err != nil || afterExpiry.Returned != 0 {
t.Fatalf("Extract(after expiry) = %+v, %v", afterExpiry, err)
}
}
func TestMemoryPoolAllOrNothingDoesNotConsumePartialInventory(t *testing.T) {
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
pool := poolWithOneProxy(t, now)
result, err := pool.Extract(context.Background(), extractionDomain.Command{
Requested: 2, Fulfillment: extractionDomain.AllOrNothing, Now: now,
})
if !errors.Is(err, extractionDomain.ErrInsufficientProxies) || result.Returned != 0 {
t.Fatalf("all-or-nothing Extract() = %+v, %v", result, err)
}
partial, err := pool.Extract(context.Background(), extractionDomain.Command{
Requested: 1, Fulfillment: extractionDomain.Partial, Now: now,
})
if err != nil || partial.Returned != 1 {
t.Fatalf("partial Extract() after rollback = %+v, %v", partial, err)
}
}
func TestMemoryPoolAssignRecoversExpiredLeaseWithoutSeparateSweep(t *testing.T) {
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
pool := poolWithOneProxy(t, now)
proxyID := pool.Snapshot(now)[0].Proxy.ID
first, err := pool.Assign(now, proxyID, "worker-a", 5*time.Second)
if err != nil {
t.Fatalf("Assign(first): %v", err)
}
second, err := pool.Assign(now.Add(5*time.Second), proxyID, "worker-b", 5*time.Second)
if err != nil {
t.Fatalf("Assign(after lease expiry): %v", err)
}
if second.Epoch <= first.Epoch || second.WorkerID != "worker-b" {
t.Fatalf("second assignment = %+v, first=%+v", second, first)
}
}
func TestMemoryPoolMakesOwnershipAndExtractionMutuallyExclusive(t *testing.T) {
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
for iteration := range 100 {
pool := poolWithOneProxy(t, now)
start := make(chan struct{})
assigned := make(chan bool, 1)
extracted := make(chan bool, 1)
go func() {
<-start
_, err := pool.Assign(now, pool.Snapshot(now)[0].Proxy.ID, "worker-a", time.Minute)
if err != nil && !errors.Is(err, ownershipDomain.ErrOwnershipUnavailable) {
t.Errorf("Assign(): %v", err)
}
assigned <- err == nil
}()
go func() {
<-start
result, err := pool.Extract(context.Background(), extractionDomain.Command{
ClientID: "client-a", Requested: 1, Fulfillment: extractionDomain.Partial, Now: now,
})
if err != nil {
t.Errorf("Extract(): %v", err)
}
extracted <- result.Returned == 1
}()
close(start)
wins := 0
if <-assigned {
wins++
}
if <-extracted {
wins++
}
if wins != 1 {
t.Fatalf("iteration %d winners = %d, want 1", iteration, wins)
}
}
}
func poolWithOneProxy(t *testing.T, now time.Time) *MemoryPool {
t.Helper()
pool := NewMemoryPool()
checkedAt := now
result, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{
ObservedAt: now,
ConfiguredTTL: 30 * time.Second,
AllocationSafetyMargin: 3 * time.Second,
Proxies: []proxyDomain.Proxy{{
Scheme: proxyDomain.SchemeHTTP,
Host: "192.0.2.10",
Port: 8080,
State: proxyDomain.StateAvailable,
LastCheckedAt: &checkedAt,
}},
})
if err != nil || result.Inserted != 1 {
t.Fatalf("UpsertFetched() = %+v, %v", result, err)
}
return pool
}

View File

@ -3,11 +3,7 @@ package extraction
import (
"context"
"errors"
"sort"
"sync"
"time"
ownershipDomain "github.com/proxy-pool/proxy-pool/internal/domain/ownership"
)
type Fulfillment string
@ -52,6 +48,7 @@ type Command struct {
ClientID string
SourceIP string
IdempotencyKey string
IdempotencyTTL time.Duration
Requested int
Fulfillment Fulfillment
Now time.Time
@ -64,16 +61,6 @@ type Command struct {
Upstreams []string
}
type Record struct {
ProxyID string
ClientID string
SourceIP string
RequestID string
Upstream string
ExtractedAt time.Time
ExpiresAt time.Time
}
type Result struct {
Requested int
Returned int
@ -81,294 +68,8 @@ type Result struct {
Items []Candidate
}
// Store atomically consumes proxies from a bounded, ephemeral activity pool.
// Implementations must not require durable per-proxy extraction records.
type Store interface {
Extract(context.Context, Command) (Result, error)
}
type MemoryStore struct {
mu sync.Mutex
candidates map[string]Candidate
records []Record
idempotent map[string]idempotencyEntry
nextEpoch uint64
ownership map[string]ownershipDomain.Assignment
}
var _ ownershipDomain.Repository = (*MemoryStore)(nil)
type idempotencyEntry struct {
command Command
result Result
}
func NewMemoryStore(candidates []Candidate) *MemoryStore {
items := make(map[string]Candidate, len(candidates))
for _, candidate := range candidates {
items[candidate.ID] = candidate
}
return &MemoryStore{
candidates: items,
idempotent: make(map[string]idempotencyEntry),
ownership: make(map[string]ownershipDomain.Assignment),
}
}
func (s *MemoryStore) Extract(ctx context.Context, command Command) (Result, error) {
result := Result{Requested: command.Requested}
if err := ctx.Err(); err != nil {
return result, err
}
if command.Requested < 0 || command.ReserveForGateway < 0 ||
command.MinRemainingTTL < 0 || command.MaxHealthCheckAge < 0 ||
(command.Fulfillment != Partial && command.Fulfillment != AllOrNothing) {
return result, ErrInvalidCommand
}
s.mu.Lock()
defer s.mu.Unlock()
if err := ctx.Err(); err != nil {
return result, err
}
idempotencyKey := command.ClientID + "\x00" + command.IdempotencyKey
if command.IdempotencyKey != "" {
if committed, ok := s.idempotent[idempotencyKey]; ok {
if !sameIdempotentRequest(committed.command, command) {
return result, ErrIdempotencyConflict
}
return cloneResult(committed.result), nil
}
}
if command.Requested <= 0 {
return result, nil
}
eligible := make([]Candidate, 0, len(s.candidates))
for _, candidate := range s.candidates {
if eligibleForExtraction(candidate, command) {
eligible = append(eligible, candidate)
}
}
sort.Slice(eligible, func(i, j int) bool {
return eligible[i].ExpiresAt.After(eligible[j].ExpiresAt)
})
available := len(eligible) - command.ReserveForGateway
if available < 0 {
available = 0
}
if command.Fulfillment == AllOrNothing && available < command.Requested {
return result, ErrInsufficientProxies
}
count := command.Requested
if count > available {
count = available
}
for i := 0; i < count; i++ {
candidate := eligible[i]
candidate.State = Extracted
s.candidates[candidate.ID] = candidate
result.Items = append(result.Items, candidate)
s.records = append(s.records, Record{
ProxyID: candidate.ID,
ClientID: command.ClientID,
SourceIP: command.SourceIP,
RequestID: command.RequestID,
Upstream: candidate.Upstream,
ExtractedAt: command.Now,
ExpiresAt: candidate.ExpiresAt,
})
}
result.Returned = len(result.Items)
if result.Returned > 0 {
result.ExtractedAt = command.Now
}
if command.IdempotencyKey != "" {
s.idempotent[idempotencyKey] = idempotencyEntry{
command: cloneCommand(command),
result: cloneResult(result),
}
}
return result, nil
}
func (s *MemoryStore) Assign(now time.Time, proxyID, workerID string, ttl time.Duration) (ownershipDomain.Assignment, error) {
if proxyID == "" || workerID == "" || ttl <= 0 {
return ownershipDomain.Assignment{}, ownershipDomain.ErrInvalidOwnership
}
s.mu.Lock()
defer s.mu.Unlock()
candidate, ok := s.candidates[proxyID]
if current, exists := s.ownership[proxyID]; exists && current.ExpiresAt.After(now) {
return ownershipDomain.Assignment{}, ownershipDomain.ErrAlreadyOwned
} else if exists {
if candidate.OwnerWorkerID == current.WorkerID {
candidate.OwnerWorkerID = ""
}
delete(s.ownership, proxyID)
}
if !ok || candidate.State != Available || candidate.OwnerWorkerID != "" {
return ownershipDomain.Assignment{}, ownershipDomain.ErrOwnershipUnavailable
}
s.nextEpoch++
assignment := ownershipDomain.Assignment{
ProxyID: proxyID, WorkerID: workerID, Epoch: s.nextEpoch, Version: 1,
ExpiresAt: now.UTC().Add(ttl),
}
candidate.OwnerWorkerID = workerID
s.candidates[proxyID] = candidate
s.ownership[proxyID] = assignment
return assignment, nil
}
func (s *MemoryStore) Renew(now time.Time, proxyID, workerID string, epoch uint64, ttl time.Duration) (ownershipDomain.Assignment, error) {
if ttl <= 0 {
return ownershipDomain.Assignment{}, ownershipDomain.ErrInvalidOwnership
}
s.mu.Lock()
defer s.mu.Unlock()
assignment, ok := s.ownership[proxyID]
if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch || !assignment.ExpiresAt.After(now) {
return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment
}
assignment.ExpiresAt = now.UTC().Add(ttl)
assignment.Version++
s.ownership[proxyID] = assignment
return assignment, nil
}
func (s *MemoryStore) BeginDrain(proxyID, workerID string, epoch uint64) (ownershipDomain.Assignment, error) {
s.mu.Lock()
defer s.mu.Unlock()
assignment, ok := s.ownership[proxyID]
if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch {
return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment
}
if assignment.Draining {
return assignment, nil
}
assignment.Draining = true
assignment.Version++
s.ownership[proxyID] = assignment
return assignment, nil
}
func (s *MemoryStore) AcknowledgeDrain(proxyID, workerID string, epoch uint64, active, reserved int64) error {
if active < 0 || reserved < 0 {
return ownershipDomain.ErrInvalidOwnership
}
s.mu.Lock()
defer s.mu.Unlock()
assignment, ok := s.ownership[proxyID]
if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch {
return ownershipDomain.ErrStaleAssignment
}
if !assignment.Draining {
return ownershipDomain.ErrNotDraining
}
if active > 0 || reserved > 0 {
return ownershipDomain.ErrDrainNotReady
}
if candidate, exists := s.candidates[proxyID]; exists && candidate.OwnerWorkerID == workerID {
candidate.OwnerWorkerID = ""
s.candidates[proxyID] = candidate
}
delete(s.ownership, proxyID)
return nil
}
func (s *MemoryStore) Get(proxyID string) (ownershipDomain.Assignment, bool) {
s.mu.Lock()
defer s.mu.Unlock()
assignment, ok := s.ownership[proxyID]
return assignment, ok
}
func (s *MemoryStore) Expire(now time.Time) []ownershipDomain.Assignment {
s.mu.Lock()
defer s.mu.Unlock()
expired := make([]ownershipDomain.Assignment, 0)
for proxyID, assignment := range s.ownership {
if assignment.ExpiresAt.After(now) {
continue
}
if candidate, ok := s.candidates[proxyID]; ok && candidate.OwnerWorkerID == assignment.WorkerID {
candidate.OwnerWorkerID = ""
s.candidates[proxyID] = candidate
}
expired = append(expired, assignment)
delete(s.ownership, proxyID)
}
sort.Slice(expired, func(i, j int) bool { return expired[i].ProxyID < expired[j].ProxyID })
return expired
}
func (s *MemoryStore) Records() []Record {
s.mu.Lock()
defer s.mu.Unlock()
return append([]Record(nil), s.records...)
}
func cloneResult(result Result) Result {
result.Items = append([]Candidate(nil), result.Items...)
return result
}
func cloneCommand(command Command) Command {
command.Protocols = append([]string(nil), command.Protocols...)
command.Regions = append([]string(nil), command.Regions...)
command.Carriers = append([]string(nil), command.Carriers...)
command.Upstreams = append([]string(nil), command.Upstreams...)
return command
}
func sameIdempotentRequest(left, right Command) bool {
return left.Requested == right.Requested &&
left.Fulfillment == right.Fulfillment &&
equalSet(left.Protocols, right.Protocols) &&
equalSet(left.Regions, right.Regions) &&
equalSet(left.Carriers, right.Carriers) &&
equalSet(left.Upstreams, right.Upstreams)
}
func equalSet(left, right []string) bool {
if len(left) != len(right) {
return false
}
counts := make(map[string]int, len(left))
for _, value := range left {
counts[value]++
}
for _, value := range right {
counts[value]--
if counts[value] < 0 {
return false
}
}
return true
}
func eligibleForExtraction(candidate Candidate, command Command) bool {
if candidate.State != Available || candidate.OwnerWorkerID != "" {
return false
}
if !candidate.ExpiresAt.IsZero() && candidate.ExpiresAt.Sub(command.Now) < command.MinRemainingTTL {
return false
}
if command.MaxHealthCheckAge > 0 && command.Now.Sub(candidate.LastCheckedAt) > command.MaxHealthCheckAge {
return false
}
return matches(command.Protocols, candidate.Protocol) &&
matches(command.Regions, candidate.Region) &&
matches(command.Carriers, candidate.Carrier) &&
matches(command.Upstreams, candidate.Upstream)
}
func matches(allowed []string, value string) bool {
if len(allowed) == 0 {
return true
}
for _, candidate := range allowed {
if candidate == value {
return true
}
}
return false
}

View File

@ -1,246 +1,12 @@
package extraction
import (
"context"
"errors"
"sync"
"testing"
"time"
)
import "testing"
func TestMemoryStoreNeverExtractsProxyTwice(t *testing.T) {
now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC)
store := NewMemoryStore([]Candidate{
{ID: "p1", State: Available, ExpiresAt: now.Add(time.Minute), LastCheckedAt: now},
})
var wg sync.WaitGroup
results := make(chan string, 1000)
for range 1000 {
wg.Add(1)
go func() {
defer wg.Done()
result, err := store.Extract(context.Background(), Command{
Requested: 1,
Fulfillment: Partial,
Now: now,
MinRemainingTTL: 30 * time.Second,
MaxHealthCheckAge: 10 * time.Second,
})
if err != nil {
t.Errorf("Extract(): %v", err)
return
func TestWireValuesRemainStable(t *testing.T) {
if Partial != "partial" || AllOrNothing != "allOrNothing" {
t.Fatalf("fulfillment values = %q, %q", Partial, AllOrNothing)
}
for _, item := range result.Items {
results <- item.ID
}
}()
}
wg.Wait()
close(results)
count := 0
for id := range results {
if id != "p1" {
t.Fatalf("unexpected proxy %q", id)
}
count++
}
if count != 1 {
t.Fatalf("proxy extracted %d times, want exactly once", count)
}
}
func TestAllOrNothingDoesNotConsumePartialInventory(t *testing.T) {
now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC)
store := NewMemoryStore([]Candidate{
{ID: "p1", State: Available, ExpiresAt: now.Add(time.Minute), LastCheckedAt: now},
})
result, err := store.Extract(context.Background(), Command{
Requested: 2,
Fulfillment: AllOrNothing,
Now: now,
MinRemainingTTL: 30 * time.Second,
MaxHealthCheckAge: 10 * time.Second,
})
if err != ErrInsufficientProxies {
t.Fatalf("Extract() error = %v, want ErrInsufficientProxies", err)
}
if len(result.Items) != 0 {
t.Fatalf("Extract() returned %d items, want 0", len(result.Items))
}
partial, err := store.Extract(context.Background(), Command{
Requested: 1,
Fulfillment: Partial,
Now: now,
MinRemainingTTL: 30 * time.Second,
MaxHealthCheckAge: 10 * time.Second,
})
if err != nil || len(partial.Items) != 1 {
t.Fatalf("inventory was consumed by failed all-or-nothing: result=%+v err=%v", partial, err)
}
}
func TestMemoryStoreCommitsAuditWithExtraction(t *testing.T) {
now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC)
store := NewMemoryStore([]Candidate{{
ID: "p1",
Upstream: "provider-a",
State: Available,
ExpiresAt: now.Add(time.Minute),
LastCheckedAt: now,
}})
_, err := store.Extract(context.Background(), Command{
RequestID: "req-1",
ClientID: "client-1",
SourceIP: "192.0.2.30",
Requested: 1,
Fulfillment: Partial,
Now: now,
MinRemainingTTL: 30 * time.Second,
MaxHealthCheckAge: 10 * time.Second,
})
if err != nil {
t.Fatalf("Extract(): %v", err)
}
records := store.Records()
if len(records) != 1 {
t.Fatalf("audit records = %d, want 1", len(records))
}
record := records[0]
if record.ProxyID != "p1" || record.ClientID != "client-1" || record.RequestID != "req-1" {
t.Fatalf("audit record = %+v", record)
}
if !record.ExtractedAt.Equal(now) || !record.ExpiresAt.Equal(now.Add(time.Minute)) {
t.Fatalf("audit timestamps = %+v", record)
}
}
func TestMemoryStoreReplaysCommittedIdempotentResult(t *testing.T) {
now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC)
store := NewMemoryStore([]Candidate{{
ID: "p1",
State: Available,
ExpiresAt: now.Add(time.Minute),
LastCheckedAt: now,
}})
command := Command{
RequestID: "req-1",
ClientID: "client-1",
IdempotencyKey: "idem-12345678",
Requested: 1,
Fulfillment: Partial,
Now: now,
MinRemainingTTL: 30 * time.Second,
MaxHealthCheckAge: 10 * time.Second,
}
first, err := store.Extract(context.Background(), command)
if err != nil {
t.Fatalf("first Extract(): %v", err)
}
command.RequestID = "req-2"
second, err := store.Extract(context.Background(), command)
if err != nil {
t.Fatalf("second Extract(): %v", err)
}
if len(first.Items) != 1 || len(second.Items) != 1 || second.Items[0].ID != first.Items[0].ID {
t.Fatalf("idempotent results: first=%+v second=%+v", first, second)
}
if got := len(store.Records()); got != 1 {
t.Fatalf("audit records = %d, want 1", got)
}
}
func TestMemoryStoreRejectsIdempotencyKeyReuseWithDifferentRequest(t *testing.T) {
now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC)
store := NewMemoryStore([]Candidate{{
ID: "p1",
State: Available,
ExpiresAt: now.Add(time.Minute),
LastCheckedAt: now,
}})
command := Command{
RequestID: "req-1",
ClientID: "client-1",
IdempotencyKey: "idem-12345678",
Requested: 1,
Fulfillment: Partial,
Now: now,
MinRemainingTTL: 30 * time.Second,
MaxHealthCheckAge: 10 * time.Second,
}
if _, err := store.Extract(context.Background(), command); err != nil {
t.Fatalf("first Extract(): %v", err)
}
command.Requested = 2
if _, err := store.Extract(context.Background(), command); err != ErrIdempotencyConflict {
t.Fatalf("second Extract() error = %v, want ErrIdempotencyConflict", err)
}
}
func TestMemoryStoreExtractsOnlyUnownedProxy(t *testing.T) {
now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC)
store := NewMemoryStore([]Candidate{
{ID: "owned", OwnerWorkerID: "worker-1", State: Available, ExpiresAt: now.Add(time.Minute), LastCheckedAt: now},
{ID: "unowned", State: Available, ExpiresAt: now.Add(time.Minute), LastCheckedAt: now},
})
result, err := store.Extract(context.Background(), Command{
RequestID: "req-1",
ClientID: "client-1",
Requested: 2,
Fulfillment: Partial,
Now: now,
MinRemainingTTL: 30 * time.Second,
MaxHealthCheckAge: 10 * time.Second,
})
if err != nil {
t.Fatalf("Extract(): %v", err)
}
if len(result.Items) != 1 || result.Items[0].ID != "unowned" {
t.Fatalf("extracted items = %+v, want only unowned", result.Items)
}
}
func TestMemoryStoreValidatesCommandAndHonorsCancellation(t *testing.T) {
store := NewMemoryStore([]Candidate{{ID: "p1", State: Available}})
if _, err := store.Extract(context.Background(), Command{
Requested: 1, Fulfillment: Partial, ReserveForGateway: -1,
}); !errors.Is(err, ErrInvalidCommand) {
t.Fatalf("Extract(negative reserve) error = %v, want ErrInvalidCommand", err)
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if _, err := store.Extract(ctx, Command{Requested: 1, Fulfillment: Partial}); !errors.Is(err, context.Canceled) {
t.Fatalf("Extract(canceled) error = %v, want context.Canceled", err)
}
result, err := store.Extract(context.Background(), Command{Requested: 1, Fulfillment: Partial})
if err != nil || result.Returned != 1 {
t.Fatalf("candidate changed after canceled request: result=%+v err=%v", result, err)
}
}
func TestMemoryStoreClonesIdempotencyCommandFilters(t *testing.T) {
store := NewMemoryStore([]Candidate{{ID: "p1", Protocol: "http", State: Available}})
protocols := []string{"http"}
command := Command{
ClientID: "client-1", IdempotencyKey: "idem-1", Requested: 1,
Fulfillment: Partial, Protocols: protocols,
}
if _, err := store.Extract(context.Background(), command); err != nil {
t.Fatalf("first Extract(): %v", err)
}
protocols[0] = "socks5"
command.Protocols = []string{"http"}
if _, err := store.Extract(context.Background(), command); err != nil {
t.Fatalf("idempotent replay after caller mutation: %v", err)
if Available != "AVAILABLE" || Extracted != "EXTRACTED" {
t.Fatalf("state values = %q, %q", Available, Extracted)
}
}

View File

@ -27,6 +27,7 @@ type Proxy struct {
SourceUpstream string
CreatedAt time.Time
ExpiresAt *time.Time
UsableUntil *time.Time
LastCheckedAt *time.Time
LastSuccessAt *time.Time
Latency time.Duration

View File

@ -3,7 +3,7 @@ package upstream
import (
"time"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
proxyDomain "proxy-pool/internal/domain/proxy"
)
type ProxyCapacity struct {

View File

@ -4,7 +4,7 @@ import (
"testing"
"time"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
proxyDomain "proxy-pool/internal/domain/proxy"
)
func TestInventoryAvailableSlotsUsesOnlyAllocatableCapacity(t *testing.T) {

View File

@ -5,8 +5,8 @@ import (
"sync/atomic"
"time"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
"github.com/proxy-pool/proxy-pool/internal/gateway/snapshot"
proxyDomain "proxy-pool/internal/domain/proxy"
"proxy-pool/internal/gateway/snapshot"
)
var ErrNoCandidate = errors.New("no local proxy candidate is available")

View File

@ -8,8 +8,8 @@ import (
"testing"
"time"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
"github.com/proxy-pool/proxy-pool/internal/gateway/snapshot"
proxyDomain "proxy-pool/internal/domain/proxy"
"proxy-pool/internal/gateway/snapshot"
)
func TestAcquireFiltersAndReservesLocalCapacity(t *testing.T) {

View File

@ -4,10 +4,10 @@ import (
"fmt"
"time"
"github.com/proxy-pool/proxy-pool/internal/config"
"github.com/proxy-pool/proxy-pool/internal/gateway/policy"
platformAdmission "github.com/proxy-pool/proxy-pool/internal/platform/admission"
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
"proxy-pool/internal/config"
"proxy-pool/internal/gateway/policy"
platformAdmission "proxy-pool/internal/platform/admission"
"proxy-pool/internal/platform/httpsecurity"
)
type Protection struct {

View File

@ -7,9 +7,9 @@ import (
"net/http/httptest"
"testing"
"github.com/proxy-pool/proxy-pool/internal/config"
"github.com/proxy-pool/proxy-pool/internal/gateway/policy"
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
"proxy-pool/internal/config"
"proxy-pool/internal/gateway/policy"
"proxy-pool/internal/platform/httpsecurity"
)
func TestBuildProtectionFromListenerConfig(t *testing.T) {

View File

@ -16,11 +16,11 @@ import (
"testing"
"time"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
"github.com/proxy-pool/proxy-pool/internal/gateway/dispatch"
"github.com/proxy-pool/proxy-pool/internal/gateway/policy"
"github.com/proxy-pool/proxy-pool/internal/gateway/snapshot"
transportDomain "github.com/proxy-pool/proxy-pool/internal/gateway/transport"
proxyDomain "proxy-pool/internal/domain/proxy"
"proxy-pool/internal/gateway/dispatch"
"proxy-pool/internal/gateway/policy"
"proxy-pool/internal/gateway/snapshot"
transportDomain "proxy-pool/internal/gateway/transport"
)
func TestHTTPProxyEndToEndRetriesDialFailure(t *testing.T) {

View File

@ -14,11 +14,11 @@ import (
"sync/atomic"
"time"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
"github.com/proxy-pool/proxy-pool/internal/gateway/dispatch"
"github.com/proxy-pool/proxy-pool/internal/gateway/policy"
transportDomain "github.com/proxy-pool/proxy-pool/internal/gateway/transport"
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
proxyDomain "proxy-pool/internal/domain/proxy"
"proxy-pool/internal/gateway/dispatch"
"proxy-pool/internal/gateway/policy"
transportDomain "proxy-pool/internal/gateway/transport"
"proxy-pool/internal/platform/httpsecurity"
)
type Config struct {

View File

@ -15,11 +15,11 @@ import (
"testing"
"time"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
"github.com/proxy-pool/proxy-pool/internal/gateway/dispatch"
"github.com/proxy-pool/proxy-pool/internal/gateway/policy"
"github.com/proxy-pool/proxy-pool/internal/gateway/snapshot"
transportDomain "github.com/proxy-pool/proxy-pool/internal/gateway/transport"
proxyDomain "proxy-pool/internal/domain/proxy"
"proxy-pool/internal/gateway/dispatch"
"proxy-pool/internal/gateway/policy"
"proxy-pool/internal/gateway/snapshot"
transportDomain "proxy-pool/internal/gateway/transport"
)
func TestHandlerRunsProtectionAndTargetPolicyBeforeRouting(t *testing.T) {

View File

@ -6,8 +6,8 @@ import (
"fmt"
"net/http"
"github.com/proxy-pool/proxy-pool/internal/gateway/policy"
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
"proxy-pool/internal/gateway/policy"
"proxy-pool/internal/platform/httpsecurity"
)
type HTTPError struct {

View File

@ -7,8 +7,8 @@ import (
"net/http"
"strings"
"github.com/proxy-pool/proxy-pool/internal/domain/routing"
"github.com/proxy-pool/proxy-pool/internal/gateway/dispatch"
"proxy-pool/internal/domain/routing"
"proxy-pool/internal/gateway/dispatch"
)
var (

View File

@ -7,7 +7,7 @@ import (
"reflect"
"testing"
"github.com/proxy-pool/proxy-pool/internal/domain/routing"
"proxy-pool/internal/domain/routing"
)
func TestRulesRouterReturnsMatchedUpstreams(t *testing.T) {

View File

@ -11,7 +11,7 @@ import (
"sync/atomic"
"time"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
proxyDomain "proxy-pool/internal/domain/proxy"
)
var (
@ -270,6 +270,9 @@ func matchesQuery(candidate proxyDomain.Proxy, query Query) bool {
if _, excluded := query.Exclude[candidate.ID]; excluded {
return false
}
if candidate.UsableUntil != nil && !candidate.UsableUntil.After(query.Now) {
return false
}
if candidate.ExpiresAt != nil && !candidate.ExpiresAt.After(query.Now.Add(query.SafetyMargin)) {
return false
}

View File

@ -7,7 +7,7 @@ import (
"testing"
"time"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
proxyDomain "proxy-pool/internal/domain/proxy"
)
func TestStoreAppliesCompleteSnapshotsInOrder(t *testing.T) {
@ -119,6 +119,34 @@ func TestViewSelectFiltersBySchemeUpstreamTagAndExclude(t *testing.T) {
}
}
func TestViewSelectRejectsProxyAfterProviderUsableDeadline(t *testing.T) {
now := time.Date(2026, 7, 29, 12, 0, 0, 0, time.UTC)
expiresAt := now.Add(30 * time.Second)
usableUntil := now.Add(27 * time.Second)
proxy := proxyDomain.Proxy{
ID: "short-lived", Scheme: proxyDomain.SchemeHTTP, State: proxyDomain.StateAvailable,
MaxConcurrency: 1, ExpiresAt: &expiresAt, UsableUntil: &usableUntil,
}
store := NewStore("cluster-a", "worker-a")
envelope := Envelope{
ClusterID: "cluster-a", WorkerID: "worker-a", Epoch: 1, Version: 1, Full: true,
Proxies: []proxyDomain.Proxy{proxy},
}
envelope.Checksum = Checksum(envelope.Proxies)
if err := store.Apply(envelope); err != nil {
t.Fatalf("Apply(): %v", err)
}
before := store.Current().Select(Query{Now: now.Add(26 * time.Second)})
if _, ok := before.EntryAt(0); !ok {
t.Fatal("EntryAt(before usable deadline) = false, want true")
}
after := store.Current().Select(Query{Now: now.Add(27 * time.Second)})
if _, ok := after.EntryAt(0); ok {
t.Fatal("EntryAt(at usable deadline) = true, want false")
}
}
func TestStoreApplyReusesCapacityAcrossVersionsAndEpochs(t *testing.T) {
store := NewStore("cluster-a", "worker-a")
firstProxies := []proxyDomain.Proxy{{

View File

@ -4,8 +4,8 @@ import (
"context"
"errors"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
platformCredentials "github.com/proxy-pool/proxy-pool/internal/platform/credentials"
proxyDomain "proxy-pool/internal/domain/proxy"
platformCredentials "proxy-pool/internal/platform/credentials"
)
var (

View File

@ -7,8 +7,8 @@ import (
"strings"
"testing"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
platformCredentials "github.com/proxy-pool/proxy-pool/internal/platform/credentials"
proxyDomain "proxy-pool/internal/domain/proxy"
platformCredentials "proxy-pool/internal/platform/credentials"
)
func TestCredentialsFormattingRedactsPassword(t *testing.T) {

View File

@ -16,7 +16,7 @@ import (
"sync/atomic"
"time"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
proxyDomain "proxy-pool/internal/domain/proxy"
)
const (

View File

@ -15,7 +15,7 @@ import (
"testing"
"time"
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
proxyDomain "proxy-pool/internal/domain/proxy"
)
func TestRoundTripForwardsHTTPViaSelectedProxy(t *testing.T) {

View File

@ -4,8 +4,8 @@ import (
"fmt"
"time"
"github.com/proxy-pool/proxy-pool/internal/config"
platformAdmission "github.com/proxy-pool/proxy-pool/internal/platform/admission"
"proxy-pool/internal/config"
platformAdmission "proxy-pool/internal/platform/admission"
)
func BuildFromListener(listener config.Listener, clientIdentification string, semantics Semantics) (*Protection, error) {

View File

@ -6,7 +6,7 @@ import (
"net/http/httptest"
"testing"
"github.com/proxy-pool/proxy-pool/internal/config"
"proxy-pool/internal/config"
)
func TestNewFromListenerMapsResolvedConfiguration(t *testing.T) {

View File

@ -4,7 +4,7 @@ import (
"errors"
"net/http"
"github.com/proxy-pool/proxy-pool/internal/platform/httpapi"
"proxy-pool/internal/platform/httpapi"
)
func WriteProblem(writer http.ResponseWriter, requestID string, err error) bool {

View File

@ -8,7 +8,14 @@
`httpsecurity`API 401 与代理 407 语义保持分离。
- 部署配置已使用 `${VAR}` 真实环境变量解析ConfigMap 与本地配置通过
`LoadResolved` 回归测试。
- PostgreSQL/Redis Adapter、生产命令入口与代表性 100,000 QPS 集群压测仍待实现。
- 已新增公用 `activitypool` 契约和并发安全内存参考实现Provider 按各供应商
TTL 与安全余量写入,独占提取、短期幂等和 Worker 所有权在同一原子边界内。
- Proxy 明细不写 PostgreSQLPostgreSQL 仅保存配置版本、管理状态、Admin
审计、管理 outbox 和可选聚合指标。
- `usableUntil` 已进入 Worker Snapshot 契约Gateway 在供应商硬过期前按安全
余量停止新分配。
- 生产 Redis Adapter、PostgreSQL 管理面 Adapter、生产命令入口与代表性
100,000 QPS 集群压测仍待实现。
## 2026-07-28
@ -37,8 +44,8 @@
Retry-After、原子 FetchBudget 以及 Pool Reconciler。
- 已强制 Worker 所有权与 Exclusive Extraction 注入同一权威 Repository
覆盖 100 轮并发竞争、续期、Drain ACK 和 Worker 过期回收。
- 已实现提取服务层策略映射、幂等审计、来源身份准入和公共 FixedWindow
全局/Client 限流器。
- 已实现提取服务层策略映射、Redis 短期幂等语义、来源身份准入和公共
FixedWindow 全局/Client 限流器;不保存逐代理提取审计记录
- 已为 100k Proxy 快照建立 scheme/upstream/tag 索引;本机持续基准中
`Acquire` 为 640 ns/op、256 B/op、2 allocs/op。该数据仅证明本地选择热路径
不代表 100k QPS 集群端到端容量。

View File

@ -24,6 +24,8 @@
7. [已完成] 执行单元测试、静态检查、构建和静态部署/契约验证;本机因
`CGO_ENABLED=0` 且无 C 编译器未运行 race保留给 Linux CI
8. [已完成] 按需求矩阵逐项审计并生成版本化文档包
9. [已完成] 将 Proxy 明细、独占提取、短期幂等和 Worker 所有权统一到
TTL 活动池契约PostgreSQL 退出代理数据路径
## 串并行关系
@ -44,5 +46,5 @@
- Docker Compose 配置与 Kubernetes Kustomize 已完成静态渲染验证;未启动
目标运行拓扑。
- `cmd/proxy-*`Provider 调度、PostgreSQL/Redis 适配器、Gateway Transport
Checker 运行时属于后续实施范围,见完成审计。
- `cmd/proxy-*`生产 Redis 活动池 Adapter、PostgreSQL 管理面 Adapter 与
Checker 运行时属于后续实施范围,见完成审计。