feat: add ephemeral proxy activity pool
This commit is contained in:
parent
ee7fc85031
commit
4de3ffb85f
19
CONTEXT.md
19
CONTEXT.md
@ -8,13 +8,21 @@
|
|||||||
- **Upstream**:一个供应商配置及其聚合代理池。Upstream 运行时全局共享,
|
- **Upstream**:一个供应商配置及其聚合代理池。Upstream 运行时全局共享,
|
||||||
不随 Routing 重复创建。
|
不随 Routing 重复创建。
|
||||||
- **Provider**:Upstream 背后的外部代理供应商及其获取接口能力。
|
- **Provider**:Upstream 背后的外部代理供应商及其获取接口能力。
|
||||||
- **Proxy**:从 Provider 获取并标准化后的代理资源,不等同于简单 IP。
|
- **Proxy**:从 Provider 获取并标准化后的短效代理资源,不等同于简单 IP。
|
||||||
|
Proxy 明细不进入 PostgreSQL,只存在于带 TTL 的 Redis 活动池和持有其快照的
|
||||||
|
节点内存;丢失后由 Provider 重新获取。
|
||||||
- **Worker**:承载 Gateway 流量的数据面节点,仅使用本地快照选路。
|
- **Worker**:承载 Gateway 流量的数据面节点,仅使用本地快照选路。
|
||||||
- **Controller**:集中管理 Provider 获取、代理生命周期、Routing 状态、
|
- **Controller**:集中管理 Provider 获取、代理生命周期、Routing 状态、
|
||||||
所有权和快照发布的控制面节点。
|
所有权和快照发布的控制面节点。
|
||||||
- **Checker**:执行基础、出口和目标级健康检查的可扩缩执行节点。
|
- **Checker**:执行基础、出口和目标级健康检查的可扩缩执行节点。
|
||||||
- **Extraction**:Distribution API 对一个或一组 Proxy 的一次性独占发放。
|
- **Extraction**:Distribution API 对一个或一组 Proxy 的一次性独占发放。
|
||||||
- **Extraction Record**:Extraction 的审计事实,不承担租约或释放语义。
|
- **Activity Pool**:Redis 中带 TTL 的短效 Proxy 运行时集合,是分配、所有权
|
||||||
|
和独占提取的原子操作位置,可在丢失后由 Provider 重建。
|
||||||
|
- **Extraction Result**:Redis 中短期保存的幂等响应,不承担租约、释放或
|
||||||
|
长期审计语义。
|
||||||
|
- **Persistent Management State**:PostgreSQL 中的配置版本、Upstream/Routing
|
||||||
|
管理状态、Admin 审计与 Outbox,以及可选聚合指标;不包含 Proxy 明细或逐次
|
||||||
|
提取记录。
|
||||||
|
|
||||||
## 状态与计数
|
## 状态与计数
|
||||||
|
|
||||||
@ -33,8 +41,10 @@
|
|||||||
|
|
||||||
- **Gateway Allocation**:Worker 原子预留本地所有 Proxy 容量,建连成功后
|
- **Gateway Allocation**:Worker 原子预留本地所有 Proxy 容量,建连成功后
|
||||||
转为 Active,结束后释放。
|
转为 Active,结束后释放。
|
||||||
- **Exclusive Extraction**:控制面原子把可提取 Proxy 从 `AVAILABLE` 改为
|
- **Exclusive Extraction**:控制面通过 Redis 原子操作把可提取 Proxy 从
|
||||||
`EXTRACTED`,成功后该 Proxy 永不再次被系统分配。
|
`AVAILABLE` 改为不可再次分配,并在有限 TTL 内保存幂等结果;成功后该
|
||||||
|
Proxy 在当前 TTL 生命周期内不再被系统分配。Redis 整体丢失后的 Provider
|
||||||
|
重建属于新的活动池代次,不承诺延续已丢失代次的排他状态。
|
||||||
- **Drain**:停止新分配,等待 Reserved 与 Active 归零后转换状态或撤销所有权。
|
- **Drain**:停止新分配,等待 Reserved 与 Active 归零后转换状态或撤销所有权。
|
||||||
- **Empty Fetch**:不是错误、不是重复,而是有效 Provider 响应中没有任何
|
- **Empty Fetch**:不是错误、不是重复,而是有效 Provider 响应中没有任何
|
||||||
合法 Proxy 候选。
|
合法 Proxy 候选。
|
||||||
@ -51,4 +61,3 @@
|
|||||||
- **reserveForGateway**:共享池中不能被 Distribution 提取的最低可用数量。
|
- **reserveForGateway**:共享池中不能被 Distribution 提取的最低可用数量。
|
||||||
- **fulfillment.partial**:尽量返回,允许少于请求数量。
|
- **fulfillment.partial**:尽量返回,允许少于请求数量。
|
||||||
- **fulfillment.allOrNothing**:不足时一个也不提取。
|
- **fulfillment.allOrNothing**:不足时一个也不提取。
|
||||||
|
|
||||||
|
|||||||
@ -4,9 +4,10 @@ info:
|
|||||||
version: 1.0.0
|
version: 1.0.0
|
||||||
description: |
|
description: |
|
||||||
Distribution API performs one-time exclusive extraction. A successful
|
Distribution API performs one-time exclusive extraction. A successful
|
||||||
operation atomically transitions every returned proxy from AVAILABLE to
|
Redis operation atomically removes or marks every returned proxy as
|
||||||
EXTRACTED. Extracted proxies are never allocated again and there is no
|
extracted in the TTL activity pool. Proxy details and per-extraction
|
||||||
release, renew, or lease API.
|
records are not persisted in PostgreSQL. There is no release, renew, or
|
||||||
|
lease API.
|
||||||
servers:
|
servers:
|
||||||
- url: http://127.0.0.1:8081
|
- url: http://127.0.0.1:8081
|
||||||
description: Distribution API
|
description: Distribution API
|
||||||
@ -20,13 +21,14 @@ paths:
|
|||||||
operationId: extractProxies
|
operationId: extractProxies
|
||||||
summary: 一次性独占提取代理
|
summary: 一次性独占提取代理
|
||||||
description: |
|
description: |
|
||||||
服务端先完成筛选、行锁定、AVAILABLE -> EXTRACTED 状态更新和审计记录
|
服务端通过单个 Redis 原子操作完成筛选、TTL/所有权复核、Gateway 预留、
|
||||||
写入,再返回代理。相同代理不会返回给两个成功请求。
|
从可分配池移除所选条目和短期幂等结果写入,再返回代理。相同代理不会
|
||||||
|
返回给两个成功请求,提取路径不访问 PostgreSQL。
|
||||||
|
|
||||||
`partial` 允许实际返回数量小于请求数量;`allOrNothing` 数量不足时不
|
`partial` 允许实际返回数量小于请求数量;`allOrNothing` 数量不足时不
|
||||||
提取任何代理并返回 409。未传 `fulfillment` 时使用服务端配置,默认
|
提取任何代理并返回 409。未传 `fulfillment` 时使用服务端配置,默认
|
||||||
为 `partial`。`Idempotency-Key` 可避免客户端因响应丢失重试而再次消耗
|
为 `partial`。`Idempotency-Key` 在 Redis 的有界 TTL 窗口内避免客户端因
|
||||||
库存。
|
响应丢失重试而再次消耗库存。
|
||||||
security:
|
security:
|
||||||
- ApiKeyAuth: []
|
- ApiKeyAuth: []
|
||||||
- BasicAuth: []
|
- BasicAuth: []
|
||||||
@ -57,7 +59,7 @@ paths:
|
|||||||
allowedUpstreams: [provider-a, provider-b]
|
allowedUpstreams: [provider-a, provider-b]
|
||||||
responses:
|
responses:
|
||||||
'200':
|
'200':
|
||||||
description: 提取事务已提交;返回的代理已永久退出可分配池
|
description: Redis 原子提取已提交;返回的代理已退出当前可分配池
|
||||||
headers:
|
headers:
|
||||||
X-Request-ID:
|
X-Request-ID:
|
||||||
$ref: '#/components/headers/RequestID'
|
$ref: '#/components/headers/RequestID'
|
||||||
@ -117,7 +119,7 @@ paths:
|
|||||||
tags: [Health]
|
tags: [Health]
|
||||||
operationId: getReadiness
|
operationId: getReadiness
|
||||||
summary: Distribution 就绪探针
|
summary: Distribution 就绪探针
|
||||||
description: PostgreSQL 不可用或权威状态不可写时返回 503。
|
description: Redis TTL 活动池不可用或无法保证原子提取时返回 503。
|
||||||
security: []
|
security: []
|
||||||
responses:
|
responses:
|
||||||
'200':
|
'200':
|
||||||
@ -375,7 +377,7 @@ components:
|
|||||||
schema:
|
schema:
|
||||||
$ref: '#/components/schemas/Problem'
|
$ref: '#/components/schemas/Problem'
|
||||||
ServiceUnavailable:
|
ServiceUnavailable:
|
||||||
description: 权威存储不可用或服务正在排空
|
description: Redis TTL 活动池不可用、原子提取不可执行或服务正在排空
|
||||||
headers:
|
headers:
|
||||||
X-Request-ID:
|
X-Request-ID:
|
||||||
$ref: '#/components/headers/RequestID'
|
$ref: '#/components/headers/RequestID'
|
||||||
|
|||||||
@ -2,7 +2,7 @@ syntax = "proto3";
|
|||||||
|
|
||||||
package proxy_pool.controlplane.v1;
|
package proxy_pool.controlplane.v1;
|
||||||
|
|
||||||
option go_package = "github.com/proxy-pool/proxy-pool/gen/controlplane/v1;controlplanev1";
|
option go_package = "proxy-pool/gen/controlplane/v1;controlplanev1";
|
||||||
|
|
||||||
import "google/protobuf/duration.proto";
|
import "google/protobuf/duration.proto";
|
||||||
import "google/protobuf/empty.proto";
|
import "google/protobuf/empty.proto";
|
||||||
@ -125,6 +125,7 @@ message OwnedProxy {
|
|||||||
uint32 max_concurrency = 10;
|
uint32 max_concurrency = 10;
|
||||||
map<string, string> tags = 11;
|
map<string, string> tags = 11;
|
||||||
uint64 ownership_epoch = 12;
|
uint64 ownership_epoch = 12;
|
||||||
|
google.protobuf.Timestamp usable_until = 13;
|
||||||
}
|
}
|
||||||
|
|
||||||
enum ProxyProtocol {
|
enum ProxyProtocol {
|
||||||
|
|||||||
@ -72,6 +72,7 @@ distribution:
|
|||||||
minRemainingTTL: 30s
|
minRemainingTTL: 30s
|
||||||
maxHealthCheckAge: 15s
|
maxHealthCheckAge: 15s
|
||||||
reserveForGateway: 100
|
reserveForGateway: 100
|
||||||
|
idempotencyTTL: 5m
|
||||||
|
|
||||||
admin:
|
admin:
|
||||||
enabled: true
|
enabled: true
|
||||||
|
|||||||
@ -46,6 +46,7 @@ distribution:
|
|||||||
minRemainingTTL: 30s
|
minRemainingTTL: 30s
|
||||||
maxHealthCheckAge: 30s
|
maxHealthCheckAge: 30s
|
||||||
reserveForGateway: 1000
|
reserveForGateway: 1000
|
||||||
|
idempotencyTTL: 5m
|
||||||
|
|
||||||
admin:
|
admin:
|
||||||
enabled: true
|
enabled: true
|
||||||
|
|||||||
@ -50,6 +50,7 @@ data:
|
|||||||
minRemainingTTL: 30s
|
minRemainingTTL: 30s
|
||||||
maxHealthCheckAge: 30s
|
maxHealthCheckAge: 30s
|
||||||
reserveForGateway: 5000
|
reserveForGateway: 5000
|
||||||
|
idempotencyTTL: 5m
|
||||||
admin:
|
admin:
|
||||||
enabled: true
|
enabled: true
|
||||||
listen: 0.0.0.0:8082
|
listen: 0.0.0.0:8082
|
||||||
|
|||||||
@ -5,7 +5,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/config"
|
"proxy-pool/internal/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
|
|||||||
@ -245,16 +245,16 @@ flowchart TB
|
|||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
sequenceDiagram
|
sequenceDiagram
|
||||||
participant DB as PostgreSQL
|
participant R as Redis Activity Pool
|
||||||
participant C as Controller
|
participant C as Controller
|
||||||
participant W as Worker
|
participant W as Worker
|
||||||
DB->>C: outbox revision 42
|
R->>C: activity or ownership revision 42
|
||||||
C->>C: build worker snapshot
|
C->>C: build worker snapshot
|
||||||
C->>W: epoch 8 version 42 checksum
|
C->>W: epoch 8 version 42 checksum
|
||||||
W->>W: validate and build indexes
|
W->>W: validate and build indexes
|
||||||
W->>W: atomic swap
|
W->>W: atomic swap
|
||||||
W-->>C: ACK epoch 8 version 42
|
W-->>C: ACK epoch 8 version 42
|
||||||
C->>DB: mark outbox delivered
|
C->>R: record worker ACK with TTL
|
||||||
```
|
```
|
||||||
|
|
||||||
## 17 Snapshot 缺口恢复
|
## 17 Snapshot 缺口恢复
|
||||||
@ -316,12 +316,10 @@ flowchart LR
|
|||||||
sequenceDiagram
|
sequenceDiagram
|
||||||
participant C as Client
|
participant C as Client
|
||||||
participant API as Distribution
|
participant API as Distribution
|
||||||
participant DB as PostgreSQL
|
participant R as Redis Activity Pool
|
||||||
C->>API: count 10 fulfillment partial
|
C->>API: count 10 fulfillment partial
|
||||||
API->>DB: lock eligible rows
|
API->>R: atomic extract with filters and reserve
|
||||||
DB-->>API: 6 rows
|
R-->>API: mark 6 EXTRACTED and return them
|
||||||
API->>DB: update 6 to EXTRACTED and audit
|
|
||||||
DB-->>API: commit
|
|
||||||
API-->>C: requested 10 returned 6
|
API-->>C: requested 10 returned 6
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -331,11 +329,10 @@ sequenceDiagram
|
|||||||
sequenceDiagram
|
sequenceDiagram
|
||||||
participant C as Client
|
participant C as Client
|
||||||
participant API as Distribution
|
participant API as Distribution
|
||||||
participant DB as PostgreSQL
|
participant R as Redis Activity Pool
|
||||||
C->>API: count 10 fulfillment allOrNothing
|
C->>API: count 10 fulfillment allOrNothing
|
||||||
API->>DB: lock eligible rows
|
API->>R: atomic extract with filters and reserve
|
||||||
DB-->>API: only 6 rows
|
R-->>API: insufficient inventory and no mutation
|
||||||
API->>DB: rollback entire transaction
|
|
||||||
API-->>C: insufficient inventory and returned 0
|
API-->>C: insufficient inventory and returned 0
|
||||||
```
|
```
|
||||||
|
|
||||||
@ -345,13 +342,12 @@ sequenceDiagram
|
|||||||
sequenceDiagram
|
sequenceDiagram
|
||||||
participant C as Controller
|
participant C as Controller
|
||||||
participant W as Gateway Worker
|
participant W as Gateway Worker
|
||||||
participant DB as PostgreSQL
|
participant R as Redis Activity Pool
|
||||||
C->>W: mark Proxy DRAINING at epoch 13
|
C->>W: mark Proxy DRAINING at epoch 13
|
||||||
W->>W: stop new allocations
|
W->>W: stop new allocations
|
||||||
W-->>C: ACK active 0 reserved 0
|
W-->>C: ACK active 0 reserved 0
|
||||||
C->>DB: clear worker ownership
|
C->>R: atomically clear ownership and extract
|
||||||
C->>DB: AVAILABLE to EXTRACTED plus audit
|
R-->>C: committed exclusive result
|
||||||
DB-->>C: committed exclusive result
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## 24 reserveForGateway 不变量
|
## 24 reserveForGateway 不变量
|
||||||
@ -367,18 +363,16 @@ flowchart TD
|
|||||||
Result -->|allOrNothing insufficient| Rollback[Return zero]
|
Result -->|allOrNothing insufficient| Rollback[Return zero]
|
||||||
```
|
```
|
||||||
|
|
||||||
## 25 PostgreSQL 独占事务
|
## 25 Redis 独占原子操作
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
flowchart TD
|
flowchart TD
|
||||||
Begin[BEGIN] --> Select[SELECT eligible FOR UPDATE SKIP LOCKED]
|
Begin[Lua script or Redis Function] --> Select[Filter eligible TTL entries]
|
||||||
Select --> Enough{Quantity satisfies mode?}
|
Select --> Enough{Quantity satisfies mode?}
|
||||||
Enough -->|no allOrNothing| Rollback[ROLLBACK]
|
Enough -->|no allOrNothing| Noop[Return zero without mutation]
|
||||||
Enough -->|yes or partial| Update[UPDATE AVAILABLE to EXTRACTED]
|
Enough -->|yes or partial| Update[Mark selected entries EXTRACTED]
|
||||||
Update --> Audit[INSERT extraction records]
|
Update --> Idempotency[Optionally cache bounded idempotency result]
|
||||||
Audit --> Outbox[INSERT outbox]
|
Idempotency --> Return[Return proxies with expiry]
|
||||||
Outbox --> Commit[COMMIT]
|
|
||||||
Commit --> Return[Return proxies with expiry]
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## 26 并发提取互斥
|
## 26 并发提取互斥
|
||||||
@ -386,17 +380,16 @@ flowchart TD
|
|||||||
```mermaid
|
```mermaid
|
||||||
sequenceDiagram
|
sequenceDiagram
|
||||||
participant A as Extract request A
|
participant A as Extract request A
|
||||||
participant DB as PostgreSQL
|
participant R as Redis Activity Pool
|
||||||
participant B as Extract request B
|
participant B as Extract request B
|
||||||
A->>DB: lock rows 1 to 10
|
A->>R: execute atomic extraction
|
||||||
B->>DB: skip locked rows 1 to 10
|
R-->>A: commit EXTRACTED 1 to 10
|
||||||
B->>DB: lock rows 11 to 20
|
B->>R: execute atomic extraction
|
||||||
A->>DB: commit EXTRACTED 1 to 10
|
R-->>B: commit EXTRACTED 11 to 20
|
||||||
B->>DB: commit EXTRACTED 11 to 20
|
|
||||||
Note over A,B: no Proxy returned twice
|
Note over A,B: no Proxy returned twice
|
||||||
```
|
```
|
||||||
|
|
||||||
## 27 Outbox 一致性
|
## 27 管理面 Outbox 一致性
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
sequenceDiagram
|
sequenceDiagram
|
||||||
@ -404,7 +397,7 @@ sequenceDiagram
|
|||||||
participant DB as PostgreSQL
|
participant DB as PostgreSQL
|
||||||
participant P as Publisher
|
participant P as Publisher
|
||||||
participant W as Worker
|
participant W as Worker
|
||||||
C->>DB: state change plus outbox
|
C->>DB: management state change plus outbox
|
||||||
DB-->>C: atomic commit
|
DB-->>C: atomic commit
|
||||||
P->>DB: read undelivered event
|
P->>DB: read undelivered event
|
||||||
P->>W: publish versioned event
|
P->>W: publish versioned event
|
||||||
@ -533,4 +526,3 @@ flowchart TD
|
|||||||
Failure --> Verify[Verify SLO and all invariants]
|
Failure --> Verify[Verify SLO and all invariants]
|
||||||
Verify --> Evidence[Archive raw metrics config and image digest]
|
Verify --> Evidence[Archive raw metrics config and image digest]
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@ -12,9 +12,9 @@ Gateway 只依赖本地不可变 Snapshot;Provider、数据库、配置重载
|
|||||||
|
|
||||||
**状态:** 接受并覆盖早期 Lease 方案。
|
**状态:** 接受并覆盖早期 Lease 方案。
|
||||||
|
|
||||||
成功响应前在一个事务中执行 `AVAILABLE -> EXTRACTED`。系统保存审计事实,
|
成功响应前,通过 Redis 原子操作完成候选选择、`AVAILABLE -> EXTRACTED` 和
|
||||||
但不提供 release、renew 或使用跟踪。这样契合“拿走真实代理后平台不再管理”
|
短期幂等结果写入;不提供 release、renew 或使用跟踪。这样契合“拿走真实
|
||||||
的最终产品语义,并消除重复发放。
|
代理后平台不再管理”的最终产品语义,并消除并发重复发放。
|
||||||
|
|
||||||
## ADR-003:单 Worker 所有权
|
## ADR-003:单 Worker 所有权
|
||||||
|
|
||||||
@ -24,11 +24,14 @@ Gateway 只依赖本地不可变 Snapshot;Provider、数据库、配置重载
|
|||||||
Distribution 只提取无所有权 Proxy;回收时执行 drain/ACK/归零/解除所有权。
|
Distribution 只提取无所有权 Proxy;回收时执行 drain/ACK/归零/解除所有权。
|
||||||
该选择避免每请求访问 Redis 做全局并发计数。
|
该选择避免每请求访问 Redis 做全局并发计数。
|
||||||
|
|
||||||
## ADR-004:PostgreSQL 权威、Redis 可重建
|
## ADR-004:PostgreSQL 保存控制面事实,Redis 承载短效活动池
|
||||||
|
|
||||||
**状态:** 接受。
|
**状态:** 接受。
|
||||||
|
|
||||||
Proxy 状态、Extraction、配置版本和 Outbox 由 PostgreSQL 持久化。Redis 只
|
Proxy 明细不进入 PostgreSQL。短效 Proxy 及其状态、所有权和过期时间只存在于
|
||||||
承担 Leader、短期速率与心跳等协调;Redis 丢失后可从权威状态恢复,避免
|
带 TTL 的 Redis 活动池及节点内存;Redis 同时承担独占提取、短期幂等结果、
|
||||||
双写状态成为不可判定的事实源。
|
Provider Leader、分布式速率和 Worker 心跳。Redis 丢失时活动池作废,由
|
||||||
|
Provider 重新获取并重建,不从 PostgreSQL 恢复原 Proxy。
|
||||||
|
|
||||||
|
PostgreSQL 只持久化配置版本、Upstream/Routing 管理状态、Admin 审计与 Outbox,
|
||||||
|
以及可选的无 Proxy 明细聚合指标。两类存储不双写 Proxy,也不建立跨存储事务。
|
||||||
|
|||||||
@ -10,7 +10,7 @@ Proto 源文件位于 `api/proto/controlplane/v1/controlplane.proto`,包含两
|
|||||||
- `CheckerControlPlane`:健康检查任务流和 Observation 批量上报。
|
- `CheckerControlPlane`:健康检查任务流和 Observation 批量上报。
|
||||||
|
|
||||||
该协议不承载 Client 的独占提取,也没有 extraction lease/release。Proxy 的
|
该协议不承载 Client 的独占提取,也没有 extraction lease/release。Proxy 的
|
||||||
`AVAILABLE -> EXTRACTED` 只在 Controller 权威事务中完成。
|
`AVAILABLE -> EXTRACTED` 只在 Controller 调用的 Redis 原子操作中完成。
|
||||||
|
|
||||||
## 2. Worker 会话
|
## 2. Worker 会话
|
||||||
|
|
||||||
@ -40,9 +40,14 @@ sequenceDiagram
|
|||||||
|
|
||||||
- 单调 `version`、`ownership_epoch`、生成时间和有效期。
|
- 单调 `version`、`ownership_epoch`、生成时间和有效期。
|
||||||
- 对该 Worker 可见的有序 Routing。
|
- 对该 Worker 可见的有序 Routing。
|
||||||
- 仅归该 Worker 所有的 Proxy 与每个 Proxy 的容量。
|
- 仅归该 Worker 所有的 Proxy、每个 Proxy 的容量、硬过期时间 `expires_at`
|
||||||
|
和停止新分配的 `usable_until`。
|
||||||
- 内容 `checksum`。
|
- 内容 `checksum`。
|
||||||
|
|
||||||
|
`usable_until = expires_at - allocationSafetyMargin`。Worker 必须以
|
||||||
|
`usable_until` 作为最后可分配时刻;达到该时间后即使尚未到 `expires_at`,
|
||||||
|
也不得再为新请求选择该 Proxy。
|
||||||
|
|
||||||
Delta 声明 `base_version`。Worker 只有在本地版本恰好等于 base 且 checksum
|
Delta 声明 `base_version`。Worker 只有在本地版本恰好等于 base 且 checksum
|
||||||
验证成功时才能应用;否则丢弃 Delta 并请求完整 Snapshot。构建在后台完成,
|
验证成功时才能应用;否则丢弃 Delta 并请求完整 Snapshot。构建在后台完成,
|
||||||
热路径只读取一次原子指针。
|
热路径只读取一次原子指针。
|
||||||
@ -57,11 +62,11 @@ Delta 声明 `base_version`。Worker 只有在本地版本恰好等于 base 且
|
|||||||
1. 新 Snapshot 标记或移除该 Proxy,使 Worker 停止新预留。
|
1. 新 Snapshot 标记或移除该 Proxy,使 Worker 停止新预留。
|
||||||
2. Worker 上报 `draining=true` 以及 Active/Reserved。
|
2. Worker 上报 `draining=true` 以及 Active/Reserved。
|
||||||
3. 两个计数都归零后 Controller 清除所有权。
|
3. 两个计数都归零后 Controller 清除所有权。
|
||||||
4. 无所有权 Proxy 才能进入 Distribution 提取事务。
|
4. 无所有权 Proxy 才能进入 Distribution 的 Redis 原子提取操作。
|
||||||
|
|
||||||
Worker 崩溃时必须等待所有权 epoch/有效期失效后再转移,避免双主。Proto 中
|
Worker 崩溃时必须等待所有权 epoch/有效期失效后再转移,避免双主。Proto 中
|
||||||
`ReportRuntimeResponse.revoke_proxy_ids` 是加速 Drain 的控制信号,不绕过
|
`ReportRuntimeResponse.revoke_proxy_ids` 是加速 Drain 的控制信号,不绕过
|
||||||
Snapshot 版本和权威持久化。
|
Snapshot 版本和 Redis 中的权威所有权状态。
|
||||||
|
|
||||||
## 5. Outcome 上报
|
## 5. Outcome 上报
|
||||||
|
|
||||||
@ -73,8 +78,9 @@ Outcome 按 Worker 单调 `sequence` 批量上报。Controller 返回已接受
|
|||||||
- `RESPONSE_HEADERS`:目标响应头前失败。
|
- `RESPONSE_HEADERS`:目标响应头前失败。
|
||||||
- `TUNNEL`:隧道建立后结束或失败。
|
- `TUNNEL`:隧道建立后结束或失败。
|
||||||
|
|
||||||
Outcome 是 Observation,不直接让 Worker 修改 PostgreSQL 状态。异步上报队列
|
Outcome 是 Observation,不直接让 Worker 修改 Redis 活动池状态,也不创建
|
||||||
必须有界;队列满时丢弃低价值样本并计指标,不能反压 Gateway 热路径。
|
PostgreSQL Proxy 明细。异步上报队列必须有界;队列满时丢弃低价值样本并计
|
||||||
|
指标,不能反压 Gateway 热路径。
|
||||||
|
|
||||||
## 6. Checker 任务
|
## 6. Checker 任务
|
||||||
|
|
||||||
@ -85,7 +91,8 @@ Checker 注册自身最大并发与支持层级,Controller 发送有 deadline
|
|||||||
- `TARGET`:针对 Routing/目标组的可达性。
|
- `TARGET`:针对 Routing/目标组的可达性。
|
||||||
|
|
||||||
Checker 只返回 `HealthObservation`。Controller reducer 按 Proxy、检查层级和
|
Checker 只返回 `HealthObservation`。Controller reducer 按 Proxy、检查层级和
|
||||||
Routing 决定 AVAILABLE、SUSPECT 或 UNHEALTHY,避免多个 Checker 并发写状态。
|
Routing 决定 AVAILABLE、SUSPECT 或 UNHEALTHY,并更新 Redis 活动池,避免
|
||||||
|
多个 Checker 并发写状态。
|
||||||
|
|
||||||
## 7. 兼容与演进
|
## 7. 兼容与演进
|
||||||
|
|
||||||
|
|||||||
@ -5,18 +5,17 @@
|
|||||||
Distribution API 只提供一次性独占提取:
|
Distribution API 只提供一次性独占提取:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
筛选 AVAILABLE
|
读取 Redis TTL 活动池中的 AVAILABLE 候选
|
||||||
-> 锁定候选
|
-> 校验 Gateway 预留、TTL、健康、所有权与过滤条件
|
||||||
-> 校验 Gateway 预留、TTL、健康与过滤条件
|
-> Redis 原子移出可分配池并写入短期幂等结果
|
||||||
-> 原子 AVAILABLE -> EXTRACTED
|
|
||||||
-> 写 Extraction Record
|
|
||||||
-> 提交事务
|
|
||||||
-> 返回真实代理地址
|
-> 返回真实代理地址
|
||||||
```
|
```
|
||||||
|
|
||||||
事务提交前不得把地址写给 Client。返回成功后,该 Proxy 不再参与 Gateway、
|
Redis 原子操作成功前不得把地址写给 Client。返回成功后,该 Proxy 在当前 TTL
|
||||||
再次提取或可用库存统计。系统不跟踪 Client 是否使用、使用并发或何时停止,
|
生命周期中不再参与 Gateway、再次提取或可用库存统计。系统不跟踪 Client 是否
|
||||||
也不提供 release、renew、status 或租约端点。
|
使用、使用并发或何时停止,也不提供 release、renew、status 或租约端点。短效
|
||||||
|
Proxy 明细只存在于 Redis TTL 活动池和节点内存;PostgreSQL 不保存 Proxy 明细
|
||||||
|
或逐次提取记录。
|
||||||
|
|
||||||
HTTP 契约源文件:`api/openapi/proxy-pool.yaml`。
|
HTTP 契约源文件:`api/openapi/proxy-pool.yaml`。
|
||||||
|
|
||||||
@ -105,7 +104,7 @@ Request ID,不记录地址或凭据。
|
|||||||
|
|
||||||
### 4.1 partial
|
### 4.1 partial
|
||||||
|
|
||||||
锁定的符合条件数量少于 `count` 时,提交实际数量:
|
符合条件数量少于 `count` 时,原子提交实际数量:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
requested=10, eligible=6, reserve=0 -> returned=6
|
requested=10, eligible=6, reserve=0 -> returned=6
|
||||||
@ -126,13 +125,14 @@ requested=10, eligible=6, reserve=0 -> returned=6
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
冲突响应后,之前被该请求临时锁定的 Proxy 仍为 AVAILABLE。
|
冲突响应后,候选 Proxy 仍在 Redis TTL 活动池中保持 AVAILABLE。
|
||||||
|
|
||||||
## 5. 资格过滤与 Gateway 预留
|
## 5. 资格过滤与 Gateway 预留
|
||||||
|
|
||||||
候选必须同时满足:
|
候选必须同时满足:
|
||||||
|
|
||||||
1. 权威状态是 AVAILABLE。
|
1. Redis TTL 活动池条目仍为 AVAILABLE,且当前时间早于按供应商安全余量计算的
|
||||||
|
`usableUntil`。
|
||||||
2. 没有 Worker 所有权,或已完成 Drain 且 Active/Reserved 均为零。
|
2. 没有 Worker 所有权,或已完成 Drain 且 Active/Reserved 均为零。
|
||||||
3. 剩余 TTL 不低于 `minRemainingTTL`。
|
3. 剩余 TTL 不低于 `minRemainingTTL`。
|
||||||
4. 最近健康检查不早于 `maxHealthCheckAge`。
|
4. 最近健康检查不早于 `maxHealthCheckAge`。
|
||||||
@ -140,29 +140,33 @@ requested=10, eligible=6, reserve=0 -> returned=6
|
|||||||
6. 提取后符合条件的共享库存不低于 `reserveForGateway`。
|
6. 提取后符合条件的共享库存不低于 `reserveForGateway`。
|
||||||
|
|
||||||
Worker-owned Proxy 不得直接提取。Controller 需要先发布 DRAINING,等待 Worker
|
Worker-owned Proxy 不得直接提取。Controller 需要先发布 DRAINING,等待 Worker
|
||||||
确认没有 Active/Reserved,再清除所有权并进入提取事务。快照延迟时仍禁止
|
确认没有 Active/Reserved,再清除 Redis 中的所有权并进入原子提取。快照延迟时
|
||||||
Gateway 与 Client 同时获得同一 Proxy。
|
仍禁止 Gateway 与 Client 同时获得同一 Proxy。
|
||||||
|
|
||||||
## 6. 并发与事务
|
## 6. 并发与原子性
|
||||||
|
|
||||||
PostgreSQL Adapter 应在一个事务内使用 `FOR UPDATE SKIP LOCKED` 获取候选,
|
Redis Adapter 应通过单个 Lua 脚本、Redis Function 或等价的原子原语完成候选
|
||||||
更新状态并写审计记录。所有状态更新必须包含 `state = 'AVAILABLE'` 前置条件。
|
筛选、TTL/健康/所有权复核、Gateway 预留计算、满足模式判断、从可分配池移除
|
||||||
|
所选条目,并写入短期幂等结果。提取过程中不访问 PostgreSQL。
|
||||||
|
|
||||||
关键不变量:
|
关键不变量:
|
||||||
|
|
||||||
- 两个并发成功响应的 Proxy ID 集合交集为空。
|
- 两个并发成功响应的 Proxy ID 集合交集为空。
|
||||||
- 状态更新或审计写入任一步失败,整个批次不返回。
|
- 原子操作失败时整个批次不返回,也不得留下部分移除结果。
|
||||||
- `allOrNothing` 不足时零行变为 EXTRACTED。
|
- `allOrNothing` 不足时零个条目退出活动池。
|
||||||
- PostgreSQL 不可写时返回 503,不以内存结果冒充成功。
|
- Redis 活动池不可用时返回 503,不以内存副本冒充成功。
|
||||||
|
- PostgreSQL 不可用不阻断提取;需要 PostgreSQL 的 Admin 管理写入单独降级。
|
||||||
|
|
||||||
## 7. 幂等
|
## 7. 幂等
|
||||||
|
|
||||||
提取是消耗库存的写操作。客户端收到超时后盲目重试可能再次提取一批不同
|
提取是消耗库存的写操作。客户端收到超时后盲目重试可能再次提取一批不同
|
||||||
Proxy,因此自动重试应提供稳定的 `Idempotency-Key`。
|
Proxy,因此自动重试应提供稳定的 `Idempotency-Key`。
|
||||||
|
|
||||||
服务端幂等记录至少包含:Client ID、Key、请求体摘要、提交结果和过期时间。
|
服务端在 Redis 中保存有界 TTL 的幂等记录,至少包含 Client ID、Key、请求体
|
||||||
同一 Client、同一 Key、相同摘要返回首次结果;摘要不同返回 409。幂等记录和
|
摘要、提交结果和过期时间。同一 Client、同一 Key、相同摘要返回首次结果;
|
||||||
Extraction Record 必须与状态更新处在相同事务边界或由同一权威恢复流程保证。
|
摘要不同返回 409。幂等结果必须和活动池状态变更处于同一个 Redis 原子操作。
|
||||||
|
TTL 到期或 Redis 数据丢失后不再保证旧 Key 去重,系统不回退到 PostgreSQL
|
||||||
|
保存逐次提取事实。
|
||||||
|
|
||||||
## 8. 认证、识别与限制
|
## 8. 认证、识别与限制
|
||||||
|
|
||||||
@ -186,28 +190,30 @@ Extraction Record 必须与状态更新处在相同事务边界或由同一权
|
|||||||
- `415`:请求体不是 `application/json`。
|
- `415`:请求体不是 `application/json`。
|
||||||
- `422`:数量、枚举或过滤组合违反业务约束。
|
- `422`:数量、枚举或过滤组合违反业务约束。
|
||||||
- `429`:全局或 Client 速率限制,响应 `Retry-After`。
|
- `429`:全局或 Client 速率限制,响应 `Retry-After`。
|
||||||
- `503`:PostgreSQL 不可写、服务排空或权威状态不可用。
|
- `503`:Redis 活动池不可用、原子提取不可执行或服务正在排空。
|
||||||
- `500`:未分类的内部错误;响应不包含底层错误文本。
|
- `500`:未分类的内部错误;响应不包含底层错误文本。
|
||||||
|
|
||||||
错误响应不得包含 Provider Secret、Proxy 凭据、SQL 或内部拓扑。
|
错误响应不得包含 Provider Secret、Proxy 凭据、SQL 或内部拓扑。
|
||||||
|
|
||||||
## 10. 审计记录
|
## 10. 短期运行记录与数据最小化
|
||||||
|
|
||||||
每个被提交的 Proxy 对应一条 Extraction Record:
|
Redis 幂等结果只在配置的 TTL 窗口内保留重放响应所需的数据:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
proxyId, clientId, sourceIP, requestId, upstream, extractedAt, expiresAt
|
clientId, idempotencyKey, requestDigest, response, expiresAt
|
||||||
```
|
```
|
||||||
|
|
||||||
无认证时 `clientId` 使用可信代理链解析后的规范化来源 IP 稳定标识,并保留
|
无认证时 `clientId` 使用可信代理链解析后的规范化来源 IP 稳定标识。应用日志
|
||||||
`sourceIP`。记录只用于审计、排错和计费事实,不承担资源归还语义。Proxy
|
仅记录 Request ID、数量、结果码和过滤摘要,不记录 Proxy 地址或凭据。需要长期
|
||||||
到期后可以清理运行记录,但 Extraction Record 按审计保留策略归档。
|
分析时只向 PostgreSQL 写入不含 Proxy 明细的可选聚合指标;系统不创建或归档
|
||||||
|
逐个 Proxy 的 Extraction Record。
|
||||||
|
|
||||||
## 11. 运行时实现边界
|
## 11. 运行时实现边界
|
||||||
|
|
||||||
`distribution.Handler` 是薄 HTTP Adapter,只负责严格解码、Header/DTO 校验、
|
`distribution.Handler` 是薄 HTTP Adapter,只负责严格解码、Header/DTO 校验、
|
||||||
身份结果注入、错误映射和健康探针。独占提取、TTL、Gateway 预留及幂等事务
|
身份结果注入、错误映射和健康探针。独占提取、TTL、Gateway 预留及幂等事务
|
||||||
继续由 `extraction.Service` 和持久化 Store 承担。
|
继续由 `extraction.Service` 和 Redis 活动池 Adapter 承担。当前内存 Store 仅作为
|
||||||
|
原子行为的参考实现与测试替身,不是生产持久化层。
|
||||||
|
|
||||||
请求体解码、Request ID 与 Problem JSON 统一复用 `platform/httpapi`。认证、
|
请求体解码、Request ID 与 Problem JSON 统一复用 `platform/httpapi`。认证、
|
||||||
可信代理、来源控制、Client ID 和准入限流由必需的 `IdentityResolver` 注入,
|
可信代理、来源控制、Client ID 和准入限流由必需的 `IdentityResolver` 注入,
|
||||||
|
|||||||
@ -145,20 +145,25 @@ distribution:
|
|||||||
minRemainingTTL: 30s
|
minRemainingTTL: 30s
|
||||||
maxHealthCheckAge: 15s
|
maxHealthCheckAge: 15s
|
||||||
reserveForGateway: 5
|
reserveForGateway: 5
|
||||||
|
idempotencyTTL: 5m
|
||||||
```
|
```
|
||||||
|
|
||||||
Extraction 是固定的一次性独占行为,**没有** `mode`、`leaseDuration`、
|
Extraction 是固定的一次性独占行为,**没有** `mode`、`leaseDuration`、
|
||||||
`release` 或 `renew` 配置。事务提交后执行 `AVAILABLE -> EXTRACTED`,该 Proxy
|
`release` 或 `renew` 配置。Redis 在一个原子操作内选择候选、执行
|
||||||
不再由 Gateway 或 Distribution 分配。
|
`AVAILABLE -> EXTRACTED` 并短期保存幂等结果;成功后该 Proxy 不再由 Gateway
|
||||||
|
或 Distribution 在当前 TTL 生命周期中分配。
|
||||||
|
|
||||||
- `fulfillment`:`partial` 或 `allOrNothing`,默认语义为 `partial`。
|
- `fulfillment`:`partial` 或 `allOrNothing`,默认语义为 `partial`。
|
||||||
- `maxCountPerRequest`:单次请求硬上限。
|
- `maxCountPerRequest`:单次请求硬上限。
|
||||||
- `minRemainingTTL`:剩余寿命低于此值时不参与提取。
|
- `minRemainingTTL`:剩余寿命低于此值时不参与提取。
|
||||||
- `maxHealthCheckAge`:最近检查早于此窗口时不参与提取。
|
- `maxHealthCheckAge`:最近检查早于此窗口时不参与提取。
|
||||||
- `reserveForGateway`:提取后必须留给 Gateway 的最低符合条件库存数量。
|
- `reserveForGateway`:提取后必须留给 Gateway 的最低符合条件库存数量。
|
||||||
|
- `idempotencyTTL`:Redis 幂等结果的最长保留时间;省略时为 5 分钟,实际
|
||||||
|
保留时间不会超过本次返回代理中最早的 `expiresAt`。
|
||||||
|
|
||||||
`partial` 会提交实际可得数量;`allOrNothing` 数量不足时事务回滚,一个也不
|
`partial` 会原子提取实际可得数量;`allOrNothing` 数量不足时不改变任何候选,
|
||||||
提取。认证关闭时仍应使用 `sourceIP` 识别匿名 Client 并执行全局/来源限流。
|
一个也不提取。认证关闭时仍应使用 `sourceIP` 识别匿名 Client 并执行全局/
|
||||||
|
来源限流。
|
||||||
|
|
||||||
`clientIdentification.mode` 支持:
|
`clientIdentification.mode` 支持:
|
||||||
|
|
||||||
@ -326,9 +331,14 @@ storage:
|
|||||||
redisURL: "${REDIS_URL}"
|
redisURL: "${REDIS_URL}"
|
||||||
```
|
```
|
||||||
|
|
||||||
PostgreSQL 是 Proxy 生命周期、Routing 选择、Worker 所有权和 Extraction Record
|
Proxy 明细不写入 PostgreSQL。Redis 是短效 Proxy 活动池的运行时事实源,使用
|
||||||
的权威存储。Redis 只承载可重建的短期协调状态,不能成为独占提取的唯一事实
|
TTL 保存 Proxy 状态、所有权和过期时间,并原子执行独占提取及短期幂等结果
|
||||||
来源。Metrics 标签禁止 Proxy IP、Client ID、Session、完整 URL 和 Request ID。
|
读写;Redis 丢失后由 Provider 重新补池,不从 PostgreSQL 恢复原 Proxy。
|
||||||
|
|
||||||
|
PostgreSQL 只保存配置版本、Upstream/Routing 管理状态、Admin 审计与 Outbox,
|
||||||
|
以及可选的无 Proxy 明细聚合指标。Distribution 提取不依赖 PostgreSQL,因而
|
||||||
|
PostgreSQL 故障本身不应使 Redis 中可完成的 Extract 返回 `503`。Metrics 标签
|
||||||
|
禁止 Proxy IP、Client ID、Session、完整 URL 和 Request ID。
|
||||||
|
|
||||||
## 9. 启动前校验清单
|
## 9. 启动前校验清单
|
||||||
|
|
||||||
|
|||||||
@ -67,7 +67,7 @@ Provider、Pool、Routing、Distribution 在首版需要共享事务和一致性
|
|||||||
- Routing Sequential 当前选择和原子切换。
|
- Routing Sequential 当前选择和原子切换。
|
||||||
- Worker 所有权/容量切片、Snapshot 发布、ACK 和重同步。
|
- Worker 所有权/容量切片、Snapshot 发布、ACK 和重同步。
|
||||||
- Distribution 与 Admin HTTP 接口。
|
- Distribution 与 Admin HTTP 接口。
|
||||||
- PostgreSQL 权威状态和 Redis 短期协调。
|
- PostgreSQL 管理面持久化和 Redis TTL 活动池。
|
||||||
|
|
||||||
### 3.3 proxy-checker
|
### 3.3 proxy-checker
|
||||||
|
|
||||||
@ -88,7 +88,8 @@ domain/
|
|||||||
proxy Proxy、状态机、TTL、唯一键、容量
|
proxy Proxy、状态机、TTL、唯一键、容量
|
||||||
routing 规则、匹配、策略与 Sequential 状态
|
routing 规则、匹配、策略与 Sequential 状态
|
||||||
upstream Provider 能力、Fetch 分类与 Pool 需求
|
upstream Provider 能力、Fetch 分类与 Pool 需求
|
||||||
extraction 独占提取命令、结果和审计事实
|
activitypool 短效 Proxy、TTL、所有权和独占消费
|
||||||
|
extraction 独占提取命令与结果
|
||||||
client 认证主体、权限与限制
|
client 认证主体、权限与限制
|
||||||
|
|
||||||
gateway/
|
gateway/
|
||||||
@ -102,7 +103,7 @@ controller/
|
|||||||
pool Pool reconcile、生命周期和所有权
|
pool Pool reconcile、生命周期和所有权
|
||||||
health 任务计划和 Observation reducer
|
health 任务计划和 Observation reducer
|
||||||
routing Sequential 原子切换和恢复
|
routing Sequential 原子切换和恢复
|
||||||
extraction 批量原子提取与审计
|
extraction Redis 活动池批量原子提取与短期幂等
|
||||||
distribution Worker 注册、Snapshot/Delta 与结果上报
|
distribution Worker 注册、Snapshot/Delta 与结果上报
|
||||||
|
|
||||||
adapters/
|
adapters/
|
||||||
@ -126,6 +127,7 @@ type Proxy struct {
|
|||||||
SourceUpstream UpstreamID
|
SourceUpstream UpstreamID
|
||||||
CreatedAt time.Time
|
CreatedAt time.Time
|
||||||
ExpiresAt *time.Time
|
ExpiresAt *time.Time
|
||||||
|
UsableUntil *time.Time
|
||||||
LastCheckedAt *time.Time
|
LastCheckedAt *time.Time
|
||||||
LastSuccessAt *time.Time
|
LastSuccessAt *time.Time
|
||||||
Latency time.Duration
|
Latency time.Duration
|
||||||
@ -136,6 +138,8 @@ type Proxy struct {
|
|||||||
```
|
```
|
||||||
|
|
||||||
密码不参与日志可见唯一键;`CredentialVersion` 区分同一用户名的凭据轮换。
|
密码不参与日志可见唯一键;`CredentialVersion` 区分同一用户名的凭据轮换。
|
||||||
|
唯一键不包含 Upstream:多个供应商返回同一唯一键时,当前生命周期的首个来源
|
||||||
|
保持归属,其他来源只计重复且不得覆盖 TTL;原来源硬过期淘汰后可重新归属。
|
||||||
运行态 `active` 与 `reserved` 存在 Worker 本地、按 Proxy ID 分片,不写入
|
运行态 `active` 与 `reserved` 存在 Worker 本地、按 Proxy ID 分片,不写入
|
||||||
不可变 Snapshot。
|
不可变 Snapshot。
|
||||||
|
|
||||||
@ -279,27 +283,26 @@ sequenceDiagram
|
|||||||
participant API as Distribution API
|
participant API as Distribution API
|
||||||
participant CP as Controller
|
participant CP as Controller
|
||||||
participant W as Gateway Worker
|
participant W as Gateway Worker
|
||||||
participant DB as PostgreSQL
|
participant R as Redis Activity Pool
|
||||||
|
|
||||||
C->>API: POST extract(count, filters)
|
C->>API: POST extract(count, filters)
|
||||||
API->>API: auth + access + limits
|
API->>API: auth + access + limits
|
||||||
API->>CP: Extract command
|
API->>CP: Extract command
|
||||||
CP->>DB: select eligible unowned proxies
|
CP->>R: atomic select eligible unowned proxies
|
||||||
alt candidate owned by Worker
|
alt candidate owned by Worker
|
||||||
CP->>W: revoke and drain
|
CP->>W: revoke and drain
|
||||||
W-->>CP: active=0, reserved=0, ownership released
|
W-->>CP: active=0, reserved=0, ownership released
|
||||||
end
|
end
|
||||||
CP->>DB: transaction AVAILABLE -> EXTRACTED
|
CP->>R: AVAILABLE -> EXTRACTED + bounded idempotency result
|
||||||
DB->>DB: insert extraction records
|
R-->>CP: consumed proxies
|
||||||
DB-->>CP: committed rows
|
|
||||||
CP-->>API: proxy + expiry
|
CP-->>API: proxy + expiry
|
||||||
API-->>C: requested/returned/proxies
|
API-->>C: requested/returned/proxies
|
||||||
```
|
```
|
||||||
|
|
||||||
### 10.2 原子批量提取
|
### 10.2 原子批量提取
|
||||||
|
|
||||||
PostgreSQL Adapter 使用事务和 `FOR UPDATE SKIP LOCKED` 选择满足以下条件的
|
Redis Adapter 使用单个 Lua 脚本或等价原子命令,选择满足以下条件的活动池
|
||||||
行,并在同一事务更新状态与写审计记录:
|
条目,并同时完成状态迁移和带 TTL 的幂等结果写入:
|
||||||
|
|
||||||
- State 为 AVAILABLE。
|
- State 为 AVAILABLE。
|
||||||
- 未分配 Worker 所有权,Reserved/Active 为 0。
|
- 未分配 Worker 所有权,Reserved/Active 为 0。
|
||||||
@ -308,7 +311,8 @@ PostgreSQL Adapter 使用事务和 `FOR UPDATE SKIP LOCKED` 选择满足以下
|
|||||||
- 提取后仍保留 `reserveForGateway`。
|
- 提取后仍保留 `reserveForGateway`。
|
||||||
- 符合 protocol、region、carrier 与 allowedUpstreams。
|
- 符合 protocol、region、carrier 与 allowedUpstreams。
|
||||||
|
|
||||||
`partial` 提交实际可得数量;`allOrNothing` 在锁定数量不足时回滚。
|
`partial` 原子消费实际可得数量;`allOrNothing` 数量不足时不改变任何条目。
|
||||||
|
Proxy 地址、凭据和逐次提取明细不写 PostgreSQL。
|
||||||
|
|
||||||
### 10.3 与 Gateway 共池
|
### 10.3 与 Gateway 共池
|
||||||
|
|
||||||
@ -332,7 +336,9 @@ flowchart LR
|
|||||||
- Worker 仅对自身 Proxy 做本地原子计数。
|
- Worker 仅对自身 Proxy 做本地原子计数。
|
||||||
- 所有权租约有 epoch 与过期时间;Worker 失联后先等待租约过期,再分配给
|
- 所有权租约有 epoch 与过期时间;Worker 失联后先等待租约过期,再分配给
|
||||||
其他 Worker,避免双主。
|
其他 Worker,避免双主。
|
||||||
- Snapshot 包含 Worker 专属 Proxy 集、Routing、策略和版本校验和。
|
- Snapshot 包含 Worker 专属 Proxy 集、Routing、策略、`ExpiresAt`、
|
||||||
|
`UsableUntil` 和版本校验和。Worker 在 `UsableUntil` 到达后立即停止新分配,
|
||||||
|
不等待供应商硬过期时间 `ExpiresAt`。
|
||||||
- Worker 断开控制面后在 `maxStaleAge` 内使用最后快照;超限停止接收新流量,
|
- Worker 断开控制面后在 `maxStaleAge` 内使用最后快照;超限停止接收新流量,
|
||||||
已有隧道排空。
|
已有隧道排空。
|
||||||
|
|
||||||
@ -340,18 +346,22 @@ flowchart LR
|
|||||||
|
|
||||||
### 12.1 PostgreSQL
|
### 12.1 PostgreSQL
|
||||||
|
|
||||||
权威保存:配置版本、Upstream、Proxy 生命周期、Routing 当前选择、Worker
|
持久化配置版本、Upstream/Routing 管理状态、Admin 审计和管理事件 outbox,
|
||||||
所有权、Extraction Record、Client、审计与 outbox。
|
以及可选的不含 Proxy 明细的聚合指标。PostgreSQL 不保存 Proxy 地址、凭据、
|
||||||
|
生命周期、Worker 所有权或逐次提取记录。
|
||||||
|
|
||||||
### 12.2 Redis
|
### 12.2 Redis
|
||||||
|
|
||||||
保存可重建短期状态:Provider Leader 租约、分布式限流、singleflight 信号、
|
保存带 TTL 的短效 Proxy 活动池、生命周期、Worker 所有权、独占提取状态和
|
||||||
短期 Client 限流和 Worker 心跳。Redis 不保存唯一权威业务状态。
|
短期幂等结果,并承载 Provider Leader、分布式限流、singleflight 信号、
|
||||||
|
短期 Client 限流和 Worker 心跳。Redis 丢失后由 Provider 重新获取代理重建,
|
||||||
|
不会从 PostgreSQL 恢复旧 Proxy。
|
||||||
|
|
||||||
### 12.3 Outbox
|
### 12.3 Outbox
|
||||||
|
|
||||||
任何需要发布 Snapshot/事件的 PostgreSQL 状态更新同时写 outbox。发布成功
|
管理状态变更在 PostgreSQL 事务内同时写 outbox。活动池变更通过 Redis 事件流
|
||||||
后标记;Worker 使用 epoch/version 幂等应用,缺口触发完整 Snapshot。
|
通知 Controller 构建 Snapshot,不跨 PostgreSQL/Redis 双写 Proxy;Worker 使用
|
||||||
|
epoch/version 幂等应用,缺口时从当前活动池构建完整 Snapshot。
|
||||||
|
|
||||||
## 13. 健康模型
|
## 13. 健康模型
|
||||||
|
|
||||||
@ -404,7 +414,7 @@ flowchart LR
|
|||||||
- Routing 当前 Upstream、切换、无可用和策略选择。
|
- Routing 当前 Upstream、切换、无可用和策略选择。
|
||||||
- Upstream Proxy 状态、Available Slots、Active、Reserved、Pending Fetch。
|
- Upstream Proxy 状态、Available Slots、Active、Reserved、Pending Fetch。
|
||||||
- Provider Fetch 成功、Empty、Duplicate、Error、429、退避和耗时。
|
- Provider Fetch 成功、Empty、Duplicate、Error、429、退避和耗时。
|
||||||
- Extraction requested、returned、insufficient、atomic conflict 和审计失败。
|
- Extraction requested、returned、insufficient、atomic conflict 和幂等缓存失败。
|
||||||
- Snapshot epoch/version、陈旧时长、应用耗时和重同步。
|
- Snapshot epoch/version、陈旧时长、应用耗时和重同步。
|
||||||
|
|
||||||
Prometheus 标签不包含 Proxy IP、Client ID、session、完整 URL 或 request ID。
|
Prometheus 标签不包含 Proxy IP、Client ID、session、完整 URL 或 request ID。
|
||||||
@ -443,8 +453,8 @@ CPU、内存、网络、Go 版本、配置和上游响应模型下测得。
|
|||||||
|---|---|
|
|---|---|
|
||||||
| Provider 超时/500 | 计 Error、退避;不计 Empty,不影响已有 Proxy |
|
| Provider 超时/500 | 计 Error、退避;不计 Empty,不影响已有 Proxy |
|
||||||
| Provider 合法空响应 | Empty++;达到阈值触发相关 Routing 原子切换 |
|
| Provider 合法空响应 | Empty++;达到阈值触发相关 Routing 原子切换 |
|
||||||
| Redis 不可用 | Gateway 不受影响;Controller 使用本地退化并停止高风险写操作 |
|
| Redis 不可用 | Gateway 暂用未过期快照;停止 Fetch 入池、Extract 和所有权变更 |
|
||||||
| PostgreSQL 不可用 | Gateway 使用最后快照;停止 Extract 和权威状态变更 |
|
| PostgreSQL 不可用 | Gateway 与 Redis Extract 不受影响;停止管理状态变更和 Admin 审计 |
|
||||||
| Controller 断线 | Worker 在 maxStaleAge 内继续;超限拒绝新流量并排空 |
|
| Controller 断线 | Worker 在 maxStaleAge 内继续;超限拒绝新流量并排空 |
|
||||||
| Worker 崩溃 | 所有权租约过期后重新分配;期间不双重所有 |
|
| Worker 崩溃 | 所有权租约过期后重新分配;期间不双重所有 |
|
||||||
| Checker 积压 | 降低普通复检频率,优先新 Proxy 与 SUSPECT,不无限排队 |
|
| Checker 积压 | 降低普通复检频率,优先新 Proxy 与 SUSPECT,不无限排队 |
|
||||||
|
|||||||
@ -16,7 +16,7 @@ Proxy Pool 把多个供应商的动态代理统一成一个可运营的资源系
|
|||||||
- **业务调用方**:通过稳定入口使用代理,不感知供应商差异。
|
- **业务调用方**:通过稳定入口使用代理,不感知供应商差异。
|
||||||
- **代理直提调用方**:按协议、区域、运营商或 Upstream 条件独占提取。
|
- **代理直提调用方**:按协议、区域、运营商或 Upstream 条件独占提取。
|
||||||
- **平台管理员**:管理 Provider、Routing、容量、健康和故障切换。
|
- **平台管理员**:管理 Provider、Routing、容量、健康和故障切换。
|
||||||
- **SRE**:依据低基数指标、审计记录和运行手册进行容量与故障管理。
|
- **SRE**:依据低基数指标、管理面审计和运行手册进行容量与故障管理。
|
||||||
|
|
||||||
## 3. 核心价值
|
## 3. 核心价值
|
||||||
|
|
||||||
@ -30,11 +30,11 @@ Provider Adapter 负责请求格式、认证和响应解析。标准化后,Rou
|
|||||||
Gateway Worker 只读取本地不可变快照并维护本地容量计数。供应商延迟、
|
Gateway Worker 只读取本地不可变快照并维护本地容量计数。供应商延迟、
|
||||||
数据库抖动和控制面重载不会成为每请求依赖。
|
数据库抖动和控制面重载不会成为每请求依赖。
|
||||||
|
|
||||||
### 3.3 明确且可审计的资源语义
|
### 3.3 明确的资源语义
|
||||||
|
|
||||||
系统区分“短时使用容量”和“一次性独占提取”。每次 Extraction 保存请求、
|
系统区分“短时使用容量”和“一次性独占提取”。Extraction 只在 Redis 中短期
|
||||||
调用方、来源、Upstream、提取时间和过期时间审计事实,但不追踪提取后的
|
保存有界幂等结果,不向 PostgreSQL 写逐个 Proxy 的提取事实;系统不追踪
|
||||||
实际使用,也不存在归还接口。
|
提取后的实际使用,也不存在归还接口。
|
||||||
|
|
||||||
## 4. 主要流程
|
## 4. 主要流程
|
||||||
|
|
||||||
@ -51,7 +51,8 @@ Gateway Worker 只读取本地不可变快照并维护本地容量计数。供
|
|||||||
|
|
||||||
1. 调用方提交数量、过滤条件和 fulfillment。
|
1. 调用方提交数量、过滤条件和 fulfillment。
|
||||||
2. Controller 校验调用方限额、TTL、健康新鲜度及 Gateway 保留量。
|
2. Controller 校验调用方限额、TTL、健康新鲜度及 Gateway 保留量。
|
||||||
3. 在一个数据库事务中锁定候选并执行 `AVAILABLE -> EXTRACTED`。
|
3. 在一个 Redis Lua 脚本、Redis Function 或等价原子操作中筛选候选并执行
|
||||||
|
`AVAILABLE -> EXTRACTED`。
|
||||||
4. `partial` 尽量返回;`allOrNothing` 数量不足时零提取。
|
4. `partial` 尽量返回;`allOrNothing` 数量不足时零提取。
|
||||||
5. 响应返回代理 URL、`expiresAt` 和 `remainingTtlSeconds`。
|
5. 响应返回代理 URL、`expiresAt` 和 `remainingTtlSeconds`。
|
||||||
|
|
||||||
@ -88,6 +89,6 @@ required_workers = ceil(peak_qps / (measured_worker_qps * target_utilization))
|
|||||||
## 7. 首版范围
|
## 7. 首版范围
|
||||||
|
|
||||||
首版包含 HTTP 正向代理、HTTPS CONNECT、REST Distribution/Admin、Provider
|
首版包含 HTTP 正向代理、HTTPS CONNECT、REST Distribution/Admin、Provider
|
||||||
适配、健康与生命周期、Sequential 等路由策略、PostgreSQL 权威状态、Redis
|
适配、健康与生命周期、Sequential 等路由策略、PostgreSQL 管理面持久化、
|
||||||
可重建协调和集群快照。SOCKS5、跨地域主动主动和高级成本优化保留扩展边界。
|
Redis TTL 活动池和集群快照。SOCKS5、跨地域主动主动和高级成本优化保留扩展
|
||||||
|
边界。
|
||||||
|
|||||||
@ -37,8 +37,8 @@ proxy-pool/
|
|||||||
### proxy-controller
|
### proxy-controller
|
||||||
|
|
||||||
Controller 是首版模块化单体。Provider、Pool、Routing 和 Extraction 共享
|
Controller 是首版模块化单体。Provider、Pool、Routing 和 Extraction 共享
|
||||||
事务边界和状态演进,避免过早拆成分布式事务。对外端口定义在领域或控制器
|
运行时策略和状态演进,但不建立 PostgreSQL/Redis 跨存储事务。对外端口定义
|
||||||
模块,具体 PostgreSQL/Redis/HTTP 实现在 `adapters`。
|
在领域或控制器模块,具体 PostgreSQL/Redis/HTTP 实现在 `adapters`。
|
||||||
|
|
||||||
### proxy-checker
|
### proxy-checker
|
||||||
|
|
||||||
@ -71,20 +71,28 @@ platform -------------------------> standard library / observability SDK
|
|||||||
|
|
||||||
## 4. 数据所有权
|
## 4. 数据所有权
|
||||||
|
|
||||||
- PostgreSQL:Proxy 生命周期、Extraction 审计、配置版本和 Outbox 的权威源。
|
- PostgreSQL:配置版本、Upstream/Routing 管理状态、Admin 审计与 Outbox,
|
||||||
- Redis:Leader、分布式速率、Worker 心跳等可丢失且可重建状态。
|
以及可选的无 Proxy 明细聚合指标。
|
||||||
|
- Redis:带 TTL 的短效 Proxy 活动池及其状态、所有权和过期时间;同时保存
|
||||||
|
Provider Leader、分布式速率、Worker 心跳和短期提取幂等结果。
|
||||||
- Worker:仅拥有分配给自己的 Proxy 本地容量计数和不可变快照。
|
- Worker:仅拥有分配给自己的 Proxy 本地容量计数和不可变快照。
|
||||||
- Controller:拥有 Provider 调度、Routing 运行态与 Worker 所有权编排。
|
- Controller:拥有 Provider 调度、Routing 运行态与 Worker 所有权编排。
|
||||||
- Checker:不拥有 Proxy 状态,只拥有执行中的检查任务。
|
- Checker:不拥有 Proxy 状态,只拥有执行中的检查任务。
|
||||||
|
|
||||||
|
Proxy 明细不进入 PostgreSQL。Redis 活动池丢失后由 Provider 重新获取并重建,
|
||||||
|
节点内存中的旧快照随版本或有效期失效,不能把 PostgreSQL 当成恢复来源。
|
||||||
|
|
||||||
## 5. 一致性边界
|
## 5. 一致性边界
|
||||||
|
|
||||||
- Extraction 使用 PostgreSQL 单事务和行锁跳过锁定候选,提交后才响应。
|
- Extraction 在 Redis 内原子完成候选筛选、排他状态迁移和短期幂等结果写入,
|
||||||
|
成功后才响应;该路径不访问 PostgreSQL。
|
||||||
- Worker 所有权采用 `worker + epoch + version + expiry`,同一 Proxy 至多归属
|
- Worker 所有权采用 `worker + epoch + version + expiry`,同一 Proxy 至多归属
|
||||||
一个 Worker。
|
一个 Worker。
|
||||||
- 从 Worker 回收 Proxy 时先 Drain,等待 ACK 且 Active/Reserved 为零,再
|
- 从 Worker 回收 Proxy 时先 Drain,等待 ACK 且 Active/Reserved 为零,再
|
||||||
解除所有权;只有无所有权 Proxy 能被 Distribution 提取。
|
解除所有权;只有无所有权 Proxy 能被 Distribution 提取。
|
||||||
- Snapshot 为整代不可变对象,通过校验和和严格版本序列原子替换。
|
- Snapshot 为整代不可变对象,通过校验和和严格版本序列原子替换。
|
||||||
|
- PostgreSQL 故障会影响管理状态变更和 Admin 审计,但不应阻断 Redis 中可完成
|
||||||
|
的独占提取;Redis 故障则使活动池暂不可用,并触发 Provider 重建。
|
||||||
|
|
||||||
## 6. 扩展规则
|
## 6. 扩展规则
|
||||||
|
|
||||||
|
|||||||
@ -12,7 +12,9 @@ semantics in `对话内容.md`.
|
|||||||
**Architecture:** Separate Gateway, Controller, Checker, and Loadgen commands. Domain
|
**Architecture:** Separate Gateway, Controller, Checker, and Loadgen commands. Domain
|
||||||
packages remain transport-free. Gateway reads immutable local snapshots. Controller
|
packages remain transport-free. Gateway reads immutable local snapshots. Controller
|
||||||
owns provider fetch, pool lifecycle, routing state, extraction, persistence, and worker
|
owns provider fetch, pool lifecycle, routing state, extraction, persistence, and worker
|
||||||
distribution. PostgreSQL is authoritative; Redis stores rebuildable coordination state.
|
distribution. PostgreSQL persists management state and Admin audit/outbox. Redis owns
|
||||||
|
the rebuildable TTL Proxy activity pool and atomic extraction; Gateway hot paths remain
|
||||||
|
memory-only.
|
||||||
|
|
||||||
**Tech Stack:** Go 1.26, `go.yaml.in/yaml/v4`, pgx/v5, go-redis/v9, gRPC/Protobuf,
|
**Tech Stack:** Go 1.26, `go.yaml.in/yaml/v4`, pgx/v5, go-redis/v9, gRPC/Protobuf,
|
||||||
Prometheus, PostgreSQL, Redis, Docker Compose, Kubernetes.
|
Prometheus, PostgreSQL, Redis, Docker Compose, Kubernetes.
|
||||||
@ -124,7 +126,8 @@ test/{fixtures,integration,e2e,load}/
|
|||||||
- [x] Implement POST extraction command with protocol/region/carrier/upstream filters.
|
- [x] Implement POST extraction command with protocol/region/carrier/upstream filters.
|
||||||
- [x] Enforce minRemainingTTL, maxHealthCheckAge, maxCount, client limits, and
|
- [x] Enforce minRemainingTTL, maxHealthCheckAge, maxCount, client limits, and
|
||||||
reserveForGateway.
|
reserveForGateway.
|
||||||
- [x] Atomically transition AVAILABLE to EXTRACTED and append audit records.
|
- [x] Atomically remove selected AVAILABLE entries from the allocatable set and return
|
||||||
|
the result; the current memory Store models the production Redis atomic boundary.
|
||||||
- [x] Implement partial and allOrNothing without Lease, release, or renewal concepts.
|
- [x] Implement partial and allOrNothing without Lease, release, or renewal concepts.
|
||||||
- [x] Run 1,000 concurrent claim attempts and prove every Proxy ID appears at most once.
|
- [x] Run 1,000 concurrent claim attempts and prove every Proxy ID appears at most once.
|
||||||
|
|
||||||
@ -155,11 +158,15 @@ test/{fixtures,integration,e2e,load}/
|
|||||||
**Files:** `internal/controller/distribution/*.go`, `admin/*.go`,
|
**Files:** `internal/controller/distribution/*.go`, `admin/*.go`,
|
||||||
`internal/adapters/postgres/*.go`, `internal/adapters/redis/*.go`, migrations, tests
|
`internal/adapters/postgres/*.go`, `internal/adapters/redis/*.go`, migrations, tests
|
||||||
|
|
||||||
- [ ] Define repository ports for Proxy, RoutingRuntime, Ownership, ExtractionRecord,
|
- [ ] Define PostgreSQL ports for ConfigVersion, Upstream/Routing management state,
|
||||||
Client, ConfigVersion, and Outbox.
|
AdminAudit, Outbox, and optional aggregate metrics; never persist Proxy details or
|
||||||
- [ ] Implement PostgreSQL extraction with one transaction and `FOR UPDATE SKIP LOCKED`.
|
per-extraction records.
|
||||||
- [ ] Implement Redis coordination for Provider leader, distributed rate, Client limit,
|
- [ ] Implement the Redis TTL activity pool and one atomic extraction operation covering
|
||||||
and Worker heartbeat; keep all state rebuildable.
|
candidate eligibility, Gateway reserve, ownership, removal, and short-lived idempotency.
|
||||||
|
- [ ] Implement Redis Provider leader, distributed rate, Client limit, and Worker
|
||||||
|
heartbeat/ownership; rebuild short-lived Proxy inventory from Providers after loss.
|
||||||
|
- [ ] Keep Provider output in Redis TTL activity state and node memory only; keep the
|
||||||
|
Gateway request path on immutable local snapshots with no Redis/PostgreSQL calls.
|
||||||
- [ ] Expose Distribution extraction/status and Admin status/enable/disable/switch/reload.
|
- [ ] Expose Distribution extraction/status and Admin status/enable/disable/switch/reload.
|
||||||
- [ ] Add integration tests using Compose-backed PostgreSQL/Redis.
|
- [ ] Add integration tests using Compose-backed PostgreSQL/Redis.
|
||||||
|
|
||||||
@ -170,8 +177,14 @@ extract/live/ready Handler 与 Admin status/enable/disable/switch/reload Handler
|
|||||||
Bearer/CIDR、可信代理、Client ID、本地准入和 API 401/Gateway 407 差异,并作为
|
Bearer/CIDR、可信代理、Client ID、本地准入和 API 401/Gateway 407 差异,并作为
|
||||||
Admin/Distribution 必需依赖。共享 `platform/httpserver` 与
|
Admin/Distribution 必需依赖。共享 `platform/httpserver` 与
|
||||||
`controller/runtime` 已完成 Distribution/Admin 独立监听器、首错联动关闭和
|
`controller/runtime` 已完成 Distribution/Admin 独立监听器、首错联动关闭和
|
||||||
有界优雅停机;端点正式勾选仍等待 PostgreSQL/Redis Adapter、命令入口与
|
有界优雅停机;端点正式勾选仍等待 Redis 活动池/原子提取 Adapter、PostgreSQL
|
||||||
Compose 集成测试。
|
管理面 Adapter、命令入口与 Compose 集成测试。
|
||||||
|
|
||||||
|
已新增公用 `domain/activitypool` 契约及并发安全内存参考实现,Provider
|
||||||
|
Reconciler 通过 `UpsertFetched` 写入带供应商 TTL 和分配安全余量的批次;已覆盖
|
||||||
|
`usableUntil` 向 Worker Snapshot 的传播与 Gateway 本地截止过滤、
|
||||||
|
重复刷新、过期淘汰、独占提取、短期幂等及 Worker ownership 互斥。生产 Redis
|
||||||
|
Lua/Function Adapter 和多节点集成测试仍待实现。
|
||||||
|
|
||||||
## Task 11: Checker and Health Reducer
|
## Task 11: Checker and Health Reducer
|
||||||
|
|
||||||
|
|||||||
@ -8,8 +8,9 @@
|
|||||||
- [ ] Sequential 并发 Empty 只切换一次,旧 Upstream Proxy 自然耗尽。
|
- [ ] Sequential 并发 Empty 只切换一次,旧 Upstream Proxy 自然耗尽。
|
||||||
- [ ] `pool.maxSize` 与 `fetch.maxTotal` 分别按当前库存和累计获取计数。
|
- [ ] `pool.maxSize` 与 `fetch.maxTotal` 分别按当前库存和累计获取计数。
|
||||||
- [ ] Extract 只有 `AVAILABLE -> EXTRACTED`,OpenAPI 不存在 release/renew。
|
- [ ] Extract 只有 `AVAILABLE -> EXTRACTED`,OpenAPI 不存在 release/renew。
|
||||||
- [ ] Extract 状态更新和审计记录位于同一数据库事务。
|
- [ ] Extract 状态更新和短期幂等结果位于同一个 Redis 原子操作。
|
||||||
- [ ] `partial` 和 `allOrNothing` 均通过并发事务测试。
|
- [ ] PostgreSQL 中不存在 Proxy 明细或逐次提取记录。
|
||||||
|
- [ ] `partial` 和 `allOrNothing` 均通过 Redis 原子并发测试。
|
||||||
- [ ] `reserveForGateway` 在所有提取路径上统一执行。
|
- [ ] `reserveForGateway` 在所有提取路径上统一执行。
|
||||||
|
|
||||||
## 安全
|
## 安全
|
||||||
@ -25,7 +26,8 @@
|
|||||||
|
|
||||||
## 可用性
|
## 可用性
|
||||||
|
|
||||||
- [ ] PostgreSQL 和 Redis 跨可用区,有监控、备份和恢复演练证据。
|
- [ ] PostgreSQL 跨可用区,有监控、备份和恢复演练证据。
|
||||||
|
- [ ] Redis 跨可用区并受监控,完成丢失后由 Provider 重建活动池的演练。
|
||||||
- [ ] Gateway、Controller、Checker 均跨主机/可用区分散。
|
- [ ] Gateway、Controller、Checker 均跨主机/可用区分散。
|
||||||
- [ ] PDB、优雅终止与最大连接时长的组合经过驱逐测试。
|
- [ ] PDB、优雅终止与最大连接时长的组合经过驱逐测试。
|
||||||
- [ ] Controller 断线时 Gateway 在 `maxStaleAge` 内继续,超限拒绝新请求。
|
- [ ] Controller 断线时 Gateway 在 `maxStaleAge` 内继续,超限拒绝新请求。
|
||||||
@ -58,4 +60,3 @@
|
|||||||
- [ ] 100k QPS 验证报告包含环境、命令、版本、场景、原始指标和结论。
|
- [ ] 100k QPS 验证报告包含环境、命令、版本、场景、原始指标和结论。
|
||||||
|
|
||||||
任一关键一致性、安全或恢复项未完成时,不标记生产就绪。
|
任一关键一致性、安全或恢复项未完成时,不标记生产就绪。
|
||||||
|
|
||||||
|
|||||||
@ -3,12 +3,15 @@
|
|||||||
## 1. 运行边界
|
## 1. 运行边界
|
||||||
|
|
||||||
- Gateway 是数据面,正常请求热路径不访问 PostgreSQL、Redis 或 Provider。
|
- Gateway 是数据面,正常请求热路径不访问 PostgreSQL、Redis 或 Provider。
|
||||||
- Controller 是权威控制面,负责 Fetch、生命周期、所有权、Snapshot、Extract
|
- Controller 编排 Fetch、生命周期、所有权、Snapshot 与 Extract;多个副本只有
|
||||||
和审计;多个副本只有一个 Provider 逻辑 Leader。
|
一个 Provider 逻辑 Leader。短效 Proxy 明细只存在于 Redis TTL 活动池和节点
|
||||||
|
内存,可由 Provider 重建。
|
||||||
|
- PostgreSQL 只持久化配置版本、Upstream/Routing 管理状态、Admin 审计与
|
||||||
|
outbox,以及可选聚合指标;不保存 Proxy 明细或逐次提取记录。
|
||||||
- Checker 执行有界健康探测,只上报 Observation,最终状态由 Controller
|
- Checker 执行有界健康探测,只上报 Observation,最终状态由 Controller
|
||||||
reducer 决定。
|
reducer 决定。
|
||||||
- Extract 是一次性独占发放。提交后状态为 `EXTRACTED`,没有 Lease、续租或
|
- Extract 是一次性独占发放。Redis 在单次原子操作中校验并从可分配活动池移除
|
||||||
Release 接口。
|
候选,同时写入带 TTL 的幂等结果;没有 Lease、续租或 Release 接口。
|
||||||
- `reserveForGateway` 是共享池硬约束,Extract 不得把 Gateway 库存清空。
|
- `reserveForGateway` 是共享池硬约束,Extract 不得把 Gateway 库存清空。
|
||||||
- 集群峰值 100,000 QPS 是设计目标,只有完成本文容量验收后才能作为已验证
|
- 集群峰值 100,000 QPS 是设计目标,只有完成本文容量验收后才能作为已验证
|
||||||
能力对外承诺。
|
能力对外承诺。
|
||||||
@ -88,7 +91,8 @@ kubectl -n proxy-pool rollout status deployment/proxy-gateway --timeout=10m
|
|||||||
liveness 失败。
|
liveness 失败。
|
||||||
- `/readyz`:进程可以接收新工作。Gateway 只有在持有未超过 `maxStaleAge`
|
- `/readyz`:进程可以接收新工作。Gateway 只有在持有未超过 `maxStaleAge`
|
||||||
的完整 Snapshot 且仍有准入能力时才 Ready。
|
的完整 Snapshot 且仍有准入能力时才 Ready。
|
||||||
- Controller 只有在配置有效、存储可用、迁移兼容且控制接口已监听时才 Ready。
|
- Controller 按能力判定就绪:Distribution/Fetch 依赖 Redis 活动池,Admin 持久化
|
||||||
|
写依赖 PostgreSQL 与兼容迁移。PostgreSQL 故障不得单独使 Extract 返回 503。
|
||||||
- Checker 在任务消费与结果上报通道可用时 Ready。
|
- Checker 在任务消费与结果上报通道可用时 Ready。
|
||||||
- `/metrics`:独立于业务入口,NetworkPolicy 仅允许监控命名空间访问。
|
- `/metrics`:独立于业务入口,NetworkPolicy 仅允许监控命名空间访问。
|
||||||
|
|
||||||
@ -140,9 +144,9 @@ HPA 使用 CPU/内存作为保护性信号;生产环境建议通过 Prometheus
|
|||||||
3. Available Slots、Reserved、Active、各 Proxy 状态数量。
|
3. Available Slots、Reserved、Active、各 Proxy 状态数量。
|
||||||
4. Provider Success、Empty、Duplicate-only、Error、429 与退避。
|
4. Provider Success、Empty、Duplicate-only、Error、429 与退避。
|
||||||
5. Checker 队列、SUSPECT 数、检查延迟和目标级失败。
|
5. Checker 队列、SUSPECT 数、检查延迟和目标级失败。
|
||||||
6. Extract requested/returned、insufficient、冲突和审计写入。
|
6. Extract requested/returned、insufficient、冲突和幂等命中。
|
||||||
7. PostgreSQL 连接、锁等待、事务失败、WAL 与备份。
|
7. PostgreSQL 连接、管理事务失败、outbox backlog、WAL 与备份。
|
||||||
8. Redis 延迟、内存、主从状态和 Leader 租约抖动。
|
8. Redis 延迟、内存、TTL 淘汰、原子操作错误、主从状态和 Leader 租约抖动。
|
||||||
|
|
||||||
Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、request ID。
|
Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、request ID。
|
||||||
需要逐请求调查时使用受控、脱敏且采样的结构化日志。
|
需要逐请求调查时使用受控、脱敏且采样的结构化日志。
|
||||||
@ -160,8 +164,8 @@ Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、requ
|
|||||||
|
|
||||||
1. 停止接收新的 Extract/Admin 写请求。
|
1. 停止接收新的 Extract/Admin 写请求。
|
||||||
2. 停止发起 Fetch,释放 Provider Leader 租约。
|
2. 停止发起 Fetch,释放 Provider Leader 租约。
|
||||||
3. 完成已进入数据库事务的 Extract 或回滚。
|
3. 等待已提交 Redis 原子操作返回;未确认请求的客户端必须使用相同幂等键重试。
|
||||||
4. 刷新 outbox、审计和 Worker ACK,再关闭连接池。
|
4. 刷新 Admin outbox、审计和 Worker ACK,再关闭 PostgreSQL/Redis 连接池。
|
||||||
|
|
||||||
### Checker
|
### Checker
|
||||||
|
|
||||||
@ -184,16 +188,23 @@ Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、requ
|
|||||||
### 7.2 PostgreSQL 不可用
|
### 7.2 PostgreSQL 不可用
|
||||||
|
|
||||||
1. Gateway 继续使用最后有效 Snapshot。
|
1. Gateway 继续使用最后有效 Snapshot。
|
||||||
2. Controller 将 Distribution 和权威写操作置为不可用,避免返回未提交代理。
|
2. Redis 健康且运行配置有效时,Distribution 继续执行原子 Extract,Provider
|
||||||
3. Provider Fetch 停止写入;已有流量不受影响。
|
继续刷新 TTL 活动池。
|
||||||
4. 恢复后核对迁移、事务回滚、outbox backlog 与 Extract 审计连续性。
|
3. 拒绝配置版本、Upstream/Routing 管理状态和其他需要 Admin 审计/outbox 的写入;
|
||||||
|
不得把 Proxy 明细临时落入 PostgreSQL。
|
||||||
|
4. 恢复后核对迁移、管理事务回滚、Admin 审计与 outbox backlog;不存在 Proxy
|
||||||
|
明细或逐次提取记录恢复步骤。
|
||||||
|
|
||||||
### 7.3 Redis 不可用
|
### 7.3 Redis 不可用
|
||||||
|
|
||||||
1. Gateway 不受影响。
|
1. Gateway 继续读取节点内最后有效 Snapshot,直到 `maxStaleAge`;请求热路径不
|
||||||
2. Controller 停止需要分布式互斥的高风险工作,防止多个 Fetch Leader。
|
回查 PostgreSQL 或 Provider。
|
||||||
3. 本地限流只作为临时降级,不能声称满足全局额度。
|
2. Distribution 立即失败关闭并返回 503,禁止本地内存提取或 PostgreSQL 兜底。
|
||||||
4. 恢复后确认 Leader 唯一、租约 epoch 单调和重复 Fetch 去重。
|
3. Controller 停止活动池写入和需要分布式互斥的工作,防止多个 Fetch Leader;
|
||||||
|
本地限流不能声称满足全局额度。
|
||||||
|
4. 恢复后确认 Leader 唯一和租约 epoch 单调,由 Provider 重新 Fetch 并构建 TTL
|
||||||
|
活动池,再恢复 Distribution。Redis 整体丢失会终止原活动池代次的排他状态和
|
||||||
|
短期幂等窗口;高可用、持久化、监控和告警必须明确并降低该风险。
|
||||||
|
|
||||||
### 7.4 Provider 故障
|
### 7.4 Provider 故障
|
||||||
|
|
||||||
@ -213,11 +224,11 @@ Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、requ
|
|||||||
|
|
||||||
### 7.6 Extract 库存不足
|
### 7.6 Extract 库存不足
|
||||||
|
|
||||||
1. `partial` 返回实际数量;`allOrNothing` 不足时整批回滚。
|
1. `partial` 返回实际数量;`allOrNothing` 不足时不改变 Redis 活动池。
|
||||||
2. 检查 TTL、health age、filter、Worker ownership 和 `reserveForGateway`。
|
2. 检查 TTL、health age、filter、Worker ownership 和 `reserveForGateway`。
|
||||||
3. 不得降低 `reserveForGateway` 到导致 Gateway 容量告警的水平。
|
3. 不得降低 `reserveForGateway` 到导致 Gateway 容量告警的水平。
|
||||||
4. 回收 Worker-owned Proxy 必须先 DRAINING、等待 active/reserved 为零、清除
|
4. 回收 Worker-owned Proxy 必须先 DRAINING、等待 active/reserved 为零、清除
|
||||||
ownership,再执行 `AVAILABLE -> EXTRACTED` 事务。
|
ownership,再由 Redis 原子操作将其从可分配活动池移除并写入短期幂等结果。
|
||||||
5. 已提取代理没有 Release;客户端归还请求只记录为无效调用,不恢复库存。
|
5. 已提取代理没有 Release;客户端归还请求只记录为无效调用,不恢复库存。
|
||||||
|
|
||||||
### 7.7 Checker 积压
|
### 7.7 Checker 积压
|
||||||
@ -229,14 +240,17 @@ Prometheus 标签禁止包含 Proxy IP、Client ID、Session、完整 URL、requ
|
|||||||
|
|
||||||
## 8. 备份与恢复
|
## 8. 备份与恢复
|
||||||
|
|
||||||
- PostgreSQL:每日全量、连续 WAL/PITR,至少每季度做恢复演练。
|
- PostgreSQL:每日全量、连续 WAL/PITR,保护配置版本、Upstream/Routing 管理
|
||||||
- Redis:仅保存可重建协调状态;不得把 Redis 备份当权威业务备份。
|
状态、Admin 审计与 outbox;至少每季度做恢复演练。
|
||||||
|
- Redis:保存可由 Provider 重建的 TTL 活动池、所有权/Leader 协调和短期幂等
|
||||||
|
结果;使用高可用与持久化降低窗口丢失风险,但不把它当长期业务档案。
|
||||||
- 配置:版本化保存校验通过的不可变 Revision 与校验和。
|
- 配置:版本化保存校验通过的不可变 Revision 与校验和。
|
||||||
- Secret:由密钥平台版本化,日志和备份中不得出现明文。
|
- Secret:由密钥平台版本化,日志和备份中不得出现明文。
|
||||||
|
|
||||||
恢复顺序:PostgreSQL -> Redis -> Controller -> Checker -> Gateway。恢复后验证
|
管理面恢复顺序:PostgreSQL -> Controller/Admin;代理运行面恢复顺序:Redis ->
|
||||||
Proxy 状态、Extraction Record、Worker ownership epoch、outbox 和配置 Revision
|
Provider 重建活动池 -> Controller/Checker -> 发布新 Snapshot。验证 Worker
|
||||||
单调一致,再开放 Gateway 与 Distribution。
|
ownership epoch、活动池 TTL、短期幂等窗口、outbox 和配置 Revision 一致后,再
|
||||||
|
开放相应能力;Gateway 在旧 Snapshot 未超过 `maxStaleAge` 时无需等待数据库恢复。
|
||||||
|
|
||||||
## 9. Secret 轮换
|
## 9. Secret 轮换
|
||||||
|
|
||||||
|
|||||||
@ -11,15 +11,14 @@
|
|||||||
- 产品设计、总体架构、项目结构、四项 ADR。
|
- 产品设计、总体架构、项目结构、四项 ADR。
|
||||||
- 开发、配置、Distribution/Admin API、控制面协议、安全、测试、运维文档。
|
- 开发、配置、Distribution/Admin API、控制面协议、安全、测试、运维文档。
|
||||||
- 20 个配置场景和 35 张 Mermaid 架构/流程/状态/故障图。
|
- 20 个配置场景和 35 张 Mermaid 架构/流程/状态/故障图。
|
||||||
- 版本化文档包 `proxy-pool-docs-v1.0.zip`,包含 50 个条目。
|
|
||||||
|
|
||||||
### 机器契约
|
### 机器契约
|
||||||
|
|
||||||
- Distribution OpenAPI:一次性独占提取、partial/allOrNothing、幂等键、
|
- Distribution OpenAPI:一次性独占提取、partial/allOrNothing、幂等键、
|
||||||
TTL/健康过滤结果与标准错误。
|
Redis TTL 活动池原子语义、TTL/健康过滤结果与标准错误。
|
||||||
- Admin OpenAPI:状态、Upstream 启停、Routing 切换和配置重载。
|
- Admin OpenAPI:状态、Upstream 启停、Routing 切换和配置重载。
|
||||||
- Protobuf:Worker 注册、全量/增量 Snapshot、ACK、运行态/结果上报、Checker
|
- Protobuf:Worker 注册、全量/增量 Snapshot、`usable_until`、ACK、运行态/
|
||||||
任务与 Observation。
|
结果上报、Checker 任务与 Observation。
|
||||||
|
|
||||||
### 核心参考实现
|
### 核心参考实现
|
||||||
|
|
||||||
@ -29,8 +28,8 @@
|
|||||||
原子计数;1,000 goroutine 不超卖测试。
|
原子计数;1,000 goroutine 不超卖测试。
|
||||||
- `ROUTE-001 / ROUTE-004`:首条命中规则与 Concurrent Sequential 单次切换。
|
- `ROUTE-001 / ROUTE-004`:首条命中规则与 Concurrent Sequential 单次切换。
|
||||||
- `FETCH-005 / FETCH-006`:Valid、Empty、DuplicateOnly、Error 分类。
|
- `FETCH-005 / FETCH-006`:Valid、Empty、DuplicateOnly、Error 分类。
|
||||||
- `DIST-001..003 / DIST-006..007`:内存事务模型验证独占提取、满足模式、TTL、
|
- `DIST-001..003 / DIST-006..007`:内存活动池参考实现验证独占提取、满足模式、
|
||||||
健康时效与 Gateway 保留量;1,000 并发不重复。
|
TTL、健康时效与 Gateway 保留量;1,000 并发不重复。
|
||||||
- `OPS-001`:完整 Snapshot 目标、epoch/version、校验和验证及原子替换。
|
- `OPS-001`:完整 Snapshot 目标、epoch/version、校验和验证及原子替换。
|
||||||
- `CAP-001 / GW 热路径边界`:本地 Dispatch 条件过滤与原子容量预留。
|
- `CAP-001 / GW 热路径边界`:本地 Dispatch 条件过滤与原子容量预留。
|
||||||
- `GW-*`:HTTP 正向代理、HTTPS CONNECT、双向 Tunnel、超时、重试、保护链与
|
- `GW-*`:HTTP 正向代理、HTTPS CONNECT、双向 Tunnel、超时、重试、保护链与
|
||||||
@ -65,8 +64,12 @@ CI 已配置 Linux race job。Docker/Kubernetes 仅完成静态验证,没有
|
|||||||
1. `cmd/proxy-gateway/controller/checker/loadgen` 进程装配。
|
1. `cmd/proxy-gateway/controller/checker/loadgen` 进程装配。
|
||||||
2. Gateway 进程装配、生产连接池调优与代表性流量压测。
|
2. Gateway 进程装配、生产连接池调优与代表性流量压测。
|
||||||
3. Provider 分布式 singleflight/Leader、长期凭据回收和累计额度执行器。
|
3. Provider 分布式 singleflight/Leader、长期凭据回收和累计额度执行器。
|
||||||
4. PostgreSQL repository、Extraction 行锁事务、Outbox 和迁移。
|
4. PostgreSQL 配置版本、Upstream/Routing 管理状态、Admin 审计/Outbox
|
||||||
5. Redis Leader、速率限制、心跳与可重建协调适配器。
|
repository 和迁移;可选聚合指标不包含 Proxy 明细。
|
||||||
|
5. Redis TTL 活动池、原子独占提取、短期幂等结果、Leader、速率限制、心跳与
|
||||||
|
所有权适配器;活动池可由 Provider 重建。接线时必须验证进入
|
||||||
|
`allocationSafetyMargin` 的不可分配条目不会长期占用补池额度,同时不得
|
||||||
|
突破 `pool.maxSize` 硬上限。
|
||||||
6. Worker ownership drain/ACK/过期回收和网络快照流。
|
6. Worker ownership drain/ACK/过期回收和网络快照流。
|
||||||
7. Checker 调度、探测器和健康 reducer。
|
7. Checker 调度、探测器和健康 reducer。
|
||||||
8. Admin/Distribution 细粒度授权、分布式限流和审计查询。
|
8. Admin/Distribution 细粒度授权、分布式限流和审计查询。
|
||||||
|
|||||||
@ -5,7 +5,8 @@
|
|||||||
- Gateway、Distribution、Admin 和 Metrics 为四个独立监听边界。
|
- Gateway、Distribution、Admin 和 Metrics 为四个独立监听边界。
|
||||||
- Provider API 和上游 Proxy 属于外部不可信网络。
|
- Provider API 和上游 Proxy 属于外部不可信网络。
|
||||||
- Worker 与 Controller 通道必须进行双向身份校验并绑定 cluster/worker。
|
- Worker 与 Controller 通道必须进行双向身份校验并绑定 cluster/worker。
|
||||||
- PostgreSQL 保存权威状态;Redis 数据默认按可重建缓存与协调信息处理。
|
- PostgreSQL 保存管理面持久状态;Redis 保存可重建的短效 Proxy 活动池与协调
|
||||||
|
信息,Proxy 地址和凭据不写入 PostgreSQL。
|
||||||
|
|
||||||
## 2. 入口控制
|
## 2. 入口控制
|
||||||
|
|
||||||
|
|||||||
@ -4,7 +4,9 @@
|
|||||||
|
|
||||||
- 先在隔离环境建立 15 分钟稳定基线,再注入单一故障。
|
- 先在隔离环境建立 15 分钟稳定基线,再注入单一故障。
|
||||||
- 每次只改变一个变量,记录开始/恢复时间与所有指标。
|
- 每次只改变一个变量,记录开始/恢复时间与所有指标。
|
||||||
- 任何数据库写故障后都核对 Proxy 状态、审计和 outbox,而非只看 HTTP 状态码。
|
- PostgreSQL 写故障后核对配置版本、Upstream/Routing 管理状态、Admin 审计和
|
||||||
|
outbox;Redis 写故障后核对活动池、独占提取和短期幂等键,不能只看 HTTP
|
||||||
|
状态码。
|
||||||
- 故障恢复后至少观察两个健康检查周期和一个 Snapshot 完整发布周期。
|
- 故障恢复后至少观察两个健康检查周期和一个 Snapshot 完整发布周期。
|
||||||
|
|
||||||
## 场景
|
## 场景
|
||||||
@ -22,15 +24,18 @@ duplicate-only 不触发 Sequential 切换。
|
|||||||
|
|
||||||
- 断开 60 秒、连接池耗尽、锁等待、事务提交失败、只读切换。
|
- 断开 60 秒、连接池耗尽、锁等待、事务提交失败、只读切换。
|
||||||
|
|
||||||
预期:Gateway 继续使用最后 Snapshot;Extract 不返回未提交记录;恢复后 outbox
|
预期:Gateway 继续使用最后 Snapshot;Redis 健康时 Fetch、活动池更新和 Extract
|
||||||
补发且不重复应用。
|
不被 PostgreSQL 故障阻断;依赖持久化的配置/Admin 写操作失败关闭。恢复后
|
||||||
|
outbox 补发且不重复应用,PostgreSQL 中没有 Proxy 明细或逐次提取记录需要恢复。
|
||||||
|
|
||||||
### Redis
|
### Redis
|
||||||
|
|
||||||
- 断开、延迟 2 秒、Leader key 丢失、主从切换。
|
- 断开、延迟 2 秒、Leader key 丢失、活动池与幂等键丢失、主从切换。
|
||||||
|
|
||||||
预期:Gateway 无影响;Provider Fetch 不出现多个有效 Leader;全局限流明确降级;
|
预期:Gateway 在 Snapshot 新鲜度窗口内无影响;Distribution 失败关闭并返回 503,
|
||||||
恢复后 epoch 单调。
|
不得改写 PostgreSQL 兜底;Provider Fetch 不出现多个有效 Leader;全局限流明确
|
||||||
|
降级。恢复后 epoch 单调,由 Provider 重新拉取并构建 TTL 活动池;已经丢失的短期
|
||||||
|
幂等窗口和旧活动池排他状态不伪装成仍然有效,重建结果视为新的活动池代次。
|
||||||
|
|
||||||
### Controller
|
### Controller
|
||||||
|
|
||||||
@ -55,10 +60,12 @@ Fetch;Delta 缺口触发完整 Snapshot。
|
|||||||
|
|
||||||
### Distribution
|
### Distribution
|
||||||
|
|
||||||
- 100 个并发请求争用相同 Proxy;数据库在事务提交时断开;Gateway 同时满载。
|
- 100 个并发请求争用相同 Proxy;Redis 在原子操作执行或主从切换时断开;Gateway
|
||||||
|
同时满载。
|
||||||
|
|
||||||
预期:每个 Proxy 最多返回一次;提交失败不返回代理;allOrNothing 整批回滚;
|
预期:同一 Redis 活动池代次内每个 Proxy 最多返回一次;该代次仍保留时,未确认
|
||||||
池中始终剩余 `reserveForGateway`;没有 Release 恢复路径。
|
成功的客户端使用同一幂等键重试并得到已缓存结果或明确失败,不出现部分可见状态;
|
||||||
|
allOrNothing 整批不变;池中始终剩余 `reserveForGateway`;没有 Release 恢复路径。
|
||||||
|
|
||||||
### 配置与 Secret
|
### 配置与 Secret
|
||||||
|
|
||||||
@ -66,4 +73,3 @@ Fetch;Delta 缺口触发完整 Snapshot。
|
|||||||
|
|
||||||
预期:新 Revision 整体拒绝,旧不可变 Snapshot 继续;认证错误计 Error 而不是
|
预期:新 Revision 整体拒绝,旧不可变 Snapshot 继续;认证错误计 Error 而不是
|
||||||
Empty;日志不出现 Secret。
|
Empty;日志不出现 Secret。
|
||||||
|
|
||||||
|
|||||||
@ -5,6 +5,8 @@
|
|||||||
- 先证明领域不变量,再证明 Adapter 契约,最后证明跨进程行为。
|
- 先证明领域不变量,再证明 Adapter 契约,最后证明跨进程行为。
|
||||||
- 并发测试必须在 race detector 下运行,不能只依赖单线程示例。
|
- 并发测试必须在 race detector 下运行,不能只依赖单线程示例。
|
||||||
- 时间、随机、网络、Provider 和存储均通过可替换接口或 fixture 控制。
|
- 时间、随机、网络、Provider 和存储均通过可替换接口或 fixture 控制。
|
||||||
|
- 生产 Adapter 必须证明数据边界:Proxy 明细只进入 Redis TTL 活动池和节点
|
||||||
|
内存,PostgreSQL 不得出现 Proxy 明细或逐次提取记录。
|
||||||
- 100,000 QPS 是待验证的集群目标,不以架构图、副本数或短时峰值替代证据。
|
- 100,000 QPS 是待验证的集群目标,不以架构图、副本数或短时峰值替代证据。
|
||||||
- 性能通过与正确性通过相互独立;高 QPS 下出现超卖、重复 Extract 或状态
|
- 性能通过与正确性通过相互独立;高 QPS 下出现超卖、重复 Extract 或状态
|
||||||
回退时,结果一律失败。
|
回退时,结果一律失败。
|
||||||
@ -24,9 +26,11 @@
|
|||||||
### 契约测试
|
### 契约测试
|
||||||
|
|
||||||
- Provider 响应模板的大小、超时、函数白名单和解析边界。
|
- Provider 响应模板的大小、超时、函数白名单和解析边界。
|
||||||
- PostgreSQL `FOR UPDATE SKIP LOCKED` 并发批量提取。
|
- Redis Lua/Function 对候选筛选、库存预留、独占移除和短期幂等结果执行单次
|
||||||
- Outbox 状态更新、发布与幂等重放。
|
原子操作;并发与主从切换下不得部分提交。
|
||||||
- Redis Leader 租约、限流和失联恢复。
|
- PostgreSQL 配置版本、Upstream/Routing 管理状态、Admin 审计与 outbox 的事务
|
||||||
|
更新及幂等重放;测试库断言不包含 Proxy 明细或逐次提取记录。
|
||||||
|
- Redis TTL 活动池、Leader 租约、限流、短期幂等窗口和失联恢复。
|
||||||
- Snapshot/Delta/ACK/Report 的版本与校验和兼容性。
|
- Snapshot/Delta/ACK/Report 的版本与校验和兼容性。
|
||||||
- OpenAPI 错误模型、认证矩阵、批量 fulfillment。
|
- OpenAPI 错误模型、认证矩阵、批量 fulfillment。
|
||||||
|
|
||||||
@ -34,7 +38,8 @@
|
|||||||
|
|
||||||
- HTTP 正向代理成功、上游连接前失败和安全重试。
|
- HTTP 正向代理成功、上游连接前失败和安全重试。
|
||||||
- HTTPS CONNECT 建立后不透明重放。
|
- HTTPS CONNECT 建立后不透明重放。
|
||||||
- Controller Fetch -> Check -> AVAILABLE -> Worker Snapshot -> Gateway 转发。
|
- Controller Fetch -> Redis TTL 活动池 -> Check -> AVAILABLE -> Worker Snapshot ->
|
||||||
|
Gateway 转发;Gateway 请求线程始终只读节点内存。
|
||||||
- Distribution 原子提取后 Gateway 不再分配同一 Proxy。
|
- Distribution 原子提取后 Gateway 不再分配同一 Proxy。
|
||||||
- 配置热更新失败保留旧 Revision,成功后新请求使用新 Snapshot。
|
- 配置热更新失败保留旧 Revision,成功后新请求使用新 Snapshot。
|
||||||
|
|
||||||
@ -51,9 +56,10 @@
|
|||||||
重置 Empty;Success 重置 Empty。
|
重置 Empty;Success 重置 Empty。
|
||||||
6. **Sequential 竞态**:达到阈值时多协程只能将 A 切到 B 一次,不能越过 B。
|
6. **Sequential 竞态**:达到阈值时多协程只能将 A 切到 B 一次,不能越过 B。
|
||||||
7. **Drain**:切换或禁用 Upstream 后停止新分配,已有连接完成后才回收。
|
7. **Drain**:切换或禁用 Upstream 后停止新分配,已有连接完成后才回收。
|
||||||
8. **Extract 竞态**:多个请求并发提取同一候选集合,每个 Proxy 最多返回一次。
|
8. **Extract 竞态**:多个请求并发提取同一候选集合,同一 Redis 活动池代次内
|
||||||
9. **Extract 批量语义**:partial 提交实际数量;allOrNothing 不足时状态和审计
|
每个 Proxy 最多返回一次。
|
||||||
全回滚;始终保留 `reserveForGateway`。
|
9. **Extract 批量语义**:partial 提交实际数量;allOrNothing 不足时 Redis
|
||||||
|
活动池和幂等结果均不变;始终保留 `reserveForGateway`。
|
||||||
10. **Worker ownership**:回收过程严格经过 DRAINING、ACK、active/reserved=0、
|
10. **Worker ownership**:回收过程严格经过 DRAINING、ACK、active/reserved=0、
|
||||||
unowned,旧 Snapshot 不可再分配。
|
unowned,旧 Snapshot 不可再分配。
|
||||||
11. **控制面故障**:Redis、PostgreSQL、Controller、Checker 与 Provider 分别
|
11. **控制面故障**:Redis、PostgreSQL、Controller、Checker 与 Provider 分别
|
||||||
@ -70,8 +76,9 @@ go vet ./...
|
|||||||
go build ./...
|
go build ./...
|
||||||
```
|
```
|
||||||
|
|
||||||
需要 PostgreSQL/Redis 的测试使用独立数据库和短生命周期容器,不复用开发数据。
|
需要 PostgreSQL/Redis 的测试使用独立实例和短生命周期容器,不复用开发数据。
|
||||||
测试结束后验证没有残留 Worker ownership、Leader 租约或未提交 Extraction Record。
|
测试结束后验证没有残留 Worker ownership、Leader 租约、活动池条目或幂等键,
|
||||||
|
并检查 PostgreSQL 中不存在 Proxy 明细和逐次提取记录。
|
||||||
|
|
||||||
## 5. 负载模型
|
## 5. 负载模型
|
||||||
|
|
||||||
@ -109,7 +116,7 @@ go build ./...
|
|||||||
|
|
||||||
业务 SLO 由产品最终确认,但至少满足以下工程门槛:
|
业务 SLO 由产品最终确认,但至少满足以下工程门槛:
|
||||||
|
|
||||||
- 无容量超卖、负计数、重复 Extract、审计缺失或状态非法回退。
|
- 无容量超卖、负计数、幂等窗口内重复 Extract、Admin 审计缺失或状态非法回退。
|
||||||
- 100k 峰值期间无进程 OOM、FD 耗尽、无界队列或全局锁热点。
|
- 100k 峰值期间无进程 OOM、FD 耗尽、无界队列或全局锁热点。
|
||||||
- Gateway 热路径在 PostgreSQL、Redis、Provider 失效时不发起同步访问。
|
- Gateway 热路径在 PostgreSQL、Redis、Provider 失效时不发起同步访问。
|
||||||
- p99 Dispatch 小于 100 微秒的设计预算需要在 100k Proxy Snapshot 下单独证明。
|
- p99 Dispatch 小于 100 微秒的设计预算需要在 100k Proxy Snapshot 下单独证明。
|
||||||
@ -125,6 +132,6 @@ go build ./...
|
|||||||
配置:Snapshot 规模、Proxy 容量、路由、重试、日志级别
|
配置:Snapshot 规模、Proxy 容量、路由、重试、日志级别
|
||||||
场景:协议、连接复用、响应体、持续时间、升压曲线、故障注入
|
场景:协议、连接复用、响应体、持续时间、升压曲线、故障注入
|
||||||
结果:QPS、建连速率、active、p50/p95/p99、错误、CPU、RSS、GC、FD、网络
|
结果:QPS、建连速率、active、p50/p95/p99、错误、CPU、RSS、GC、FD、网络
|
||||||
不变量:capacity、ownership、extraction、audit、reserve 检查结果
|
不变量:capacity、ownership、extraction、idempotency、admin audit、reserve 检查结果
|
||||||
结论:通过/失败,以及适用边界
|
结论:通过/失败,以及适用边界
|
||||||
```
|
```
|
||||||
|
|||||||
@ -19,6 +19,7 @@ upstreams:
|
|||||||
proxyAuth: {type: response}
|
proxyAuth: {type: response}
|
||||||
pool: {maxSize: 1000}
|
pool: {maxSize: 1000}
|
||||||
capacity: {maxConcurrencyPerProxy: 10}
|
capacity: {maxConcurrencyPerProxy: 10}
|
||||||
|
lifecycle: {ttl: 2m, allocationSafetyMargin: 15s}
|
||||||
fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1}
|
fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1}
|
||||||
provider-b:
|
provider-b:
|
||||||
enabled: true
|
enabled: true
|
||||||
@ -28,4 +29,5 @@ upstreams:
|
|||||||
proxyAuth: {type: response}
|
proxyAuth: {type: response}
|
||||||
pool: {maxSize: 1000}
|
pool: {maxSize: 1000}
|
||||||
capacity: {maxConcurrencyPerProxy: 10}
|
capacity: {maxConcurrencyPerProxy: 10}
|
||||||
|
lifecycle: {ttl: 2m, allocationSafetyMargin: 15s}
|
||||||
fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1}
|
fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1}
|
||||||
|
|||||||
@ -19,6 +19,7 @@ upstreams:
|
|||||||
proxyAuth: {type: response}
|
proxyAuth: {type: response}
|
||||||
pool: {maxSize: 1000}
|
pool: {maxSize: 1000}
|
||||||
capacity: {maxConcurrencyPerProxy: 10}
|
capacity: {maxConcurrencyPerProxy: 10}
|
||||||
|
lifecycle: {ttl: 2m, allocationSafetyMargin: 15s}
|
||||||
fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1}
|
fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1}
|
||||||
provider-b:
|
provider-b:
|
||||||
enabled: true
|
enabled: true
|
||||||
@ -28,4 +29,5 @@ upstreams:
|
|||||||
proxyAuth: {type: response}
|
proxyAuth: {type: response}
|
||||||
pool: {maxSize: 1000}
|
pool: {maxSize: 1000}
|
||||||
capacity: {maxConcurrencyPerProxy: 10}
|
capacity: {maxConcurrencyPerProxy: 10}
|
||||||
|
lifecycle: {ttl: 2m, allocationSafetyMargin: 15s}
|
||||||
fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1}
|
fetch: {requestInterval: 1s, timeout: 3s, maxAttempts: 3, maxInFlight: 1}
|
||||||
|
|||||||
@ -6,7 +6,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
projectconfig "github.com/proxy-pool/proxy-pool/internal/config"
|
projectconfig "proxy-pool/internal/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestAllExamplesLoadStrictly(t *testing.T) {
|
func TestAllExamplesLoadStrictly(t *testing.T) {
|
||||||
|
|||||||
@ -40,8 +40,8 @@ Extract 请求使用哪些 Upstream。
|
|||||||
- 代理分配必须原子预留容量,防止并发超卖。
|
- 代理分配必须原子预留容量,防止并发超卖。
|
||||||
- 代理唯一键为 `scheme + host + port + username + credentialVersion`;日志
|
- 代理唯一键为 `scheme + host + port + username + credentialVersion`;日志
|
||||||
和指标不得暴露密码。
|
和指标不得暴露密码。
|
||||||
- TTL 优先级为响应 `expiresAt`、响应 `ttl`、配置固定 TTL、不过期;内部
|
- TTL 优先级为响应 `expiresAt`、响应 `ttl`、配置固定 TTL;活动池中的每个
|
||||||
时间统一 UTC。
|
Proxy 必须有明确过期时间,内部时间统一 UTC。
|
||||||
- 健康检查至少区分全局健康和 Routing/目标健康,并使用抖动和并发上限。
|
- 健康检查至少区分全局健康和 Routing/目标健康,并使用抖动和并发上限。
|
||||||
- GET/HEAD 可按配置安全重试;非幂等方法默认不自动重试;CONNECT 建立后
|
- GET/HEAD 可按配置安全重试;非幂等方法默认不自动重试;CONNECT 建立后
|
||||||
不透明重放。
|
不透明重放。
|
||||||
@ -55,8 +55,8 @@ Extract 请求使用哪些 Upstream。
|
|||||||
- 数据面采用多 Worker,本地不可变代理快照和本地容量计数。
|
- 数据面采用多 Worker,本地不可变代理快照和本地容量计数。
|
||||||
- 同一代理必须由单个 Worker 所有,或由控制面下发容量切片;禁止每请求
|
- 同一代理必须由单个 Worker 所有,或由控制面下发容量切片;禁止每请求
|
||||||
访问 Redis 做全局并发计数。
|
访问 Redis 做全局并发计数。
|
||||||
- 控制面集中 Provider 获取、独立限流、singleflight、Leader 选举、状态
|
- 控制面集中 Provider 获取、独立限流、singleflight、Leader 选举和快照分发。
|
||||||
持久化和快照分发。
|
Proxy 明细只进入 Redis TTL 活动池,不写 PostgreSQL。
|
||||||
- 副本数必须由单 Worker 实测能力、目标利用率和故障域余量计算。
|
- 副本数必须由单 Worker 实测能力、目标利用率和故障域余量计算。
|
||||||
|
|
||||||
## 配置模型
|
## 配置模型
|
||||||
@ -82,4 +82,3 @@ Routing 自上而下匹配,首条命中停止;支持 Gateway 与 Extract 两
|
|||||||
- Provider 超时不得计入 Empty Fetch。
|
- Provider 超时不得计入 Empty Fetch。
|
||||||
- 重复代理不得重复入池,也不得触发空结果切换。
|
- 重复代理不得重复入池,也不得触发空结果切换。
|
||||||
- 所有 Upstream 不可用时按显式策略执行。
|
- 所有 Upstream 不可用时按显式策略执行。
|
||||||
|
|
||||||
|
|||||||
2
go.mod
2
go.mod
@ -1,4 +1,4 @@
|
|||||||
module github.com/proxy-pool/proxy-pool
|
module proxy-pool
|
||||||
|
|
||||||
go 1.26.0
|
go 1.26.0
|
||||||
|
|
||||||
|
|||||||
@ -5,10 +5,10 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/adapters/providerapi"
|
"proxy-pool/internal/adapters/providerapi"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/config"
|
"proxy-pool/internal/config"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/transport"
|
"proxy-pool/internal/gateway/transport"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/credentials"
|
"proxy-pool/internal/platform/credentials"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestParsedCredentialsResolveInGatewayTransport(t *testing.T) {
|
func TestParsedCredentialsResolveInGatewayTransport(t *testing.T) {
|
||||||
|
|||||||
@ -12,8 +12,8 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/config"
|
"proxy-pool/internal/config"
|
||||||
controllerProvider "github.com/proxy-pool/proxy-pool/internal/controller/provider"
|
controllerProvider "proxy-pool/internal/controller/provider"
|
||||||
)
|
)
|
||||||
|
|
||||||
const defaultRetryAfterMax = 30 * time.Second
|
const defaultRetryAfterMax = 30 * time.Second
|
||||||
|
|||||||
@ -12,8 +12,8 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/config"
|
"proxy-pool/internal/config"
|
||||||
controllerProvider "github.com/proxy-pool/proxy-pool/internal/controller/provider"
|
controllerProvider "proxy-pool/internal/controller/provider"
|
||||||
)
|
)
|
||||||
|
|
||||||
var _ controllerProvider.ProviderAdapter = (*HTTPAdapter)(nil)
|
var _ controllerProvider.ProviderAdapter = (*HTTPAdapter)(nil)
|
||||||
|
|||||||
@ -4,7 +4,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/config"
|
"proxy-pool/internal/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestTemplateParserRejectsNestedRanges(t *testing.T) {
|
func TestTemplateParserRejectsNestedRanges(t *testing.T) {
|
||||||
|
|||||||
@ -11,9 +11,9 @@ import (
|
|||||||
"text/template"
|
"text/template"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/config"
|
"proxy-pool/internal/config"
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/credentials"
|
"proxy-pool/internal/platform/credentials"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
|
|||||||
@ -10,10 +10,10 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/config"
|
"proxy-pool/internal/config"
|
||||||
controllerProvider "github.com/proxy-pool/proxy-pool/internal/controller/provider"
|
controllerProvider "proxy-pool/internal/controller/provider"
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/credentials"
|
"proxy-pool/internal/platform/credentials"
|
||||||
)
|
)
|
||||||
|
|
||||||
var _ controllerProvider.Parser = (*TemplateParser)(nil)
|
var _ controllerProvider.Parser = (*TemplateParser)(nil)
|
||||||
|
|||||||
@ -113,6 +113,7 @@ type Extraction struct {
|
|||||||
MinRemainingTTL Duration `yaml:"minRemainingTTL"`
|
MinRemainingTTL Duration `yaml:"minRemainingTTL"`
|
||||||
MaxHealthCheckAge Duration `yaml:"maxHealthCheckAge"`
|
MaxHealthCheckAge Duration `yaml:"maxHealthCheckAge"`
|
||||||
ReserveForGateway int `yaml:"reserveForGateway"`
|
ReserveForGateway int `yaml:"reserveForGateway"`
|
||||||
|
IdempotencyTTL Duration `yaml:"idempotencyTTL"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type Metrics struct {
|
type Metrics struct {
|
||||||
|
|||||||
@ -6,6 +6,7 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
"go.yaml.in/yaml/v4"
|
"go.yaml.in/yaml/v4"
|
||||||
)
|
)
|
||||||
@ -352,6 +353,22 @@ func TestValidateRejectsInvalidConfigurationMatrix(t *testing.T) {
|
|||||||
},
|
},
|
||||||
want: "reserveForGateway",
|
want: "reserveForGateway",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: "negative extraction idempotency ttl",
|
||||||
|
mutate: func(cfg *Config) {
|
||||||
|
cfg.Distribution.Extraction.IdempotencyTTL = Duration(-time.Second)
|
||||||
|
},
|
||||||
|
want: "idempotencyTTL",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "enabled upstream without ttl",
|
||||||
|
mutate: func(cfg *Config) {
|
||||||
|
upstream := cfg.Upstreams["provider-a"]
|
||||||
|
upstream.Lifecycle.TTL = 0
|
||||||
|
cfg.Upstreams["provider-a"] = upstream
|
||||||
|
},
|
||||||
|
want: "lifecycle.ttl",
|
||||||
|
},
|
||||||
{
|
{
|
||||||
name: "no enabled upstream",
|
name: "no enabled upstream",
|
||||||
mutate: func(cfg *Config) {
|
mutate: func(cfg *Config) {
|
||||||
|
|||||||
@ -81,6 +81,9 @@ func Validate(cfg *Config) error {
|
|||||||
if err := requireNonNegative("distribution.reserveForGateway", cfg.Distribution.Extraction.ReserveForGateway); err != nil {
|
if err := requireNonNegative("distribution.reserveForGateway", cfg.Distribution.Extraction.ReserveForGateway); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
if err := requireNonNegative("distribution.idempotencyTTL", cfg.Distribution.Extraction.IdempotencyTTL); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@ -312,7 +315,7 @@ func validateUpstream(name string, upstream Upstream) error {
|
|||||||
if err := requirePositive(scope+" capacity.maxConcurrencyPerProxy", upstream.Capacity.MaxConcurrencyPerProxy); err != nil {
|
if err := requirePositive(scope+" capacity.maxConcurrencyPerProxy", upstream.Capacity.MaxConcurrencyPerProxy); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err := requireNonNegative(scope+" lifecycle.ttl", upstream.Lifecycle.TTL); err != nil {
|
if err := requirePositive(scope+" lifecycle.ttl", upstream.Lifecycle.TTL); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err := requireNonNegative(scope+" lifecycle.allocationSafetyMargin", upstream.Lifecycle.AllocationSafetyMargin); err != nil {
|
if err := requireNonNegative(scope+" lifecycle.allocationSafetyMargin", upstream.Lifecycle.AllocationSafetyMargin); err != nil {
|
||||||
|
|||||||
@ -7,8 +7,8 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpapi"
|
"proxy-pool/internal/platform/httpapi"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
|
"proxy-pool/internal/platform/httpsecurity"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
|
|||||||
@ -9,8 +9,8 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpapi"
|
"proxy-pool/internal/platform/httpapi"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
|
"proxy-pool/internal/platform/httpsecurity"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestHandlerReturnsStatusWithoutSensitiveDetails(t *testing.T) {
|
func TestHandlerReturnsStatusWithoutSensitiveDetails(t *testing.T) {
|
||||||
|
|||||||
@ -7,10 +7,10 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
controllerExtraction "github.com/proxy-pool/proxy-pool/internal/controller/extraction"
|
controllerExtraction "proxy-pool/internal/controller/extraction"
|
||||||
domainExtraction "github.com/proxy-pool/proxy-pool/internal/domain/extraction"
|
domainExtraction "proxy-pool/internal/domain/extraction"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpapi"
|
"proxy-pool/internal/platform/httpapi"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
|
"proxy-pool/internal/platform/httpsecurity"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
|
|||||||
@ -11,10 +11,10 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
controllerExtraction "github.com/proxy-pool/proxy-pool/internal/controller/extraction"
|
controllerExtraction "proxy-pool/internal/controller/extraction"
|
||||||
domainExtraction "github.com/proxy-pool/proxy-pool/internal/domain/extraction"
|
domainExtraction "proxy-pool/internal/domain/extraction"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpapi"
|
"proxy-pool/internal/platform/httpapi"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
|
"proxy-pool/internal/platform/httpsecurity"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestNewHandlerValidatesDependenciesAndBodyLimit(t *testing.T) {
|
func TestNewHandlerValidatesDependenciesAndBodyLimit(t *testing.T) {
|
||||||
|
|||||||
@ -6,7 +6,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
domain "github.com/proxy-pool/proxy-pool/internal/domain/extraction"
|
domain "proxy-pool/internal/domain/extraction"
|
||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
@ -24,6 +24,7 @@ type Policy struct {
|
|||||||
MinRemainingTTL time.Duration
|
MinRemainingTTL time.Duration
|
||||||
MaxHealthCheckAge time.Duration
|
MaxHealthCheckAge time.Duration
|
||||||
ReserveForGateway int
|
ReserveForGateway int
|
||||||
|
IdempotencyTTL time.Duration
|
||||||
}
|
}
|
||||||
|
|
||||||
type Filters struct {
|
type Filters struct {
|
||||||
@ -85,7 +86,7 @@ func NewService(store domain.Store, policy Policy, admission Admission, now func
|
|||||||
return nil, fmt.Errorf("%w: store is required", ErrInvalidServicePolicy)
|
return nil, fmt.Errorf("%w: store is required", ErrInvalidServicePolicy)
|
||||||
}
|
}
|
||||||
if policy.MaxCountPerRequest <= 0 || policy.MinRemainingTTL < 0 ||
|
if policy.MaxCountPerRequest <= 0 || policy.MinRemainingTTL < 0 ||
|
||||||
policy.MaxHealthCheckAge < 0 || policy.ReserveForGateway < 0 {
|
policy.MaxHealthCheckAge < 0 || policy.ReserveForGateway < 0 || policy.IdempotencyTTL < 0 {
|
||||||
return nil, ErrInvalidServicePolicy
|
return nil, ErrInvalidServicePolicy
|
||||||
}
|
}
|
||||||
if policy.DefaultFulfillment != domain.Partial && policy.DefaultFulfillment != domain.AllOrNothing {
|
if policy.DefaultFulfillment != domain.Partial && policy.DefaultFulfillment != domain.AllOrNothing {
|
||||||
@ -119,13 +120,14 @@ func (s *Service) Extract(ctx context.Context, request Request) (Response, error
|
|||||||
now := s.now().UTC()
|
now := s.now().UTC()
|
||||||
clientID := request.ClientID
|
clientID := request.ClientID
|
||||||
if clientID == "" {
|
if clientID == "" {
|
||||||
clientID = "anonymous"
|
clientID = request.SourceIP
|
||||||
}
|
}
|
||||||
result, err := s.store.Extract(ctx, domain.Command{
|
result, err := s.store.Extract(ctx, domain.Command{
|
||||||
RequestID: request.RequestID,
|
RequestID: request.RequestID,
|
||||||
ClientID: clientID,
|
ClientID: clientID,
|
||||||
SourceIP: request.SourceIP,
|
SourceIP: request.SourceIP,
|
||||||
IdempotencyKey: request.IdempotencyKey,
|
IdempotencyKey: request.IdempotencyKey,
|
||||||
|
IdempotencyTTL: s.policy.IdempotencyTTL,
|
||||||
Requested: request.Count,
|
Requested: request.Count,
|
||||||
Fulfillment: fulfillment,
|
Fulfillment: fulfillment,
|
||||||
Now: now,
|
Now: now,
|
||||||
|
|||||||
@ -6,7 +6,9 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
domain "github.com/proxy-pool/proxy-pool/internal/domain/extraction"
|
"proxy-pool/internal/domain/activitypool"
|
||||||
|
domain "proxy-pool/internal/domain/extraction"
|
||||||
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestServiceAppliesPolicyAndBuildsResponse(t *testing.T) {
|
func TestServiceAppliesPolicyAndBuildsResponse(t *testing.T) {
|
||||||
@ -30,6 +32,7 @@ func TestServiceAppliesPolicyAndBuildsResponse(t *testing.T) {
|
|||||||
MinRemainingTTL: 30 * time.Second,
|
MinRemainingTTL: 30 * time.Second,
|
||||||
MaxHealthCheckAge: 15 * time.Second,
|
MaxHealthCheckAge: 15 * time.Second,
|
||||||
ReserveForGateway: 5,
|
ReserveForGateway: 5,
|
||||||
|
IdempotencyTTL: 2 * time.Minute,
|
||||||
}, allowAllAdmission{}, func() time.Time { return now })
|
}, allowAllAdmission{}, func() time.Time { return now })
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewService(): %v", err)
|
t.Fatalf("NewService(): %v", err)
|
||||||
@ -53,7 +56,10 @@ func TestServiceAppliesPolicyAndBuildsResponse(t *testing.T) {
|
|||||||
t.Fatalf("store command policy = %+v", store.command)
|
t.Fatalf("store command policy = %+v", store.command)
|
||||||
}
|
}
|
||||||
if store.command.RequestID != "req-1" || store.command.ClientID != "client-1" {
|
if store.command.RequestID != "req-1" || store.command.ClientID != "client-1" {
|
||||||
t.Fatalf("store command audit context = %+v", store.command)
|
t.Fatalf("store command request context = %+v", store.command)
|
||||||
|
}
|
||||||
|
if store.command.IdempotencyTTL != 2*time.Minute {
|
||||||
|
t.Fatalf("store command idempotency TTL = %s, want 2m", store.command.IdempotencyTTL)
|
||||||
}
|
}
|
||||||
if response.RequestID != "req-1" || response.Requested != 2 || response.Returned != 1 {
|
if response.RequestID != "req-1" || response.Requested != 2 || response.Returned != 1 {
|
||||||
t.Fatalf("response = %+v", response)
|
t.Fatalf("response = %+v", response)
|
||||||
@ -116,15 +122,45 @@ func TestServiceAppliesAdmissionBeforeStoreUsingStableIdentity(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestServiceUsesSourceIdentityForEphemeralIdempotency(t *testing.T) {
|
||||||
|
store := &recordingStore{}
|
||||||
|
service, err := NewService(store, Policy{
|
||||||
|
MaxCountPerRequest: 1,
|
||||||
|
DefaultFulfillment: domain.Partial,
|
||||||
|
}, allowAllAdmission{}, time.Now)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewService(): %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = service.Extract(context.Background(), Request{
|
||||||
|
RequestID: "req-1",
|
||||||
|
SourceIP: "192.0.2.30",
|
||||||
|
Count: 1,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Extract(): %v", err)
|
||||||
|
}
|
||||||
|
if store.command.ClientID != "192.0.2.30" {
|
||||||
|
t.Fatalf("store command ClientID = %q, want source IP", store.command.ClientID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestServiceIdempotentReplayKeepsOriginalExtractionTime(t *testing.T) {
|
func TestServiceIdempotentReplayKeepsOriginalExtractionTime(t *testing.T) {
|
||||||
firstTime := time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC)
|
firstTime := time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC)
|
||||||
secondTime := firstTime.Add(time.Minute)
|
secondTime := firstTime.Add(time.Minute)
|
||||||
store := domain.NewMemoryStore([]domain.Candidate{{
|
store := activitypool.NewMemoryPool()
|
||||||
ID: "p1",
|
checkedAt := firstTime
|
||||||
State: domain.Available,
|
upserted, err := store.UpsertFetched(context.Background(), "provider-a", activitypool.FetchedBatch{
|
||||||
ExpiresAt: firstTime.Add(2 * time.Minute),
|
ObservedAt: firstTime,
|
||||||
LastCheckedAt: firstTime,
|
ConfiguredTTL: 2 * time.Minute,
|
||||||
}})
|
Proxies: []proxyDomain.Proxy{{
|
||||||
|
ID: "p1", Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8080,
|
||||||
|
State: proxyDomain.StateAvailable, LastCheckedAt: &checkedAt,
|
||||||
|
}},
|
||||||
|
})
|
||||||
|
if err != nil || upserted.Inserted != 1 {
|
||||||
|
t.Fatalf("UpsertFetched() = %+v, %v", upserted, err)
|
||||||
|
}
|
||||||
times := []time.Time{firstTime, secondTime}
|
times := []time.Time{firstTime, secondTime}
|
||||||
service, err := NewService(store, Policy{
|
service, err := NewService(store, Policy{
|
||||||
MaxCountPerRequest: 1,
|
MaxCountPerRequest: 1,
|
||||||
|
|||||||
@ -5,7 +5,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"sync"
|
"sync"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/domain/upstream"
|
"proxy-pool/internal/domain/upstream"
|
||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
|
|||||||
@ -3,7 +3,7 @@ package pool
|
|||||||
import (
|
import (
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
ownershipDomain "github.com/proxy-pool/proxy-pool/internal/domain/ownership"
|
ownershipDomain "proxy-pool/internal/domain/ownership"
|
||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
|
|||||||
@ -9,7 +9,9 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
extractionDomain "github.com/proxy-pool/proxy-pool/internal/domain/extraction"
|
"proxy-pool/internal/domain/activitypool"
|
||||||
|
extractionDomain "proxy-pool/internal/domain/extraction"
|
||||||
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestOwnershipManagerPreventsDualAssignment(t *testing.T) {
|
func TestOwnershipManagerPreventsDualAssignment(t *testing.T) {
|
||||||
@ -64,9 +66,7 @@ func TestOwnershipManagerRenewsOnlyCurrentAssignment(t *testing.T) {
|
|||||||
func TestSharedRepositoryMakesOwnershipAndExtractionMutuallyExclusive(t *testing.T) {
|
func TestSharedRepositoryMakesOwnershipAndExtractionMutuallyExclusive(t *testing.T) {
|
||||||
for iteration := range 100 {
|
for iteration := range 100 {
|
||||||
now := time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC)
|
now := time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC)
|
||||||
store := extractionDomain.NewMemoryStore([]extractionDomain.Candidate{{
|
store := newTestActivityPool(t, now, "proxy-1")
|
||||||
ID: "proxy-1", State: extractionDomain.Available,
|
|
||||||
}})
|
|
||||||
manager, err := NewOwnershipManager(store)
|
manager, err := NewOwnershipManager(store)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("iteration %d NewOwnershipManager(): %v", iteration, err)
|
t.Fatalf("iteration %d NewOwnershipManager(): %v", iteration, err)
|
||||||
@ -159,15 +159,29 @@ func TestOwnershipManagerExpiresCrashedWorkerAssignment(t *testing.T) {
|
|||||||
|
|
||||||
func newTestOwnershipManager(t *testing.T, proxyIDs ...string) *OwnershipManager {
|
func newTestOwnershipManager(t *testing.T, proxyIDs ...string) *OwnershipManager {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
candidates := make([]extractionDomain.Candidate, 0, len(proxyIDs))
|
now := time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC)
|
||||||
for _, proxyID := range proxyIDs {
|
manager, err := NewOwnershipManager(newTestActivityPool(t, now, proxyIDs...))
|
||||||
candidates = append(candidates, extractionDomain.Candidate{
|
|
||||||
ID: proxyID, State: extractionDomain.Available,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
manager, err := NewOwnershipManager(extractionDomain.NewMemoryStore(candidates))
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("NewOwnershipManager(): %v", err)
|
t.Fatalf("NewOwnershipManager(): %v", err)
|
||||||
}
|
}
|
||||||
return manager
|
return manager
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func newTestActivityPool(t *testing.T, now time.Time, proxyIDs ...string) *activitypool.MemoryPool {
|
||||||
|
t.Helper()
|
||||||
|
proxies := make([]proxyDomain.Proxy, 0, len(proxyIDs))
|
||||||
|
for index, proxyID := range proxyIDs {
|
||||||
|
proxies = append(proxies, proxyDomain.Proxy{
|
||||||
|
ID: proxyID, Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.1",
|
||||||
|
Port: uint16(8000 + index), State: proxyDomain.StateAvailable,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
store := activitypool.NewMemoryPool()
|
||||||
|
result, err := store.UpsertFetched(context.Background(), "provider-a", activitypool.FetchedBatch{
|
||||||
|
ObservedAt: now, ConfiguredTTL: 10 * time.Minute, Proxies: proxies,
|
||||||
|
})
|
||||||
|
if err != nil || result.Inserted != len(proxyIDs) {
|
||||||
|
t.Fatalf("UpsertFetched() = %+v, %v", result, err)
|
||||||
|
}
|
||||||
|
return store
|
||||||
|
}
|
||||||
|
|||||||
@ -4,7 +4,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/domain/upstream"
|
"proxy-pool/internal/domain/upstream"
|
||||||
)
|
)
|
||||||
|
|
||||||
var ErrInvalidReconcilePolicy = errors.New("invalid pool reconcile policy")
|
var ErrInvalidReconcilePolicy = errors.New("invalid pool reconcile policy")
|
||||||
|
|||||||
@ -4,8 +4,8 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/domain/upstream"
|
"proxy-pool/internal/domain/upstream"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestPoolReconcilerSignalsOnlyWhenSlotsAreLowAndWholeFetchFits(t *testing.T) {
|
func TestPoolReconcilerSignalsOnlyWhenSlotsAreLowAndWholeFetchFits(t *testing.T) {
|
||||||
|
|||||||
@ -4,8 +4,9 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
"proxy-pool/internal/domain/activitypool"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/domain/upstream"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
|
"proxy-pool/internal/domain/upstream"
|
||||||
)
|
)
|
||||||
|
|
||||||
type FetchResponse struct {
|
type FetchResponse struct {
|
||||||
@ -31,13 +32,6 @@ type Parser interface {
|
|||||||
Parse(context.Context, []byte) ([]proxyDomain.Proxy, error)
|
Parse(context.Context, []byte) ([]proxyDomain.Proxy, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
// CandidateSink owns deduplication, is concurrency-safe, and returns the new
|
|
||||||
// count. On error it must not retain candidates; a successful count must be
|
|
||||||
// between zero and len(candidates).
|
|
||||||
type CandidateSink interface {
|
|
||||||
Add(context.Context, string, []proxyDomain.Proxy) (int, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
type Result struct {
|
type Result struct {
|
||||||
UpstreamID string
|
UpstreamID string
|
||||||
Class upstream.FetchClass
|
Class upstream.FetchClass
|
||||||
@ -53,9 +47,9 @@ type ResultRecorder interface {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type Ports struct {
|
type Ports struct {
|
||||||
Adapter ProviderAdapter
|
Adapter ProviderAdapter
|
||||||
Parser Parser
|
Parser Parser
|
||||||
Candidates CandidateSink
|
Activity activitypool.Upserter
|
||||||
Results ResultRecorder
|
Results ResultRecorder
|
||||||
Capacity upstream.FetchCapacity
|
Capacity upstream.FetchCapacity
|
||||||
}
|
}
|
||||||
|
|||||||
@ -8,17 +8,20 @@ import (
|
|||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/domain/upstream"
|
"proxy-pool/internal/domain/activitypool"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/coalesce"
|
"proxy-pool/internal/domain/upstream"
|
||||||
|
"proxy-pool/internal/platform/coalesce"
|
||||||
)
|
)
|
||||||
|
|
||||||
type Config struct {
|
type Config struct {
|
||||||
UpstreamID string
|
UpstreamID string
|
||||||
RequestInterval time.Duration
|
RequestInterval time.Duration
|
||||||
Timeout time.Duration
|
Timeout time.Duration
|
||||||
MaxAttempts int
|
MaxAttempts int
|
||||||
MaxInFlight int
|
MaxInFlight int
|
||||||
Retry RetryConfig
|
TTL time.Duration
|
||||||
|
AllocationSafetyMargin time.Duration
|
||||||
|
Retry RetryConfig
|
||||||
}
|
}
|
||||||
|
|
||||||
type RetryConfig struct {
|
type RetryConfig struct {
|
||||||
@ -62,6 +65,10 @@ func NewReconciler(config Config, ports Ports, runtimes ...Runtime) (*Reconciler
|
|||||||
if config.RequestInterval < 0 || config.Timeout <= 0 || config.MaxAttempts <= 0 || config.MaxInFlight <= 0 {
|
if config.RequestInterval < 0 || config.Timeout <= 0 || config.MaxAttempts <= 0 || config.MaxInFlight <= 0 {
|
||||||
return nil, fmt.Errorf("new provider reconciler: fetch limits must be positive")
|
return nil, fmt.Errorf("new provider reconciler: fetch limits must be positive")
|
||||||
}
|
}
|
||||||
|
if config.TTL < 0 || config.AllocationSafetyMargin < 0 ||
|
||||||
|
(config.TTL > 0 && config.AllocationSafetyMargin >= config.TTL) {
|
||||||
|
return nil, fmt.Errorf("new provider reconciler: lifecycle settings are invalid")
|
||||||
|
}
|
||||||
if config.Retry.Initial < 0 || config.Retry.Max < 0 || config.Retry.Jitter < 0 || config.Retry.Jitter > 100 {
|
if config.Retry.Initial < 0 || config.Retry.Max < 0 || config.Retry.Jitter < 0 || config.Retry.Jitter > 100 {
|
||||||
return nil, fmt.Errorf("new provider reconciler: retry settings are invalid")
|
return nil, fmt.Errorf("new provider reconciler: retry settings are invalid")
|
||||||
}
|
}
|
||||||
@ -71,7 +78,7 @@ func NewReconciler(config Config, ports Ports, runtimes ...Runtime) (*Reconciler
|
|||||||
if config.Retry.Max > 0 && config.Retry.Initial > config.Retry.Max {
|
if config.Retry.Max > 0 && config.Retry.Initial > config.Retry.Max {
|
||||||
return nil, fmt.Errorf("new provider reconciler: retry initial delay exceeds maximum")
|
return nil, fmt.Errorf("new provider reconciler: retry initial delay exceeds maximum")
|
||||||
}
|
}
|
||||||
if ports.Adapter == nil || ports.Parser == nil || ports.Candidates == nil ||
|
if ports.Adapter == nil || ports.Parser == nil || ports.Activity == nil ||
|
||||||
ports.Results == nil || ports.Capacity == nil {
|
ports.Results == nil || ports.Capacity == nil {
|
||||||
return nil, fmt.Errorf("new provider reconciler: all ports are required")
|
return nil, fmt.Errorf("new provider reconciler: all ports are required")
|
||||||
}
|
}
|
||||||
@ -202,7 +209,14 @@ func (r *Reconciler) fetchAttempt(ctx context.Context, attempt int) (FetchRespon
|
|||||||
if expected := permit.Expected(); expected < len(retained) {
|
if expected := permit.Expected(); expected < len(retained) {
|
||||||
retained = retained[:expected]
|
retained = retained[:expected]
|
||||||
}
|
}
|
||||||
newCount, candidateErr = r.ports.Candidates.Add(callCtx, r.config.UpstreamID, retained)
|
upserted, err := r.ports.Activity.UpsertFetched(callCtx, r.config.UpstreamID, activitypool.FetchedBatch{
|
||||||
|
ObservedAt: r.runtime.Clock.Now().UTC(),
|
||||||
|
ConfiguredTTL: r.config.TTL,
|
||||||
|
AllocationSafetyMargin: r.config.AllocationSafetyMargin,
|
||||||
|
Proxies: retained,
|
||||||
|
})
|
||||||
|
candidateErr = err
|
||||||
|
newCount = upserted.Inserted
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if callErr == nil && parseErr == nil && candidateErr == nil {
|
if callErr == nil && parseErr == nil && candidateErr == nil {
|
||||||
|
|||||||
@ -8,10 +8,47 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
"proxy-pool/internal/domain/activitypool"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/domain/upstream"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
|
"proxy-pool/internal/domain/upstream"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func TestReconcilerWritesProviderTTLPolicyToEphemeralPool(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 29, 12, 0, 0, 0, time.UTC)
|
||||||
|
clock := newFakeClock(now)
|
||||||
|
results := make(chan Result, 1)
|
||||||
|
batches := make(chan activitypool.FetchedBatch, 1)
|
||||||
|
ports := successfulPorts(func() {}, results)
|
||||||
|
ports.Activity = activitySinkFunc(func(_ context.Context, upstreamID string, batch activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
|
||||||
|
if upstreamID != "provider-a" {
|
||||||
|
t.Errorf("upstream ID = %q, want provider-a", upstreamID)
|
||||||
|
}
|
||||||
|
batches <- batch
|
||||||
|
return activitypool.UpsertResult{Accepted: 1, Inserted: 1}, nil
|
||||||
|
})
|
||||||
|
reconciler, err := NewReconciler(Config{
|
||||||
|
UpstreamID: "provider-a",
|
||||||
|
Timeout: time.Second,
|
||||||
|
MaxAttempts: 1,
|
||||||
|
MaxInFlight: 1,
|
||||||
|
TTL: 30 * time.Second,
|
||||||
|
AllocationSafetyMargin: 3 * time.Second,
|
||||||
|
}, ports, Runtime{Clock: clock})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewReconciler(): %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
result := runSingleReconcile(t, reconciler, results)
|
||||||
|
if result.ValidCount != 1 || result.NewCount != 1 {
|
||||||
|
t.Fatalf("result = %+v, want valid=1 new=1", result)
|
||||||
|
}
|
||||||
|
batch := <-batches
|
||||||
|
if !batch.ObservedAt.Equal(now) || batch.ConfiguredTTL != 30*time.Second ||
|
||||||
|
batch.AllocationSafetyMargin != 3*time.Second {
|
||||||
|
t.Fatalf("activity batch = %+v", batch)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestReconcilerCoalescesConcurrentNotifications(t *testing.T) {
|
func TestReconcilerCoalescesConcurrentNotifications(t *testing.T) {
|
||||||
var calls atomic.Int64
|
var calls atomic.Int64
|
||||||
result := make(chan Result, 1)
|
result := make(chan Result, 1)
|
||||||
@ -28,8 +65,8 @@ func TestReconcilerCoalescesConcurrentNotifications(t *testing.T) {
|
|||||||
Parser: parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) {
|
Parser: parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) {
|
||||||
return []proxyDomain.Proxy{{ID: "proxy-1"}}, nil
|
return []proxyDomain.Proxy{{ID: "proxy-1"}}, nil
|
||||||
}),
|
}),
|
||||||
Candidates: candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) {
|
Activity: activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
|
||||||
return 1, nil
|
return activitypool.UpsertResult{Accepted: 1, Inserted: 1}, nil
|
||||||
}),
|
}),
|
||||||
Results: resultRecorderFunc(func(got Result) { result <- got }),
|
Results: resultRecorderFunc(func(got Result) { result <- got }),
|
||||||
Capacity: unlimitedFetchCapacity{},
|
Capacity: unlimitedFetchCapacity{},
|
||||||
@ -122,8 +159,8 @@ func TestReconcilerRetriesErrorsWithExponentialBackoffAndJitter(t *testing.T) {
|
|||||||
}
|
}
|
||||||
return FetchResponse{Body: []byte("fixture")}, nil
|
return FetchResponse{Body: []byte("fixture")}, nil
|
||||||
})
|
})
|
||||||
ports.Candidates = candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) {
|
ports.Activity = activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
|
||||||
return 0, nil
|
return activitypool.UpsertResult{Accepted: 1}, nil
|
||||||
})
|
})
|
||||||
reconciler, err := NewReconciler(Config{
|
reconciler, err := NewReconciler(Config{
|
||||||
UpstreamID: "provider-a",
|
UpstreamID: "provider-a",
|
||||||
@ -196,8 +233,8 @@ func TestReconcilerClassifiesFetchResults(t *testing.T) {
|
|||||||
Parser: parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) {
|
Parser: parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) {
|
||||||
return make([]proxyDomain.Proxy, tt.candidates), tt.parseErr
|
return make([]proxyDomain.Proxy, tt.candidates), tt.parseErr
|
||||||
}),
|
}),
|
||||||
Candidates: candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) {
|
Activity: activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
|
||||||
return tt.newCount, nil
|
return activitypool.UpsertResult{Accepted: tt.candidates, Inserted: tt.newCount}, nil
|
||||||
}),
|
}),
|
||||||
Results: resultRecorderFunc(func(got Result) { results <- got }),
|
Results: resultRecorderFunc(func(got Result) { results <- got }),
|
||||||
Capacity: unlimitedFetchCapacity{},
|
Capacity: unlimitedFetchCapacity{},
|
||||||
@ -325,9 +362,9 @@ func TestReconcilerAppliesAttemptTimeoutToEveryPort(t *testing.T) {
|
|||||||
assertDeadline(ctx, "Parser.Parse")
|
assertDeadline(ctx, "Parser.Parse")
|
||||||
return []proxyDomain.Proxy{{ID: "proxy-1"}}, nil
|
return []proxyDomain.Proxy{{ID: "proxy-1"}}, nil
|
||||||
})
|
})
|
||||||
ports.Candidates = candidateSinkFunc(func(ctx context.Context, _ string, _ []proxyDomain.Proxy) (int, error) {
|
ports.Activity = activitySinkFunc(func(ctx context.Context, _ string, _ activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
|
||||||
assertDeadline(ctx, "CandidateSink.Add")
|
assertDeadline(ctx, "ActivitySink.UpsertFetched")
|
||||||
return 1, nil
|
return activitypool.UpsertResult{Accepted: 1, Inserted: 1}, nil
|
||||||
})
|
})
|
||||||
reconciler, err := NewReconciler(Config{
|
reconciler, err := NewReconciler(Config{
|
||||||
UpstreamID: "provider-a",
|
UpstreamID: "provider-a",
|
||||||
@ -493,14 +530,14 @@ func TestReconcilerUsesConfiguredMaxInFlight(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestReconcilerDoesNotRefetchWhenCandidateSinkFails(t *testing.T) {
|
func TestReconcilerDoesNotRefetchWhenActivitySinkFails(t *testing.T) {
|
||||||
clock := newFakeClock(time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC))
|
clock := newFakeClock(time.Date(2026, 7, 28, 12, 0, 0, 0, time.UTC))
|
||||||
sleeper := &fakeSleeper{clock: clock}
|
sleeper := &fakeSleeper{clock: clock}
|
||||||
results := make(chan Result, 3)
|
results := make(chan Result, 3)
|
||||||
var calls atomic.Int64
|
var calls atomic.Int64
|
||||||
ports := successfulPorts(func() { calls.Add(1) }, results)
|
ports := successfulPorts(func() { calls.Add(1) }, results)
|
||||||
ports.Candidates = candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) {
|
ports.Activity = activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
|
||||||
return 0, errors.New("candidate store unavailable")
|
return activitypool.UpsertResult{}, errors.New("activity pool unavailable")
|
||||||
})
|
})
|
||||||
reconciler, err := NewReconciler(Config{
|
reconciler, err := NewReconciler(Config{
|
||||||
UpstreamID: "provider-a",
|
UpstreamID: "provider-a",
|
||||||
@ -597,11 +634,11 @@ func TestReconcilerCompletesFetchPermitWithFetchedAndRetainedCounts(t *testing.T
|
|||||||
ports.Parser = parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) {
|
ports.Parser = parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) {
|
||||||
return []proxyDomain.Proxy{{ID: "p1"}, {ID: "p2"}, {ID: "p3"}}, nil
|
return []proxyDomain.Proxy{{ID: "p1"}, {ID: "p2"}, {ID: "p3"}}, nil
|
||||||
})
|
})
|
||||||
ports.Candidates = candidateSinkFunc(func(_ context.Context, _ string, candidates []proxyDomain.Proxy) (int, error) {
|
ports.Activity = activitySinkFunc(func(_ context.Context, _ string, batch activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
|
||||||
if len(candidates) != 2 {
|
if len(batch.Proxies) != 2 {
|
||||||
t.Errorf("CandidateSink candidates = %d, want permit limit 2", len(candidates))
|
t.Errorf("activity batch proxies = %d, want permit limit 2", len(batch.Proxies))
|
||||||
}
|
}
|
||||||
return 1, nil
|
return activitypool.UpsertResult{Accepted: 2, Inserted: 1}, nil
|
||||||
})
|
})
|
||||||
ports.Capacity = fetchCapacityFunc(func(string) (upstream.FetchPermit, bool, error) {
|
ports.Capacity = fetchCapacityFunc(func(string) (upstream.FetchPermit, bool, error) {
|
||||||
return &recordingFetchPermit{expected: 2, completed: completed}, true, nil
|
return &recordingFetchPermit{expected: 2, completed: completed}, true, nil
|
||||||
@ -720,8 +757,8 @@ func successfulPorts(onFetch func(), results chan<- Result) Ports {
|
|||||||
Parser: parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) {
|
Parser: parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) {
|
||||||
return []proxyDomain.Proxy{{ID: "proxy-1"}}, nil
|
return []proxyDomain.Proxy{{ID: "proxy-1"}}, nil
|
||||||
}),
|
}),
|
||||||
Candidates: candidateSinkFunc(func(context.Context, string, []proxyDomain.Proxy) (int, error) {
|
Activity: activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
|
||||||
return 1, nil
|
return activitypool.UpsertResult{Accepted: 1, Inserted: 1}, nil
|
||||||
}),
|
}),
|
||||||
Results: resultRecorderFunc(func(got Result) { results <- got }),
|
Results: resultRecorderFunc(func(got Result) { results <- got }),
|
||||||
Capacity: unlimitedFetchCapacity{},
|
Capacity: unlimitedFetchCapacity{},
|
||||||
@ -796,10 +833,10 @@ func (f parserFunc) Parse(ctx context.Context, body []byte) ([]proxyDomain.Proxy
|
|||||||
return f(ctx, body)
|
return f(ctx, body)
|
||||||
}
|
}
|
||||||
|
|
||||||
type candidateSinkFunc func(context.Context, string, []proxyDomain.Proxy) (int, error)
|
type activitySinkFunc func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error)
|
||||||
|
|
||||||
func (f candidateSinkFunc) Add(ctx context.Context, upstreamID string, candidates []proxyDomain.Proxy) (int, error) {
|
func (f activitySinkFunc) UpsertFetched(ctx context.Context, upstreamID string, batch activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
|
||||||
return f(ctx, upstreamID, candidates)
|
return f(ctx, upstreamID, batch)
|
||||||
}
|
}
|
||||||
|
|
||||||
type fetchCapacityFunc func(string) (upstream.FetchPermit, bool, error)
|
type fetchCapacityFunc func(string) (upstream.FetchPermit, bool, error)
|
||||||
|
|||||||
@ -8,11 +8,11 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/config"
|
"proxy-pool/internal/config"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/controller/admin"
|
"proxy-pool/internal/controller/admin"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/controller/distribution"
|
"proxy-pool/internal/controller/distribution"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
|
"proxy-pool/internal/platform/httpsecurity"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpserver"
|
"proxy-pool/internal/platform/httpserver"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
|
|||||||
@ -10,10 +10,10 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/config"
|
"proxy-pool/internal/config"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/controller/admin"
|
"proxy-pool/internal/controller/admin"
|
||||||
controllerExtraction "github.com/proxy-pool/proxy-pool/internal/controller/extraction"
|
controllerExtraction "proxy-pool/internal/controller/extraction"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpserver"
|
"proxy-pool/internal/platform/httpserver"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestRuntimeServesDistributionAndAdminOnIndependentListeners(t *testing.T) {
|
func TestRuntimeServesDistributionAndAdminOnIndependentListeners(t *testing.T) {
|
||||||
|
|||||||
590
internal/domain/activitypool/pool.go
Normal file
590
internal/domain/activitypool/pool.go
Normal file
@ -0,0 +1,590 @@
|
|||||||
|
package activitypool
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"errors"
|
||||||
|
"sort"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
extractionDomain "proxy-pool/internal/domain/extraction"
|
||||||
|
ownershipDomain "proxy-pool/internal/domain/ownership"
|
||||||
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
|
)
|
||||||
|
|
||||||
|
const defaultIdempotencyTTL = 5 * time.Minute
|
||||||
|
|
||||||
|
var ErrInvalidBatch = errors.New("invalid activity pool batch")
|
||||||
|
|
||||||
|
// FetchedBatch describes one ephemeral provider response. Proxies without a
|
||||||
|
// usable expiry are dropped because this pool is intentionally rebuildable.
|
||||||
|
type FetchedBatch struct {
|
||||||
|
ObservedAt time.Time
|
||||||
|
ConfiguredTTL time.Duration
|
||||||
|
AllocationSafetyMargin time.Duration
|
||||||
|
Proxies []proxyDomain.Proxy
|
||||||
|
}
|
||||||
|
|
||||||
|
type UpsertResult struct {
|
||||||
|
Accepted int
|
||||||
|
Inserted int
|
||||||
|
Refreshed int
|
||||||
|
Dropped int
|
||||||
|
}
|
||||||
|
|
||||||
|
type Upserter interface {
|
||||||
|
// A proxy unique key has one incumbent upstream for its current lifetime.
|
||||||
|
// Duplicates from other upstreams must not replace that source lifecycle.
|
||||||
|
UpsertFetched(context.Context, string, FetchedBatch) (UpsertResult, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type Entry struct {
|
||||||
|
Proxy proxyDomain.Proxy
|
||||||
|
UsableUntil time.Time
|
||||||
|
OwnerWorkerID string
|
||||||
|
State proxyDomain.State
|
||||||
|
}
|
||||||
|
|
||||||
|
type MemoryPool struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
|
||||||
|
entries map[string]Entry
|
||||||
|
keyByID map[string]string
|
||||||
|
idempotent map[string]idempotencyEntry
|
||||||
|
ownership map[string]ownershipDomain.Assignment
|
||||||
|
nextEpoch uint64
|
||||||
|
}
|
||||||
|
|
||||||
|
type idempotencyEntry struct {
|
||||||
|
command extractionDomain.Command
|
||||||
|
result extractionDomain.Result
|
||||||
|
expiresAt time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
_ Upserter = (*MemoryPool)(nil)
|
||||||
|
_ extractionDomain.Store = (*MemoryPool)(nil)
|
||||||
|
_ ownershipDomain.Repository = (*MemoryPool)(nil)
|
||||||
|
)
|
||||||
|
|
||||||
|
func NewMemoryPool() *MemoryPool {
|
||||||
|
return &MemoryPool{
|
||||||
|
entries: make(map[string]Entry),
|
||||||
|
keyByID: make(map[string]string),
|
||||||
|
idempotent: make(map[string]idempotencyEntry),
|
||||||
|
ownership: make(map[string]ownershipDomain.Assignment),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *MemoryPool) UpsertFetched(ctx context.Context, upstreamID string, batch FetchedBatch) (UpsertResult, error) {
|
||||||
|
var result UpsertResult
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return result, err
|
||||||
|
}
|
||||||
|
if p == nil || upstreamID == "" || batch.ObservedAt.IsZero() || batch.ConfiguredTTL < 0 ||
|
||||||
|
batch.AllocationSafetyMargin < 0 ||
|
||||||
|
(batch.ConfiguredTTL > 0 && batch.AllocationSafetyMargin >= batch.ConfiguredTTL) {
|
||||||
|
return result, ErrInvalidBatch
|
||||||
|
}
|
||||||
|
for _, candidate := range batch.Proxies {
|
||||||
|
if candidate.SourceUpstream != "" && candidate.SourceUpstream != upstreamID {
|
||||||
|
return result, ErrInvalidBatch
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return result, err
|
||||||
|
}
|
||||||
|
p.purgeExpiredLocked(batch.ObservedAt)
|
||||||
|
seenIDs := make(map[string]string, len(batch.Proxies))
|
||||||
|
for _, candidate := range batch.Proxies {
|
||||||
|
if !validProxyIdentity(candidate) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
key := candidate.UniqueKey()
|
||||||
|
proxyID := candidate.ID
|
||||||
|
if proxyID == "" {
|
||||||
|
proxyID = stableProxyID(key)
|
||||||
|
}
|
||||||
|
if existingKey, exists := p.keyByID[proxyID]; exists && existingKey != key {
|
||||||
|
return UpsertResult{}, ErrInvalidBatch
|
||||||
|
}
|
||||||
|
if existingKey, exists := seenIDs[proxyID]; exists && existingKey != key {
|
||||||
|
return UpsertResult{}, ErrInvalidBatch
|
||||||
|
}
|
||||||
|
seenIDs[proxyID] = key
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, candidate := range batch.Proxies {
|
||||||
|
if !validProxyIdentity(candidate) {
|
||||||
|
result.Dropped++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
candidate.SourceUpstream = upstreamID
|
||||||
|
expiresAt := proxyDomain.EffectiveExpiry(batch.ObservedAt, candidate.ExpiresAt, 0, batch.ConfiguredTTL)
|
||||||
|
if expiresAt == nil {
|
||||||
|
result.Dropped++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
usableUntil := expiresAt.Add(-batch.AllocationSafetyMargin)
|
||||||
|
if !usableUntil.After(batch.ObservedAt) {
|
||||||
|
result.Dropped++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
candidate.ExpiresAt = expiresAt
|
||||||
|
candidate.UsableUntil = &usableUntil
|
||||||
|
if candidate.CreatedAt.IsZero() {
|
||||||
|
candidate.CreatedAt = batch.ObservedAt.UTC()
|
||||||
|
}
|
||||||
|
if candidate.State == "" {
|
||||||
|
candidate.State = proxyDomain.StateFetched
|
||||||
|
}
|
||||||
|
key := candidate.UniqueKey()
|
||||||
|
result.Accepted++
|
||||||
|
|
||||||
|
if current, exists := p.entries[key]; exists {
|
||||||
|
result.Refreshed++
|
||||||
|
if current.State == proxyDomain.StateExtracted || current.Proxy.SourceUpstream != upstreamID {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
candidate.ID = current.Proxy.ID
|
||||||
|
candidate.CreatedAt = current.Proxy.CreatedAt
|
||||||
|
candidate.State = current.State
|
||||||
|
candidate.LastCheckedAt = current.Proxy.LastCheckedAt
|
||||||
|
candidate.LastSuccessAt = current.Proxy.LastSuccessAt
|
||||||
|
candidate.Latency = current.Proxy.Latency
|
||||||
|
p.entries[key] = Entry{
|
||||||
|
Proxy: cloneProxy(candidate),
|
||||||
|
UsableUntil: usableUntil,
|
||||||
|
OwnerWorkerID: current.OwnerWorkerID,
|
||||||
|
State: current.State,
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if candidate.ID == "" {
|
||||||
|
candidate.ID = stableProxyID(key)
|
||||||
|
}
|
||||||
|
p.entries[key] = Entry{
|
||||||
|
Proxy: cloneProxy(candidate),
|
||||||
|
UsableUntil: usableUntil,
|
||||||
|
State: candidate.State,
|
||||||
|
}
|
||||||
|
p.keyByID[candidate.ID] = key
|
||||||
|
result.Inserted++
|
||||||
|
}
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *MemoryPool) Snapshot(now time.Time) []Entry {
|
||||||
|
if p == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
p.purgeExpiredLocked(now)
|
||||||
|
entries := make([]Entry, 0, len(p.entries))
|
||||||
|
for _, entry := range p.entries {
|
||||||
|
entry.Proxy = cloneProxy(entry.Proxy)
|
||||||
|
entries = append(entries, entry)
|
||||||
|
}
|
||||||
|
sort.Slice(entries, func(i, j int) bool { return entries[i].Proxy.ID < entries[j].Proxy.ID })
|
||||||
|
return entries
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *MemoryPool) PurgeExpired(now time.Time) int {
|
||||||
|
if p == nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
return p.purgeExpiredLocked(now)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *MemoryPool) Extract(ctx context.Context, command extractionDomain.Command) (extractionDomain.Result, error) {
|
||||||
|
result := extractionDomain.Result{Requested: command.Requested}
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return result, err
|
||||||
|
}
|
||||||
|
if p == nil || command.Now.IsZero() || command.Requested < 0 || command.ReserveForGateway < 0 ||
|
||||||
|
command.MinRemainingTTL < 0 || command.MaxHealthCheckAge < 0 || command.IdempotencyTTL < 0 ||
|
||||||
|
(command.IdempotencyKey != "" && command.ClientID == "") ||
|
||||||
|
(command.Fulfillment != extractionDomain.Partial && command.Fulfillment != extractionDomain.AllOrNothing) {
|
||||||
|
return result, extractionDomain.ErrInvalidCommand
|
||||||
|
}
|
||||||
|
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return result, err
|
||||||
|
}
|
||||||
|
p.purgeExpiredLocked(command.Now)
|
||||||
|
|
||||||
|
idempotencyKey := command.ClientID + "\x00" + command.IdempotencyKey
|
||||||
|
if command.IdempotencyKey != "" {
|
||||||
|
if committed, ok := p.idempotent[idempotencyKey]; ok {
|
||||||
|
if !sameIdempotentRequest(committed.command, command) {
|
||||||
|
return result, extractionDomain.ErrIdempotencyConflict
|
||||||
|
}
|
||||||
|
return cloneResult(committed.result), nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if command.Requested == 0 {
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
eligible := make([]string, 0, len(p.entries))
|
||||||
|
for key, entry := range p.entries {
|
||||||
|
if eligibleForExtraction(entry, command) {
|
||||||
|
eligible = append(eligible, key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Slice(eligible, func(i, j int) bool {
|
||||||
|
return p.entries[eligible[i]].UsableUntil.After(p.entries[eligible[j]].UsableUntil)
|
||||||
|
})
|
||||||
|
available := len(eligible) - command.ReserveForGateway
|
||||||
|
if available < 0 {
|
||||||
|
available = 0
|
||||||
|
}
|
||||||
|
if command.Fulfillment == extractionDomain.AllOrNothing && available < command.Requested {
|
||||||
|
return result, extractionDomain.ErrInsufficientProxies
|
||||||
|
}
|
||||||
|
count := command.Requested
|
||||||
|
if count > available {
|
||||||
|
count = available
|
||||||
|
}
|
||||||
|
for _, key := range eligible[:count] {
|
||||||
|
entry := p.entries[key]
|
||||||
|
entry.State = proxyDomain.StateExtracted
|
||||||
|
entry.Proxy.State = proxyDomain.StateExtracted
|
||||||
|
p.entries[key] = entry
|
||||||
|
result.Items = append(result.Items, extractionCandidate(entry))
|
||||||
|
}
|
||||||
|
result.Returned = len(result.Items)
|
||||||
|
if result.Returned > 0 {
|
||||||
|
result.ExtractedAt = command.Now.UTC()
|
||||||
|
}
|
||||||
|
if command.IdempotencyKey != "" {
|
||||||
|
expiresAt := command.Now.Add(idempotencyTTL(command.IdempotencyTTL))
|
||||||
|
for _, item := range result.Items {
|
||||||
|
if !item.ExpiresAt.IsZero() && item.ExpiresAt.Before(expiresAt) {
|
||||||
|
expiresAt = item.ExpiresAt
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if expiresAt.After(command.Now) {
|
||||||
|
p.idempotent[idempotencyKey] = idempotencyEntry{
|
||||||
|
command: cloneCommand(command), result: cloneResult(result), expiresAt: expiresAt,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return result, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *MemoryPool) Assign(now time.Time, proxyID, workerID string, ttl time.Duration) (ownershipDomain.Assignment, error) {
|
||||||
|
if p == nil || now.IsZero() || proxyID == "" || workerID == "" || ttl <= 0 {
|
||||||
|
return ownershipDomain.Assignment{}, ownershipDomain.ErrInvalidOwnership
|
||||||
|
}
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
p.purgeExpiredLocked(now)
|
||||||
|
if current, exists := p.ownership[proxyID]; exists {
|
||||||
|
if current.ExpiresAt.After(now) {
|
||||||
|
return ownershipDomain.Assignment{}, ownershipDomain.ErrAlreadyOwned
|
||||||
|
}
|
||||||
|
if entry, ok := p.entryByIDLocked(proxyID); ok && entry.OwnerWorkerID == current.WorkerID {
|
||||||
|
entry.OwnerWorkerID = ""
|
||||||
|
p.setEntryByIDLocked(proxyID, entry)
|
||||||
|
}
|
||||||
|
delete(p.ownership, proxyID)
|
||||||
|
}
|
||||||
|
entry, ok := p.entryByIDLocked(proxyID)
|
||||||
|
if !ok || entry.State != proxyDomain.StateAvailable || entry.OwnerWorkerID != "" || !entry.UsableUntil.After(now) {
|
||||||
|
return ownershipDomain.Assignment{}, ownershipDomain.ErrOwnershipUnavailable
|
||||||
|
}
|
||||||
|
p.nextEpoch++
|
||||||
|
expiresAt := minTime(now.UTC().Add(ttl), entry.UsableUntil)
|
||||||
|
assignment := ownershipDomain.Assignment{
|
||||||
|
ProxyID: proxyID, WorkerID: workerID, Epoch: p.nextEpoch, Version: 1, ExpiresAt: expiresAt,
|
||||||
|
}
|
||||||
|
entry.OwnerWorkerID = workerID
|
||||||
|
p.setEntryByIDLocked(proxyID, entry)
|
||||||
|
p.ownership[proxyID] = assignment
|
||||||
|
return assignment, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *MemoryPool) Renew(now time.Time, proxyID, workerID string, epoch uint64, ttl time.Duration) (ownershipDomain.Assignment, error) {
|
||||||
|
if p == nil || now.IsZero() || ttl <= 0 {
|
||||||
|
return ownershipDomain.Assignment{}, ownershipDomain.ErrInvalidOwnership
|
||||||
|
}
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
p.purgeExpiredLocked(now)
|
||||||
|
assignment, ok := p.ownership[proxyID]
|
||||||
|
if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch || !assignment.ExpiresAt.After(now) {
|
||||||
|
return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment
|
||||||
|
}
|
||||||
|
entry, ok := p.entryByIDLocked(proxyID)
|
||||||
|
if !ok {
|
||||||
|
return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment
|
||||||
|
}
|
||||||
|
assignment.ExpiresAt = minTime(now.UTC().Add(ttl), entry.UsableUntil)
|
||||||
|
assignment.Version++
|
||||||
|
p.ownership[proxyID] = assignment
|
||||||
|
return assignment, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *MemoryPool) BeginDrain(proxyID, workerID string, epoch uint64) (ownershipDomain.Assignment, error) {
|
||||||
|
if p == nil {
|
||||||
|
return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment
|
||||||
|
}
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
assignment, ok := p.ownership[proxyID]
|
||||||
|
if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch {
|
||||||
|
return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment
|
||||||
|
}
|
||||||
|
if !assignment.Draining {
|
||||||
|
assignment.Draining = true
|
||||||
|
assignment.Version++
|
||||||
|
p.ownership[proxyID] = assignment
|
||||||
|
}
|
||||||
|
return assignment, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *MemoryPool) AcknowledgeDrain(proxyID, workerID string, epoch uint64, active, reserved int64) error {
|
||||||
|
if p == nil || active < 0 || reserved < 0 {
|
||||||
|
return ownershipDomain.ErrInvalidOwnership
|
||||||
|
}
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
assignment, ok := p.ownership[proxyID]
|
||||||
|
if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch {
|
||||||
|
return ownershipDomain.ErrStaleAssignment
|
||||||
|
}
|
||||||
|
if !assignment.Draining {
|
||||||
|
return ownershipDomain.ErrNotDraining
|
||||||
|
}
|
||||||
|
if active > 0 || reserved > 0 {
|
||||||
|
return ownershipDomain.ErrDrainNotReady
|
||||||
|
}
|
||||||
|
if entry, exists := p.entryByIDLocked(proxyID); exists && entry.OwnerWorkerID == workerID {
|
||||||
|
entry.OwnerWorkerID = ""
|
||||||
|
p.setEntryByIDLocked(proxyID, entry)
|
||||||
|
}
|
||||||
|
delete(p.ownership, proxyID)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *MemoryPool) Get(proxyID string) (ownershipDomain.Assignment, bool) {
|
||||||
|
if p == nil {
|
||||||
|
return ownershipDomain.Assignment{}, false
|
||||||
|
}
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
assignment, ok := p.ownership[proxyID]
|
||||||
|
return assignment, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *MemoryPool) Expire(now time.Time) []ownershipDomain.Assignment {
|
||||||
|
if p == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
expired := make([]ownershipDomain.Assignment, 0)
|
||||||
|
for proxyID, assignment := range p.ownership {
|
||||||
|
entry, exists := p.entryByIDLocked(proxyID)
|
||||||
|
if exists && assignment.ExpiresAt.After(now) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if exists && entry.OwnerWorkerID == assignment.WorkerID {
|
||||||
|
entry.OwnerWorkerID = ""
|
||||||
|
p.setEntryByIDLocked(proxyID, entry)
|
||||||
|
}
|
||||||
|
expired = append(expired, assignment)
|
||||||
|
delete(p.ownership, proxyID)
|
||||||
|
}
|
||||||
|
p.purgeExpiredLocked(now)
|
||||||
|
sort.Slice(expired, func(i, j int) bool { return expired[i].ProxyID < expired[j].ProxyID })
|
||||||
|
return expired
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *MemoryPool) purgeExpiredLocked(now time.Time) int {
|
||||||
|
removed := 0
|
||||||
|
for key, entry := range p.entries {
|
||||||
|
if entry.Proxy.ExpiresAt == nil || entry.Proxy.ExpiresAt.After(now) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
delete(p.ownership, entry.Proxy.ID)
|
||||||
|
delete(p.keyByID, entry.Proxy.ID)
|
||||||
|
delete(p.entries, key)
|
||||||
|
removed++
|
||||||
|
}
|
||||||
|
for key, entry := range p.idempotent {
|
||||||
|
if !entry.expiresAt.After(now) {
|
||||||
|
delete(p.idempotent, key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return removed
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *MemoryPool) entryByIDLocked(proxyID string) (Entry, bool) {
|
||||||
|
key, ok := p.keyByID[proxyID]
|
||||||
|
if !ok {
|
||||||
|
return Entry{}, false
|
||||||
|
}
|
||||||
|
entry, ok := p.entries[key]
|
||||||
|
return entry, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *MemoryPool) setEntryByIDLocked(proxyID string, entry Entry) {
|
||||||
|
if key, ok := p.keyByID[proxyID]; ok {
|
||||||
|
p.entries[key] = entry
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func eligibleForExtraction(entry Entry, command extractionDomain.Command) bool {
|
||||||
|
if entry.State != proxyDomain.StateAvailable || entry.OwnerWorkerID != "" || !entry.UsableUntil.After(command.Now) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if entry.Proxy.ExpiresAt != nil && entry.Proxy.ExpiresAt.Sub(command.Now) < command.MinRemainingTTL {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if command.MaxHealthCheckAge > 0 {
|
||||||
|
if entry.Proxy.LastCheckedAt == nil || command.Now.Sub(*entry.Proxy.LastCheckedAt) > command.MaxHealthCheckAge {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return matches(command.Protocols, string(entry.Proxy.Scheme)) &&
|
||||||
|
matches(command.Regions, entry.Proxy.Tags["region"]) &&
|
||||||
|
matches(command.Carriers, entry.Proxy.Tags["carrier"]) &&
|
||||||
|
matches(command.Upstreams, entry.Proxy.SourceUpstream)
|
||||||
|
}
|
||||||
|
|
||||||
|
func extractionCandidate(entry Entry) extractionDomain.Candidate {
|
||||||
|
expiresAt := time.Time{}
|
||||||
|
if entry.Proxy.ExpiresAt != nil {
|
||||||
|
expiresAt = *entry.Proxy.ExpiresAt
|
||||||
|
}
|
||||||
|
checkedAt := time.Time{}
|
||||||
|
if entry.Proxy.LastCheckedAt != nil {
|
||||||
|
checkedAt = *entry.Proxy.LastCheckedAt
|
||||||
|
}
|
||||||
|
return extractionDomain.Candidate{
|
||||||
|
ID: entry.Proxy.ID, Protocol: string(entry.Proxy.Scheme), Host: entry.Proxy.Host,
|
||||||
|
Port: entry.Proxy.Port, Username: entry.Proxy.Username, Region: entry.Proxy.Tags["region"],
|
||||||
|
Carrier: entry.Proxy.Tags["carrier"], Upstream: entry.Proxy.SourceUpstream,
|
||||||
|
OwnerWorkerID: entry.OwnerWorkerID, State: extractionDomain.Available,
|
||||||
|
ExpiresAt: expiresAt, LastCheckedAt: checkedAt,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func stableProxyID(key string) string {
|
||||||
|
digest := sha256.Sum256([]byte(key))
|
||||||
|
return "px_" + hex.EncodeToString(digest[:12])
|
||||||
|
}
|
||||||
|
|
||||||
|
func validProxyIdentity(candidate proxyDomain.Proxy) bool {
|
||||||
|
if candidate.Host == "" || candidate.Port == 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
switch candidate.Scheme {
|
||||||
|
case proxyDomain.SchemeHTTP, proxyDomain.SchemeHTTPS, proxyDomain.SchemeSOCKS5:
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneProxy(candidate proxyDomain.Proxy) proxyDomain.Proxy {
|
||||||
|
if candidate.ExpiresAt != nil {
|
||||||
|
value := *candidate.ExpiresAt
|
||||||
|
candidate.ExpiresAt = &value
|
||||||
|
}
|
||||||
|
if candidate.UsableUntil != nil {
|
||||||
|
value := *candidate.UsableUntil
|
||||||
|
candidate.UsableUntil = &value
|
||||||
|
}
|
||||||
|
if candidate.LastCheckedAt != nil {
|
||||||
|
value := *candidate.LastCheckedAt
|
||||||
|
candidate.LastCheckedAt = &value
|
||||||
|
}
|
||||||
|
if candidate.LastSuccessAt != nil {
|
||||||
|
value := *candidate.LastSuccessAt
|
||||||
|
candidate.LastSuccessAt = &value
|
||||||
|
}
|
||||||
|
if candidate.Tags != nil {
|
||||||
|
tags := make(map[string]string, len(candidate.Tags))
|
||||||
|
for key, value := range candidate.Tags {
|
||||||
|
tags[key] = value
|
||||||
|
}
|
||||||
|
candidate.Tags = tags
|
||||||
|
}
|
||||||
|
return candidate
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneResult(result extractionDomain.Result) extractionDomain.Result {
|
||||||
|
result.Items = append([]extractionDomain.Candidate(nil), result.Items...)
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneCommand(command extractionDomain.Command) extractionDomain.Command {
|
||||||
|
command.Protocols = append([]string(nil), command.Protocols...)
|
||||||
|
command.Regions = append([]string(nil), command.Regions...)
|
||||||
|
command.Carriers = append([]string(nil), command.Carriers...)
|
||||||
|
command.Upstreams = append([]string(nil), command.Upstreams...)
|
||||||
|
return command
|
||||||
|
}
|
||||||
|
|
||||||
|
func sameIdempotentRequest(left, right extractionDomain.Command) bool {
|
||||||
|
return left.Requested == right.Requested && left.Fulfillment == right.Fulfillment &&
|
||||||
|
equalSet(left.Protocols, right.Protocols) && equalSet(left.Regions, right.Regions) &&
|
||||||
|
equalSet(left.Carriers, right.Carriers) && equalSet(left.Upstreams, right.Upstreams)
|
||||||
|
}
|
||||||
|
|
||||||
|
func equalSet(left, right []string) bool {
|
||||||
|
if len(left) != len(right) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
counts := make(map[string]int, len(left))
|
||||||
|
for _, value := range left {
|
||||||
|
counts[value]++
|
||||||
|
}
|
||||||
|
for _, value := range right {
|
||||||
|
counts[value]--
|
||||||
|
if counts[value] < 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
func matches(allowed []string, value string) bool {
|
||||||
|
if len(allowed) == 0 {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
for _, candidate := range allowed {
|
||||||
|
if candidate == value {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func idempotencyTTL(configured time.Duration) time.Duration {
|
||||||
|
if configured > 0 {
|
||||||
|
return configured
|
||||||
|
}
|
||||||
|
return defaultIdempotencyTTL
|
||||||
|
}
|
||||||
|
|
||||||
|
func minTime(left, right time.Time) time.Time {
|
||||||
|
if left.Before(right) {
|
||||||
|
return left
|
||||||
|
}
|
||||||
|
return right
|
||||||
|
}
|
||||||
359
internal/domain/activitypool/pool_test.go
Normal file
359
internal/domain/activitypool/pool_test.go
Normal file
@ -0,0 +1,359 @@
|
|||||||
|
package activitypool
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
extractionDomain "proxy-pool/internal/domain/extraction"
|
||||||
|
ownershipDomain "proxy-pool/internal/domain/ownership"
|
||||||
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestMemoryPoolUpsertAppliesProviderTTLAndRefreshesWithoutGrowth(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
|
||||||
|
pool := NewMemoryPool()
|
||||||
|
batch := FetchedBatch{
|
||||||
|
ObservedAt: now,
|
||||||
|
ConfiguredTTL: 30 * time.Second,
|
||||||
|
AllocationSafetyMargin: 3 * time.Second,
|
||||||
|
Proxies: []proxyDomain.Proxy{{
|
||||||
|
Scheme: proxyDomain.SchemeHTTP,
|
||||||
|
Host: "192.0.2.10",
|
||||||
|
Port: 8080,
|
||||||
|
State: proxyDomain.StateAvailable,
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
|
||||||
|
first, err := pool.UpsertFetched(context.Background(), "provider-a", batch)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UpsertFetched(first): %v", err)
|
||||||
|
}
|
||||||
|
if first.Inserted != 1 || first.Refreshed != 0 || first.Dropped != 0 {
|
||||||
|
t.Fatalf("first result = %+v", first)
|
||||||
|
}
|
||||||
|
|
||||||
|
entries := pool.Snapshot(now)
|
||||||
|
if len(entries) != 1 {
|
||||||
|
t.Fatalf("Snapshot() entries = %d, want 1", len(entries))
|
||||||
|
}
|
||||||
|
if !entries[0].Proxy.ExpiresAt.Equal(now.Add(30 * time.Second)) {
|
||||||
|
t.Fatalf("ExpiresAt = %v, want %v", entries[0].Proxy.ExpiresAt, now.Add(30*time.Second))
|
||||||
|
}
|
||||||
|
if !entries[0].UsableUntil.Equal(now.Add(27 * time.Second)) {
|
||||||
|
t.Fatalf("UsableUntil = %v, want %v", entries[0].UsableUntil, now.Add(27*time.Second))
|
||||||
|
}
|
||||||
|
if entries[0].Proxy.UsableUntil == nil || !entries[0].Proxy.UsableUntil.Equal(now.Add(27*time.Second)) {
|
||||||
|
t.Fatalf("Proxy.UsableUntil = %v, want %v", entries[0].Proxy.UsableUntil, now.Add(27*time.Second))
|
||||||
|
}
|
||||||
|
|
||||||
|
batch.ObservedAt = now.Add(10 * time.Second)
|
||||||
|
second, err := pool.UpsertFetched(context.Background(), "provider-a", batch)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UpsertFetched(refresh): %v", err)
|
||||||
|
}
|
||||||
|
if second.Inserted != 0 || second.Refreshed != 1 || len(pool.Snapshot(batch.ObservedAt)) != 1 {
|
||||||
|
t.Fatalf("refresh result = %+v, entries=%d", second, len(pool.Snapshot(batch.ObservedAt)))
|
||||||
|
}
|
||||||
|
if got := pool.Snapshot(batch.ObservedAt)[0].Proxy.ExpiresAt; !got.Equal(now.Add(40 * time.Second)) {
|
||||||
|
t.Fatalf("refreshed ExpiresAt = %v, want %v", got, now.Add(40*time.Second))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMemoryPoolCrossProviderDuplicateDoesNotReplaceSourceLifecycle(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
|
||||||
|
pool := NewMemoryPool()
|
||||||
|
proxy := proxyDomain.Proxy{
|
||||||
|
Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8080,
|
||||||
|
State: proxyDomain.StateAvailable,
|
||||||
|
}
|
||||||
|
first, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{
|
||||||
|
ObservedAt: now, ConfiguredTTL: 30 * time.Second,
|
||||||
|
AllocationSafetyMargin: 3 * time.Second, Proxies: []proxyDomain.Proxy{proxy},
|
||||||
|
})
|
||||||
|
if err != nil || first.Inserted != 1 {
|
||||||
|
t.Fatalf("UpsertFetched(provider-a) = %+v, %v", first, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
duplicate, err := pool.UpsertFetched(context.Background(), "provider-b", FetchedBatch{
|
||||||
|
ObservedAt: now.Add(time.Second), ConfiguredTTL: 5 * time.Minute,
|
||||||
|
AllocationSafetyMargin: 10 * time.Second, Proxies: []proxyDomain.Proxy{proxy},
|
||||||
|
})
|
||||||
|
if err != nil || duplicate.Inserted != 0 || duplicate.Refreshed != 1 {
|
||||||
|
t.Fatalf("UpsertFetched(provider-b) = %+v, %v", duplicate, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
entries := pool.Snapshot(now.Add(time.Second))
|
||||||
|
if len(entries) != 1 {
|
||||||
|
t.Fatalf("Snapshot() entries = %d, want 1", len(entries))
|
||||||
|
}
|
||||||
|
entry := entries[0]
|
||||||
|
if entry.Proxy.SourceUpstream != "provider-a" {
|
||||||
|
t.Fatalf("SourceUpstream = %q, want provider-a", entry.Proxy.SourceUpstream)
|
||||||
|
}
|
||||||
|
if !entry.Proxy.ExpiresAt.Equal(now.Add(30*time.Second)) ||
|
||||||
|
!entry.UsableUntil.Equal(now.Add(27*time.Second)) {
|
||||||
|
t.Fatalf("lifecycle = expires %v usable %v, want provider-a lifecycle", entry.Proxy.ExpiresAt, entry.UsableUntil)
|
||||||
|
}
|
||||||
|
|
||||||
|
afterExpiry, err := pool.UpsertFetched(context.Background(), "provider-b", FetchedBatch{
|
||||||
|
ObservedAt: now.Add(31 * time.Second), ConfiguredTTL: 5 * time.Minute,
|
||||||
|
AllocationSafetyMargin: 10 * time.Second, Proxies: []proxyDomain.Proxy{proxy},
|
||||||
|
})
|
||||||
|
if err != nil || afterExpiry.Inserted != 1 {
|
||||||
|
t.Fatalf("UpsertFetched(provider-b after expiry) = %+v, %v", afterExpiry, err)
|
||||||
|
}
|
||||||
|
replacement := pool.Snapshot(now.Add(31 * time.Second))[0]
|
||||||
|
if replacement.Proxy.SourceUpstream != "provider-b" {
|
||||||
|
t.Fatalf("replacement SourceUpstream = %q, want provider-b", replacement.Proxy.SourceUpstream)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMemoryPoolRefreshPreservesRuntimeStateAndHealth(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
|
||||||
|
checkedAt := now.Add(-time.Second)
|
||||||
|
pool := NewMemoryPool()
|
||||||
|
first, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{
|
||||||
|
ObservedAt: now, ConfiguredTTL: 30 * time.Second,
|
||||||
|
Proxies: []proxyDomain.Proxy{{
|
||||||
|
Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8080,
|
||||||
|
State: proxyDomain.StateAvailable, LastCheckedAt: &checkedAt,
|
||||||
|
}},
|
||||||
|
})
|
||||||
|
if err != nil || first.Inserted != 1 {
|
||||||
|
t.Fatalf("UpsertFetched(first) = %+v, %v", first, err)
|
||||||
|
}
|
||||||
|
initial := pool.Snapshot(now)[0]
|
||||||
|
|
||||||
|
second, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{
|
||||||
|
ObservedAt: now.Add(10 * time.Second), ConfiguredTTL: 30 * time.Second,
|
||||||
|
Proxies: []proxyDomain.Proxy{{
|
||||||
|
Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8080,
|
||||||
|
State: proxyDomain.StateFetched,
|
||||||
|
}},
|
||||||
|
})
|
||||||
|
if err != nil || second.Refreshed != 1 {
|
||||||
|
t.Fatalf("UpsertFetched(refresh) = %+v, %v", second, err)
|
||||||
|
}
|
||||||
|
refreshed := pool.Snapshot(now.Add(10 * time.Second))[0]
|
||||||
|
if refreshed.Proxy.ID != initial.Proxy.ID || refreshed.State != proxyDomain.StateAvailable ||
|
||||||
|
refreshed.Proxy.LastCheckedAt == nil || !refreshed.Proxy.LastCheckedAt.Equal(checkedAt) ||
|
||||||
|
!refreshed.Proxy.CreatedAt.Equal(initial.Proxy.CreatedAt) {
|
||||||
|
t.Fatalf("refreshed entry lost runtime state: initial=%+v refreshed=%+v", initial, refreshed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMemoryPoolDropsCandidatesWithoutUsableTTL(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
|
||||||
|
expired := now.Add(-time.Second)
|
||||||
|
tooShort := now.Add(2 * time.Second)
|
||||||
|
pool := NewMemoryPool()
|
||||||
|
|
||||||
|
result, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{
|
||||||
|
ObservedAt: now,
|
||||||
|
AllocationSafetyMargin: 3 * time.Second,
|
||||||
|
Proxies: []proxyDomain.Proxy{
|
||||||
|
{Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8001},
|
||||||
|
{Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.11", Port: 8002, ExpiresAt: &expired},
|
||||||
|
{Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.12", Port: 8003, ExpiresAt: &tooShort},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("UpsertFetched(): %v", err)
|
||||||
|
}
|
||||||
|
if result.Dropped != 3 || result.Accepted != 0 || len(pool.Snapshot(now)) != 0 {
|
||||||
|
t.Fatalf("result = %+v, entries=%d", result, len(pool.Snapshot(now)))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMemoryPoolRejectsInvalidBatchAtomically(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
|
||||||
|
pool := NewMemoryPool()
|
||||||
|
result, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{
|
||||||
|
ObservedAt: now,
|
||||||
|
ConfiguredTTL: time.Minute,
|
||||||
|
Proxies: []proxyDomain.Proxy{
|
||||||
|
{Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.10", Port: 8001},
|
||||||
|
{Scheme: proxyDomain.SchemeHTTP, Host: "192.0.2.11", Port: 8002, SourceUpstream: "provider-b"},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if !errors.Is(err, ErrInvalidBatch) {
|
||||||
|
t.Fatalf("UpsertFetched() error = %v, want ErrInvalidBatch", err)
|
||||||
|
}
|
||||||
|
if result != (UpsertResult{}) || len(pool.Snapshot(now)) != 0 {
|
||||||
|
t.Fatalf("failed batch retained state: result=%+v entries=%+v", result, pool.Snapshot(now))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMemoryPoolNeverExtractsProxyTwiceAndDoesNotWriteAuditRecords(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
|
||||||
|
pool := poolWithOneProxy(t, now)
|
||||||
|
|
||||||
|
var wait sync.WaitGroup
|
||||||
|
results := make(chan string, 1000)
|
||||||
|
for range 1000 {
|
||||||
|
wait.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wait.Done()
|
||||||
|
result, err := pool.Extract(context.Background(), extractionDomain.Command{
|
||||||
|
Requested: 1,
|
||||||
|
Fulfillment: extractionDomain.Partial,
|
||||||
|
Now: now,
|
||||||
|
MinRemainingTTL: 10 * time.Second,
|
||||||
|
MaxHealthCheckAge: time.Minute,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("Extract(): %v", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, item := range result.Items {
|
||||||
|
results <- item.ID
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
wait.Wait()
|
||||||
|
close(results)
|
||||||
|
|
||||||
|
count := 0
|
||||||
|
for range results {
|
||||||
|
count++
|
||||||
|
}
|
||||||
|
if count != 1 {
|
||||||
|
t.Fatalf("proxy extracted %d times, want exactly once", count)
|
||||||
|
}
|
||||||
|
if got := pool.Snapshot(now); len(got) != 1 || got[0].State != proxyDomain.StateExtracted {
|
||||||
|
t.Fatalf("post-extraction snapshot = %+v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMemoryPoolIdempotencyIsBoundedByProxyExpiry(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
|
||||||
|
pool := poolWithOneProxy(t, now)
|
||||||
|
command := extractionDomain.Command{
|
||||||
|
ClientID: "client-a",
|
||||||
|
IdempotencyKey: "idem-a",
|
||||||
|
IdempotencyTTL: 5 * time.Minute,
|
||||||
|
Requested: 1,
|
||||||
|
Fulfillment: extractionDomain.Partial,
|
||||||
|
Now: now,
|
||||||
|
MinRemainingTTL: 10 * time.Second,
|
||||||
|
MaxHealthCheckAge: time.Minute,
|
||||||
|
}
|
||||||
|
|
||||||
|
first, err := pool.Extract(context.Background(), command)
|
||||||
|
if err != nil || first.Returned != 1 {
|
||||||
|
t.Fatalf("first Extract() = %+v, %v", first, err)
|
||||||
|
}
|
||||||
|
command.Now = now.Add(20 * time.Second)
|
||||||
|
replayed, err := pool.Extract(context.Background(), command)
|
||||||
|
if err != nil || replayed.Returned != 1 || replayed.Items[0].ID != first.Items[0].ID {
|
||||||
|
t.Fatalf("replayed Extract() = %+v, %v", replayed, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if purged := pool.PurgeExpired(now.Add(31 * time.Second)); purged != 1 {
|
||||||
|
t.Fatalf("PurgeExpired() = %d, want 1", purged)
|
||||||
|
}
|
||||||
|
command.Now = now.Add(31 * time.Second)
|
||||||
|
afterExpiry, err := pool.Extract(context.Background(), command)
|
||||||
|
if err != nil || afterExpiry.Returned != 0 {
|
||||||
|
t.Fatalf("Extract(after expiry) = %+v, %v", afterExpiry, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMemoryPoolAllOrNothingDoesNotConsumePartialInventory(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
|
||||||
|
pool := poolWithOneProxy(t, now)
|
||||||
|
result, err := pool.Extract(context.Background(), extractionDomain.Command{
|
||||||
|
Requested: 2, Fulfillment: extractionDomain.AllOrNothing, Now: now,
|
||||||
|
})
|
||||||
|
if !errors.Is(err, extractionDomain.ErrInsufficientProxies) || result.Returned != 0 {
|
||||||
|
t.Fatalf("all-or-nothing Extract() = %+v, %v", result, err)
|
||||||
|
}
|
||||||
|
partial, err := pool.Extract(context.Background(), extractionDomain.Command{
|
||||||
|
Requested: 1, Fulfillment: extractionDomain.Partial, Now: now,
|
||||||
|
})
|
||||||
|
if err != nil || partial.Returned != 1 {
|
||||||
|
t.Fatalf("partial Extract() after rollback = %+v, %v", partial, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMemoryPoolAssignRecoversExpiredLeaseWithoutSeparateSweep(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
|
||||||
|
pool := poolWithOneProxy(t, now)
|
||||||
|
proxyID := pool.Snapshot(now)[0].Proxy.ID
|
||||||
|
first, err := pool.Assign(now, proxyID, "worker-a", 5*time.Second)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Assign(first): %v", err)
|
||||||
|
}
|
||||||
|
second, err := pool.Assign(now.Add(5*time.Second), proxyID, "worker-b", 5*time.Second)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Assign(after lease expiry): %v", err)
|
||||||
|
}
|
||||||
|
if second.Epoch <= first.Epoch || second.WorkerID != "worker-b" {
|
||||||
|
t.Fatalf("second assignment = %+v, first=%+v", second, first)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMemoryPoolMakesOwnershipAndExtractionMutuallyExclusive(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 29, 10, 0, 0, 0, time.UTC)
|
||||||
|
for iteration := range 100 {
|
||||||
|
pool := poolWithOneProxy(t, now)
|
||||||
|
start := make(chan struct{})
|
||||||
|
assigned := make(chan bool, 1)
|
||||||
|
extracted := make(chan bool, 1)
|
||||||
|
go func() {
|
||||||
|
<-start
|
||||||
|
_, err := pool.Assign(now, pool.Snapshot(now)[0].Proxy.ID, "worker-a", time.Minute)
|
||||||
|
if err != nil && !errors.Is(err, ownershipDomain.ErrOwnershipUnavailable) {
|
||||||
|
t.Errorf("Assign(): %v", err)
|
||||||
|
}
|
||||||
|
assigned <- err == nil
|
||||||
|
}()
|
||||||
|
go func() {
|
||||||
|
<-start
|
||||||
|
result, err := pool.Extract(context.Background(), extractionDomain.Command{
|
||||||
|
ClientID: "client-a", Requested: 1, Fulfillment: extractionDomain.Partial, Now: now,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Errorf("Extract(): %v", err)
|
||||||
|
}
|
||||||
|
extracted <- result.Returned == 1
|
||||||
|
}()
|
||||||
|
close(start)
|
||||||
|
|
||||||
|
wins := 0
|
||||||
|
if <-assigned {
|
||||||
|
wins++
|
||||||
|
}
|
||||||
|
if <-extracted {
|
||||||
|
wins++
|
||||||
|
}
|
||||||
|
if wins != 1 {
|
||||||
|
t.Fatalf("iteration %d winners = %d, want 1", iteration, wins)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func poolWithOneProxy(t *testing.T, now time.Time) *MemoryPool {
|
||||||
|
t.Helper()
|
||||||
|
pool := NewMemoryPool()
|
||||||
|
checkedAt := now
|
||||||
|
result, err := pool.UpsertFetched(context.Background(), "provider-a", FetchedBatch{
|
||||||
|
ObservedAt: now,
|
||||||
|
ConfiguredTTL: 30 * time.Second,
|
||||||
|
AllocationSafetyMargin: 3 * time.Second,
|
||||||
|
Proxies: []proxyDomain.Proxy{{
|
||||||
|
Scheme: proxyDomain.SchemeHTTP,
|
||||||
|
Host: "192.0.2.10",
|
||||||
|
Port: 8080,
|
||||||
|
State: proxyDomain.StateAvailable,
|
||||||
|
LastCheckedAt: &checkedAt,
|
||||||
|
}},
|
||||||
|
})
|
||||||
|
if err != nil || result.Inserted != 1 {
|
||||||
|
t.Fatalf("UpsertFetched() = %+v, %v", result, err)
|
||||||
|
}
|
||||||
|
return pool
|
||||||
|
}
|
||||||
@ -3,11 +3,7 @@ package extraction
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"sort"
|
|
||||||
"sync"
|
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
ownershipDomain "github.com/proxy-pool/proxy-pool/internal/domain/ownership"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
type Fulfillment string
|
type Fulfillment string
|
||||||
@ -52,6 +48,7 @@ type Command struct {
|
|||||||
ClientID string
|
ClientID string
|
||||||
SourceIP string
|
SourceIP string
|
||||||
IdempotencyKey string
|
IdempotencyKey string
|
||||||
|
IdempotencyTTL time.Duration
|
||||||
Requested int
|
Requested int
|
||||||
Fulfillment Fulfillment
|
Fulfillment Fulfillment
|
||||||
Now time.Time
|
Now time.Time
|
||||||
@ -64,16 +61,6 @@ type Command struct {
|
|||||||
Upstreams []string
|
Upstreams []string
|
||||||
}
|
}
|
||||||
|
|
||||||
type Record struct {
|
|
||||||
ProxyID string
|
|
||||||
ClientID string
|
|
||||||
SourceIP string
|
|
||||||
RequestID string
|
|
||||||
Upstream string
|
|
||||||
ExtractedAt time.Time
|
|
||||||
ExpiresAt time.Time
|
|
||||||
}
|
|
||||||
|
|
||||||
type Result struct {
|
type Result struct {
|
||||||
Requested int
|
Requested int
|
||||||
Returned int
|
Returned int
|
||||||
@ -81,294 +68,8 @@ type Result struct {
|
|||||||
Items []Candidate
|
Items []Candidate
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Store atomically consumes proxies from a bounded, ephemeral activity pool.
|
||||||
|
// Implementations must not require durable per-proxy extraction records.
|
||||||
type Store interface {
|
type Store interface {
|
||||||
Extract(context.Context, Command) (Result, error)
|
Extract(context.Context, Command) (Result, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
type MemoryStore struct {
|
|
||||||
mu sync.Mutex
|
|
||||||
candidates map[string]Candidate
|
|
||||||
records []Record
|
|
||||||
idempotent map[string]idempotencyEntry
|
|
||||||
nextEpoch uint64
|
|
||||||
ownership map[string]ownershipDomain.Assignment
|
|
||||||
}
|
|
||||||
|
|
||||||
var _ ownershipDomain.Repository = (*MemoryStore)(nil)
|
|
||||||
|
|
||||||
type idempotencyEntry struct {
|
|
||||||
command Command
|
|
||||||
result Result
|
|
||||||
}
|
|
||||||
|
|
||||||
func NewMemoryStore(candidates []Candidate) *MemoryStore {
|
|
||||||
items := make(map[string]Candidate, len(candidates))
|
|
||||||
for _, candidate := range candidates {
|
|
||||||
items[candidate.ID] = candidate
|
|
||||||
}
|
|
||||||
return &MemoryStore{
|
|
||||||
candidates: items,
|
|
||||||
idempotent: make(map[string]idempotencyEntry),
|
|
||||||
ownership: make(map[string]ownershipDomain.Assignment),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *MemoryStore) Extract(ctx context.Context, command Command) (Result, error) {
|
|
||||||
result := Result{Requested: command.Requested}
|
|
||||||
if err := ctx.Err(); err != nil {
|
|
||||||
return result, err
|
|
||||||
}
|
|
||||||
if command.Requested < 0 || command.ReserveForGateway < 0 ||
|
|
||||||
command.MinRemainingTTL < 0 || command.MaxHealthCheckAge < 0 ||
|
|
||||||
(command.Fulfillment != Partial && command.Fulfillment != AllOrNothing) {
|
|
||||||
return result, ErrInvalidCommand
|
|
||||||
}
|
|
||||||
s.mu.Lock()
|
|
||||||
defer s.mu.Unlock()
|
|
||||||
if err := ctx.Err(); err != nil {
|
|
||||||
return result, err
|
|
||||||
}
|
|
||||||
|
|
||||||
idempotencyKey := command.ClientID + "\x00" + command.IdempotencyKey
|
|
||||||
if command.IdempotencyKey != "" {
|
|
||||||
if committed, ok := s.idempotent[idempotencyKey]; ok {
|
|
||||||
if !sameIdempotentRequest(committed.command, command) {
|
|
||||||
return result, ErrIdempotencyConflict
|
|
||||||
}
|
|
||||||
return cloneResult(committed.result), nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if command.Requested <= 0 {
|
|
||||||
return result, nil
|
|
||||||
}
|
|
||||||
eligible := make([]Candidate, 0, len(s.candidates))
|
|
||||||
for _, candidate := range s.candidates {
|
|
||||||
if eligibleForExtraction(candidate, command) {
|
|
||||||
eligible = append(eligible, candidate)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
sort.Slice(eligible, func(i, j int) bool {
|
|
||||||
return eligible[i].ExpiresAt.After(eligible[j].ExpiresAt)
|
|
||||||
})
|
|
||||||
available := len(eligible) - command.ReserveForGateway
|
|
||||||
if available < 0 {
|
|
||||||
available = 0
|
|
||||||
}
|
|
||||||
if command.Fulfillment == AllOrNothing && available < command.Requested {
|
|
||||||
return result, ErrInsufficientProxies
|
|
||||||
}
|
|
||||||
count := command.Requested
|
|
||||||
if count > available {
|
|
||||||
count = available
|
|
||||||
}
|
|
||||||
for i := 0; i < count; i++ {
|
|
||||||
candidate := eligible[i]
|
|
||||||
candidate.State = Extracted
|
|
||||||
s.candidates[candidate.ID] = candidate
|
|
||||||
result.Items = append(result.Items, candidate)
|
|
||||||
s.records = append(s.records, Record{
|
|
||||||
ProxyID: candidate.ID,
|
|
||||||
ClientID: command.ClientID,
|
|
||||||
SourceIP: command.SourceIP,
|
|
||||||
RequestID: command.RequestID,
|
|
||||||
Upstream: candidate.Upstream,
|
|
||||||
ExtractedAt: command.Now,
|
|
||||||
ExpiresAt: candidate.ExpiresAt,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
result.Returned = len(result.Items)
|
|
||||||
if result.Returned > 0 {
|
|
||||||
result.ExtractedAt = command.Now
|
|
||||||
}
|
|
||||||
if command.IdempotencyKey != "" {
|
|
||||||
s.idempotent[idempotencyKey] = idempotencyEntry{
|
|
||||||
command: cloneCommand(command),
|
|
||||||
result: cloneResult(result),
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return result, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *MemoryStore) Assign(now time.Time, proxyID, workerID string, ttl time.Duration) (ownershipDomain.Assignment, error) {
|
|
||||||
if proxyID == "" || workerID == "" || ttl <= 0 {
|
|
||||||
return ownershipDomain.Assignment{}, ownershipDomain.ErrInvalidOwnership
|
|
||||||
}
|
|
||||||
s.mu.Lock()
|
|
||||||
defer s.mu.Unlock()
|
|
||||||
candidate, ok := s.candidates[proxyID]
|
|
||||||
if current, exists := s.ownership[proxyID]; exists && current.ExpiresAt.After(now) {
|
|
||||||
return ownershipDomain.Assignment{}, ownershipDomain.ErrAlreadyOwned
|
|
||||||
} else if exists {
|
|
||||||
if candidate.OwnerWorkerID == current.WorkerID {
|
|
||||||
candidate.OwnerWorkerID = ""
|
|
||||||
}
|
|
||||||
delete(s.ownership, proxyID)
|
|
||||||
}
|
|
||||||
if !ok || candidate.State != Available || candidate.OwnerWorkerID != "" {
|
|
||||||
return ownershipDomain.Assignment{}, ownershipDomain.ErrOwnershipUnavailable
|
|
||||||
}
|
|
||||||
s.nextEpoch++
|
|
||||||
assignment := ownershipDomain.Assignment{
|
|
||||||
ProxyID: proxyID, WorkerID: workerID, Epoch: s.nextEpoch, Version: 1,
|
|
||||||
ExpiresAt: now.UTC().Add(ttl),
|
|
||||||
}
|
|
||||||
candidate.OwnerWorkerID = workerID
|
|
||||||
s.candidates[proxyID] = candidate
|
|
||||||
s.ownership[proxyID] = assignment
|
|
||||||
return assignment, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *MemoryStore) Renew(now time.Time, proxyID, workerID string, epoch uint64, ttl time.Duration) (ownershipDomain.Assignment, error) {
|
|
||||||
if ttl <= 0 {
|
|
||||||
return ownershipDomain.Assignment{}, ownershipDomain.ErrInvalidOwnership
|
|
||||||
}
|
|
||||||
s.mu.Lock()
|
|
||||||
defer s.mu.Unlock()
|
|
||||||
assignment, ok := s.ownership[proxyID]
|
|
||||||
if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch || !assignment.ExpiresAt.After(now) {
|
|
||||||
return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment
|
|
||||||
}
|
|
||||||
assignment.ExpiresAt = now.UTC().Add(ttl)
|
|
||||||
assignment.Version++
|
|
||||||
s.ownership[proxyID] = assignment
|
|
||||||
return assignment, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *MemoryStore) BeginDrain(proxyID, workerID string, epoch uint64) (ownershipDomain.Assignment, error) {
|
|
||||||
s.mu.Lock()
|
|
||||||
defer s.mu.Unlock()
|
|
||||||
assignment, ok := s.ownership[proxyID]
|
|
||||||
if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch {
|
|
||||||
return ownershipDomain.Assignment{}, ownershipDomain.ErrStaleAssignment
|
|
||||||
}
|
|
||||||
if assignment.Draining {
|
|
||||||
return assignment, nil
|
|
||||||
}
|
|
||||||
assignment.Draining = true
|
|
||||||
assignment.Version++
|
|
||||||
s.ownership[proxyID] = assignment
|
|
||||||
return assignment, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *MemoryStore) AcknowledgeDrain(proxyID, workerID string, epoch uint64, active, reserved int64) error {
|
|
||||||
if active < 0 || reserved < 0 {
|
|
||||||
return ownershipDomain.ErrInvalidOwnership
|
|
||||||
}
|
|
||||||
s.mu.Lock()
|
|
||||||
defer s.mu.Unlock()
|
|
||||||
assignment, ok := s.ownership[proxyID]
|
|
||||||
if !ok || assignment.WorkerID != workerID || assignment.Epoch != epoch {
|
|
||||||
return ownershipDomain.ErrStaleAssignment
|
|
||||||
}
|
|
||||||
if !assignment.Draining {
|
|
||||||
return ownershipDomain.ErrNotDraining
|
|
||||||
}
|
|
||||||
if active > 0 || reserved > 0 {
|
|
||||||
return ownershipDomain.ErrDrainNotReady
|
|
||||||
}
|
|
||||||
if candidate, exists := s.candidates[proxyID]; exists && candidate.OwnerWorkerID == workerID {
|
|
||||||
candidate.OwnerWorkerID = ""
|
|
||||||
s.candidates[proxyID] = candidate
|
|
||||||
}
|
|
||||||
delete(s.ownership, proxyID)
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *MemoryStore) Get(proxyID string) (ownershipDomain.Assignment, bool) {
|
|
||||||
s.mu.Lock()
|
|
||||||
defer s.mu.Unlock()
|
|
||||||
assignment, ok := s.ownership[proxyID]
|
|
||||||
return assignment, ok
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *MemoryStore) Expire(now time.Time) []ownershipDomain.Assignment {
|
|
||||||
s.mu.Lock()
|
|
||||||
defer s.mu.Unlock()
|
|
||||||
expired := make([]ownershipDomain.Assignment, 0)
|
|
||||||
for proxyID, assignment := range s.ownership {
|
|
||||||
if assignment.ExpiresAt.After(now) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
if candidate, ok := s.candidates[proxyID]; ok && candidate.OwnerWorkerID == assignment.WorkerID {
|
|
||||||
candidate.OwnerWorkerID = ""
|
|
||||||
s.candidates[proxyID] = candidate
|
|
||||||
}
|
|
||||||
expired = append(expired, assignment)
|
|
||||||
delete(s.ownership, proxyID)
|
|
||||||
}
|
|
||||||
sort.Slice(expired, func(i, j int) bool { return expired[i].ProxyID < expired[j].ProxyID })
|
|
||||||
return expired
|
|
||||||
}
|
|
||||||
|
|
||||||
func (s *MemoryStore) Records() []Record {
|
|
||||||
s.mu.Lock()
|
|
||||||
defer s.mu.Unlock()
|
|
||||||
return append([]Record(nil), s.records...)
|
|
||||||
}
|
|
||||||
|
|
||||||
func cloneResult(result Result) Result {
|
|
||||||
result.Items = append([]Candidate(nil), result.Items...)
|
|
||||||
return result
|
|
||||||
}
|
|
||||||
|
|
||||||
func cloneCommand(command Command) Command {
|
|
||||||
command.Protocols = append([]string(nil), command.Protocols...)
|
|
||||||
command.Regions = append([]string(nil), command.Regions...)
|
|
||||||
command.Carriers = append([]string(nil), command.Carriers...)
|
|
||||||
command.Upstreams = append([]string(nil), command.Upstreams...)
|
|
||||||
return command
|
|
||||||
}
|
|
||||||
|
|
||||||
func sameIdempotentRequest(left, right Command) bool {
|
|
||||||
return left.Requested == right.Requested &&
|
|
||||||
left.Fulfillment == right.Fulfillment &&
|
|
||||||
equalSet(left.Protocols, right.Protocols) &&
|
|
||||||
equalSet(left.Regions, right.Regions) &&
|
|
||||||
equalSet(left.Carriers, right.Carriers) &&
|
|
||||||
equalSet(left.Upstreams, right.Upstreams)
|
|
||||||
}
|
|
||||||
|
|
||||||
func equalSet(left, right []string) bool {
|
|
||||||
if len(left) != len(right) {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
counts := make(map[string]int, len(left))
|
|
||||||
for _, value := range left {
|
|
||||||
counts[value]++
|
|
||||||
}
|
|
||||||
for _, value := range right {
|
|
||||||
counts[value]--
|
|
||||||
if counts[value] < 0 {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
func eligibleForExtraction(candidate Candidate, command Command) bool {
|
|
||||||
if candidate.State != Available || candidate.OwnerWorkerID != "" {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
if !candidate.ExpiresAt.IsZero() && candidate.ExpiresAt.Sub(command.Now) < command.MinRemainingTTL {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
if command.MaxHealthCheckAge > 0 && command.Now.Sub(candidate.LastCheckedAt) > command.MaxHealthCheckAge {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
return matches(command.Protocols, candidate.Protocol) &&
|
|
||||||
matches(command.Regions, candidate.Region) &&
|
|
||||||
matches(command.Carriers, candidate.Carrier) &&
|
|
||||||
matches(command.Upstreams, candidate.Upstream)
|
|
||||||
}
|
|
||||||
|
|
||||||
func matches(allowed []string, value string) bool {
|
|
||||||
if len(allowed) == 0 {
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
for _, candidate := range allowed {
|
|
||||||
if candidate == value {
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|||||||
@ -1,246 +1,12 @@
|
|||||||
package extraction
|
package extraction
|
||||||
|
|
||||||
import (
|
import "testing"
|
||||||
"context"
|
|
||||||
"errors"
|
|
||||||
"sync"
|
|
||||||
"testing"
|
|
||||||
"time"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestMemoryStoreNeverExtractsProxyTwice(t *testing.T) {
|
func TestWireValuesRemainStable(t *testing.T) {
|
||||||
now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC)
|
if Partial != "partial" || AllOrNothing != "allOrNothing" {
|
||||||
store := NewMemoryStore([]Candidate{
|
t.Fatalf("fulfillment values = %q, %q", Partial, AllOrNothing)
|
||||||
{ID: "p1", State: Available, ExpiresAt: now.Add(time.Minute), LastCheckedAt: now},
|
|
||||||
})
|
|
||||||
|
|
||||||
var wg sync.WaitGroup
|
|
||||||
results := make(chan string, 1000)
|
|
||||||
for range 1000 {
|
|
||||||
wg.Add(1)
|
|
||||||
go func() {
|
|
||||||
defer wg.Done()
|
|
||||||
result, err := store.Extract(context.Background(), Command{
|
|
||||||
Requested: 1,
|
|
||||||
Fulfillment: Partial,
|
|
||||||
Now: now,
|
|
||||||
MinRemainingTTL: 30 * time.Second,
|
|
||||||
MaxHealthCheckAge: 10 * time.Second,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("Extract(): %v", err)
|
|
||||||
return
|
|
||||||
}
|
|
||||||
for _, item := range result.Items {
|
|
||||||
results <- item.ID
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
}
|
}
|
||||||
wg.Wait()
|
if Available != "AVAILABLE" || Extracted != "EXTRACTED" {
|
||||||
close(results)
|
t.Fatalf("state values = %q, %q", Available, Extracted)
|
||||||
|
|
||||||
count := 0
|
|
||||||
for id := range results {
|
|
||||||
if id != "p1" {
|
|
||||||
t.Fatalf("unexpected proxy %q", id)
|
|
||||||
}
|
|
||||||
count++
|
|
||||||
}
|
|
||||||
if count != 1 {
|
|
||||||
t.Fatalf("proxy extracted %d times, want exactly once", count)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestAllOrNothingDoesNotConsumePartialInventory(t *testing.T) {
|
|
||||||
now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC)
|
|
||||||
store := NewMemoryStore([]Candidate{
|
|
||||||
{ID: "p1", State: Available, ExpiresAt: now.Add(time.Minute), LastCheckedAt: now},
|
|
||||||
})
|
|
||||||
|
|
||||||
result, err := store.Extract(context.Background(), Command{
|
|
||||||
Requested: 2,
|
|
||||||
Fulfillment: AllOrNothing,
|
|
||||||
Now: now,
|
|
||||||
MinRemainingTTL: 30 * time.Second,
|
|
||||||
MaxHealthCheckAge: 10 * time.Second,
|
|
||||||
})
|
|
||||||
if err != ErrInsufficientProxies {
|
|
||||||
t.Fatalf("Extract() error = %v, want ErrInsufficientProxies", err)
|
|
||||||
}
|
|
||||||
if len(result.Items) != 0 {
|
|
||||||
t.Fatalf("Extract() returned %d items, want 0", len(result.Items))
|
|
||||||
}
|
|
||||||
|
|
||||||
partial, err := store.Extract(context.Background(), Command{
|
|
||||||
Requested: 1,
|
|
||||||
Fulfillment: Partial,
|
|
||||||
Now: now,
|
|
||||||
MinRemainingTTL: 30 * time.Second,
|
|
||||||
MaxHealthCheckAge: 10 * time.Second,
|
|
||||||
})
|
|
||||||
if err != nil || len(partial.Items) != 1 {
|
|
||||||
t.Fatalf("inventory was consumed by failed all-or-nothing: result=%+v err=%v", partial, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestMemoryStoreCommitsAuditWithExtraction(t *testing.T) {
|
|
||||||
now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC)
|
|
||||||
store := NewMemoryStore([]Candidate{{
|
|
||||||
ID: "p1",
|
|
||||||
Upstream: "provider-a",
|
|
||||||
State: Available,
|
|
||||||
ExpiresAt: now.Add(time.Minute),
|
|
||||||
LastCheckedAt: now,
|
|
||||||
}})
|
|
||||||
|
|
||||||
_, err := store.Extract(context.Background(), Command{
|
|
||||||
RequestID: "req-1",
|
|
||||||
ClientID: "client-1",
|
|
||||||
SourceIP: "192.0.2.30",
|
|
||||||
Requested: 1,
|
|
||||||
Fulfillment: Partial,
|
|
||||||
Now: now,
|
|
||||||
MinRemainingTTL: 30 * time.Second,
|
|
||||||
MaxHealthCheckAge: 10 * time.Second,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Extract(): %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
records := store.Records()
|
|
||||||
if len(records) != 1 {
|
|
||||||
t.Fatalf("audit records = %d, want 1", len(records))
|
|
||||||
}
|
|
||||||
record := records[0]
|
|
||||||
if record.ProxyID != "p1" || record.ClientID != "client-1" || record.RequestID != "req-1" {
|
|
||||||
t.Fatalf("audit record = %+v", record)
|
|
||||||
}
|
|
||||||
if !record.ExtractedAt.Equal(now) || !record.ExpiresAt.Equal(now.Add(time.Minute)) {
|
|
||||||
t.Fatalf("audit timestamps = %+v", record)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestMemoryStoreReplaysCommittedIdempotentResult(t *testing.T) {
|
|
||||||
now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC)
|
|
||||||
store := NewMemoryStore([]Candidate{{
|
|
||||||
ID: "p1",
|
|
||||||
State: Available,
|
|
||||||
ExpiresAt: now.Add(time.Minute),
|
|
||||||
LastCheckedAt: now,
|
|
||||||
}})
|
|
||||||
command := Command{
|
|
||||||
RequestID: "req-1",
|
|
||||||
ClientID: "client-1",
|
|
||||||
IdempotencyKey: "idem-12345678",
|
|
||||||
Requested: 1,
|
|
||||||
Fulfillment: Partial,
|
|
||||||
Now: now,
|
|
||||||
MinRemainingTTL: 30 * time.Second,
|
|
||||||
MaxHealthCheckAge: 10 * time.Second,
|
|
||||||
}
|
|
||||||
|
|
||||||
first, err := store.Extract(context.Background(), command)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("first Extract(): %v", err)
|
|
||||||
}
|
|
||||||
command.RequestID = "req-2"
|
|
||||||
second, err := store.Extract(context.Background(), command)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("second Extract(): %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if len(first.Items) != 1 || len(second.Items) != 1 || second.Items[0].ID != first.Items[0].ID {
|
|
||||||
t.Fatalf("idempotent results: first=%+v second=%+v", first, second)
|
|
||||||
}
|
|
||||||
if got := len(store.Records()); got != 1 {
|
|
||||||
t.Fatalf("audit records = %d, want 1", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestMemoryStoreRejectsIdempotencyKeyReuseWithDifferentRequest(t *testing.T) {
|
|
||||||
now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC)
|
|
||||||
store := NewMemoryStore([]Candidate{{
|
|
||||||
ID: "p1",
|
|
||||||
State: Available,
|
|
||||||
ExpiresAt: now.Add(time.Minute),
|
|
||||||
LastCheckedAt: now,
|
|
||||||
}})
|
|
||||||
command := Command{
|
|
||||||
RequestID: "req-1",
|
|
||||||
ClientID: "client-1",
|
|
||||||
IdempotencyKey: "idem-12345678",
|
|
||||||
Requested: 1,
|
|
||||||
Fulfillment: Partial,
|
|
||||||
Now: now,
|
|
||||||
MinRemainingTTL: 30 * time.Second,
|
|
||||||
MaxHealthCheckAge: 10 * time.Second,
|
|
||||||
}
|
|
||||||
if _, err := store.Extract(context.Background(), command); err != nil {
|
|
||||||
t.Fatalf("first Extract(): %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
command.Requested = 2
|
|
||||||
if _, err := store.Extract(context.Background(), command); err != ErrIdempotencyConflict {
|
|
||||||
t.Fatalf("second Extract() error = %v, want ErrIdempotencyConflict", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestMemoryStoreExtractsOnlyUnownedProxy(t *testing.T) {
|
|
||||||
now := time.Date(2026, 7, 28, 10, 0, 0, 0, time.UTC)
|
|
||||||
store := NewMemoryStore([]Candidate{
|
|
||||||
{ID: "owned", OwnerWorkerID: "worker-1", State: Available, ExpiresAt: now.Add(time.Minute), LastCheckedAt: now},
|
|
||||||
{ID: "unowned", State: Available, ExpiresAt: now.Add(time.Minute), LastCheckedAt: now},
|
|
||||||
})
|
|
||||||
|
|
||||||
result, err := store.Extract(context.Background(), Command{
|
|
||||||
RequestID: "req-1",
|
|
||||||
ClientID: "client-1",
|
|
||||||
Requested: 2,
|
|
||||||
Fulfillment: Partial,
|
|
||||||
Now: now,
|
|
||||||
MinRemainingTTL: 30 * time.Second,
|
|
||||||
MaxHealthCheckAge: 10 * time.Second,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("Extract(): %v", err)
|
|
||||||
}
|
|
||||||
if len(result.Items) != 1 || result.Items[0].ID != "unowned" {
|
|
||||||
t.Fatalf("extracted items = %+v, want only unowned", result.Items)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestMemoryStoreValidatesCommandAndHonorsCancellation(t *testing.T) {
|
|
||||||
store := NewMemoryStore([]Candidate{{ID: "p1", State: Available}})
|
|
||||||
if _, err := store.Extract(context.Background(), Command{
|
|
||||||
Requested: 1, Fulfillment: Partial, ReserveForGateway: -1,
|
|
||||||
}); !errors.Is(err, ErrInvalidCommand) {
|
|
||||||
t.Fatalf("Extract(negative reserve) error = %v, want ErrInvalidCommand", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
ctx, cancel := context.WithCancel(context.Background())
|
|
||||||
cancel()
|
|
||||||
if _, err := store.Extract(ctx, Command{Requested: 1, Fulfillment: Partial}); !errors.Is(err, context.Canceled) {
|
|
||||||
t.Fatalf("Extract(canceled) error = %v, want context.Canceled", err)
|
|
||||||
}
|
|
||||||
result, err := store.Extract(context.Background(), Command{Requested: 1, Fulfillment: Partial})
|
|
||||||
if err != nil || result.Returned != 1 {
|
|
||||||
t.Fatalf("candidate changed after canceled request: result=%+v err=%v", result, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestMemoryStoreClonesIdempotencyCommandFilters(t *testing.T) {
|
|
||||||
store := NewMemoryStore([]Candidate{{ID: "p1", Protocol: "http", State: Available}})
|
|
||||||
protocols := []string{"http"}
|
|
||||||
command := Command{
|
|
||||||
ClientID: "client-1", IdempotencyKey: "idem-1", Requested: 1,
|
|
||||||
Fulfillment: Partial, Protocols: protocols,
|
|
||||||
}
|
|
||||||
if _, err := store.Extract(context.Background(), command); err != nil {
|
|
||||||
t.Fatalf("first Extract(): %v", err)
|
|
||||||
}
|
|
||||||
protocols[0] = "socks5"
|
|
||||||
command.Protocols = []string{"http"}
|
|
||||||
if _, err := store.Extract(context.Background(), command); err != nil {
|
|
||||||
t.Fatalf("idempotent replay after caller mutation: %v", err)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@ -27,6 +27,7 @@ type Proxy struct {
|
|||||||
SourceUpstream string
|
SourceUpstream string
|
||||||
CreatedAt time.Time
|
CreatedAt time.Time
|
||||||
ExpiresAt *time.Time
|
ExpiresAt *time.Time
|
||||||
|
UsableUntil *time.Time
|
||||||
LastCheckedAt *time.Time
|
LastCheckedAt *time.Time
|
||||||
LastSuccessAt *time.Time
|
LastSuccessAt *time.Time
|
||||||
Latency time.Duration
|
Latency time.Duration
|
||||||
|
|||||||
@ -3,7 +3,7 @@ package upstream
|
|||||||
import (
|
import (
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
)
|
)
|
||||||
|
|
||||||
type ProxyCapacity struct {
|
type ProxyCapacity struct {
|
||||||
|
|||||||
@ -4,7 +4,7 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestInventoryAvailableSlotsUsesOnlyAllocatableCapacity(t *testing.T) {
|
func TestInventoryAvailableSlotsUsesOnlyAllocatableCapacity(t *testing.T) {
|
||||||
|
|||||||
@ -5,8 +5,8 @@ import (
|
|||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/snapshot"
|
"proxy-pool/internal/gateway/snapshot"
|
||||||
)
|
)
|
||||||
|
|
||||||
var ErrNoCandidate = errors.New("no local proxy candidate is available")
|
var ErrNoCandidate = errors.New("no local proxy candidate is available")
|
||||||
|
|||||||
@ -8,8 +8,8 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/snapshot"
|
"proxy-pool/internal/gateway/snapshot"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestAcquireFiltersAndReservesLocalCapacity(t *testing.T) {
|
func TestAcquireFiltersAndReservesLocalCapacity(t *testing.T) {
|
||||||
|
|||||||
@ -4,10 +4,10 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/config"
|
"proxy-pool/internal/config"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/policy"
|
"proxy-pool/internal/gateway/policy"
|
||||||
platformAdmission "github.com/proxy-pool/proxy-pool/internal/platform/admission"
|
platformAdmission "proxy-pool/internal/platform/admission"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
|
"proxy-pool/internal/platform/httpsecurity"
|
||||||
)
|
)
|
||||||
|
|
||||||
type Protection struct {
|
type Protection struct {
|
||||||
|
|||||||
@ -7,9 +7,9 @@ import (
|
|||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/config"
|
"proxy-pool/internal/config"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/policy"
|
"proxy-pool/internal/gateway/policy"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
|
"proxy-pool/internal/platform/httpsecurity"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestBuildProtectionFromListenerConfig(t *testing.T) {
|
func TestBuildProtectionFromListenerConfig(t *testing.T) {
|
||||||
|
|||||||
@ -16,11 +16,11 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/dispatch"
|
"proxy-pool/internal/gateway/dispatch"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/policy"
|
"proxy-pool/internal/gateway/policy"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/snapshot"
|
"proxy-pool/internal/gateway/snapshot"
|
||||||
transportDomain "github.com/proxy-pool/proxy-pool/internal/gateway/transport"
|
transportDomain "proxy-pool/internal/gateway/transport"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestHTTPProxyEndToEndRetriesDialFailure(t *testing.T) {
|
func TestHTTPProxyEndToEndRetriesDialFailure(t *testing.T) {
|
||||||
|
|||||||
@ -14,11 +14,11 @@ import (
|
|||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/dispatch"
|
"proxy-pool/internal/gateway/dispatch"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/policy"
|
"proxy-pool/internal/gateway/policy"
|
||||||
transportDomain "github.com/proxy-pool/proxy-pool/internal/gateway/transport"
|
transportDomain "proxy-pool/internal/gateway/transport"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
|
"proxy-pool/internal/platform/httpsecurity"
|
||||||
)
|
)
|
||||||
|
|
||||||
type Config struct {
|
type Config struct {
|
||||||
|
|||||||
@ -15,11 +15,11 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/dispatch"
|
"proxy-pool/internal/gateway/dispatch"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/policy"
|
"proxy-pool/internal/gateway/policy"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/snapshot"
|
"proxy-pool/internal/gateway/snapshot"
|
||||||
transportDomain "github.com/proxy-pool/proxy-pool/internal/gateway/transport"
|
transportDomain "proxy-pool/internal/gateway/transport"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestHandlerRunsProtectionAndTargetPolicyBeforeRouting(t *testing.T) {
|
func TestHandlerRunsProtectionAndTargetPolicyBeforeRouting(t *testing.T) {
|
||||||
|
|||||||
@ -6,8 +6,8 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/policy"
|
"proxy-pool/internal/gateway/policy"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpsecurity"
|
"proxy-pool/internal/platform/httpsecurity"
|
||||||
)
|
)
|
||||||
|
|
||||||
type HTTPError struct {
|
type HTTPError struct {
|
||||||
|
|||||||
@ -7,8 +7,8 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/domain/routing"
|
"proxy-pool/internal/domain/routing"
|
||||||
"github.com/proxy-pool/proxy-pool/internal/gateway/dispatch"
|
"proxy-pool/internal/gateway/dispatch"
|
||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
|
|||||||
@ -7,7 +7,7 @@ import (
|
|||||||
"reflect"
|
"reflect"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/domain/routing"
|
"proxy-pool/internal/domain/routing"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestRulesRouterReturnsMatchedUpstreams(t *testing.T) {
|
func TestRulesRouterReturnsMatchedUpstreams(t *testing.T) {
|
||||||
|
|||||||
@ -11,7 +11,7 @@ import (
|
|||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
@ -270,6 +270,9 @@ func matchesQuery(candidate proxyDomain.Proxy, query Query) bool {
|
|||||||
if _, excluded := query.Exclude[candidate.ID]; excluded {
|
if _, excluded := query.Exclude[candidate.ID]; excluded {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
if candidate.UsableUntil != nil && !candidate.UsableUntil.After(query.Now) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
if candidate.ExpiresAt != nil && !candidate.ExpiresAt.After(query.Now.Add(query.SafetyMargin)) {
|
if candidate.ExpiresAt != nil && !candidate.ExpiresAt.After(query.Now.Add(query.SafetyMargin)) {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|||||||
@ -7,7 +7,7 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestStoreAppliesCompleteSnapshotsInOrder(t *testing.T) {
|
func TestStoreAppliesCompleteSnapshotsInOrder(t *testing.T) {
|
||||||
@ -119,6 +119,34 @@ func TestViewSelectFiltersBySchemeUpstreamTagAndExclude(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestViewSelectRejectsProxyAfterProviderUsableDeadline(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 29, 12, 0, 0, 0, time.UTC)
|
||||||
|
expiresAt := now.Add(30 * time.Second)
|
||||||
|
usableUntil := now.Add(27 * time.Second)
|
||||||
|
proxy := proxyDomain.Proxy{
|
||||||
|
ID: "short-lived", Scheme: proxyDomain.SchemeHTTP, State: proxyDomain.StateAvailable,
|
||||||
|
MaxConcurrency: 1, ExpiresAt: &expiresAt, UsableUntil: &usableUntil,
|
||||||
|
}
|
||||||
|
store := NewStore("cluster-a", "worker-a")
|
||||||
|
envelope := Envelope{
|
||||||
|
ClusterID: "cluster-a", WorkerID: "worker-a", Epoch: 1, Version: 1, Full: true,
|
||||||
|
Proxies: []proxyDomain.Proxy{proxy},
|
||||||
|
}
|
||||||
|
envelope.Checksum = Checksum(envelope.Proxies)
|
||||||
|
if err := store.Apply(envelope); err != nil {
|
||||||
|
t.Fatalf("Apply(): %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
before := store.Current().Select(Query{Now: now.Add(26 * time.Second)})
|
||||||
|
if _, ok := before.EntryAt(0); !ok {
|
||||||
|
t.Fatal("EntryAt(before usable deadline) = false, want true")
|
||||||
|
}
|
||||||
|
after := store.Current().Select(Query{Now: now.Add(27 * time.Second)})
|
||||||
|
if _, ok := after.EntryAt(0); ok {
|
||||||
|
t.Fatal("EntryAt(at usable deadline) = true, want false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestStoreApplyReusesCapacityAcrossVersionsAndEpochs(t *testing.T) {
|
func TestStoreApplyReusesCapacityAcrossVersionsAndEpochs(t *testing.T) {
|
||||||
store := NewStore("cluster-a", "worker-a")
|
store := NewStore("cluster-a", "worker-a")
|
||||||
firstProxies := []proxyDomain.Proxy{{
|
firstProxies := []proxyDomain.Proxy{{
|
||||||
|
|||||||
@ -4,8 +4,8 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
platformCredentials "github.com/proxy-pool/proxy-pool/internal/platform/credentials"
|
platformCredentials "proxy-pool/internal/platform/credentials"
|
||||||
)
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
|
|||||||
@ -7,8 +7,8 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
platformCredentials "github.com/proxy-pool/proxy-pool/internal/platform/credentials"
|
platformCredentials "proxy-pool/internal/platform/credentials"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestCredentialsFormattingRedactsPassword(t *testing.T) {
|
func TestCredentialsFormattingRedactsPassword(t *testing.T) {
|
||||||
|
|||||||
@ -16,7 +16,7 @@ import (
|
|||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
)
|
)
|
||||||
|
|
||||||
const (
|
const (
|
||||||
|
|||||||
@ -15,7 +15,7 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
proxyDomain "github.com/proxy-pool/proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestRoundTripForwardsHTTPViaSelectedProxy(t *testing.T) {
|
func TestRoundTripForwardsHTTPViaSelectedProxy(t *testing.T) {
|
||||||
|
|||||||
@ -4,8 +4,8 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/config"
|
"proxy-pool/internal/config"
|
||||||
platformAdmission "github.com/proxy-pool/proxy-pool/internal/platform/admission"
|
platformAdmission "proxy-pool/internal/platform/admission"
|
||||||
)
|
)
|
||||||
|
|
||||||
func BuildFromListener(listener config.Listener, clientIdentification string, semantics Semantics) (*Protection, error) {
|
func BuildFromListener(listener config.Listener, clientIdentification string, semantics Semantics) (*Protection, error) {
|
||||||
|
|||||||
@ -6,7 +6,7 @@ import (
|
|||||||
"net/http/httptest"
|
"net/http/httptest"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/config"
|
"proxy-pool/internal/config"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestNewFromListenerMapsResolvedConfiguration(t *testing.T) {
|
func TestNewFromListenerMapsResolvedConfiguration(t *testing.T) {
|
||||||
|
|||||||
@ -4,7 +4,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
|
||||||
"github.com/proxy-pool/proxy-pool/internal/platform/httpapi"
|
"proxy-pool/internal/platform/httpapi"
|
||||||
)
|
)
|
||||||
|
|
||||||
func WriteProblem(writer http.ResponseWriter, requestID string, err error) bool {
|
func WriteProblem(writer http.ResponseWriter, requestID string, err error) bool {
|
||||||
|
|||||||
13
progress.md
13
progress.md
@ -8,7 +8,14 @@
|
|||||||
`httpsecurity`,API 401 与代理 407 语义保持分离。
|
`httpsecurity`,API 401 与代理 407 语义保持分离。
|
||||||
- 部署配置已使用 `${VAR}` 真实环境变量解析,ConfigMap 与本地配置通过
|
- 部署配置已使用 `${VAR}` 真实环境变量解析,ConfigMap 与本地配置通过
|
||||||
`LoadResolved` 回归测试。
|
`LoadResolved` 回归测试。
|
||||||
- PostgreSQL/Redis Adapter、生产命令入口与代表性 100,000 QPS 集群压测仍待实现。
|
- 已新增公用 `activitypool` 契约和并发安全内存参考实现;Provider 按各供应商
|
||||||
|
TTL 与安全余量写入,独占提取、短期幂等和 Worker 所有权在同一原子边界内。
|
||||||
|
- Proxy 明细不写 PostgreSQL;PostgreSQL 仅保存配置版本、管理状态、Admin
|
||||||
|
审计、管理 outbox 和可选聚合指标。
|
||||||
|
- `usableUntil` 已进入 Worker Snapshot 契约,Gateway 在供应商硬过期前按安全
|
||||||
|
余量停止新分配。
|
||||||
|
- 生产 Redis Adapter、PostgreSQL 管理面 Adapter、生产命令入口与代表性
|
||||||
|
100,000 QPS 集群压测仍待实现。
|
||||||
|
|
||||||
## 2026-07-28
|
## 2026-07-28
|
||||||
|
|
||||||
@ -37,8 +44,8 @@
|
|||||||
Retry-After、原子 FetchBudget 以及 Pool Reconciler。
|
Retry-After、原子 FetchBudget 以及 Pool Reconciler。
|
||||||
- 已强制 Worker 所有权与 Exclusive Extraction 注入同一权威 Repository,
|
- 已强制 Worker 所有权与 Exclusive Extraction 注入同一权威 Repository,
|
||||||
覆盖 100 轮并发竞争、续期、Drain ACK 和 Worker 过期回收。
|
覆盖 100 轮并发竞争、续期、Drain ACK 和 Worker 过期回收。
|
||||||
- 已实现提取服务层策略映射、幂等审计、来源身份准入和公共 FixedWindow
|
- 已实现提取服务层策略映射、Redis 短期幂等语义、来源身份准入和公共
|
||||||
全局/Client 限流器。
|
FixedWindow 全局/Client 限流器;不保存逐代理提取审计记录。
|
||||||
- 已为 100k Proxy 快照建立 scheme/upstream/tag 索引;本机持续基准中
|
- 已为 100k Proxy 快照建立 scheme/upstream/tag 索引;本机持续基准中
|
||||||
`Acquire` 为 640 ns/op、256 B/op、2 allocs/op。该数据仅证明本地选择热路径,
|
`Acquire` 为 640 ns/op、256 B/op、2 allocs/op。该数据仅证明本地选择热路径,
|
||||||
不代表 100k QPS 集群端到端容量。
|
不代表 100k QPS 集群端到端容量。
|
||||||
|
|||||||
@ -24,6 +24,8 @@
|
|||||||
7. [已完成] 执行单元测试、静态检查、构建和静态部署/契约验证;本机因
|
7. [已完成] 执行单元测试、静态检查、构建和静态部署/契约验证;本机因
|
||||||
`CGO_ENABLED=0` 且无 C 编译器未运行 race,保留给 Linux CI
|
`CGO_ENABLED=0` 且无 C 编译器未运行 race,保留给 Linux CI
|
||||||
8. [已完成] 按需求矩阵逐项审计并生成版本化文档包
|
8. [已完成] 按需求矩阵逐项审计并生成版本化文档包
|
||||||
|
9. [已完成] 将 Proxy 明细、独占提取、短期幂等和 Worker 所有权统一到
|
||||||
|
TTL 活动池契约;PostgreSQL 退出代理数据路径
|
||||||
|
|
||||||
## 串并行关系
|
## 串并行关系
|
||||||
|
|
||||||
@ -44,5 +46,5 @@
|
|||||||
|
|
||||||
- Docker Compose 配置与 Kubernetes Kustomize 已完成静态渲染验证;未启动
|
- Docker Compose 配置与 Kubernetes Kustomize 已完成静态渲染验证;未启动
|
||||||
目标运行拓扑。
|
目标运行拓扑。
|
||||||
- `cmd/proxy-*`、Provider 调度、PostgreSQL/Redis 适配器、Gateway Transport
|
- `cmd/proxy-*`、生产 Redis 活动池 Adapter、PostgreSQL 管理面 Adapter 与
|
||||||
与 Checker 运行时属于后续实施范围,见完成审计。
|
Checker 运行时属于后续实施范围,见完成审计。
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user