feat: report gateway proxy outcomes
This commit is contained in:
parent
155411ef23
commit
6766097ea7
15
README.md
15
README.md
@ -21,7 +21,7 @@ Proxy Pool 用 Controller 协调这些变化,并让 Gateway 数据面只消费
|
|||||||
|
|
||||||
- **Gateway**:调用方连接平台,由平台选择上游代理并转发 HTTP 或 HTTPS
|
- **Gateway**:调用方连接平台,由平台选择上游代理并转发 HTTP 或 HTTPS
|
||||||
CONNECT。`proxy-gateway` 已装配本地监听、指标探针和控制面 Register/Watch/ACK/
|
CONNECT。`proxy-gateway` 已装配本地监听、指标探针和控制面 Register/Watch/ACK/
|
||||||
Runtime 会话;带凭据 Proxy 分发仍待闭环。
|
Runtime/Outcome 会话;带凭据 Proxy 的分发与当前内存 View 已闭环。
|
||||||
- **Distribution**:调用方按条件提取真实代理;成功提取即独占消费,不支持归还、
|
- **Distribution**:调用方按条件提取真实代理;成功提取即独占消费,不支持归还、
|
||||||
续租或状态查询。
|
续租或状态查询。
|
||||||
- **Admin**:运维人员查询状态、启停 Upstream、切换 Routing,并触发严格配置
|
- **Admin**:运维人员查询状态、启停 Upstream、切换 Routing,并触发严格配置
|
||||||
@ -43,7 +43,9 @@ Proxy Pool 用 Controller 协调这些变化,并让 Gateway 数据面只消费
|
|||||||
审计与 Outbox;不保存 Proxy 明细或逐次提取记录。
|
审计与 Outbox;不保存 Proxy 明细或逐次提取记录。
|
||||||
- **Gateway 组件**:HTTP 正向代理、HTTPS CONNECT、双向 Tunnel、重试、超时、
|
- **Gateway 组件**:HTTP 正向代理、HTTPS CONNECT、双向 Tunnel、重试、超时、
|
||||||
目的地址保护、本地快照存储、容量调度和 Worker 控制面 Register/Watch/ACK/
|
目的地址保护、本地快照存储、容量调度和 Worker 控制面 Register/Watch/ACK/
|
||||||
Runtime 会话组件已有实现与定向测试。`SessionSupervisor` 会为可恢复控制面中断执行
|
Runtime/Outcome 会话组件已有实现与定向测试。每次代理尝试只向本地有界队列写入
|
||||||
|
Outcome,微批确认失败会重发同一序列;队列满时丢弃观测样本,不阻塞转发请求。
|
||||||
|
`SessionSupervisor` 会为可恢复控制面中断执行
|
||||||
有界退避重连。Controller 可向 Worker 下发已归属 Proxy、Gateway Routing 与按引用去重的
|
有界退避重连。Controller 可向 Worker 下发已归属 Proxy、Gateway Routing 与按引用去重的
|
||||||
凭据材料快照。Gateway 会将 Routing、Proxy 与凭据原子编译为同一内存 View,并只按当前未过期
|
凭据材料快照。Gateway 会将 Routing、Proxy 与凭据原子编译为同一内存 View,并只按当前未过期
|
||||||
View 匹配请求,并在内存中按 Sequential、Random、Round Robin、Weighted 或 Least
|
View 匹配请求,并在内存中按 Sequential、Random、Round Robin、Weighted 或 Least
|
||||||
@ -78,18 +80,17 @@ flowchart LR
|
|||||||
|
|
||||||
## 当前完成度
|
## 当前完成度
|
||||||
|
|
||||||
截至 **2026-07-31**,实施计划检查项为 **55 / 74(74.3%)**。详情见
|
截至 **2026-07-31**,实施计划检查项为 **56 / 74(75.7%)**。详情见
|
||||||
[实施计划](docs/development/implementation-plan.md)和
|
[实施计划](docs/development/implementation-plan.md)和
|
||||||
[交付完成度审计](docs/requirements/completion-audit.md)。
|
[交付完成度审计](docs/requirements/completion-audit.md)。
|
||||||
|
|
||||||
- **已完成**:严格配置、Provider 获取与协调、Redis 活动池、Distribution 原子
|
- **已完成**:严格配置、Provider 获取与协调、Redis 活动池、Distribution 原子
|
||||||
提取与限流、Controller 的 Admin/Distribution/Metrics 监听,以及 PostgreSQL
|
提取与限流、Controller 的 Admin/Distribution/Metrics 监听,以及 PostgreSQL
|
||||||
管理状态;WorkerControlPlane 的 Register、Snapshot ACK、Runtime 心跳接收和
|
管理状态;WorkerControlPlane 的 Register、Snapshot ACK、Runtime 心跳接收和
|
||||||
Redis 会话栅栏。
|
Redis 会话栅栏,以及 Gateway Outcome 上报的有界队列、序列确认与重试。
|
||||||
- **部分完成**:Gateway Outcome 上报、Checker 调度与健康状态链、Docker Compose/
|
- **部分完成**:Checker 调度与健康状态链、Docker Compose/
|
||||||
Kubernetes 运行时 mTLS Overlay,以及 protobuf 契约。
|
Kubernetes 运行时 mTLS Overlay,以及 protobuf 契约。
|
||||||
- **待完成**:Outcome 上报、Checker 调度与健康状态链,
|
- **待完成**:Checker 调度与健康状态链,以及 loadgen 和代表性集群压测。
|
||||||
以及 loadgen 和代表性集群压测。
|
|
||||||
|
|
||||||
检查项数量不等于生产就绪度。静态部署清单与 protobuf descriptor 验证也不代表
|
检查项数量不等于生产就绪度。静态部署清单与 protobuf descriptor 验证也不代表
|
||||||
端到端拓扑已经完成;`100,000 QPS` 仍只是待验证的集群设计目标。
|
端到端拓扑已经完成;`100,000 QPS` 仍只是待验证的集群设计目标。
|
||||||
|
|||||||
@ -35,10 +35,13 @@ Gateway 已将该 payload 编译并原子发布到与 Proxy 相同版本的本
|
|||||||
当前未过期 View;派发器已按五种策略从该 View 选择上游,且在 Proxy 容量耗尽时只在该
|
当前未过期 View;派发器已按五种策略从该 View 选择上游,且在 Proxy 容量耗尽时只在该
|
||||||
View 的其余候选中回退。`wait_timeout` 随 `on_unavailable=WAIT` 下发并在 Gateway 作为有界
|
View 的其余候选中回退。`wait_timeout` 随 `on_unavailable=WAIT` 下发并在 Gateway 作为有界
|
||||||
本地容量等待使用;`DIRECT` 仍先经过 TargetPolicy 再建立 HTTP/CONNECT 直连。`proxy-gateway`
|
本地容量等待使用;`DIRECT` 仍先经过 TargetPolicy 再建立 HTTP/CONNECT 直连。`proxy-gateway`
|
||||||
已装配 Register/Watch/ACK/Runtime 会话、HTTP 代理监听和 Snapshot 就绪探针;控制面中断时
|
已装配 Register/Watch/ACK/Runtime/Outcome 会话、HTTP 代理监听和 Snapshot 就绪探针;控制面中断时
|
||||||
保持进程运行并以有界退避重连,未取得有效 Snapshot 的 Worker 不会 Ready。增量、带凭据
|
保持进程运行并以有界退避重连,未取得有效 Snapshot 的 Worker 不会 Ready。`ReportOutcomes`
|
||||||
Outcome 与 Checker 闭环尚未实现。`ReportOutcomes` 仍明确返回 `Unimplemented`;
|
已实现为客户端流:Controller 校验流内固定的 Worker/Session 身份,对每个批次以
|
||||||
`100,000 QPS` 仍是未验证的设计目标。
|
`(session_id, sequence, SHA-256)` 建立 Redis 原子栅栏,并只返回最后确认的序列。
|
||||||
|
Redis 只保留每个当前会话的一条序列和摘要;原始 Outcome、代理明细与逐请求记录均不写入
|
||||||
|
Redis 或 PostgreSQL。Gateway 只将结果写入本地有界队列,队列满时丢弃样本,不等待控制面
|
||||||
|
或存储。Checker 闭环尚未实现;`100,000 QPS` 仍是未验证的设计目标。
|
||||||
|
|
||||||
`WatchSnapshots` 建立时校验当前 session;每次签发快照引用时也把 `session_id`
|
`WatchSnapshots` 建立时校验当前 session;每次签发快照引用时也把 `session_id`
|
||||||
交给 Redis 原子校验。重复 Register 会同时清除旧 Runtime 和已签发引用,因此迟到的
|
交给 Redis 原子校验。重复 Register 会同时清除旧 Runtime 和已签发引用,因此迟到的
|
||||||
@ -137,8 +140,15 @@ Outcome 按 Worker 单调 `sequence` 批量上报。Controller 返回已接受
|
|||||||
- `TUNNEL`:隧道建立后结束或失败。
|
- `TUNNEL`:隧道建立后结束或失败。
|
||||||
|
|
||||||
Outcome 是 Observation,不直接让 Worker 修改 Redis 活动池状态,也不创建
|
Outcome 是 Observation,不直接让 Worker 修改 Redis 活动池状态,也不创建
|
||||||
PostgreSQL Proxy 明细。异步上报队列必须有界;队列满时丢弃低价值样本并计
|
PostgreSQL Proxy 明细。Gateway 将每次尝试写入进程内有界队列,当前默认容量为
|
||||||
指标,不能反压 Gateway 热路径。
|
`65536` 个事件,单批最多为 `min(512, maxRuntimeCounters)`;一次 ReportOutcomes RPC
|
||||||
|
最多合并 16 个已就绪批次。队列满时只递增本地丢弃计数,不反压 Gateway 热路径。
|
||||||
|
|
||||||
|
同一 session 内,Controller 仅接受递增 `sequence`;相同序列且摘要一致视为幂等
|
||||||
|
重放,相同序列且摘要不同返回 `AlreadyExists`,较小序列返回 `Aborted`。Gateway 在
|
||||||
|
未收到确认时保留并重发完全相同的批次。暂态传输错误在当前 session 内按有上限的
|
||||||
|
退避重试;会话栅栏错误交由 session supervisor 重建 session。栅栏随 session 替换
|
||||||
|
或过期清理,因此长期在线 Worker 不会因独立 Outcome TTL 接受旧序列。
|
||||||
|
|
||||||
## 6. Checker 任务
|
## 6. Checker 任务
|
||||||
|
|
||||||
|
|||||||
@ -190,6 +190,11 @@ Gateway 连接 Controller 时使用独立启动参数而非 `controlPlane.listen
|
|||||||
模板保持 `controlPlane.enabled: false`,环境 Overlay 挂载 mTLS 证书并启用后才可启动
|
模板保持 `controlPlane.enabled: false`,环境 Overlay 挂载 mTLS 证书并启用后才可启动
|
||||||
Gateway。
|
Gateway。
|
||||||
|
|
||||||
|
`maxRuntimeCounters` 同时限制单个 Runtime 报告和单个 Outcome 批次的条目数。Gateway
|
||||||
|
在本地维护容量为 `65536` 的非阻塞 Outcome 队列,默认微批上限为 `512`,实际取二者中
|
||||||
|
较小值;该队列与其序列确认状态仅存在于 Gateway 进程内。Controller 的 Redis 状态只保存
|
||||||
|
当前 Worker session 的最后确认序列和摘要,PostgreSQL 不保存 Outcome、代理明细或逐请求记录。
|
||||||
|
|
||||||
## 5. Gateway
|
## 5. Gateway
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
|
|||||||
@ -223,7 +223,7 @@ payload 已按配置顺序和 Admin revision/current 状态发布并覆盖 check
|
|||||||
Proxy 原子编译为同版本 View,动态 Router 只匹配该未过期 View。派发器的五种上游选择已
|
Proxy 原子编译为同版本 View,动态 Router 只匹配该未过期 View。派发器的五种上游选择已
|
||||||
接入该 View,并在容量耗尽时在同版本候选中回退;`onUnavailable` 的 reject、wait 与 direct
|
接入该 View,并在容量耗尽时在同版本候选中回退;`onUnavailable` 的 reject、wait 与 direct
|
||||||
已接入 Gateway;`proxy-gateway` 已装配本地 HTTP/Metrics 监听、快照就绪探针和
|
已接入 Gateway;`proxy-gateway` 已装配本地 HTTP/Metrics 监听、快照就绪探针和
|
||||||
控制面重连与快照凭据分发,Outcome 上报仍未实现。
|
控制面重连与快照凭据分发,以及有界 Outcome 上报、序列确认和精确重试。
|
||||||
|
|
||||||
已新增公用 `domain/activitypool` 契约及并发安全内存参考实现,Provider
|
已新增公用 `domain/activitypool` 契约及并发安全内存参考实现,Provider
|
||||||
Reconciler 通过 `UpsertFetched` 写入带供应商 TTL 和分配安全余量的批次;已覆盖
|
Reconciler 通过 `UpsertFetched` 写入带供应商 TTL 和分配安全余量的批次;已覆盖
|
||||||
@ -261,7 +261,8 @@ Controller 多副本共享同一计数。Client 身份只以 SHA-256 摘要进
|
|||||||
Gateway 请求热路径仍只使用本地准入,不增加 Redis/PostgreSQL 调用。
|
Gateway 请求热路径仍只使用本地准入,不增加 Redis/PostgreSQL 调用。
|
||||||
WorkerControlPlane gRPC 接收端、session 签发/心跳、Snapshot ACK 账本、基础
|
WorkerControlPlane gRPC 接收端、session 签发/心跳、Snapshot ACK 账本、基础
|
||||||
Snapshot 流、Gateway 会话客户端与快照凭据分发已完成;权威 Proxy/Routing 发布、
|
Snapshot 流、Gateway 会话客户端与快照凭据分发已完成;权威 Proxy/Routing 发布、
|
||||||
Outcome 和健康执行链仍待完成,因此 Task 10 尚未全部完成。
|
Outcome 上报已完成为有界队列、批次序列/摘要栅栏和确认重试;健康执行链仍待完成,
|
||||||
|
因此 Task 10 尚未全部完成。
|
||||||
|
|
||||||
## Task 11: Checker and Health Reducer
|
## Task 11: Checker and Health Reducer
|
||||||
|
|
||||||
|
|||||||
@ -3,6 +3,9 @@
|
|||||||
## 1. 运行边界
|
## 1. 运行边界
|
||||||
|
|
||||||
- Gateway 是数据面,正常请求热路径不访问 PostgreSQL、Redis 或 Provider。
|
- Gateway 是数据面,正常请求热路径不访问 PostgreSQL、Redis 或 Provider。
|
||||||
|
- Gateway 的 Proxy Outcome 仅进入进程内有界队列;Controller 只维护当前 session
|
||||||
|
的最后确认序列和摘要,原始 Outcome 不写入 Redis 或 PostgreSQL。队列满时丢弃
|
||||||
|
观测样本,代理转发和 Redis session TTL 不受影响。
|
||||||
- Controller 编排 Fetch、生命周期、所有权、Snapshot 与 Extract;多个副本只有
|
- Controller 编排 Fetch、生命周期、所有权、Snapshot 与 Extract;多个副本只有
|
||||||
一个 Provider 逻辑 Leader。短效 Proxy 明细只存在于 Redis TTL 活动池和节点
|
一个 Provider 逻辑 Leader。短效 Proxy 明细只存在于 Redis TTL 活动池和节点
|
||||||
内存,可由 Provider 重建。
|
内存,可由 Provider 重建。
|
||||||
|
|||||||
@ -38,7 +38,7 @@
|
|||||||
- `CAP-001 / GW 热路径边界`:本地 Dispatch 条件过滤与原子容量预留。
|
- `CAP-001 / GW 热路径边界`:本地 Dispatch 条件过滤与原子容量预留。
|
||||||
- `GW-*`:HTTP 正向代理、HTTPS CONNECT、双向 Tunnel、超时、重试、保护链与
|
- `GW-*`:HTTP 正向代理、HTTPS CONNECT、双向 Tunnel、超时、重试、保护链与
|
||||||
优雅停机 Handler 已实现并通过定向测试。`proxy-gateway` 已装配本地 HTTP/Metrics
|
优雅停机 Handler 已实现并通过定向测试。`proxy-gateway` 已装配本地 HTTP/Metrics
|
||||||
监听、快照就绪探针、显式控制面拨号和 Register/Watch/ACK/Runtime 会话;本地 gRPC
|
监听、快照就绪探针、显式控制面拨号和 Register/Watch/ACK/Runtime/Outcome 会话;本地 gRPC
|
||||||
集成测试验证收到有效 Snapshot 后才 Ready,并可完成 direct 路由。
|
集成测试验证收到有效 Snapshot 后才 Ready,并可完成 direct 路由。
|
||||||
- `PROVIDER-*`:Provider HTTP Client、严格响应上限、模板解析安全边界、凭据
|
- `PROVIDER-*`:Provider HTTP Client、严格响应上限、模板解析安全边界、凭据
|
||||||
引用 Store 与 Reconciler Adapter 已实现。
|
引用 Store 与 Reconciler Adapter 已实现。
|
||||||
@ -56,7 +56,8 @@
|
|||||||
索引构建已归属 Proxy payload,并以租约收紧可用期;凭据材料按引用去重并与 Proxy/
|
索引构建已归属 Proxy payload,并以租约收紧可用期;凭据材料按引用去重并与 Proxy/
|
||||||
Routing 一起纳入 Snapshot checksum,仅在 mTLS 和 Gateway 当前内存 View 中处理。Gateway
|
Routing 一起纳入 Snapshot checksum,仅在 mTLS 和 Gateway 当前内存 View 中处理。Gateway
|
||||||
Routing payload 已由配置顺序和 Admin 当前状态合成并覆盖 Snapshot checksum,其运行时消费、
|
Routing payload 已由配置顺序和 Admin 当前状态合成并覆盖 Snapshot checksum,其运行时消费、
|
||||||
Outcome 和 Checker 尚未闭环。Snapshot 签发在 Redis 中原子匹配当前
|
Outcome 已实现为 Gateway 本地有界队列、微批确认重试和 Controller 的 session/sequence/
|
||||||
|
摘要 Redis 栅栏;原始事件不落 Redis 或 PostgreSQL。Checker 尚未闭环。Snapshot 签发在 Redis 中原子匹配当前
|
||||||
`session_id`,重注册会清除旧引用,迟到旧 Stream 不会覆盖新 session。Controller
|
`session_id`,重注册会清除旧引用,迟到旧 Stream 不会覆盖新 session。Controller
|
||||||
在最近成功下发的 Snapshot `valid_until` 到达时关闭流;Gateway 的公用
|
在最近成功下发的 Snapshot `valid_until` 到达时关闭流;Gateway 的公用
|
||||||
`SessionSupervisor` 已实现可恢复错误的有界退避重连。Gateway 会校验并执行 Snapshot
|
`SessionSupervisor` 已实现可恢复错误的有界退避重连。Gateway 会校验并执行 Snapshot
|
||||||
@ -105,7 +106,7 @@ Controller/Gateway 入口,完整 mTLS 运行时拓扑仍只有静态验证。
|
|||||||
分布式请求额度、Distribution Client 限制和 Provider 状态丢失重建已完成。
|
分布式请求额度、Distribution Client 限制和 Provider 状态丢失重建已完成。
|
||||||
6. Worker 基础网络快照流、Proxy/Gateway Routing/凭据 Snapshot payload、Gateway Snapshot
|
6. Worker 基础网络快照流、Proxy/Gateway Routing/凭据 Snapshot payload、Gateway Snapshot
|
||||||
客户端和进程装配、同版本 Routing 编译/动态匹配、五种策略上游选择与 reject/wait/direct 已完成;
|
客户端和进程装配、同版本 Routing 编译/动态匹配、五种策略上游选择与 reject/wait/direct 已完成;
|
||||||
Outcome 上报仍待完成。Redis ownership drain/ACK/过期回收及按 Worker 的可下发索引已完成。
|
Outcome 上报已完成基础观测链;Redis ownership drain/ACK/过期回收及按 Worker 的可下发索引已完成。
|
||||||
7. Checker 调度、探测器和健康 reducer。
|
7. Checker 调度、探测器和健康 reducer。
|
||||||
8. Admin/Distribution 细粒度授权和审计查询;Distribution 分布式限流已完成。
|
8. Admin/Distribution 细粒度授权和审计查询;Distribution 分布式限流已完成。
|
||||||
9. 真实 Compose/Kubernetes 集成、故障演练和代表性集群负载测试。
|
9. 真实 Compose/Kubernetes 集成、故障演练和代表性集群负载测试。
|
||||||
|
|||||||
@ -7,8 +7,8 @@
|
|||||||
|
|
||||||
| ID | 最终需求 | 来源 | 验证证据 |
|
| ID | 最终需求 | 来源 | 验证证据 |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| ARCH-001 | 数据面 Worker 与控制面 Controller 分离 | 1-70 | 包、协议和部署拓扑已分离;Controller 已运行 Worker Register/Watch/ACK/Runtime gRPC,并发布 Proxy/Gateway Routing/按引用去重凭据完整快照;Gateway 已将它们编译为同版本动态 View,并由独立进程维护控制面会话。Checker/Loadgen 构建产物待实现 |
|
| ARCH-001 | 数据面 Worker 与控制面 Controller 分离 | 1-70 | 包、协议和部署拓扑已分离;Controller 已运行 Worker Register/Watch/ACK/Runtime/Outcome gRPC,并发布 Proxy/Gateway Routing/按引用去重凭据完整快照;Gateway 已将它们编译为同版本动态 View,并由独立进程维护控制面会话。Checker/Loadgen 构建产物待实现 |
|
||||||
| ARCH-002 | 热路径只做认证、本地路由和网络转发 | 1-70, 380-430 | Gateway bootstrap 集成测试验证启动期控制面会话与快照就绪,HTTP 请求只走本地 Snapshot/Dispatch;代表性性能剖析待完成 |
|
| ARCH-002 | 热路径只做认证、本地路由和网络转发 | 1-70, 380-430 | Gateway bootstrap 集成测试验证启动期控制面会话与快照就绪,HTTP 请求只走本地 Snapshot/Dispatch;Outcome 仅写入有界非阻塞本地队列,代表性性能剖析待完成 |
|
||||||
| ARCH-003 | Gateway、Distribution、Admin、Metrics 独立入口 | 8904-8958 | Controller 命令已装配 Distribution/Admin/Metrics 三个独立监听及联动停机;Gateway 命令已装配代理与 Metrics 监听,运行时 mTLS 部署 Overlay 待完成 |
|
| ARCH-003 | Gateway、Distribution、Admin、Metrics 独立入口 | 8904-8958 | Controller 命令已装配 Distribution/Admin/Metrics 三个独立监听及联动停机;Gateway 命令已装配代理与 Metrics 监听,运行时 mTLS 部署 Overlay 待完成 |
|
||||||
| ARCH-004 | Controller 集中 Provider 获取与切换 | 1403-1580 | Redis Leader、动态 Provider Supervisor 与 Bootstrap 生产装配已完成;Admin disable/reload 驱动取消替换,多副本按权威 HMAC 指纹和 revision 栅栏收敛并拒绝旧配置换主;Routing 切换到 Drain 的编排待完成 |
|
| ARCH-004 | Controller 集中 Provider 获取与切换 | 1403-1580 | Redis Leader、动态 Provider Supervisor 与 Bootstrap 生产装配已完成;Admin disable/reload 驱动取消替换,多副本按权威 HMAC 指纹和 revision 栅栏收敛并拒绝旧配置换主;Routing 切换到 Drain 的编排待完成 |
|
||||||
| ARCH-005 | 100k QPS 峰值使用多 Worker 集群 | 当前会话 | 未验证设计目标;待代表性集群负载报告 |
|
| ARCH-005 | 100k QPS 峰值使用多 Worker 集群 | 当前会话 | 未验证设计目标;待代表性集群负载报告 |
|
||||||
@ -60,6 +60,7 @@
|
|||||||
| GW-003 | CONNECT 建立后不得透明重放 | 221-300 | `handler_test.go` 200 前重试、200 后中继故障不重放、半关闭测试 |
|
| GW-003 | CONNECT 建立后不得透明重放 | 221-300 | `handler_test.go` 200 前重试、200 后中继故障不重放、半关闭测试 |
|
||||||
| GW-004 | Client 认证可关闭,但访问控制、身份识别和限流独立 | 8112-8441 | `protection_test.go`、`bootstrap_test.go` 与入口并发上限测试 |
|
| GW-004 | Client 认证可关闭,但访问控制、身份识别和限流独立 | 8112-8441 | `protection_test.go`、`bootstrap_test.go` 与入口并发上限测试 |
|
||||||
| GW-005 | 防私网、回环、链路本地、保留/元数据地址、任意 CONNECT 端口和 DNS Rebinding | 8904-8931 | `policy/target_test.go` 解析前后校验、端口白名单、混合 DNS 结果与已验证 IP 绑定测试 |
|
| GW-005 | 防私网、回环、链路本地、保留/元数据地址、任意 CONNECT 端口和 DNS Rebinding | 8904-8931 | `policy/target_test.go` 解析前后校验、端口白名单、混合 DNS 结果与已验证 IP 绑定测试 |
|
||||||
|
| GW-006 | Proxy Outcome 不反压转发并按会话序列去重 | 当前会话 | `gateway/outcome` 队列溢出与微批测试、`controlplane/outcome_reporter_test.go` 精确重试测试、Worker gRPC 和 Memory/Redis 共享栅栏契约 |
|
||||||
|
|
||||||
## Distribution
|
## Distribution
|
||||||
|
|
||||||
|
|||||||
@ -46,7 +46,7 @@
|
|||||||
- Distribution 原子提取后 Gateway 不再分配同一 Proxy。
|
- Distribution 原子提取后 Gateway 不再分配同一 Proxy。
|
||||||
- 配置热更新失败保留旧 Revision,成功后新请求使用新 Snapshot。
|
- 配置热更新失败保留旧 Revision,成功后新请求使用新 Snapshot。
|
||||||
|
|
||||||
## 3. 必测的 11 类场景
|
## 3. 必测的 12 类场景
|
||||||
|
|
||||||
1. **并发容量**:1000 协程争用同一 Proxy,始终满足
|
1. **并发容量**:1000 协程争用同一 Proxy,始终满足
|
||||||
`active + reserved <= effectiveMaxConcurrency`。
|
`active + reserved <= effectiveMaxConcurrency`。
|
||||||
@ -68,6 +68,8 @@
|
|||||||
unowned,旧 Snapshot 不可再分配。
|
unowned,旧 Snapshot 不可再分配。
|
||||||
11. **控制面故障**:Redis、PostgreSQL、Controller、Checker 与 Provider 分别
|
11. **控制面故障**:Redis、PostgreSQL、Controller、Checker 与 Provider 分别
|
||||||
失效时,行为与 Runbook 一致,Gateway 热路径不被同步依赖拖垮。
|
失效时,行为与 Runbook 一致,Gateway 热路径不被同步依赖拖垮。
|
||||||
|
12. **Outcome 栅栏**:队列写入不阻塞转发;同一 session 的相同序列/摘要可重放,
|
||||||
|
相同序列的不同摘要冲突,较小序列拒绝;Session 替换后旧 Worker 结果拒绝。
|
||||||
|
|
||||||
## 4. 测试命令
|
## 4. 测试命令
|
||||||
|
|
||||||
|
|||||||
@ -26,6 +26,7 @@ type keyspace struct {
|
|||||||
workerSnapshotExpiry string
|
workerSnapshotExpiry string
|
||||||
workerRuntime string
|
workerRuntime string
|
||||||
workerRuntimeExpiry string
|
workerRuntimeExpiry string
|
||||||
|
workerOutcomes string
|
||||||
}
|
}
|
||||||
|
|
||||||
func newKeyspace(namespace string) keyspace {
|
func newKeyspace(namespace string) keyspace {
|
||||||
@ -48,6 +49,7 @@ func newKeyspace(namespace string) keyspace {
|
|||||||
workerSnapshotExpiry: prefix + ":worker-snapshot-expiry",
|
workerSnapshotExpiry: prefix + ":worker-snapshot-expiry",
|
||||||
workerRuntime: prefix + ":worker-runtime",
|
workerRuntime: prefix + ":worker-runtime",
|
||||||
workerRuntimeExpiry: prefix + ":worker-runtime-expiry",
|
workerRuntimeExpiry: prefix + ":worker-runtime-expiry",
|
||||||
|
workerOutcomes: prefix + ":worker-outcomes",
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@ -20,6 +20,7 @@ const (
|
|||||||
runtimeRecordSnapshot = "record_snapshot"
|
runtimeRecordSnapshot = "record_snapshot"
|
||||||
runtimeAcknowledge = "acknowledge_snapshot"
|
runtimeAcknowledge = "acknowledge_snapshot"
|
||||||
runtimeReplaceReport = "replace_report"
|
runtimeReplaceReport = "replace_report"
|
||||||
|
runtimeRecordOutcomes = "record_outcomes"
|
||||||
runtimeRead = "read"
|
runtimeRead = "read"
|
||||||
)
|
)
|
||||||
|
|
||||||
@ -84,6 +85,13 @@ type runtimeReportWire struct {
|
|||||||
Counters []runtimeCounterWire `json:"counters"`
|
Counters []runtimeCounterWire `json:"counters"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type runtimeOutcomeReferenceWire struct {
|
||||||
|
Version int `json:"version"`
|
||||||
|
WorkerID string `json:"workerId"`
|
||||||
|
SessionID string `json:"sessionId"`
|
||||||
|
Sequence string `json:"sequence"`
|
||||||
|
}
|
||||||
|
|
||||||
type runtimeOwnedProxyWire struct {
|
type runtimeOwnedProxyWire struct {
|
||||||
ProxyID string `json:"proxyId"`
|
ProxyID string `json:"proxyId"`
|
||||||
WorkerID string `json:"workerId"`
|
WorkerID string `json:"workerId"`
|
||||||
@ -103,6 +111,7 @@ var (
|
|||||||
_ workerruntime.SessionWriter = (*Adapter)(nil)
|
_ workerruntime.SessionWriter = (*Adapter)(nil)
|
||||||
_ workerruntime.ReportWriter = (*Adapter)(nil)
|
_ workerruntime.ReportWriter = (*Adapter)(nil)
|
||||||
_ workerruntime.RuntimeReader = (*Adapter)(nil)
|
_ workerruntime.RuntimeReader = (*Adapter)(nil)
|
||||||
|
_ workerruntime.OutcomeWriter = (*Adapter)(nil)
|
||||||
)
|
)
|
||||||
|
|
||||||
func (a *Adapter) CurrentOwnershipEpoch(ctx context.Context) (uint64, error) {
|
func (a *Adapter) CurrentOwnershipEpoch(ctx context.Context) (uint64, error) {
|
||||||
@ -317,6 +326,48 @@ func (a *Adapter) ReplaceRuntime(ctx context.Context, report workerruntime.Repor
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RecordOutcomes persists only the monotonically increasing batch fence. The
|
||||||
|
// Gateway event payload is deliberately excluded from Redis to keep the data
|
||||||
|
// plane hot path bounded and free of per-request storage writes.
|
||||||
|
func (a *Adapter) RecordOutcomes(ctx context.Context, reference workerruntime.OutcomeReference) (uint64, error) {
|
||||||
|
if err := validateRuntimeCall(ctx, a); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
normalized, err := workerruntime.NormalizeOutcomeReference(reference)
|
||||||
|
if err != nil {
|
||||||
|
return 0, workerruntime.ErrInvalidOutcome
|
||||||
|
}
|
||||||
|
payload, err := json.Marshal(runtimeOutcomeReferenceWire{
|
||||||
|
Version: runtimeWireVersion, WorkerID: normalized.WorkerID, SessionID: normalized.SessionID,
|
||||||
|
Sequence: strconv.FormatUint(normalized.Sequence, 10),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return 0, workerruntime.ErrInvalidOutcome
|
||||||
|
}
|
||||||
|
reply, err := a.runRuntime(ctx, runtimeRecordOutcomes, 0, payload, hex.EncodeToString(normalized.Digest[:]))
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
switch reply.Status {
|
||||||
|
case scriptOK:
|
||||||
|
accepted, parseErr := strconv.ParseUint(reply.Record, 10, 64)
|
||||||
|
if parseErr != nil || accepted == 0 {
|
||||||
|
return 0, invalidScriptReply("invalid worker outcome reply")
|
||||||
|
}
|
||||||
|
return accepted, nil
|
||||||
|
case scriptInvalid:
|
||||||
|
return 0, workerruntime.ErrInvalidOutcome
|
||||||
|
case scriptStale:
|
||||||
|
return 0, workerruntime.ErrStaleOutcome
|
||||||
|
case scriptConflict:
|
||||||
|
return 0, workerruntime.ErrConflictingOutcome
|
||||||
|
case scriptUnavailable:
|
||||||
|
return 0, workerruntime.ErrStaleSession
|
||||||
|
default:
|
||||||
|
return 0, invalidScriptReply("unexpected worker outcome reply")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func (a *Adapter) ReadRuntime(ctx context.Context, proxies []workerruntime.OwnedProxy) ([]workerruntime.Snapshot, error) {
|
func (a *Adapter) ReadRuntime(ctx context.Context, proxies []workerruntime.OwnedProxy) ([]workerruntime.Snapshot, error) {
|
||||||
if err := validateRuntimeCall(ctx, a); err != nil {
|
if err := validateRuntimeCall(ctx, a); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@ -403,6 +454,7 @@ func (a *Adapter) runRuntime(
|
|||||||
a.keys.workerSessions, a.keys.workerSessionExpiry,
|
a.keys.workerSessions, a.keys.workerSessionExpiry,
|
||||||
a.keys.workerSnapshots, a.keys.workerSnapshotExpiry,
|
a.keys.workerSnapshots, a.keys.workerSnapshotExpiry,
|
||||||
a.keys.workerRuntime, a.keys.workerRuntimeExpiry, a.keys.owners, a.keys.epoch,
|
a.keys.workerRuntime, a.keys.workerRuntimeExpiry, a.keys.owners, a.keys.epoch,
|
||||||
|
a.keys.workerOutcomes,
|
||||||
}, operation, ttlMS, a.options.CleanupLimit, string(payload), digest)
|
}, operation, ttlMS, a.options.CleanupLimit, string(payload), digest)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return runtimeScriptReply{}, err
|
return runtimeScriptReply{}, err
|
||||||
|
|||||||
@ -20,7 +20,7 @@ func TestRedisWorkerControlStoreContract(t *testing.T) {
|
|||||||
t.Fatalf("Assign(): %v", err)
|
t.Fatalf("Assign(): %v", err)
|
||||||
}
|
}
|
||||||
return contracttest.Fixture{
|
return contracttest.Fixture{
|
||||||
Store: fixture.Adapter, Reader: fixture.Adapter, TTL: 100 * time.Millisecond,
|
Store: fixture.Adapter, Reader: fixture.Adapter, Outcomes: fixture.Adapter, TTL: 100 * time.Millisecond,
|
||||||
Advance: time.Sleep,
|
Advance: time.Sleep,
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|||||||
@ -6,6 +6,7 @@ local runtime_key = KEYS[5]
|
|||||||
local runtime_expiry_key = KEYS[6]
|
local runtime_expiry_key = KEYS[6]
|
||||||
local owners_key = KEYS[7]
|
local owners_key = KEYS[7]
|
||||||
local epoch_key = KEYS[8]
|
local epoch_key = KEYS[8]
|
||||||
|
local outcomes_key = KEYS[9]
|
||||||
|
|
||||||
local operation = ARGV[1]
|
local operation = ARGV[1]
|
||||||
local ttl_ms = tonumber(ARGV[2])
|
local ttl_ms = tonumber(ARGV[2])
|
||||||
@ -60,6 +61,7 @@ local function cleanup(now)
|
|||||||
for _, worker_id in ipairs(expired_sessions) do
|
for _, worker_id in ipairs(expired_sessions) do
|
||||||
redis.call('HDEL', sessions_key, worker_id)
|
redis.call('HDEL', sessions_key, worker_id)
|
||||||
redis.call('HDEL', runtime_key, worker_id)
|
redis.call('HDEL', runtime_key, worker_id)
|
||||||
|
redis.call('HDEL', outcomes_key, worker_id)
|
||||||
redis.call('ZREM', session_expiry_key, worker_id)
|
redis.call('ZREM', session_expiry_key, worker_id)
|
||||||
redis.call('ZREM', runtime_expiry_key, worker_id)
|
redis.call('ZREM', runtime_expiry_key, worker_id)
|
||||||
end
|
end
|
||||||
@ -106,6 +108,13 @@ local function valid_reference(value)
|
|||||||
string.match(value.checksum, '^[0-9a-f]+$') ~= nil
|
string.match(value.checksum, '^[0-9a-f]+$') ~= nil
|
||||||
end
|
end
|
||||||
|
|
||||||
|
local function valid_outcome(value)
|
||||||
|
return value and value.version == 1 and type(value.workerId) == 'string' and value.workerId ~= '' and
|
||||||
|
type(value.sessionId) == 'string' and value.sessionId ~= '' and valid_uint(value.sequence) and
|
||||||
|
type(value.digest) == 'string' and string.len(value.digest) == 64 and
|
||||||
|
string.match(value.digest, '^[0-9a-f]+$') ~= nil
|
||||||
|
end
|
||||||
|
|
||||||
local function compare_reference(left, right)
|
local function compare_reference(left, right)
|
||||||
local epoch_order = compare_uint(left.ownershipEpoch, right.ownershipEpoch)
|
local epoch_order = compare_uint(left.ownershipEpoch, right.ownershipEpoch)
|
||||||
if epoch_order ~= 0 then
|
if epoch_order ~= 0 then
|
||||||
@ -148,6 +157,7 @@ if operation == 'open_session' then
|
|||||||
end
|
end
|
||||||
redis.call('HDEL', runtime_key, session.workerId)
|
redis.call('HDEL', runtime_key, session.workerId)
|
||||||
redis.call('ZREM', runtime_expiry_key, session.workerId)
|
redis.call('ZREM', runtime_expiry_key, session.workerId)
|
||||||
|
redis.call('HDEL', outcomes_key, session.workerId)
|
||||||
redis.call('HDEL', snapshots_key, session.workerId)
|
redis.call('HDEL', snapshots_key, session.workerId)
|
||||||
redis.call('ZREM', snapshot_expiry_key, session.workerId)
|
redis.call('ZREM', snapshot_expiry_key, session.workerId)
|
||||||
session.expiresAtMs = now + ttl_ms
|
session.expiresAtMs = now + ttl_ms
|
||||||
@ -301,10 +311,12 @@ if operation == 'replace_session' then
|
|||||||
if epoch_order > 0 or version_order > 0 then
|
if epoch_order > 0 or version_order > 0 then
|
||||||
redis.call('HDEL', runtime_key, session.workerId)
|
redis.call('HDEL', runtime_key, session.workerId)
|
||||||
redis.call('ZREM', runtime_expiry_key, session.workerId)
|
redis.call('ZREM', runtime_expiry_key, session.workerId)
|
||||||
|
redis.call('HDEL', outcomes_key, session.workerId)
|
||||||
end
|
end
|
||||||
else
|
else
|
||||||
redis.call('HDEL', runtime_key, session.workerId)
|
redis.call('HDEL', runtime_key, session.workerId)
|
||||||
redis.call('ZREM', runtime_expiry_key, session.workerId)
|
redis.call('ZREM', runtime_expiry_key, session.workerId)
|
||||||
|
redis.call('HDEL', outcomes_key, session.workerId)
|
||||||
end
|
end
|
||||||
session.expiresAtMs = now + ttl_ms
|
session.expiresAtMs = now + ttl_ms
|
||||||
redis.call('HSET', sessions_key, session.workerId, cjson.encode(session))
|
redis.call('HSET', sessions_key, session.workerId, cjson.encode(session))
|
||||||
@ -312,6 +324,39 @@ if operation == 'replace_session' then
|
|||||||
return reply('ok')
|
return reply('ok')
|
||||||
end
|
end
|
||||||
|
|
||||||
|
if operation == 'record_outcomes' then
|
||||||
|
if type(digest) ~= 'string' or string.len(digest) ~= 64 or
|
||||||
|
string.match(digest, '^[0-9a-f]+$') == nil then
|
||||||
|
return reply('invalid')
|
||||||
|
end
|
||||||
|
local reference = decode_table(payload)
|
||||||
|
if not reference or reference.version ~= 1 or type(reference.workerId) ~= 'string' or reference.workerId == '' or
|
||||||
|
type(reference.sessionId) ~= 'string' or reference.sessionId == '' or not valid_uint(reference.sequence) then
|
||||||
|
return reply('invalid')
|
||||||
|
end
|
||||||
|
local session = decode_table(redis.call('HGET', sessions_key, reference.workerId))
|
||||||
|
if not valid_session(session) or session.workerId ~= reference.workerId or session.sessionId ~= reference.sessionId or
|
||||||
|
type(session.expiresAtMs) ~= 'number' or session.expiresAtMs <= now then
|
||||||
|
return reply('unavailable')
|
||||||
|
end
|
||||||
|
local current = decode_table(redis.call('HGET', outcomes_key, reference.workerId))
|
||||||
|
if valid_outcome(current) and current.workerId == reference.workerId and current.sessionId == reference.sessionId then
|
||||||
|
local ordering = compare_uint(reference.sequence, current.sequence)
|
||||||
|
if ordering < 0 then
|
||||||
|
return reply('stale')
|
||||||
|
end
|
||||||
|
if ordering == 0 then
|
||||||
|
if current.digest == digest then
|
||||||
|
return reply('ok', nil, current.sequence)
|
||||||
|
end
|
||||||
|
return reply('conflict')
|
||||||
|
end
|
||||||
|
end
|
||||||
|
reference.digest = digest
|
||||||
|
redis.call('HSET', outcomes_key, reference.workerId, cjson.encode(reference))
|
||||||
|
return reply('ok', nil, reference.sequence)
|
||||||
|
end
|
||||||
|
|
||||||
if operation == 'replace_report' then
|
if operation == 'replace_report' then
|
||||||
if not ttl_ms or ttl_ms <= 0 or type(digest) ~= 'string' or digest == '' then
|
if not ttl_ms or ttl_ms <= 0 or type(digest) ~= 'string' or digest == '' then
|
||||||
return reply('invalid')
|
return reply('invalid')
|
||||||
|
|||||||
@ -4,9 +4,11 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"crypto/sha256"
|
"crypto/sha256"
|
||||||
"errors"
|
"errors"
|
||||||
|
"io"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
controlplanev1 "proxy-pool/gen/controlplane/v1"
|
controlplanev1 "proxy-pool/gen/controlplane/v1"
|
||||||
|
"proxy-pool/internal/domain/outcome"
|
||||||
"proxy-pool/internal/domain/workerruntime"
|
"proxy-pool/internal/domain/workerruntime"
|
||||||
|
|
||||||
"google.golang.org/grpc"
|
"google.golang.org/grpc"
|
||||||
@ -197,6 +199,84 @@ func (handler *GRPCHandler) ReportRuntime(ctx context.Context, request *controlp
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (handler *GRPCHandler) ReportOutcomes(stream grpc.ClientStreamingServer[controlplanev1.OutcomeBatch, controlplanev1.ReportOutcomesResponse]) error {
|
||||||
|
if stream == nil || handler == nil || handler.service == nil || handler.identity == nil {
|
||||||
|
return grpcError(ErrInvalidCommand)
|
||||||
|
}
|
||||||
|
var workerID, sessionID string
|
||||||
|
var accepted uint64
|
||||||
|
batches := 0
|
||||||
|
for {
|
||||||
|
request, err := stream.Recv()
|
||||||
|
if errors.Is(err, io.EOF) {
|
||||||
|
if workerID == "" {
|
||||||
|
return grpcError(ErrInvalidCommand)
|
||||||
|
}
|
||||||
|
return stream.SendAndClose(&controlplanev1.ReportOutcomesResponse{AcceptedThroughSequence: accepted})
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if batches == outcome.MaxBatchesPerStream {
|
||||||
|
return grpcError(ErrInvalidCommand)
|
||||||
|
}
|
||||||
|
batch, err := decodeOutcomeBatch(request)
|
||||||
|
if err != nil {
|
||||||
|
return grpcError(err)
|
||||||
|
}
|
||||||
|
if workerID == "" {
|
||||||
|
if err := handler.authorize(stream.Context(), batch.WorkerID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
workerID, sessionID = batch.WorkerID, batch.SessionID
|
||||||
|
} else if batch.WorkerID != workerID || batch.SessionID != sessionID {
|
||||||
|
return grpcError(ErrInvalidCommand)
|
||||||
|
}
|
||||||
|
accepted, err = handler.service.ReportOutcomes(stream.Context(), batch)
|
||||||
|
if err != nil {
|
||||||
|
return grpcError(err)
|
||||||
|
}
|
||||||
|
batches++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeOutcomeBatch(request *controlplanev1.OutcomeBatch) (outcome.Batch, error) {
|
||||||
|
if request == nil {
|
||||||
|
return outcome.Batch{}, ErrInvalidCommand
|
||||||
|
}
|
||||||
|
events := make([]outcome.Event, len(request.GetOutcomes()))
|
||||||
|
for index, item := range request.GetOutcomes() {
|
||||||
|
if item == nil || item.GetLatency() == nil || item.GetLatency().CheckValid() != nil ||
|
||||||
|
item.GetObservedAt() == nil || item.GetObservedAt().CheckValid() != nil {
|
||||||
|
return outcome.Batch{}, ErrInvalidCommand
|
||||||
|
}
|
||||||
|
stage, ok := decodeOutcomeStage(item.GetStage())
|
||||||
|
if !ok {
|
||||||
|
return outcome.Batch{}, ErrInvalidCommand
|
||||||
|
}
|
||||||
|
events[index] = outcome.Event{
|
||||||
|
ProxyID: item.GetProxyId(), RoutingName: item.GetRoutingName(), Stage: stage, Success: item.GetSuccess(),
|
||||||
|
ErrorClass: outcome.ErrorClass(item.GetErrorClass()), Latency: item.GetLatency().AsDuration(), ObservedAt: item.GetObservedAt().AsTime(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return outcome.Batch{WorkerID: request.GetWorkerId(), SessionID: request.GetSessionId(), Sequence: request.GetSequence(), Events: events}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeOutcomeStage(stage controlplanev1.OutcomeStage) (outcome.Stage, bool) {
|
||||||
|
switch stage {
|
||||||
|
case controlplanev1.OutcomeStage_OUTCOME_STAGE_DIAL:
|
||||||
|
return outcome.StageDial, true
|
||||||
|
case controlplanev1.OutcomeStage_OUTCOME_STAGE_PROXY_HANDSHAKE:
|
||||||
|
return outcome.StageProxyHandshake, true
|
||||||
|
case controlplanev1.OutcomeStage_OUTCOME_STAGE_RESPONSE_HEADERS:
|
||||||
|
return outcome.StageResponseHeaders, true
|
||||||
|
case controlplanev1.OutcomeStage_OUTCOME_STAGE_TUNNEL:
|
||||||
|
return outcome.StageTunnel, true
|
||||||
|
default:
|
||||||
|
return outcome.StageUnspecified, false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func (handler *GRPCHandler) authorize(ctx context.Context, workerID string) error {
|
func (handler *GRPCHandler) authorize(ctx context.Context, workerID string) error {
|
||||||
if err := handler.identity.Authorize(ctx, workerID); err != nil {
|
if err := handler.identity.Authorize(ctx, workerID); err != nil {
|
||||||
return status.Error(codes.PermissionDenied, "worker identity is not authorized")
|
return status.Error(codes.PermissionDenied, "worker identity is not authorized")
|
||||||
@ -210,7 +290,8 @@ func grpcError(err error) error {
|
|||||||
return status.Error(codes.Canceled, "worker control request canceled")
|
return status.Error(codes.Canceled, "worker control request canceled")
|
||||||
case errors.Is(err, context.DeadlineExceeded):
|
case errors.Is(err, context.DeadlineExceeded):
|
||||||
return status.Error(codes.DeadlineExceeded, "worker control request deadline exceeded")
|
return status.Error(codes.DeadlineExceeded, "worker control request deadline exceeded")
|
||||||
case errors.Is(err, ErrInvalidCommand), errors.Is(err, workerruntime.ErrInvalidReport),
|
case errors.Is(err, ErrInvalidCommand), errors.Is(err, outcome.ErrInvalidBatch), errors.Is(err, workerruntime.ErrInvalidReport),
|
||||||
|
errors.Is(err, workerruntime.ErrInvalidOutcome),
|
||||||
errors.Is(err, workerruntime.ErrInvalidAcknowledgement), errors.Is(err, workerruntime.ErrInvalidSnapshotReference):
|
errors.Is(err, workerruntime.ErrInvalidAcknowledgement), errors.Is(err, workerruntime.ErrInvalidSnapshotReference):
|
||||||
return status.Error(codes.InvalidArgument, "invalid worker control request")
|
return status.Error(codes.InvalidArgument, "invalid worker control request")
|
||||||
case errors.Is(err, ErrProtocolVersion):
|
case errors.Is(err, ErrProtocolVersion):
|
||||||
@ -225,6 +306,10 @@ func grpcError(err error) error {
|
|||||||
return status.Error(codes.Aborted, "worker runtime sequence is stale")
|
return status.Error(codes.Aborted, "worker runtime sequence is stale")
|
||||||
case errors.Is(err, workerruntime.ErrConflictingReport):
|
case errors.Is(err, workerruntime.ErrConflictingReport):
|
||||||
return status.Error(codes.AlreadyExists, "worker runtime sequence conflicts")
|
return status.Error(codes.AlreadyExists, "worker runtime sequence conflicts")
|
||||||
|
case errors.Is(err, workerruntime.ErrStaleOutcome):
|
||||||
|
return status.Error(codes.Aborted, "worker outcome sequence is stale")
|
||||||
|
case errors.Is(err, workerruntime.ErrConflictingOutcome):
|
||||||
|
return status.Error(codes.AlreadyExists, "worker outcome sequence conflicts")
|
||||||
default:
|
default:
|
||||||
return status.Error(codes.Unavailable, "worker control plane unavailable")
|
return status.Error(codes.Unavailable, "worker control plane unavailable")
|
||||||
}
|
}
|
||||||
|
|||||||
@ -8,12 +8,15 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
controlplanev1 "proxy-pool/gen/controlplane/v1"
|
controlplanev1 "proxy-pool/gen/controlplane/v1"
|
||||||
|
"proxy-pool/internal/domain/outcome"
|
||||||
"proxy-pool/internal/domain/workerruntime"
|
"proxy-pool/internal/domain/workerruntime"
|
||||||
|
|
||||||
"google.golang.org/grpc"
|
"google.golang.org/grpc"
|
||||||
"google.golang.org/grpc/codes"
|
"google.golang.org/grpc/codes"
|
||||||
"google.golang.org/grpc/status"
|
"google.golang.org/grpc/status"
|
||||||
"google.golang.org/grpc/test/bufconn"
|
"google.golang.org/grpc/test/bufconn"
|
||||||
|
"google.golang.org/protobuf/proto"
|
||||||
|
"google.golang.org/protobuf/types/known/durationpb"
|
||||||
"google.golang.org/protobuf/types/known/timestamppb"
|
"google.golang.org/protobuf/types/known/timestamppb"
|
||||||
)
|
)
|
||||||
|
|
||||||
@ -45,7 +48,7 @@ func TestGRPCHandlerMapsWorkerRequests(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestGRPCHandlerMapsErrorsAndLeavesStreamsUnimplemented(t *testing.T) {
|
func TestGRPCHandlerMapsErrorsAndLeavesSnapshotStreamUnimplemented(t *testing.T) {
|
||||||
service := &grpcServiceStub{registerErr: ErrProtocolVersion}
|
service := &grpcServiceStub{registerErr: ErrProtocolVersion}
|
||||||
client, cleanup := grpcWorkerClient(t, service, allowIdentity{})
|
client, cleanup := grpcWorkerClient(t, service, allowIdentity{})
|
||||||
defer cleanup()
|
defer cleanup()
|
||||||
@ -58,11 +61,88 @@ func TestGRPCHandlerMapsErrorsAndLeavesStreamsUnimplemented(t *testing.T) {
|
|||||||
if err != nil || status.Code(streamErr) != codes.Unimplemented {
|
if err != nil || status.Code(streamErr) != codes.Unimplemented {
|
||||||
t.Fatalf("WatchSnapshots() = %v, %v", err, streamErr)
|
t.Fatalf("WatchSnapshots() = %v, %v", err, streamErr)
|
||||||
}
|
}
|
||||||
outcomes, err := client.ReportOutcomes(context.Background())
|
}
|
||||||
_, outcomesErr := outcomes.CloseAndRecv()
|
|
||||||
if err != nil || status.Code(outcomesErr) != codes.Unimplemented {
|
func TestGRPCHandlerReportsOutcomeStream(t *testing.T) {
|
||||||
t.Fatalf("ReportOutcomes() = %v, %v", err, outcomesErr)
|
service := &grpcServiceStub{}
|
||||||
|
client, cleanup := grpcWorkerClient(t, service, allowIdentity{})
|
||||||
|
defer cleanup()
|
||||||
|
stream, err := client.ReportOutcomes(context.Background())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReportOutcomes() = %v", err)
|
||||||
}
|
}
|
||||||
|
if err := stream.Send(&controlplanev1.OutcomeBatch{
|
||||||
|
WorkerId: "worker-a", SessionId: "session-a", Sequence: 3,
|
||||||
|
Outcomes: []*controlplanev1.ProxyOutcome{{
|
||||||
|
ProxyId: "proxy-a", RoutingName: "route-a", Stage: controlplanev1.OutcomeStage_OUTCOME_STAGE_DIAL,
|
||||||
|
Success: false, ErrorClass: "timeout", Latency: durationpb.New(25 * time.Millisecond), ObservedAt: timestamppb.New(time.Now()),
|
||||||
|
}},
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("Send() = %v", err)
|
||||||
|
}
|
||||||
|
response, err := stream.CloseAndRecv()
|
||||||
|
if err != nil || response.GetAcceptedThroughSequence() != 3 || service.outcome.Sequence != 3 ||
|
||||||
|
service.outcome.Events[0].ErrorClass != outcome.ErrorClassTimeout {
|
||||||
|
t.Fatalf("CloseAndRecv() = %+v, %v; outcome = %+v", response, err, service.outcome)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGRPCHandlerFencesOutcomesThroughWorkerService(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 31, 12, 0, 0, 0, time.UTC)
|
||||||
|
store, err := workerruntime.NewMemoryStore(func() time.Time { return now })
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewMemoryStore() = %v", err)
|
||||||
|
}
|
||||||
|
service, err := NewService(store, Options{
|
||||||
|
ProtocolVersion: 1, HeartbeatInterval: time.Second, SessionTTL: 3 * time.Second,
|
||||||
|
MaxStaleAge: time.Second, MaxRuntimeCounters: 4,
|
||||||
|
SessionID: func() (string, error) { return "session-a", nil },
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewService() = %v", err)
|
||||||
|
}
|
||||||
|
client, cleanup := grpcWorkerClient(t, service, allowIdentity{})
|
||||||
|
defer cleanup()
|
||||||
|
registered, err := client.RegisterWorker(context.Background(), &controlplanev1.RegisterWorkerRequest{
|
||||||
|
WorkerId: "worker-a", InstanceId: "instance-a", Zone: "zone-a", SupportedProtocolVersion: 1,
|
||||||
|
})
|
||||||
|
if err != nil || registered.GetSessionId() != "session-a" {
|
||||||
|
t.Fatalf("RegisterWorker() = %+v, %v", registered, err)
|
||||||
|
}
|
||||||
|
batch := &controlplanev1.OutcomeBatch{
|
||||||
|
WorkerId: "worker-a", SessionId: "session-a", Sequence: 1,
|
||||||
|
Outcomes: []*controlplanev1.ProxyOutcome{{
|
||||||
|
ProxyId: "proxy-a", Stage: controlplanev1.OutcomeStage_OUTCOME_STAGE_DIAL, Success: true,
|
||||||
|
Latency: durationpb.New(time.Millisecond), ObservedAt: timestamppb.New(now),
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
if accepted, err := reportOutcomeBatch(context.Background(), client, batch); err != nil || accepted != 1 {
|
||||||
|
t.Fatalf("ReportOutcomes(first) = %d, %v", accepted, err)
|
||||||
|
}
|
||||||
|
if accepted, err := reportOutcomeBatch(context.Background(), client, batch); err != nil || accepted != 1 {
|
||||||
|
t.Fatalf("ReportOutcomes(replay) = %d, %v", accepted, err)
|
||||||
|
}
|
||||||
|
conflicting := proto.Clone(batch).(*controlplanev1.OutcomeBatch)
|
||||||
|
conflicting.Outcomes[0].Success = false
|
||||||
|
conflicting.Outcomes[0].ErrorClass = "dial"
|
||||||
|
if _, err := reportOutcomeBatch(context.Background(), client, conflicting); status.Code(err) != codes.AlreadyExists {
|
||||||
|
t.Fatalf("ReportOutcomes(conflict) code = %s, want AlreadyExists; error=%v", status.Code(err), err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func reportOutcomeBatch(ctx context.Context, client controlplanev1.WorkerControlPlaneClient, batch *controlplanev1.OutcomeBatch) (uint64, error) {
|
||||||
|
stream, err := client.ReportOutcomes(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
if err := stream.Send(batch); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
response, err := stream.CloseAndRecv()
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
return response.GetAcceptedThroughSequence(), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestGRPCHandlerStreamsIssuedFullSnapshots(t *testing.T) {
|
func TestGRPCHandlerStreamsIssuedFullSnapshots(t *testing.T) {
|
||||||
@ -166,6 +246,9 @@ type grpcServiceStub struct {
|
|||||||
acknowledgeErr error
|
acknowledgeErr error
|
||||||
report workerruntime.Report
|
report workerruntime.Report
|
||||||
reportErr error
|
reportErr error
|
||||||
|
outcome outcome.Batch
|
||||||
|
outcomeAccepted uint64
|
||||||
|
outcomeErr error
|
||||||
issued workerruntime.SnapshotReference
|
issued workerruntime.SnapshotReference
|
||||||
issuedSessionID string
|
issuedSessionID string
|
||||||
validateErr error
|
validateErr error
|
||||||
@ -195,6 +278,13 @@ func (stub *grpcServiceStub) ReportRuntime(_ context.Context, report workerrunti
|
|||||||
stub.report = report
|
stub.report = report
|
||||||
return RuntimeDecision{AcceptedOwnershipEpoch: 9}, stub.reportErr
|
return RuntimeDecision{AcceptedOwnershipEpoch: 9}, stub.reportErr
|
||||||
}
|
}
|
||||||
|
func (stub *grpcServiceStub) ReportOutcomes(_ context.Context, batch outcome.Batch) (uint64, error) {
|
||||||
|
stub.outcome = batch
|
||||||
|
if stub.outcomeAccepted == 0 {
|
||||||
|
stub.outcomeAccepted = batch.Sequence
|
||||||
|
}
|
||||||
|
return stub.outcomeAccepted, stub.outcomeErr
|
||||||
|
}
|
||||||
|
|
||||||
type allowIdentity struct{}
|
type allowIdentity struct{}
|
||||||
|
|
||||||
|
|||||||
@ -8,6 +8,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"proxy-pool/internal/domain/outcome"
|
||||||
ownershipDomain "proxy-pool/internal/domain/ownership"
|
ownershipDomain "proxy-pool/internal/domain/ownership"
|
||||||
"proxy-pool/internal/domain/workerruntime"
|
"proxy-pool/internal/domain/workerruntime"
|
||||||
platformCredentials "proxy-pool/internal/platform/credentials"
|
platformCredentials "proxy-pool/internal/platform/credentials"
|
||||||
@ -71,6 +72,7 @@ type Service interface {
|
|||||||
IssueSnapshot(context.Context, string, workerruntime.SnapshotReference) error
|
IssueSnapshot(context.Context, string, workerruntime.SnapshotReference) error
|
||||||
Acknowledge(context.Context, SnapshotAcknowledgement) error
|
Acknowledge(context.Context, SnapshotAcknowledgement) error
|
||||||
ReportRuntime(context.Context, workerruntime.Report) (RuntimeDecision, error)
|
ReportRuntime(context.Context, workerruntime.Report) (RuntimeDecision, error)
|
||||||
|
ReportOutcomes(context.Context, outcome.Batch) (uint64, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (service *service) CurrentOwnershipEpoch(ctx context.Context) (uint64, error) {
|
func (service *service) CurrentOwnershipEpoch(ctx context.Context) (uint64, error) {
|
||||||
@ -119,6 +121,7 @@ func (service *service) IssueSnapshot(ctx context.Context, sessionID string, ref
|
|||||||
|
|
||||||
type service struct {
|
type service struct {
|
||||||
store workerruntime.ControlStore
|
store workerruntime.ControlStore
|
||||||
|
outcomes workerruntime.OutcomeWriter
|
||||||
options Options
|
options Options
|
||||||
snapshots SnapshotSource
|
snapshots SnapshotSource
|
||||||
}
|
}
|
||||||
@ -133,6 +136,7 @@ func NewService(store workerruntime.ControlStore, options Options) (Service, err
|
|||||||
options.SessionID = randomSessionID
|
options.SessionID = randomSessionID
|
||||||
}
|
}
|
||||||
result := &service{store: store, options: options}
|
result := &service{store: store, options: options}
|
||||||
|
result.outcomes, _ = store.(workerruntime.OutcomeWriter)
|
||||||
if options.SnapshotReader != nil {
|
if options.SnapshotReader != nil {
|
||||||
var routing []RoutingSource
|
var routing []RoutingSource
|
||||||
if options.RoutingSource != nil {
|
if options.RoutingSource != nil {
|
||||||
@ -252,6 +256,31 @@ func (service *service) ReportRuntime(ctx context.Context, report workerruntime.
|
|||||||
return RuntimeDecision{AcceptedOwnershipEpoch: epoch}, nil
|
return RuntimeDecision{AcceptedOwnershipEpoch: epoch}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ReportOutcomes records only a batch digest and sequence fence. Raw Gateway
|
||||||
|
// observations are intentionally not persisted by the control plane.
|
||||||
|
func (service *service) ReportOutcomes(ctx context.Context, batch outcome.Batch) (uint64, error) {
|
||||||
|
if ctx == nil {
|
||||||
|
return 0, ErrInvalidCommand
|
||||||
|
}
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
normalized, digest, err := outcome.NormalizeBatch(batch, service.options.MaxRuntimeCounters)
|
||||||
|
if err != nil {
|
||||||
|
return 0, errors.Join(ErrInvalidCommand, err)
|
||||||
|
}
|
||||||
|
if service.outcomes == nil {
|
||||||
|
return 0, ErrUnavailable
|
||||||
|
}
|
||||||
|
accepted, err := service.outcomes.RecordOutcomes(ctx, workerruntime.OutcomeReference{
|
||||||
|
WorkerID: normalized.WorkerID, SessionID: normalized.SessionID, Sequence: normalized.Sequence, Digest: digest,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return 0, classifyStoreError(err)
|
||||||
|
}
|
||||||
|
return accepted, nil
|
||||||
|
}
|
||||||
|
|
||||||
func randomSessionID() (string, error) {
|
func randomSessionID() (string, error) {
|
||||||
var value [16]byte
|
var value [16]byte
|
||||||
if _, err := rand.Read(value[:]); err != nil {
|
if _, err := rand.Read(value[:]); err != nil {
|
||||||
@ -266,8 +295,10 @@ func classifyStoreError(err error) error {
|
|||||||
}
|
}
|
||||||
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) ||
|
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) ||
|
||||||
errors.Is(err, workerruntime.ErrInvalidSession) || errors.Is(err, workerruntime.ErrInvalidReport) ||
|
errors.Is(err, workerruntime.ErrInvalidSession) || errors.Is(err, workerruntime.ErrInvalidReport) ||
|
||||||
|
errors.Is(err, workerruntime.ErrInvalidOutcome) ||
|
||||||
errors.Is(err, workerruntime.ErrInvalidAcknowledgement) || errors.Is(err, workerruntime.ErrStaleSession) ||
|
errors.Is(err, workerruntime.ErrInvalidAcknowledgement) || errors.Is(err, workerruntime.ErrStaleSession) ||
|
||||||
errors.Is(err, workerruntime.ErrStaleReport) || errors.Is(err, workerruntime.ErrConflictingReport) ||
|
errors.Is(err, workerruntime.ErrStaleReport) || errors.Is(err, workerruntime.ErrConflictingReport) ||
|
||||||
|
errors.Is(err, workerruntime.ErrStaleOutcome) || errors.Is(err, workerruntime.ErrConflictingOutcome) ||
|
||||||
errors.Is(err, workerruntime.ErrSnapshotMismatch) || errors.Is(err, workerruntime.ErrStaleAcknowledgement) {
|
errors.Is(err, workerruntime.ErrSnapshotMismatch) || errors.Is(err, workerruntime.ErrStaleAcknowledgement) {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|||||||
@ -7,6 +7,7 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"proxy-pool/internal/domain/outcome"
|
||||||
"proxy-pool/internal/domain/workerruntime"
|
"proxy-pool/internal/domain/workerruntime"
|
||||||
)
|
)
|
||||||
|
|
||||||
@ -51,6 +52,13 @@ func TestServiceRegistersAcknowledgesAndReportsRuntime(t *testing.T) {
|
|||||||
if err != nil || decision.RequireFullSnapshot || decision.AcceptedOwnershipEpoch != registered.OwnershipEpoch {
|
if err != nil || decision.RequireFullSnapshot || decision.AcceptedOwnershipEpoch != registered.OwnershipEpoch {
|
||||||
t.Fatalf("ReportRuntime() = %+v, %v", decision, err)
|
t.Fatalf("ReportRuntime() = %+v, %v", decision, err)
|
||||||
}
|
}
|
||||||
|
accepted, err := service.ReportOutcomes(context.Background(), outcome.Batch{
|
||||||
|
WorkerID: "worker-a", SessionID: registered.SessionID, Sequence: 1,
|
||||||
|
Events: []outcome.Event{{ProxyID: "proxy-a", Stage: outcome.StageDial, Success: true, ObservedAt: now}},
|
||||||
|
})
|
||||||
|
if err != nil || accepted != 1 {
|
||||||
|
t.Fatalf("ReportOutcomes() = %d, %v", accepted, err)
|
||||||
|
}
|
||||||
decision, err = service.ReportRuntime(context.Background(), workerruntime.Report{
|
decision, err = service.ReportRuntime(context.Background(), workerruntime.Report{
|
||||||
WorkerID: "worker-a", SessionID: registered.SessionID, Sequence: 2,
|
WorkerID: "worker-a", SessionID: registered.SessionID, Sequence: 2,
|
||||||
SnapshotVersion: 8, OwnershipEpoch: registered.OwnershipEpoch, ObservedAt: now,
|
SnapshotVersion: 8, OwnershipEpoch: registered.OwnershipEpoch, ObservedAt: now,
|
||||||
|
|||||||
110
internal/domain/outcome/outcome.go
Normal file
110
internal/domain/outcome/outcome.go
Normal file
@ -0,0 +1,110 @@
|
|||||||
|
package outcome
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"regexp"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
maximumIdentifierBytes = 128
|
||||||
|
MaxBatchesPerStream = 16
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
ErrInvalidBatch = errors.New("invalid proxy outcome batch")
|
||||||
|
identifier = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$`)
|
||||||
|
)
|
||||||
|
|
||||||
|
// Stage identifies the furthest proxy-processing boundary reached by a
|
||||||
|
// request. It is deliberately independent of a particular transport error.
|
||||||
|
type Stage uint8
|
||||||
|
|
||||||
|
const (
|
||||||
|
StageUnspecified Stage = iota
|
||||||
|
StageDial
|
||||||
|
StageProxyHandshake
|
||||||
|
StageResponseHeaders
|
||||||
|
StageTunnel
|
||||||
|
)
|
||||||
|
|
||||||
|
// ErrorClass is a bounded category suitable for aggregation and metrics.
|
||||||
|
// It must never contain raw connection, target, or credential details.
|
||||||
|
type ErrorClass string
|
||||||
|
|
||||||
|
const (
|
||||||
|
ErrorClassDial ErrorClass = "dial"
|
||||||
|
ErrorClassTimeout ErrorClass = "timeout"
|
||||||
|
ErrorClassProxyResponse ErrorClass = "proxy_response"
|
||||||
|
ErrorClassHandshake ErrorClass = "handshake"
|
||||||
|
ErrorClassRelay ErrorClass = "relay"
|
||||||
|
ErrorClassCanceled ErrorClass = "canceled"
|
||||||
|
ErrorClassInternal ErrorClass = "internal"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Event struct {
|
||||||
|
ProxyID string
|
||||||
|
RoutingName string
|
||||||
|
Stage Stage
|
||||||
|
Success bool
|
||||||
|
ErrorClass ErrorClass
|
||||||
|
Latency time.Duration
|
||||||
|
ObservedAt time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
type Batch struct {
|
||||||
|
WorkerID string
|
||||||
|
SessionID string
|
||||||
|
Sequence uint64
|
||||||
|
Events []Event
|
||||||
|
}
|
||||||
|
|
||||||
|
// NormalizeBatch validates one bounded report, deep-copies it, and returns a
|
||||||
|
// digest used by the Controller's durable idempotency fence.
|
||||||
|
func NormalizeBatch(batch Batch, maximumEvents int) (Batch, [sha256.Size]byte, error) {
|
||||||
|
if !validIdentifier(batch.WorkerID) || !validIdentifier(batch.SessionID) || batch.Sequence == 0 ||
|
||||||
|
maximumEvents <= 0 || len(batch.Events) == 0 || len(batch.Events) > maximumEvents {
|
||||||
|
return Batch{}, [sha256.Size]byte{}, ErrInvalidBatch
|
||||||
|
}
|
||||||
|
normalized := batch
|
||||||
|
normalized.Events = append([]Event(nil), batch.Events...)
|
||||||
|
for index := range normalized.Events {
|
||||||
|
event := &normalized.Events[index]
|
||||||
|
if !validIdentifier(event.ProxyID) || (event.RoutingName != "" && !validIdentifier(event.RoutingName)) ||
|
||||||
|
!validStage(event.Stage) || event.Latency < 0 || event.ObservedAt.IsZero() ||
|
||||||
|
(event.Success && event.ErrorClass != "") || (!event.Success && !validErrorClass(event.ErrorClass)) {
|
||||||
|
return Batch{}, [sha256.Size]byte{}, ErrInvalidBatch
|
||||||
|
}
|
||||||
|
event.ObservedAt = event.ObservedAt.UTC()
|
||||||
|
}
|
||||||
|
payload, err := json.Marshal(normalized)
|
||||||
|
if err != nil {
|
||||||
|
return Batch{}, [sha256.Size]byte{}, ErrInvalidBatch
|
||||||
|
}
|
||||||
|
return normalized, sha256.Sum256(payload), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validIdentifier(value string) bool {
|
||||||
|
return len(value) <= maximumIdentifierBytes && identifier.MatchString(value)
|
||||||
|
}
|
||||||
|
|
||||||
|
func validStage(stage Stage) bool {
|
||||||
|
switch stage {
|
||||||
|
case StageDial, StageProxyHandshake, StageResponseHeaders, StageTunnel:
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validErrorClass(class ErrorClass) bool {
|
||||||
|
switch class {
|
||||||
|
case ErrorClassDial, ErrorClassTimeout, ErrorClassProxyResponse, ErrorClassHandshake,
|
||||||
|
ErrorClassRelay, ErrorClassCanceled, ErrorClassInternal:
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
52
internal/domain/outcome/outcome_test.go
Normal file
52
internal/domain/outcome/outcome_test.go
Normal file
@ -0,0 +1,52 @@
|
|||||||
|
package outcome
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestNormalizeBatchCanonicalizesAndDigestsOutcomes(t *testing.T) {
|
||||||
|
observedAt := time.Date(2026, 7, 31, 12, 0, 0, 0, time.FixedZone("UTC+8", 8*60*60))
|
||||||
|
normalized, digest, err := NormalizeBatch(Batch{
|
||||||
|
WorkerID: "worker-a", SessionID: "session-a", Sequence: 7,
|
||||||
|
Events: []Event{{
|
||||||
|
ProxyID: "proxy-a", RoutingName: "gateway-default", Stage: StageResponseHeaders,
|
||||||
|
Success: false, ErrorClass: ErrorClassTimeout, Latency: 25 * time.Millisecond, ObservedAt: observedAt,
|
||||||
|
}},
|
||||||
|
}, 4)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NormalizeBatch(): %v", err)
|
||||||
|
}
|
||||||
|
if normalized.Events[0].ObservedAt.Location() != time.UTC || digest == [32]byte{} {
|
||||||
|
t.Fatalf("normalized outcome = %+v, digest = %x", normalized.Events[0], digest)
|
||||||
|
}
|
||||||
|
_, repeated, err := NormalizeBatch(normalized, 4)
|
||||||
|
if err != nil || digest != repeated {
|
||||||
|
t.Fatalf("repeated NormalizeBatch() digest = %x, %v; want %x", repeated, err, digest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizeBatchRejectsUnboundedOrUnsafeInput(t *testing.T) {
|
||||||
|
now := time.Now()
|
||||||
|
valid := Event{ProxyID: "proxy-a", Stage: StageDial, Success: false, ErrorClass: ErrorClassDial, ObservedAt: now}
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
batch Batch
|
||||||
|
limit int
|
||||||
|
}{
|
||||||
|
{name: "missing worker", batch: Batch{SessionID: "session-a", Sequence: 1, Events: []Event{valid}}, limit: 1},
|
||||||
|
{name: "zero sequence", batch: Batch{WorkerID: "worker-a", SessionID: "session-a", Events: []Event{valid}}, limit: 1},
|
||||||
|
{name: "empty batch", batch: Batch{WorkerID: "worker-a", SessionID: "session-a", Sequence: 1}, limit: 1},
|
||||||
|
{name: "exceeds limit", batch: Batch{WorkerID: "worker-a", SessionID: "session-a", Sequence: 1, Events: []Event{valid, valid}}, limit: 1},
|
||||||
|
{name: "success with error", batch: Batch{WorkerID: "worker-a", SessionID: "session-a", Sequence: 1, Events: []Event{{ProxyID: "proxy-a", Stage: StageDial, Success: true, ErrorClass: ErrorClassDial, ObservedAt: now}}}, limit: 1},
|
||||||
|
{name: "unknown error class", batch: Batch{WorkerID: "worker-a", SessionID: "session-a", Sequence: 1, Events: []Event{{ProxyID: "proxy-a", Stage: StageDial, ErrorClass: "raw network failure", ObservedAt: now}}}, limit: 1},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
if _, _, err := NormalizeBatch(test.batch, test.limit); !errors.Is(err, ErrInvalidBatch) {
|
||||||
|
t.Fatalf("NormalizeBatch() error = %v, want ErrInvalidBatch", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -17,7 +17,7 @@ func TestMemoryStoreContract(t *testing.T) {
|
|||||||
t.Fatalf("NewMemoryStore(): %v", err)
|
t.Fatalf("NewMemoryStore(): %v", err)
|
||||||
}
|
}
|
||||||
return contracttest.Fixture{
|
return contracttest.Fixture{
|
||||||
Store: store, Reader: store, TTL: time.Minute,
|
Store: store, Reader: store, Outcomes: store, TTL: time.Minute,
|
||||||
Advance: func(duration time.Duration) { now = now.Add(duration) },
|
Advance: func(duration time.Duration) { now = now.Add(duration) },
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|||||||
@ -13,6 +13,7 @@ import (
|
|||||||
type Fixture struct {
|
type Fixture struct {
|
||||||
Store workerruntime.ControlStore
|
Store workerruntime.ControlStore
|
||||||
Reader workerruntime.RuntimeReader
|
Reader workerruntime.RuntimeReader
|
||||||
|
Outcomes workerruntime.OutcomeWriter
|
||||||
TTL time.Duration
|
TTL time.Duration
|
||||||
Advance func(time.Duration)
|
Advance func(time.Duration)
|
||||||
}
|
}
|
||||||
@ -25,6 +26,8 @@ func Run(t *testing.T, factory Factory) {
|
|||||||
t.Run("acknowledged runtime lifecycle", func(t *testing.T) { runLifecycle(t, newFixture(t, factory)) })
|
t.Run("acknowledged runtime lifecycle", func(t *testing.T) { runLifecycle(t, newFixture(t, factory)) })
|
||||||
t.Run("negative acknowledgement fences runtime", func(t *testing.T) { runNegativeAck(t, newFixture(t, factory)) })
|
t.Run("negative acknowledgement fences runtime", func(t *testing.T) { runNegativeAck(t, newFixture(t, factory)) })
|
||||||
t.Run("session replacement fences issued snapshots", func(t *testing.T) { runSessionReplacementFence(t, newFixture(t, factory)) })
|
t.Run("session replacement fences issued snapshots", func(t *testing.T) { runSessionReplacementFence(t, newFixture(t, factory)) })
|
||||||
|
t.Run("outcome sequence fences retries", func(t *testing.T) { runOutcomeFence(t, newFixture(t, factory)) })
|
||||||
|
t.Run("outcome invalid call is a store error", func(t *testing.T) { runOutcomeInvalidCall(t, newFixture(t, factory)) })
|
||||||
}
|
}
|
||||||
|
|
||||||
func runLifecycle(t *testing.T, fixture Fixture) {
|
func runLifecycle(t *testing.T, fixture Fixture) {
|
||||||
@ -117,10 +120,57 @@ func runSessionReplacementFence(t *testing.T, fixture Fixture) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func runOutcomeFence(t *testing.T, fixture Fixture) {
|
||||||
|
t.Helper()
|
||||||
|
ctx := context.Background()
|
||||||
|
open(t, fixture.Store, fixture.TTL)
|
||||||
|
first := workerruntime.OutcomeReference{
|
||||||
|
WorkerID: "worker-a", SessionID: "session-a", Sequence: 1,
|
||||||
|
Digest: sha256.Sum256([]byte("outcome-1")),
|
||||||
|
}
|
||||||
|
accepted, err := fixture.Outcomes.RecordOutcomes(ctx, first)
|
||||||
|
if err != nil || accepted != first.Sequence {
|
||||||
|
t.Fatalf("RecordOutcomes(first) = %d, %v; want 1, nil", accepted, err)
|
||||||
|
}
|
||||||
|
accepted, err = fixture.Outcomes.RecordOutcomes(ctx, first)
|
||||||
|
if err != nil || accepted != first.Sequence {
|
||||||
|
t.Fatalf("RecordOutcomes(replay) = %d, %v; want 1, nil", accepted, err)
|
||||||
|
}
|
||||||
|
conflicting := first
|
||||||
|
conflicting.Digest = sha256.Sum256([]byte("outcome-conflict"))
|
||||||
|
if _, err := fixture.Outcomes.RecordOutcomes(ctx, conflicting); !errors.Is(err, workerruntime.ErrConflictingOutcome) {
|
||||||
|
t.Fatalf("RecordOutcomes(conflict) = %v, want ErrConflictingOutcome", err)
|
||||||
|
}
|
||||||
|
next := first
|
||||||
|
next.Sequence = 2
|
||||||
|
next.Digest = sha256.Sum256([]byte("outcome-2"))
|
||||||
|
if accepted, err := fixture.Outcomes.RecordOutcomes(ctx, next); err != nil || accepted != next.Sequence {
|
||||||
|
t.Fatalf("RecordOutcomes(next) = %d, %v; want 2, nil", accepted, err)
|
||||||
|
}
|
||||||
|
if _, err := fixture.Outcomes.RecordOutcomes(ctx, first); !errors.Is(err, workerruntime.ErrStaleOutcome) {
|
||||||
|
t.Fatalf("RecordOutcomes(stale) = %v, want ErrStaleOutcome", err)
|
||||||
|
}
|
||||||
|
if err := fixture.Store.OpenSession(ctx, workerruntime.Session{
|
||||||
|
WorkerID: "worker-a", InstanceID: "instance-b", SessionID: "session-b", Zone: "zone-a", ProtocolVersion: 1,
|
||||||
|
}, fixture.TTL); err != nil {
|
||||||
|
t.Fatalf("OpenSession(replacement): %v", err)
|
||||||
|
}
|
||||||
|
if _, err := fixture.Outcomes.RecordOutcomes(ctx, first); !errors.Is(err, workerruntime.ErrStaleSession) {
|
||||||
|
t.Fatalf("RecordOutcomes(old session) = %v, want ErrStaleSession", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func runOutcomeInvalidCall(t *testing.T, fixture Fixture) {
|
||||||
|
t.Helper()
|
||||||
|
if _, err := fixture.Outcomes.RecordOutcomes(nil, workerruntime.OutcomeReference{}); !errors.Is(err, workerruntime.ErrInvalidStore) {
|
||||||
|
t.Fatalf("RecordOutcomes(nil) = %v, want ErrInvalidStore", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func newFixture(t *testing.T, factory Factory) Fixture {
|
func newFixture(t *testing.T, factory Factory) Fixture {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
fixture := factory(t)
|
fixture := factory(t)
|
||||||
if fixture.Store == nil || fixture.Reader == nil || fixture.TTL <= 0 || fixture.Advance == nil {
|
if fixture.Store == nil || fixture.Reader == nil || fixture.Outcomes == nil || fixture.TTL <= 0 || fixture.Advance == nil {
|
||||||
t.Fatal("contract fixture is incomplete")
|
t.Fatal("contract fixture is incomplete")
|
||||||
}
|
}
|
||||||
return fixture
|
return fixture
|
||||||
|
|||||||
@ -14,6 +14,7 @@ type MemoryStore struct {
|
|||||||
sessions map[string]memorySession
|
sessions map[string]memorySession
|
||||||
references map[string]memoryReference
|
references map[string]memoryReference
|
||||||
reports map[string]memoryReport
|
reports map[string]memoryReport
|
||||||
|
outcomes map[string]memoryOutcome
|
||||||
}
|
}
|
||||||
|
|
||||||
type memorySession struct {
|
type memorySession struct {
|
||||||
@ -33,11 +34,16 @@ type memoryReport struct {
|
|||||||
counters map[string]Counter
|
counters map[string]Counter
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type memoryOutcome struct {
|
||||||
|
value OutcomeReference
|
||||||
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
_ ControlStore = (*MemoryStore)(nil)
|
_ ControlStore = (*MemoryStore)(nil)
|
||||||
_ SessionWriter = (*MemoryStore)(nil)
|
_ SessionWriter = (*MemoryStore)(nil)
|
||||||
_ ReportWriter = (*MemoryStore)(nil)
|
_ ReportWriter = (*MemoryStore)(nil)
|
||||||
_ RuntimeReader = (*MemoryStore)(nil)
|
_ RuntimeReader = (*MemoryStore)(nil)
|
||||||
|
_ OutcomeWriter = (*MemoryStore)(nil)
|
||||||
)
|
)
|
||||||
|
|
||||||
func NewMemoryStore(now func() time.Time) (*MemoryStore, error) {
|
func NewMemoryStore(now func() time.Time) (*MemoryStore, error) {
|
||||||
@ -46,7 +52,7 @@ func NewMemoryStore(now func() time.Time) (*MemoryStore, error) {
|
|||||||
}
|
}
|
||||||
return &MemoryStore{
|
return &MemoryStore{
|
||||||
now: now, epoch: 1, sessions: make(map[string]memorySession),
|
now: now, epoch: 1, sessions: make(map[string]memorySession),
|
||||||
references: make(map[string]memoryReference), reports: make(map[string]memoryReport),
|
references: make(map[string]memoryReference), reports: make(map[string]memoryReport), outcomes: make(map[string]memoryOutcome),
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -85,6 +91,7 @@ func (store *MemoryStore) OpenSession(ctx context.Context, session Session, ttl
|
|||||||
defer store.mu.Unlock()
|
defer store.mu.Unlock()
|
||||||
delete(store.reports, normalized.WorkerID)
|
delete(store.reports, normalized.WorkerID)
|
||||||
delete(store.references, normalized.WorkerID)
|
delete(store.references, normalized.WorkerID)
|
||||||
|
delete(store.outcomes, normalized.WorkerID)
|
||||||
store.sessions[normalized.WorkerID] = memorySession{value: normalized, expiresAt: now.Add(ttl)}
|
store.sessions[normalized.WorkerID] = memorySession{value: normalized, expiresAt: now.Add(ttl)}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@ -246,12 +253,49 @@ func (store *MemoryStore) ReplaceSession(ctx context.Context, session Session, t
|
|||||||
ackAdvanced := exists && !identityChanged && !expired && legacySessionAfter(session, current.value)
|
ackAdvanced := exists && !identityChanged && !expired && legacySessionAfter(session, current.value)
|
||||||
if identityChanged || expired || ackAdvanced {
|
if identityChanged || expired || ackAdvanced {
|
||||||
delete(store.reports, session.WorkerID)
|
delete(store.reports, session.WorkerID)
|
||||||
|
delete(store.outcomes, session.WorkerID)
|
||||||
}
|
}
|
||||||
session.RuntimeEnabled = true
|
session.RuntimeEnabled = true
|
||||||
store.sessions[session.WorkerID] = memorySession{value: session, expiresAt: now.Add(ttl)}
|
store.sessions[session.WorkerID] = memorySession{value: session, expiresAt: now.Add(ttl)}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (store *MemoryStore) RecordOutcomes(ctx context.Context, reference OutcomeReference) (uint64, error) {
|
||||||
|
if ctx == nil || store == nil {
|
||||||
|
return 0, ErrInvalidStore
|
||||||
|
}
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
normalized, err := NormalizeOutcomeReference(reference)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
now, err := store.currentTime()
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
store.mu.Lock()
|
||||||
|
defer store.mu.Unlock()
|
||||||
|
session, exists := store.sessions[normalized.WorkerID]
|
||||||
|
if !exists || !session.expiresAt.After(now) || session.value.SessionID != normalized.SessionID {
|
||||||
|
return 0, ErrStaleSession
|
||||||
|
}
|
||||||
|
current, exists := store.outcomes[normalized.WorkerID]
|
||||||
|
if exists && current.value.SessionID == normalized.SessionID {
|
||||||
|
switch {
|
||||||
|
case normalized.Sequence < current.value.Sequence:
|
||||||
|
return 0, ErrStaleOutcome
|
||||||
|
case normalized.Sequence == current.value.Sequence && normalized.Digest != current.value.Digest:
|
||||||
|
return 0, ErrConflictingOutcome
|
||||||
|
case normalized.Sequence == current.value.Sequence:
|
||||||
|
return current.value.Sequence, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
store.outcomes[normalized.WorkerID] = memoryOutcome{value: normalized}
|
||||||
|
return normalized.Sequence, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (store *MemoryStore) ReplaceRuntime(ctx context.Context, report Report, ttl time.Duration) error {
|
func (store *MemoryStore) ReplaceRuntime(ctx context.Context, report Report, ttl time.Duration) error {
|
||||||
if ctx == nil || store == nil || ttl <= 0 {
|
if ctx == nil || store == nil || ttl <= 0 {
|
||||||
return ErrInvalidReport
|
return ErrInvalidReport
|
||||||
|
|||||||
51
internal/domain/workerruntime/outcome_test.go
Normal file
51
internal/domain/workerruntime/outcome_test.go
Normal file
@ -0,0 +1,51 @@
|
|||||||
|
package workerruntime
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestMemoryStoreFencesOutcomeSequenceBySessionAndDigest(t *testing.T) {
|
||||||
|
now := time.Date(2026, 7, 31, 12, 0, 0, 0, time.UTC)
|
||||||
|
store := newRuntimeStore(t, &now)
|
||||||
|
openControlSession(t, store, "session-a", time.Minute)
|
||||||
|
first := OutcomeReference{
|
||||||
|
WorkerID: "worker-a", SessionID: "session-a", Sequence: 1, Digest: sha256.Sum256([]byte("batch-1")),
|
||||||
|
}
|
||||||
|
accepted, err := store.RecordOutcomes(context.Background(), first)
|
||||||
|
if err != nil || accepted != 1 {
|
||||||
|
t.Fatalf("RecordOutcomes(first) = %d, %v; want 1, nil", accepted, err)
|
||||||
|
}
|
||||||
|
accepted, err = store.RecordOutcomes(context.Background(), first)
|
||||||
|
if err != nil || accepted != 1 {
|
||||||
|
t.Fatalf("RecordOutcomes(replay) = %d, %v; want 1, nil", accepted, err)
|
||||||
|
}
|
||||||
|
conflicting := first
|
||||||
|
conflicting.Digest = sha256.Sum256([]byte("different"))
|
||||||
|
if _, err := store.RecordOutcomes(context.Background(), conflicting); !errors.Is(err, ErrConflictingOutcome) {
|
||||||
|
t.Fatalf("RecordOutcomes(conflicting replay) error = %v, want ErrConflictingOutcome", err)
|
||||||
|
}
|
||||||
|
if _, err := store.RecordOutcomes(context.Background(), OutcomeReference{
|
||||||
|
WorkerID: "worker-a", SessionID: "session-a", Sequence: 0, Digest: first.Digest,
|
||||||
|
}); !errors.Is(err, ErrInvalidOutcome) {
|
||||||
|
t.Fatalf("RecordOutcomes(zero sequence) error = %v, want ErrInvalidOutcome", err)
|
||||||
|
}
|
||||||
|
if _, err := store.RecordOutcomes(context.Background(), OutcomeReference{
|
||||||
|
WorkerID: "worker-a", SessionID: "session-a", Sequence: 2, Digest: sha256.Sum256([]byte("batch-2")),
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("RecordOutcomes(next) error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := store.RecordOutcomes(context.Background(), OutcomeReference{
|
||||||
|
WorkerID: "worker-a", SessionID: "session-a", Sequence: 1, Digest: first.Digest,
|
||||||
|
}); !errors.Is(err, ErrStaleOutcome) {
|
||||||
|
t.Fatalf("RecordOutcomes(stale) error = %v, want ErrStaleOutcome", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
openControlSession(t, store, "session-b", time.Minute)
|
||||||
|
if _, err := store.RecordOutcomes(context.Background(), first); !errors.Is(err, ErrStaleSession) {
|
||||||
|
t.Fatalf("RecordOutcomes(old session) error = %v, want ErrStaleSession", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -21,6 +21,9 @@ var (
|
|||||||
ErrStaleSnapshotReference = errors.New("stale worker snapshot reference")
|
ErrStaleSnapshotReference = errors.New("stale worker snapshot reference")
|
||||||
ErrConflictingSnapshotReference = errors.New("conflicting worker snapshot reference")
|
ErrConflictingSnapshotReference = errors.New("conflicting worker snapshot reference")
|
||||||
ErrStaleAcknowledgement = errors.New("stale worker snapshot acknowledgement")
|
ErrStaleAcknowledgement = errors.New("stale worker snapshot acknowledgement")
|
||||||
|
ErrInvalidOutcome = errors.New("invalid worker outcome reference")
|
||||||
|
ErrStaleOutcome = errors.New("stale worker outcome sequence")
|
||||||
|
ErrConflictingOutcome = errors.New("conflicting worker outcome sequence")
|
||||||
)
|
)
|
||||||
|
|
||||||
type Session struct {
|
type Session struct {
|
||||||
@ -84,6 +87,15 @@ type Snapshot struct {
|
|||||||
Fresh bool
|
Fresh bool
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// OutcomeReference is the durable deduplication fence for one Gateway Outcome
|
||||||
|
// batch. The event payload itself remains outside the runtime store.
|
||||||
|
type OutcomeReference struct {
|
||||||
|
WorkerID string
|
||||||
|
SessionID string
|
||||||
|
Sequence uint64
|
||||||
|
Digest [sha256.Size]byte
|
||||||
|
}
|
||||||
|
|
||||||
type SessionWriter interface {
|
type SessionWriter interface {
|
||||||
ReplaceSession(context.Context, Session, time.Duration) error
|
ReplaceSession(context.Context, Session, time.Duration) error
|
||||||
}
|
}
|
||||||
@ -97,6 +109,10 @@ type ControlStore interface {
|
|||||||
ReplaceRuntime(context.Context, Report, time.Duration) error
|
ReplaceRuntime(context.Context, Report, time.Duration) error
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type OutcomeWriter interface {
|
||||||
|
RecordOutcomes(context.Context, OutcomeReference) (uint64, error)
|
||||||
|
}
|
||||||
|
|
||||||
type ReportWriter interface {
|
type ReportWriter interface {
|
||||||
ReplaceRuntime(context.Context, Report, time.Duration) error
|
ReplaceRuntime(context.Context, Report, time.Duration) error
|
||||||
}
|
}
|
||||||
|
|||||||
@ -82,6 +82,14 @@ func NormalizeAcknowledgement(acknowledgement SnapshotAcknowledgement) (Snapshot
|
|||||||
return acknowledgement, nil
|
return acknowledgement, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func NormalizeOutcomeReference(reference OutcomeReference) (OutcomeReference, error) {
|
||||||
|
if !ValidIdentifier(reference.WorkerID) || !ValidIdentifier(reference.SessionID) || reference.Sequence == 0 ||
|
||||||
|
checksumIsZero(reference.Digest) {
|
||||||
|
return OutcomeReference{}, ErrInvalidOutcome
|
||||||
|
}
|
||||||
|
return reference, nil
|
||||||
|
}
|
||||||
|
|
||||||
// NormalizeReport returns the canonical sparse replacement and its digest.
|
// NormalizeReport returns the canonical sparse replacement and its digest.
|
||||||
func NormalizeReport(report Report) (Report, [sha256.Size]byte, error) {
|
func NormalizeReport(report Report) (Report, [sha256.Size]byte, error) {
|
||||||
if !ValidIdentifier(report.WorkerID) || !ValidIdentifier(report.SessionID) || report.Sequence == 0 ||
|
if !ValidIdentifier(report.WorkerID) || !ValidIdentifier(report.SessionID) || report.Sequence == 0 ||
|
||||||
|
|||||||
@ -27,6 +27,7 @@ import (
|
|||||||
"proxy-pool/internal/domain/workerruntime"
|
"proxy-pool/internal/domain/workerruntime"
|
||||||
"proxy-pool/internal/gateway/controlplane"
|
"proxy-pool/internal/gateway/controlplane"
|
||||||
"proxy-pool/internal/gateway/dispatch"
|
"proxy-pool/internal/gateway/dispatch"
|
||||||
|
gatewayOutcome "proxy-pool/internal/gateway/outcome"
|
||||||
"proxy-pool/internal/gateway/server"
|
"proxy-pool/internal/gateway/server"
|
||||||
"proxy-pool/internal/gateway/snapshot"
|
"proxy-pool/internal/gateway/snapshot"
|
||||||
"proxy-pool/internal/gateway/transport"
|
"proxy-pool/internal/gateway/transport"
|
||||||
@ -45,6 +46,8 @@ const (
|
|||||||
defaultReconnectInitialDelay = time.Second
|
defaultReconnectInitialDelay = time.Second
|
||||||
defaultReconnectMaxDelay = 30 * time.Second
|
defaultReconnectMaxDelay = 30 * time.Second
|
||||||
defaultReconnectJitter = 20
|
defaultReconnectJitter = 20
|
||||||
|
defaultOutcomeQueueCapacity = 65_536
|
||||||
|
defaultOutcomeBatchSize = 512
|
||||||
)
|
)
|
||||||
|
|
||||||
// Options provides process-local settings. ControlPlaneAddress is deliberately
|
// Options provides process-local settings. ControlPlaneAddress is deliberately
|
||||||
@ -147,6 +150,13 @@ func newRuntime(ctx context.Context, configuration *config.Config, options Optio
|
|||||||
return nil, fmt.Errorf("build gateway target policy: %w", err)
|
return nil, fmt.Errorf("build gateway target policy: %w", err)
|
||||||
}
|
}
|
||||||
proxyTransport := transport.New(transport.Config{}, snapshotCredentialResolver{store: store})
|
proxyTransport := transport.New(transport.Config{}, snapshotCredentialResolver{store: store})
|
||||||
|
outcomes, err := gatewayOutcome.NewQueue(gatewayOutcome.QueueOptions{
|
||||||
|
Capacity: defaultOutcomeQueueCapacity, MaxBatch: min(defaultOutcomeBatchSize, configuration.ControlPlane.MaxRuntimeCounters),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
proxyTransport.CloseIdleConnections()
|
||||||
|
return nil, fmt.Errorf("build gateway outcome queue: %w", err)
|
||||||
|
}
|
||||||
handler, err := server.New(server.ConfigFromListener(configuration.Gateway), server.Dependencies{
|
handler, err := server.New(server.ConfigFromListener(configuration.Gateway), server.Dependencies{
|
||||||
Auth: protection.Auth,
|
Auth: protection.Auth,
|
||||||
Access: protection.Access,
|
Access: protection.Access,
|
||||||
@ -155,6 +165,7 @@ func newRuntime(ctx context.Context, configuration *config.Config, options Optio
|
|||||||
Router: server.NewSnapshotRouter(store),
|
Router: server.NewSnapshotRouter(store),
|
||||||
Dispatcher: dispatch.New(store),
|
Dispatcher: dispatch.New(store),
|
||||||
Transport: proxyTransport,
|
Transport: proxyTransport,
|
||||||
|
Outcomes: outcomes,
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
proxyTransport.CloseIdleConnections()
|
proxyTransport.CloseIdleConnections()
|
||||||
@ -198,7 +209,12 @@ func newRuntime(ctx context.Context, configuration *config.Config, options Optio
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("build snapshot watcher: %w", err)
|
return nil, fmt.Errorf("build snapshot watcher: %w", err)
|
||||||
}
|
}
|
||||||
session, err := controlplane.NewSessionRunner(reporter, watcher)
|
outcomeReporter, err := controlplane.NewOutcomeReporter(controlplane.NewGeneratedOutcomeRPCClient(client), outcomes,
|
||||||
|
controlplane.OutcomeReporterOptions{WorkerID: options.WorkerID})
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("build outcome reporter: %w", err)
|
||||||
|
}
|
||||||
|
session, err := controlplane.NewSessionRunner(reporter, watcher, outcomeReporter)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("build control plane session: %w", err)
|
return nil, fmt.Errorf("build control plane session: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
254
internal/gateway/controlplane/outcome_reporter.go
Normal file
254
internal/gateway/controlplane/outcome_reporter.go
Normal file
@ -0,0 +1,254 @@
|
|||||||
|
package controlplane
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
controlplanev1 "proxy-pool/gen/controlplane/v1"
|
||||||
|
domain "proxy-pool/internal/domain/outcome"
|
||||||
|
"proxy-pool/internal/domain/workerruntime"
|
||||||
|
gatewayOutcome "proxy-pool/internal/gateway/outcome"
|
||||||
|
|
||||||
|
"google.golang.org/grpc/codes"
|
||||||
|
"google.golang.org/grpc/status"
|
||||||
|
"google.golang.org/protobuf/types/known/durationpb"
|
||||||
|
"google.golang.org/protobuf/types/known/timestamppb"
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
ErrInvalidOutcomeReporter = errors.New("invalid gateway outcome reporter")
|
||||||
|
ErrOutcomeUnconfirmed = errors.New("controller did not confirm gateway outcome batch")
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
outcomeRetryInitialDelay = 100 * time.Millisecond
|
||||||
|
outcomeRetryMaxDelay = 5 * time.Second
|
||||||
|
)
|
||||||
|
|
||||||
|
type OutcomeStream interface {
|
||||||
|
Send(*controlplanev1.OutcomeBatch) error
|
||||||
|
CloseAndRecv() (*controlplanev1.ReportOutcomesResponse, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type OutcomeRPCClient interface {
|
||||||
|
ReportOutcomes(context.Context) (OutcomeStream, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type OutcomeReporterOptions struct {
|
||||||
|
WorkerID string
|
||||||
|
MaxBatchesPerReportRPC int
|
||||||
|
}
|
||||||
|
|
||||||
|
type OutcomeReporter struct {
|
||||||
|
client OutcomeRPCClient
|
||||||
|
queue *gatewayOutcome.Queue
|
||||||
|
options OutcomeReporterOptions
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
sessionID string
|
||||||
|
sequence uint64
|
||||||
|
pending []domain.Batch
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewOutcomeReporter(client OutcomeRPCClient, queue *gatewayOutcome.Queue, options OutcomeReporterOptions) (*OutcomeReporter, error) {
|
||||||
|
if client == nil || queue == nil || !workerruntime.ValidIdentifier(options.WorkerID) {
|
||||||
|
return nil, ErrInvalidOutcomeReporter
|
||||||
|
}
|
||||||
|
if options.MaxBatchesPerReportRPC <= 0 {
|
||||||
|
options.MaxBatchesPerReportRPC = domain.MaxBatchesPerStream
|
||||||
|
}
|
||||||
|
if options.MaxBatchesPerReportRPC > domain.MaxBatchesPerStream {
|
||||||
|
return nil, ErrInvalidOutcomeReporter
|
||||||
|
}
|
||||||
|
return &OutcomeReporter{client: client, queue: queue, options: options}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// RunRegistered continuously batches local observations for a single
|
||||||
|
// Controller session. Transient delivery failures retain the exact pending
|
||||||
|
// batches and retry them in the same session with bounded backoff.
|
||||||
|
func (reporter *OutcomeReporter) RunRegistered(ctx context.Context, registration Registration) error {
|
||||||
|
if reporter == nil || ctx == nil || !workerruntime.ValidIdentifier(registration.SessionID) {
|
||||||
|
return ErrInvalidOutcomeReporter
|
||||||
|
}
|
||||||
|
reporter.mu.Lock()
|
||||||
|
reporter.sessionID = registration.SessionID
|
||||||
|
reporter.sequence = 0
|
||||||
|
reporter.pending = nil
|
||||||
|
reporter.mu.Unlock()
|
||||||
|
retryDelay := outcomeRetryInitialDelay
|
||||||
|
for {
|
||||||
|
err := reporter.Report(ctx, registration.SessionID)
|
||||||
|
if err == nil {
|
||||||
|
retryDelay = outcomeRetryInitialDelay
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return ctx.Err()
|
||||||
|
}
|
||||||
|
if terminalOutcomeError(err) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := waitOutcomeRetry(ctx, retryDelay); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
retryDelay = nextOutcomeRetryDelay(retryDelay)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func terminalOutcomeError(err error) bool {
|
||||||
|
if errors.Is(err, ErrInvalidOutcomeReporter) || errors.Is(err, ErrNotRegistered) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
switch status.Code(err) {
|
||||||
|
case codes.Aborted, codes.AlreadyExists, codes.FailedPrecondition,
|
||||||
|
codes.InvalidArgument, codes.PermissionDenied, codes.Unauthenticated:
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func waitOutcomeRetry(ctx context.Context, delay time.Duration) error {
|
||||||
|
timer := time.NewTimer(delay)
|
||||||
|
defer timer.Stop()
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
case <-timer.C:
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func nextOutcomeRetryDelay(delay time.Duration) time.Duration {
|
||||||
|
if delay >= outcomeRetryMaxDelay/2 {
|
||||||
|
return outcomeRetryMaxDelay
|
||||||
|
}
|
||||||
|
return delay * 2
|
||||||
|
}
|
||||||
|
|
||||||
|
func (reporter *OutcomeReporter) Report(ctx context.Context, sessionID string) error {
|
||||||
|
if reporter == nil || ctx == nil || !workerruntime.ValidIdentifier(sessionID) {
|
||||||
|
return ErrInvalidOutcomeReporter
|
||||||
|
}
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
reporter.mu.Lock()
|
||||||
|
defer reporter.mu.Unlock()
|
||||||
|
if reporter.sessionID != sessionID {
|
||||||
|
return ErrNotRegistered
|
||||||
|
}
|
||||||
|
if err := reporter.fillPending(ctx); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
streamCtx, cancel := context.WithCancel(ctx)
|
||||||
|
defer cancel()
|
||||||
|
stream, err := reporter.client.ReportOutcomes(streamCtx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for _, batch := range reporter.pending {
|
||||||
|
request, requestErr := outcomeRequest(batch)
|
||||||
|
if requestErr != nil {
|
||||||
|
return requestErr
|
||||||
|
}
|
||||||
|
if sendErr := stream.Send(request); sendErr != nil {
|
||||||
|
return sendErr
|
||||||
|
}
|
||||||
|
}
|
||||||
|
response, err := stream.CloseAndRecv()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
last := reporter.pending[len(reporter.pending)-1].Sequence
|
||||||
|
if response == nil || response.GetAcceptedThroughSequence() != last {
|
||||||
|
return ErrOutcomeUnconfirmed
|
||||||
|
}
|
||||||
|
reporter.sequence = last
|
||||||
|
reporter.pending = nil
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (reporter *OutcomeReporter) fillPending(ctx context.Context) error {
|
||||||
|
if len(reporter.pending) == 0 {
|
||||||
|
events, err := reporter.queue.Next(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := reporter.append(events); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for len(reporter.pending) < reporter.options.MaxBatchesPerReportRPC {
|
||||||
|
events, ok := reporter.queue.TryNext()
|
||||||
|
if !ok {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if err := reporter.append(events); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (reporter *OutcomeReporter) append(events []domain.Event) error {
|
||||||
|
batch, _, err := domain.NormalizeBatch(domain.Batch{
|
||||||
|
WorkerID: reporter.options.WorkerID, SessionID: reporter.sessionID,
|
||||||
|
Sequence: reporter.sequence + uint64(len(reporter.pending)) + 1, Events: events,
|
||||||
|
}, reporter.queue.MaxBatch())
|
||||||
|
if err != nil {
|
||||||
|
return errors.Join(ErrInvalidOutcomeReporter, err)
|
||||||
|
}
|
||||||
|
reporter.pending = append(reporter.pending, batch)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func outcomeRequest(batch domain.Batch) (*controlplanev1.OutcomeBatch, error) {
|
||||||
|
proxies := make([]*controlplanev1.ProxyOutcome, len(batch.Events))
|
||||||
|
for index, event := range batch.Events {
|
||||||
|
stage, ok := outcomeStage(event.Stage)
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("%w: unknown outcome stage", ErrInvalidOutcomeReporter)
|
||||||
|
}
|
||||||
|
proxies[index] = &controlplanev1.ProxyOutcome{
|
||||||
|
ProxyId: event.ProxyID, RoutingName: event.RoutingName, Stage: stage, Success: event.Success,
|
||||||
|
ErrorClass: string(event.ErrorClass), Latency: durationpb.New(event.Latency), ObservedAt: timestamppb.New(event.ObservedAt),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return &controlplanev1.OutcomeBatch{
|
||||||
|
WorkerId: batch.WorkerID, SessionId: batch.SessionID, Sequence: batch.Sequence, Outcomes: proxies,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func outcomeStage(stage domain.Stage) (controlplanev1.OutcomeStage, bool) {
|
||||||
|
switch stage {
|
||||||
|
case domain.StageDial:
|
||||||
|
return controlplanev1.OutcomeStage_OUTCOME_STAGE_DIAL, true
|
||||||
|
case domain.StageProxyHandshake:
|
||||||
|
return controlplanev1.OutcomeStage_OUTCOME_STAGE_PROXY_HANDSHAKE, true
|
||||||
|
case domain.StageResponseHeaders:
|
||||||
|
return controlplanev1.OutcomeStage_OUTCOME_STAGE_RESPONSE_HEADERS, true
|
||||||
|
case domain.StageTunnel:
|
||||||
|
return controlplanev1.OutcomeStage_OUTCOME_STAGE_TUNNEL, true
|
||||||
|
default:
|
||||||
|
return controlplanev1.OutcomeStage_OUTCOME_STAGE_UNSPECIFIED, false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type generatedOutcomeRPCClient struct {
|
||||||
|
client controlplanev1.WorkerControlPlaneClient
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewGeneratedOutcomeRPCClient(client controlplanev1.WorkerControlPlaneClient) OutcomeRPCClient {
|
||||||
|
return generatedOutcomeRPCClient{client: client}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (client generatedOutcomeRPCClient) ReportOutcomes(ctx context.Context) (OutcomeStream, error) {
|
||||||
|
if client.client == nil {
|
||||||
|
return nil, ErrInvalidOutcomeReporter
|
||||||
|
}
|
||||||
|
return client.client.ReportOutcomes(ctx)
|
||||||
|
}
|
||||||
175
internal/gateway/controlplane/outcome_reporter_test.go
Normal file
175
internal/gateway/controlplane/outcome_reporter_test.go
Normal file
@ -0,0 +1,175 @@
|
|||||||
|
package controlplane
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
controlplanev1 "proxy-pool/gen/controlplane/v1"
|
||||||
|
domain "proxy-pool/internal/domain/outcome"
|
||||||
|
gatewayOutcome "proxy-pool/internal/gateway/outcome"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestOutcomeReporterRetriesExactPendingBatchAfterTransportFailure(t *testing.T) {
|
||||||
|
queue, err := gatewayOutcome.NewQueue(gatewayOutcome.QueueOptions{Capacity: 4, MaxBatch: 2})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewQueue() = %v", err)
|
||||||
|
}
|
||||||
|
queue.Record(domain.Event{ProxyID: "proxy-a", Stage: domain.StageDial, Success: true, ObservedAt: time.Now()})
|
||||||
|
client := &outcomeClientStub{closeErr: errors.New("connection dropped")}
|
||||||
|
reporter, err := NewOutcomeReporter(client, queue, OutcomeReporterOptions{WorkerID: "worker-a"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewOutcomeReporter() = %v", err)
|
||||||
|
}
|
||||||
|
registration := Registration{SessionID: "session-a"}
|
||||||
|
reporter.sessionID = registration.SessionID
|
||||||
|
if err := reporter.Report(context.Background(), registration.SessionID); !errors.Is(err, client.closeErr) {
|
||||||
|
t.Fatalf("Report(first) = %v, want transport failure", err)
|
||||||
|
}
|
||||||
|
client.closeErr = nil
|
||||||
|
if err := reporter.Report(context.Background(), registration.SessionID); err != nil {
|
||||||
|
t.Fatalf("Report(retry) = %v", err)
|
||||||
|
}
|
||||||
|
if len(client.batches) != 2 || client.batches[0].GetSequence() != 1 || client.batches[1].GetSequence() != 1 ||
|
||||||
|
client.batches[0].GetOutcomes()[0].GetProxyId() != "proxy-a" {
|
||||||
|
t.Fatalf("sent batches = %+v", client.batches)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOutcomeReporterCombinesAvailableBatchesAndRequiresConfirmation(t *testing.T) {
|
||||||
|
queue, err := gatewayOutcome.NewQueue(gatewayOutcome.QueueOptions{Capacity: 8, MaxBatch: 1})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewQueue() = %v", err)
|
||||||
|
}
|
||||||
|
for _, proxyID := range []string{"proxy-a", "proxy-b"} {
|
||||||
|
queue.Record(domain.Event{ProxyID: proxyID, Stage: domain.StageDial, Success: true, ObservedAt: time.Now()})
|
||||||
|
}
|
||||||
|
client := &outcomeClientStub{}
|
||||||
|
reporter, err := NewOutcomeReporter(client, queue, OutcomeReporterOptions{WorkerID: "worker-a", MaxBatchesPerReportRPC: 2})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewOutcomeReporter() = %v", err)
|
||||||
|
}
|
||||||
|
reporter.sessionID = "session-a"
|
||||||
|
if err := reporter.Report(context.Background(), "session-a"); err != nil {
|
||||||
|
t.Fatalf("Report() = %v", err)
|
||||||
|
}
|
||||||
|
if len(client.batches) != 2 || client.batches[0].GetSequence() != 1 || client.batches[1].GetSequence() != 2 {
|
||||||
|
t.Fatalf("sent batches = %+v", client.batches)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOutcomeReporterRetriesTransientFailureWithinRegisteredSession(t *testing.T) {
|
||||||
|
queue, err := gatewayOutcome.NewQueue(gatewayOutcome.QueueOptions{Capacity: 4, MaxBatch: 2})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewQueue() = %v", err)
|
||||||
|
}
|
||||||
|
queue.Record(domain.Event{ProxyID: "proxy-a", Stage: domain.StageDial, Success: true, ObservedAt: time.Now()})
|
||||||
|
client := &retryingOutcomeClient{failures: 1, confirmed: make(chan struct{}, 1)}
|
||||||
|
reporter, err := NewOutcomeReporter(client, queue, OutcomeReporterOptions{WorkerID: "worker-a"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewOutcomeReporter() = %v", err)
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
done := make(chan error, 1)
|
||||||
|
go func() { done <- reporter.RunRegistered(ctx, Registration{SessionID: "session-a"}) }()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-client.confirmed:
|
||||||
|
case <-time.After(time.Second):
|
||||||
|
t.Fatal("outcome batch was not confirmed after transient failure")
|
||||||
|
}
|
||||||
|
if batches := client.Batches(); len(batches) != 2 || batches[0].GetSequence() != 1 ||
|
||||||
|
batches[1].GetSequence() != 1 || batches[0].GetOutcomes()[0].GetProxyId() != "proxy-a" {
|
||||||
|
t.Fatalf("sent batches = %+v, want exact retry in the same session", batches)
|
||||||
|
}
|
||||||
|
cancel()
|
||||||
|
if err := <-done; !errors.Is(err, context.Canceled) {
|
||||||
|
t.Fatalf("RunRegistered() = %v, want context canceled", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOutcomeReporterRejectsUnboundedStreamOptions(t *testing.T) {
|
||||||
|
queue, err := gatewayOutcome.NewQueue(gatewayOutcome.QueueOptions{Capacity: 1, MaxBatch: 1})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewQueue() = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := NewOutcomeReporter(&outcomeClientStub{}, queue, OutcomeReporterOptions{
|
||||||
|
WorkerID: "worker-a", MaxBatchesPerReportRPC: domain.MaxBatchesPerStream + 1,
|
||||||
|
}); !errors.Is(err, ErrInvalidOutcomeReporter) {
|
||||||
|
t.Fatalf("NewOutcomeReporter() error = %v, want ErrInvalidOutcomeReporter", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type outcomeClientStub struct {
|
||||||
|
batches []*controlplanev1.OutcomeBatch
|
||||||
|
closeErr error
|
||||||
|
accepted uint64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (client *outcomeClientStub) ReportOutcomes(context.Context) (OutcomeStream, error) {
|
||||||
|
return &outcomeStreamStub{client: client}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type outcomeStreamStub struct{ client *outcomeClientStub }
|
||||||
|
|
||||||
|
func (stream *outcomeStreamStub) Send(batch *controlplanev1.OutcomeBatch) error {
|
||||||
|
stream.client.batches = append(stream.client.batches, batch)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (stream *outcomeStreamStub) CloseAndRecv() (*controlplanev1.ReportOutcomesResponse, error) {
|
||||||
|
if stream.client.closeErr != nil {
|
||||||
|
return nil, stream.client.closeErr
|
||||||
|
}
|
||||||
|
accepted := stream.client.accepted
|
||||||
|
if accepted == 0 && len(stream.client.batches) > 0 {
|
||||||
|
accepted = stream.client.batches[len(stream.client.batches)-1].GetSequence()
|
||||||
|
}
|
||||||
|
return &controlplanev1.ReportOutcomesResponse{AcceptedThroughSequence: accepted}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type retryingOutcomeClient struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
batches []*controlplanev1.OutcomeBatch
|
||||||
|
failures int
|
||||||
|
confirmed chan struct{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (client *retryingOutcomeClient) ReportOutcomes(context.Context) (OutcomeStream, error) {
|
||||||
|
return &retryingOutcomeStream{client: client}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (client *retryingOutcomeClient) Batches() []*controlplanev1.OutcomeBatch {
|
||||||
|
client.mu.Lock()
|
||||||
|
defer client.mu.Unlock()
|
||||||
|
return append([]*controlplanev1.OutcomeBatch(nil), client.batches...)
|
||||||
|
}
|
||||||
|
|
||||||
|
type retryingOutcomeStream struct{ client *retryingOutcomeClient }
|
||||||
|
|
||||||
|
func (stream *retryingOutcomeStream) Send(batch *controlplanev1.OutcomeBatch) error {
|
||||||
|
stream.client.mu.Lock()
|
||||||
|
stream.client.batches = append(stream.client.batches, batch)
|
||||||
|
stream.client.mu.Unlock()
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (stream *retryingOutcomeStream) CloseAndRecv() (*controlplanev1.ReportOutcomesResponse, error) {
|
||||||
|
stream.client.mu.Lock()
|
||||||
|
if stream.client.failures > 0 {
|
||||||
|
stream.client.failures--
|
||||||
|
stream.client.mu.Unlock()
|
||||||
|
return nil, errors.New("temporary delivery failure")
|
||||||
|
}
|
||||||
|
accepted := stream.client.batches[len(stream.client.batches)-1].GetSequence()
|
||||||
|
confirmed := stream.client.confirmed
|
||||||
|
stream.client.mu.Unlock()
|
||||||
|
select {
|
||||||
|
case confirmed <- struct{}{}:
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
return &controlplanev1.ReportOutcomesResponse{AcceptedThroughSequence: accepted}, nil
|
||||||
|
}
|
||||||
@ -10,13 +10,18 @@ var ErrSnapshotStreamClosed = errors.New("worker snapshot stream closed")
|
|||||||
type SessionRunner struct {
|
type SessionRunner struct {
|
||||||
reporter *RuntimeReporter
|
reporter *RuntimeReporter
|
||||||
watcher *SnapshotWatcher
|
watcher *SnapshotWatcher
|
||||||
|
outcomes *OutcomeReporter
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewSessionRunner(reporter *RuntimeReporter, watcher *SnapshotWatcher) (*SessionRunner, error) {
|
func NewSessionRunner(reporter *RuntimeReporter, watcher *SnapshotWatcher, outcomes ...*OutcomeReporter) (*SessionRunner, error) {
|
||||||
if reporter == nil || watcher == nil {
|
if reporter == nil || watcher == nil || len(outcomes) > 1 || (len(outcomes) == 1 && outcomes[0] == nil) {
|
||||||
return nil, ErrInvalidOptions
|
return nil, ErrInvalidOptions
|
||||||
}
|
}
|
||||||
return &SessionRunner{reporter: reporter, watcher: watcher}, nil
|
runner := &SessionRunner{reporter: reporter, watcher: watcher}
|
||||||
|
if len(outcomes) == 1 {
|
||||||
|
runner.outcomes = outcomes[0]
|
||||||
|
}
|
||||||
|
return runner, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// Run registers exactly once, then runs snapshot intake and Runtime reporting
|
// Run registers exactly once, then runs snapshot intake and Runtime reporting
|
||||||
@ -32,21 +37,36 @@ func (runner *SessionRunner) Run(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
groupCtx, cancel := context.WithCancel(ctx)
|
groupCtx, cancel := context.WithCancel(ctx)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
results := make(chan error, 2)
|
runners := []func() error{
|
||||||
go func() { results <- runner.watcher.Watch(groupCtx, registration.SessionID) }()
|
func() error { return runner.watcher.Watch(groupCtx, registration.SessionID) },
|
||||||
go func() { results <- runner.reporter.RunRegistered(groupCtx) }()
|
func() error { return runner.reporter.RunRegistered(groupCtx) },
|
||||||
|
}
|
||||||
|
if runner.outcomes != nil {
|
||||||
|
runners = append(runners, func() error { return runner.outcomes.RunRegistered(groupCtx, registration) })
|
||||||
|
}
|
||||||
|
results := make(chan error, len(runners))
|
||||||
|
for _, run := range runners {
|
||||||
|
go func(run func() error) { results <- run() }(run)
|
||||||
|
}
|
||||||
|
|
||||||
first := <-results
|
first := <-results
|
||||||
cancel()
|
cancel()
|
||||||
second := <-results
|
others := make([]error, 0, len(runners)-1)
|
||||||
|
for range len(runners) - 1 {
|
||||||
|
others = append(others, <-results)
|
||||||
|
}
|
||||||
if ctx.Err() != nil {
|
if ctx.Err() != nil {
|
||||||
return ctx.Err()
|
return ctx.Err()
|
||||||
}
|
}
|
||||||
if first == nil {
|
if first == nil {
|
||||||
return ErrSnapshotStreamClosed
|
return ErrSnapshotStreamClosed
|
||||||
}
|
}
|
||||||
if errors.Is(first, context.Canceled) && second != nil {
|
if errors.Is(first, context.Canceled) {
|
||||||
return second
|
for _, other := range others {
|
||||||
|
if other != nil && !errors.Is(other, context.Canceled) {
|
||||||
|
return other
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return first
|
return first
|
||||||
}
|
}
|
||||||
|
|||||||
99
internal/gateway/outcome/queue.go
Normal file
99
internal/gateway/outcome/queue.go
Normal file
@ -0,0 +1,99 @@
|
|||||||
|
// Package outcome provides the Gateway's bounded, non-blocking observation
|
||||||
|
// buffer. It deliberately has no dependency on Controller or storage code.
|
||||||
|
package outcome
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"sync/atomic"
|
||||||
|
|
||||||
|
domain "proxy-pool/internal/domain/outcome"
|
||||||
|
)
|
||||||
|
|
||||||
|
var ErrInvalidQueueOptions = errors.New("invalid gateway outcome queue options")
|
||||||
|
|
||||||
|
type QueueOptions struct {
|
||||||
|
Capacity int
|
||||||
|
MaxBatch int
|
||||||
|
}
|
||||||
|
|
||||||
|
// Queue accepts observations from the request path without blocking. Events
|
||||||
|
// beyond Capacity are intentionally dropped because outcome telemetry must not
|
||||||
|
// delay proxy forwarding or consume unbounded process memory.
|
||||||
|
type Queue struct {
|
||||||
|
events chan domain.Event
|
||||||
|
maxBatch int
|
||||||
|
dropped atomic.Uint64
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewQueue(options QueueOptions) (*Queue, error) {
|
||||||
|
if options.Capacity <= 0 || options.MaxBatch <= 0 || options.MaxBatch > options.Capacity {
|
||||||
|
return nil, ErrInvalidQueueOptions
|
||||||
|
}
|
||||||
|
return &Queue{events: make(chan domain.Event, options.Capacity), maxBatch: options.MaxBatch}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (queue *Queue) Record(event domain.Event) {
|
||||||
|
if queue == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case queue.events <- event:
|
||||||
|
default:
|
||||||
|
queue.dropped.Add(1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (queue *Queue) Dropped() uint64 {
|
||||||
|
if queue == nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return queue.dropped.Load()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (queue *Queue) MaxBatch() int {
|
||||||
|
if queue == nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return queue.maxBatch
|
||||||
|
}
|
||||||
|
|
||||||
|
// Next waits for one event then drains an immediately available bounded batch.
|
||||||
|
func (queue *Queue) Next(ctx context.Context) ([]domain.Event, error) {
|
||||||
|
if queue == nil || ctx == nil {
|
||||||
|
return nil, ErrInvalidQueueOptions
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return nil, ctx.Err()
|
||||||
|
case first := <-queue.events:
|
||||||
|
return queue.drain(first), nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TryNext drains one batch only when work is already buffered.
|
||||||
|
func (queue *Queue) TryNext() ([]domain.Event, bool) {
|
||||||
|
if queue == nil {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case first := <-queue.events:
|
||||||
|
return queue.drain(first), true
|
||||||
|
default:
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (queue *Queue) drain(first domain.Event) []domain.Event {
|
||||||
|
batch := make([]domain.Event, 0, queue.maxBatch)
|
||||||
|
batch = append(batch, first)
|
||||||
|
for len(batch) < queue.maxBatch {
|
||||||
|
select {
|
||||||
|
case event := <-queue.events:
|
||||||
|
batch = append(batch, event)
|
||||||
|
default:
|
||||||
|
return batch
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return batch
|
||||||
|
}
|
||||||
41
internal/gateway/outcome/queue_test.go
Normal file
41
internal/gateway/outcome/queue_test.go
Normal file
@ -0,0 +1,41 @@
|
|||||||
|
package outcome
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
domain "proxy-pool/internal/domain/outcome"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestQueueBatchesWithoutBlockingAndCountsOverflow(t *testing.T) {
|
||||||
|
queue, err := NewQueue(QueueOptions{Capacity: 2, MaxBatch: 2})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewQueue() = %v", err)
|
||||||
|
}
|
||||||
|
queue.Record(domain.Event{ProxyID: "proxy-a"})
|
||||||
|
queue.Record(domain.Event{ProxyID: "proxy-b"})
|
||||||
|
queue.Record(domain.Event{ProxyID: "proxy-c"})
|
||||||
|
if queue.Dropped() != 1 {
|
||||||
|
t.Fatalf("Dropped() = %d, want 1", queue.Dropped())
|
||||||
|
}
|
||||||
|
batch, err := queue.Next(context.Background())
|
||||||
|
if err != nil || len(batch) != 2 || batch[0].ProxyID != "proxy-a" || batch[1].ProxyID != "proxy-b" {
|
||||||
|
t.Fatalf("Next() = %+v, %v", batch, err)
|
||||||
|
}
|
||||||
|
if batch, ok := queue.TryNext(); ok || batch != nil {
|
||||||
|
t.Fatalf("TryNext() = %+v, %t; want nil, false", batch, ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestQueueHonorsCanceledContext(t *testing.T) {
|
||||||
|
queue, err := NewQueue(QueueOptions{Capacity: 1, MaxBatch: 1})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewQueue() = %v", err)
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
if _, err := queue.Next(ctx); !errors.Is(err, context.Canceled) {
|
||||||
|
t.Fatalf("Next(canceled) = %v, want context canceled", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -14,6 +14,7 @@ import (
|
|||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
outcomeDomain "proxy-pool/internal/domain/outcome"
|
||||||
proxyDomain "proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
"proxy-pool/internal/domain/routing"
|
"proxy-pool/internal/domain/routing"
|
||||||
"proxy-pool/internal/gateway/dispatch"
|
"proxy-pool/internal/gateway/dispatch"
|
||||||
@ -78,6 +79,12 @@ type DirectTransport interface {
|
|||||||
OpenDirectTunnel(context.Context, string) (net.Conn, error)
|
OpenDirectTunnel(context.Context, string) (net.Conn, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// OutcomeRecorder accepts best-effort proxy observations. Implementations must
|
||||||
|
// return immediately; request forwarding never waits for telemetry delivery.
|
||||||
|
type OutcomeRecorder interface {
|
||||||
|
Record(outcomeDomain.Event)
|
||||||
|
}
|
||||||
|
|
||||||
type waitingDispatcher interface {
|
type waitingDispatcher interface {
|
||||||
AcquireWait(context.Context, dispatch.Request, time.Duration) (*dispatch.Lease, error)
|
AcquireWait(context.Context, dispatch.Request, time.Duration) (*dispatch.Lease, error)
|
||||||
}
|
}
|
||||||
@ -90,6 +97,7 @@ type Dependencies struct {
|
|||||||
Router Router
|
Router Router
|
||||||
Dispatcher Dispatcher
|
Dispatcher Dispatcher
|
||||||
Transport ProxyTransport
|
Transport ProxyTransport
|
||||||
|
Outcomes OutcomeRecorder
|
||||||
}
|
}
|
||||||
|
|
||||||
type Handler struct {
|
type Handler struct {
|
||||||
@ -99,6 +107,7 @@ type Handler struct {
|
|||||||
router Router
|
router Router
|
||||||
dispatcher Dispatcher
|
dispatcher Dispatcher
|
||||||
transport ProxyTransport
|
transport ProxyTransport
|
||||||
|
outcomes OutcomeRecorder
|
||||||
buffers sync.Pool
|
buffers sync.Pool
|
||||||
inFlight chan struct{}
|
inFlight chan struct{}
|
||||||
forceClose atomic.Bool
|
forceClose atomic.Bool
|
||||||
@ -136,6 +145,7 @@ func New(config Config, dependencies Dependencies) (*Handler, error) {
|
|||||||
router: dependencies.Router,
|
router: dependencies.Router,
|
||||||
dispatcher: dependencies.Dispatcher,
|
dispatcher: dependencies.Dispatcher,
|
||||||
transport: dependencies.Transport,
|
transport: dependencies.Transport,
|
||||||
|
outcomes: dependencies.Outcomes,
|
||||||
tunnels: make(map[*activeTunnel]struct{}),
|
tunnels: make(map[*activeTunnel]struct{}),
|
||||||
shutdownDone: make(chan struct{}),
|
shutdownDone: make(chan struct{}),
|
||||||
}
|
}
|
||||||
@ -230,14 +240,16 @@ func (handler *Handler) connect(
|
|||||||
lastErr = directErr
|
lastErr = directErr
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
handler.serveTunnel(writer, request, nil, upstream)
|
handler.serveTunnel(writer, request, nil, upstream, "", "")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
|
|
||||||
|
started := time.Now().UTC()
|
||||||
upstream, err := handler.transport.OpenTunnel(request.Context(), lease.Proxy, target.DialAddress())
|
upstream, err := handler.transport.OpenTunnel(request.Context(), lease.Proxy, target.DialAddress())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
handler.recordOutcome(lease.Proxy.ID, route.RoutingName, outcomeDomain.StageProxyHandshake, false, err, started)
|
||||||
finishLease(lease, false)
|
finishLease(lease, false)
|
||||||
excluded[lease.Proxy.ID] = struct{}{}
|
excluded[lease.Proxy.ID] = struct{}{}
|
||||||
var responseError *transportDomain.ProxyResponseError
|
var responseError *transportDomain.ProxyResponseError
|
||||||
@ -259,7 +271,8 @@ func (handler *Handler) connect(
|
|||||||
lastErr = err
|
lastErr = err
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
handler.serveTunnel(writer, request, func() { finishLease(lease, true) }, upstream)
|
handler.recordOutcome(lease.Proxy.ID, route.RoutingName, outcomeDomain.StageProxyHandshake, true, nil, started)
|
||||||
|
handler.serveTunnel(writer, request, func() { finishLease(lease, true) }, upstream, lease.Proxy.ID, route.RoutingName)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -279,6 +292,8 @@ func (handler *Handler) serveTunnel(
|
|||||||
request *http.Request,
|
request *http.Request,
|
||||||
finish func(),
|
finish func(),
|
||||||
upstream net.Conn,
|
upstream net.Conn,
|
||||||
|
proxyID string,
|
||||||
|
routingName string,
|
||||||
) {
|
) {
|
||||||
if finish != nil {
|
if finish != nil {
|
||||||
defer finish()
|
defer finish()
|
||||||
@ -287,11 +302,13 @@ func (handler *Handler) serveTunnel(
|
|||||||
|
|
||||||
hijacker, ok := writer.(http.Hijacker)
|
hijacker, ok := writer.(http.Hijacker)
|
||||||
if !ok {
|
if !ok {
|
||||||
|
handler.recordOutcome(proxyID, routingName, outcomeDomain.StageTunnel, false, errors.New("gateway response does not support connection hijacking"), time.Now().UTC())
|
||||||
http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
|
http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
client, readWriter, err := hijacker.Hijack()
|
client, readWriter, err := hijacker.Hijack()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
handler.recordOutcome(proxyID, routingName, outcomeDomain.StageTunnel, false, err, time.Now().UTC())
|
||||||
http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
|
http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@ -302,14 +319,18 @@ func (handler *Handler) serveTunnel(
|
|||||||
}
|
}
|
||||||
defer handler.unregisterTunnel(tunnel)
|
defer handler.unregisterTunnel(tunnel)
|
||||||
|
|
||||||
|
started := time.Now().UTC()
|
||||||
if _, err := readWriter.WriteString("HTTP/1.1 200 Connection Established\r\n\r\n"); err != nil {
|
if _, err := readWriter.WriteString("HTTP/1.1 200 Connection Established\r\n\r\n"); err != nil {
|
||||||
|
handler.recordOutcome(proxyID, routingName, outcomeDomain.StageTunnel, false, err, started)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if err := readWriter.Flush(); err != nil {
|
if err := readWriter.Flush(); err != nil {
|
||||||
|
handler.recordOutcome(proxyID, routingName, outcomeDomain.StageTunnel, false, err, started)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
bufferedClient := &bufferedClientConn{Conn: client, reader: readWriter.Reader}
|
bufferedClient := &bufferedClientConn{Conn: client, reader: readWriter.Reader}
|
||||||
_ = handler.transport.Relay(request.Context(), bufferedClient, upstream)
|
err = handler.transport.Relay(request.Context(), bufferedClient, upstream)
|
||||||
|
handler.recordOutcome(proxyID, routingName, outcomeDomain.StageTunnel, err == nil, err, started)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (handler *Handler) Shutdown(ctx context.Context) error {
|
func (handler *Handler) Shutdown(ctx context.Context) error {
|
||||||
@ -444,8 +465,10 @@ func (handler *Handler) forwardHTTP(
|
|||||||
committed.Store(true)
|
committed.Store(true)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
started := time.Now().UTC()
|
||||||
response, err := handler.transport.RoundTrip(request.Context(), lease.Proxy, attemptRequest, commit)
|
response, err := handler.transport.RoundTrip(request.Context(), lease.Proxy, attemptRequest, commit)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
handler.recordOutcome(lease.Proxy.ID, route.RoutingName, outcomeDomain.StageDial, false, err, started)
|
||||||
finishLease(lease, committed.Load())
|
finishLease(lease, committed.Load())
|
||||||
excluded[lease.Proxy.ID] = struct{}{}
|
excluded[lease.Proxy.ID] = struct{}{}
|
||||||
lastErr = err
|
lastErr = err
|
||||||
@ -455,10 +478,17 @@ func (handler *Handler) forwardHTTP(
|
|||||||
if err := commit(); err != nil {
|
if err := commit(); err != nil {
|
||||||
_ = response.Body.Close()
|
_ = response.Body.Close()
|
||||||
finishLease(lease, false)
|
finishLease(lease, false)
|
||||||
|
handler.recordOutcome(lease.Proxy.ID, route.RoutingName, outcomeDomain.StageResponseHeaders, false, err, started)
|
||||||
lastErr = err
|
lastErr = err
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if response.StatusCode == http.StatusProxyAuthRequired {
|
||||||
|
handler.recordOutcome(lease.Proxy.ID, route.RoutingName, outcomeDomain.StageResponseHeaders, false,
|
||||||
|
&transportDomain.ProxyResponseError{StatusCode: response.StatusCode}, started)
|
||||||
|
} else {
|
||||||
|
handler.recordOutcome(lease.Proxy.ID, route.RoutingName, outcomeDomain.StageResponseHeaders, true, nil, started)
|
||||||
|
}
|
||||||
handler.writeResponse(writer, response)
|
handler.writeResponse(writer, response)
|
||||||
finishLease(lease, true)
|
finishLease(lease, true)
|
||||||
return
|
return
|
||||||
@ -467,6 +497,61 @@ func (handler *Handler) forwardHTTP(
|
|||||||
writeGatewayError(writer, fmt.Errorf("forward gateway request: %w", lastErr))
|
writeGatewayError(writer, fmt.Errorf("forward gateway request: %w", lastErr))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (handler *Handler) recordOutcome(
|
||||||
|
proxyID, routingName string,
|
||||||
|
stage outcomeDomain.Stage,
|
||||||
|
success bool,
|
||||||
|
err error,
|
||||||
|
started time.Time,
|
||||||
|
) {
|
||||||
|
if handler == nil || handler.outcomes == nil || proxyID == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
observedAt := time.Now().UTC()
|
||||||
|
if started.IsZero() {
|
||||||
|
started = observedAt
|
||||||
|
}
|
||||||
|
latency := observedAt.Sub(started)
|
||||||
|
if latency < 0 {
|
||||||
|
latency = 0
|
||||||
|
}
|
||||||
|
event := outcomeDomain.Event{
|
||||||
|
ProxyID: proxyID, RoutingName: routingName, Stage: stage, Success: success,
|
||||||
|
Latency: latency, ObservedAt: observedAt,
|
||||||
|
}
|
||||||
|
if !success {
|
||||||
|
event.ErrorClass = classifyOutcomeError(stage, err)
|
||||||
|
}
|
||||||
|
handler.outcomes.Record(event)
|
||||||
|
}
|
||||||
|
|
||||||
|
func classifyOutcomeError(stage outcomeDomain.Stage, err error) outcomeDomain.ErrorClass {
|
||||||
|
switch {
|
||||||
|
case errors.Is(err, context.Canceled):
|
||||||
|
return outcomeDomain.ErrorClassCanceled
|
||||||
|
case errors.Is(err, context.DeadlineExceeded):
|
||||||
|
return outcomeDomain.ErrorClassTimeout
|
||||||
|
}
|
||||||
|
var responseError *transportDomain.ProxyResponseError
|
||||||
|
if errors.As(err, &responseError) {
|
||||||
|
return outcomeDomain.ErrorClassProxyResponse
|
||||||
|
}
|
||||||
|
var networkError net.Error
|
||||||
|
if errors.As(err, &networkError) && networkError.Timeout() {
|
||||||
|
return outcomeDomain.ErrorClassTimeout
|
||||||
|
}
|
||||||
|
switch stage {
|
||||||
|
case outcomeDomain.StageDial:
|
||||||
|
return outcomeDomain.ErrorClassDial
|
||||||
|
case outcomeDomain.StageProxyHandshake:
|
||||||
|
return outcomeDomain.ErrorClassHandshake
|
||||||
|
case outcomeDomain.StageTunnel:
|
||||||
|
return outcomeDomain.ErrorClassRelay
|
||||||
|
default:
|
||||||
|
return outcomeDomain.ErrorClassInternal
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func (handler *Handler) acquireRoute(ctx context.Context, route dispatch.Request) (*dispatch.Lease, error) {
|
func (handler *Handler) acquireRoute(ctx context.Context, route dispatch.Request) (*dispatch.Lease, error) {
|
||||||
lease, err := handler.dispatcher.Acquire(route)
|
lease, err := handler.dispatcher.Acquire(route)
|
||||||
if !errors.Is(err, dispatch.ErrNoCandidate) || route.OnUnavailable != routing.OnUnavailableWait || route.WaitTimeout <= 0 {
|
if !errors.Is(err, dispatch.ErrNoCandidate) || route.OnUnavailable != routing.OnUnavailableWait || route.WaitTimeout <= 0 {
|
||||||
|
|||||||
@ -15,6 +15,7 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
outcomeDomain "proxy-pool/internal/domain/outcome"
|
||||||
proxyDomain "proxy-pool/internal/domain/proxy"
|
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||||
"proxy-pool/internal/domain/routing"
|
"proxy-pool/internal/domain/routing"
|
||||||
"proxy-pool/internal/gateway/dispatch"
|
"proxy-pool/internal/gateway/dispatch"
|
||||||
@ -103,6 +104,50 @@ func TestHandlerRetriesGETWithAnotherProxyBeforeResponseCommit(t *testing.T) {
|
|||||||
assertNoLeakedCapacity(t, view)
|
assertNoLeakedCapacity(t, view)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestHandlerRecordsOutcomeForEachProxyAttempt(t *testing.T) {
|
||||||
|
dispatcher, _ := dispatcherWithProxies(t, "proxy-a", "proxy-b")
|
||||||
|
transport := &fakeTransport{roundTrip: func(
|
||||||
|
_ context.Context,
|
||||||
|
selected proxyDomain.Proxy,
|
||||||
|
_ *http.Request,
|
||||||
|
commit ...func() error,
|
||||||
|
) (*http.Response, error) {
|
||||||
|
if selected.ID == "proxy-a" {
|
||||||
|
return nil, errors.New("dial failed")
|
||||||
|
}
|
||||||
|
if err := commit[0](); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return &http.Response{StatusCode: http.StatusNoContent, Header: make(http.Header), Body: http.NoBody}, nil
|
||||||
|
}}
|
||||||
|
handler := newTestHandler(t, Config{MaxAttempts: 2, RetryMethods: []string{http.MethodGet}}, dispatcher, transport)
|
||||||
|
recorder := &outcomeRecorder{}
|
||||||
|
handler.outcomes = recorder
|
||||||
|
|
||||||
|
response := httptest.NewRecorder()
|
||||||
|
handler.ServeHTTP(response, httptest.NewRequest(http.MethodGet, "http://example.test/resource", nil))
|
||||||
|
events := recorder.Events()
|
||||||
|
if response.Code != http.StatusNoContent || len(events) != 2 {
|
||||||
|
t.Fatalf("status=%d events=%+v", response.Code, events)
|
||||||
|
}
|
||||||
|
if events[0].ProxyID != "proxy-a" || events[0].Stage != outcomeDomain.StageDial || events[0].Success ||
|
||||||
|
events[0].ErrorClass != outcomeDomain.ErrorClassDial || events[1].ProxyID != "proxy-b" ||
|
||||||
|
events[1].Stage != outcomeDomain.StageResponseHeaders || !events[1].Success {
|
||||||
|
t.Fatalf("outcomes = %+v", events)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHandlerClampsNegativeOutcomeLatency(t *testing.T) {
|
||||||
|
recorder := &outcomeRecorder{}
|
||||||
|
handler := &Handler{outcomes: recorder}
|
||||||
|
handler.recordOutcome("proxy-a", "route-a", outcomeDomain.StageDial, true, nil, time.Now().Add(time.Second))
|
||||||
|
|
||||||
|
events := recorder.Events()
|
||||||
|
if len(events) != 1 || events[0].Latency != 0 {
|
||||||
|
t.Fatalf("outcomes = %+v, want zero latency", events)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestHandlerDoesNotRetryPOST(t *testing.T) {
|
func TestHandlerDoesNotRetryPOST(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@ -149,6 +194,8 @@ func TestHandlerReturns407WithoutRetry(t *testing.T) {
|
|||||||
}, nil
|
}, nil
|
||||||
}}
|
}}
|
||||||
handler := newTestHandler(t, Config{MaxAttempts: 2, RetryMethods: []string{http.MethodGet}}, dispatcher, transport)
|
handler := newTestHandler(t, Config{MaxAttempts: 2, RetryMethods: []string{http.MethodGet}}, dispatcher, transport)
|
||||||
|
recorder := &outcomeRecorder{}
|
||||||
|
handler.outcomes = recorder
|
||||||
|
|
||||||
response := httptest.NewRecorder()
|
response := httptest.NewRecorder()
|
||||||
handler.ServeHTTP(response, httptest.NewRequest(http.MethodGet, "http://example.test/resource", nil))
|
handler.ServeHTTP(response, httptest.NewRequest(http.MethodGet, "http://example.test/resource", nil))
|
||||||
@ -159,6 +206,11 @@ func TestHandlerReturns407WithoutRetry(t *testing.T) {
|
|||||||
if len(transport.attempts()) != 1 {
|
if len(transport.attempts()) != 1 {
|
||||||
t.Fatalf("attempts = %v, want no retry for 407", transport.attempts())
|
t.Fatalf("attempts = %v, want no retry for 407", transport.attempts())
|
||||||
}
|
}
|
||||||
|
events := recorder.Events()
|
||||||
|
if len(events) != 1 || events[0].Success || events[0].Stage != outcomeDomain.StageResponseHeaders ||
|
||||||
|
events[0].ErrorClass != outcomeDomain.ErrorClassProxyResponse {
|
||||||
|
t.Fatalf("outcomes = %+v, want failed proxy response", events)
|
||||||
|
}
|
||||||
assertNoLeakedCapacity(t, view)
|
assertNoLeakedCapacity(t, view)
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -496,6 +548,23 @@ type fakeTransport struct {
|
|||||||
relay func(context.Context, net.Conn, net.Conn) error
|
relay func(context.Context, net.Conn, net.Conn) error
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type outcomeRecorder struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
events []outcomeDomain.Event
|
||||||
|
}
|
||||||
|
|
||||||
|
func (recorder *outcomeRecorder) Record(event outcomeDomain.Event) {
|
||||||
|
recorder.mu.Lock()
|
||||||
|
recorder.events = append(recorder.events, event)
|
||||||
|
recorder.mu.Unlock()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (recorder *outcomeRecorder) Events() []outcomeDomain.Event {
|
||||||
|
recorder.mu.Lock()
|
||||||
|
defer recorder.mu.Unlock()
|
||||||
|
return append([]outcomeDomain.Event(nil), recorder.events...)
|
||||||
|
}
|
||||||
|
|
||||||
type waitRecordingDispatcher struct{ waitTimeout time.Duration }
|
type waitRecordingDispatcher struct{ waitTimeout time.Duration }
|
||||||
|
|
||||||
func (dispatcher *waitRecordingDispatcher) Acquire(dispatch.Request) (*dispatch.Lease, error) {
|
func (dispatcher *waitRecordingDispatcher) Acquire(dispatch.Request) (*dispatch.Lease, error) {
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user