feat: report gateway proxy outcomes
Some checks are pending
ci / proto (push) Waiting to run
ci / test (ubuntu-latest) (push) Waiting to run
ci / test (windows-latest) (push) Waiting to run
ci / race (push) Waiting to run
ci / integration (push) Waiting to run

This commit is contained in:
youfak 2026-07-31 17:36:17 +08:00
parent 155411ef23
commit 6766097ea7
32 changed files with 1482 additions and 55 deletions

View File

@ -21,7 +21,7 @@ Proxy Pool 用 Controller 协调这些变化,并让 Gateway 数据面只消费
- **Gateway**:调用方连接平台,由平台选择上游代理并转发 HTTP 或 HTTPS - **Gateway**:调用方连接平台,由平台选择上游代理并转发 HTTP 或 HTTPS
CONNECT。`proxy-gateway` 已装配本地监听、指标探针和控制面 Register/Watch/ACK/ CONNECT。`proxy-gateway` 已装配本地监听、指标探针和控制面 Register/Watch/ACK/
Runtime 会话;带凭据 Proxy 分发仍待闭环。 Runtime/Outcome 会话;带凭据 Proxy 的分发与当前内存 View 已闭环。
- **Distribution**:调用方按条件提取真实代理;成功提取即独占消费,不支持归还、 - **Distribution**:调用方按条件提取真实代理;成功提取即独占消费,不支持归还、
续租或状态查询。 续租或状态查询。
- **Admin**:运维人员查询状态、启停 Upstream、切换 Routing并触发严格配置 - **Admin**:运维人员查询状态、启停 Upstream、切换 Routing并触发严格配置
@ -43,7 +43,9 @@ Proxy Pool 用 Controller 协调这些变化,并让 Gateway 数据面只消费
审计与 Outbox不保存 Proxy 明细或逐次提取记录。 审计与 Outbox不保存 Proxy 明细或逐次提取记录。
- **Gateway 组件**HTTP 正向代理、HTTPS CONNECT、双向 Tunnel、重试、超时、 - **Gateway 组件**HTTP 正向代理、HTTPS CONNECT、双向 Tunnel、重试、超时、
目的地址保护、本地快照存储、容量调度和 Worker 控制面 Register/Watch/ACK/ 目的地址保护、本地快照存储、容量调度和 Worker 控制面 Register/Watch/ACK/
Runtime 会话组件已有实现与定向测试。`SessionSupervisor` 会为可恢复控制面中断执行 Runtime/Outcome 会话组件已有实现与定向测试。每次代理尝试只向本地有界队列写入
Outcome微批确认失败会重发同一序列队列满时丢弃观测样本不阻塞转发请求。
`SessionSupervisor` 会为可恢复控制面中断执行
有界退避重连。Controller 可向 Worker 下发已归属 Proxy、Gateway Routing 与按引用去重的 有界退避重连。Controller 可向 Worker 下发已归属 Proxy、Gateway Routing 与按引用去重的
凭据材料快照。Gateway 会将 Routing、Proxy 与凭据原子编译为同一内存 View并只按当前未过期 凭据材料快照。Gateway 会将 Routing、Proxy 与凭据原子编译为同一内存 View并只按当前未过期
View 匹配请求,并在内存中按 Sequential、Random、Round Robin、Weighted 或 Least View 匹配请求,并在内存中按 Sequential、Random、Round Robin、Weighted 或 Least
@ -78,18 +80,17 @@ flowchart LR
## 当前完成度 ## 当前完成度
截至 **2026-07-31**,实施计划检查项为 **55 / 7474.3%**。详情见 截至 **2026-07-31**,实施计划检查项为 **56 / 7475.7%**。详情见
[实施计划](docs/development/implementation-plan.md)和 [实施计划](docs/development/implementation-plan.md)和
[交付完成度审计](docs/requirements/completion-audit.md)。 [交付完成度审计](docs/requirements/completion-audit.md)。
- **已完成**严格配置、Provider 获取与协调、Redis 活动池、Distribution 原子 - **已完成**严格配置、Provider 获取与协调、Redis 活动池、Distribution 原子
提取与限流、Controller 的 Admin/Distribution/Metrics 监听,以及 PostgreSQL 提取与限流、Controller 的 Admin/Distribution/Metrics 监听,以及 PostgreSQL
管理状态WorkerControlPlane 的 Register、Snapshot ACK、Runtime 心跳接收和 管理状态WorkerControlPlane 的 Register、Snapshot ACK、Runtime 心跳接收和
Redis 会话栅栏。 Redis 会话栅栏,以及 Gateway Outcome 上报的有界队列、序列确认与重试
- **部分完成**Gateway Outcome 上报、Checker 调度与健康状态链、Docker Compose/ - **部分完成**Checker 调度与健康状态链、Docker Compose/
Kubernetes 运行时 mTLS Overlay以及 protobuf 契约。 Kubernetes 运行时 mTLS Overlay以及 protobuf 契约。
- **待完成**Outcome 上报、Checker 调度与健康状态链, - **待完成**Checker 调度与健康状态链,以及 loadgen 和代表性集群压测。
以及 loadgen 和代表性集群压测。
检查项数量不等于生产就绪度。静态部署清单与 protobuf descriptor 验证也不代表 检查项数量不等于生产就绪度。静态部署清单与 protobuf descriptor 验证也不代表
端到端拓扑已经完成;`100,000 QPS` 仍只是待验证的集群设计目标。 端到端拓扑已经完成;`100,000 QPS` 仍只是待验证的集群设计目标。

View File

@ -35,10 +35,13 @@ Gateway 已将该 payload 编译并原子发布到与 Proxy 相同版本的本
当前未过期 View派发器已按五种策略从该 View 选择上游,且在 Proxy 容量耗尽时只在该 当前未过期 View派发器已按五种策略从该 View 选择上游,且在 Proxy 容量耗尽时只在该
View 的其余候选中回退。`wait_timeout` 随 `on_unavailable=WAIT` 下发并在 Gateway 作为有界 View 的其余候选中回退。`wait_timeout` 随 `on_unavailable=WAIT` 下发并在 Gateway 作为有界
本地容量等待使用;`DIRECT` 仍先经过 TargetPolicy 再建立 HTTP/CONNECT 直连。`proxy-gateway` 本地容量等待使用;`DIRECT` 仍先经过 TargetPolicy 再建立 HTTP/CONNECT 直连。`proxy-gateway`
已装配 Register/Watch/ACK/Runtime 会话、HTTP 代理监听和 Snapshot 就绪探针;控制面中断时 已装配 Register/Watch/ACK/Runtime/Outcome 会话、HTTP 代理监听和 Snapshot 就绪探针;控制面中断时
保持进程运行并以有界退避重连,未取得有效 Snapshot 的 Worker 不会 Ready。增量、带凭据 保持进程运行并以有界退避重连,未取得有效 Snapshot 的 Worker 不会 Ready。`ReportOutcomes`
Outcome 与 Checker 闭环尚未实现。`ReportOutcomes` 仍明确返回 `Unimplemented` 已实现为客户端流Controller 校验流内固定的 Worker/Session 身份,对每个批次以
`100,000 QPS` 仍是未验证的设计目标。 `(session_id, sequence, SHA-256)` 建立 Redis 原子栅栏,并只返回最后确认的序列。
Redis 只保留每个当前会话的一条序列和摘要;原始 Outcome、代理明细与逐请求记录均不写入
Redis 或 PostgreSQL。Gateway 只将结果写入本地有界队列,队列满时丢弃样本,不等待控制面
或存储。Checker 闭环尚未实现;`100,000 QPS` 仍是未验证的设计目标。
`WatchSnapshots` 建立时校验当前 session每次签发快照引用时也把 `session_id` `WatchSnapshots` 建立时校验当前 session每次签发快照引用时也把 `session_id`
交给 Redis 原子校验。重复 Register 会同时清除旧 Runtime 和已签发引用,因此迟到的 交给 Redis 原子校验。重复 Register 会同时清除旧 Runtime 和已签发引用,因此迟到的
@ -137,8 +140,15 @@ Outcome 按 Worker 单调 `sequence` 批量上报。Controller 返回已接受
- `TUNNEL`:隧道建立后结束或失败。 - `TUNNEL`:隧道建立后结束或失败。
Outcome 是 Observation不直接让 Worker 修改 Redis 活动池状态,也不创建 Outcome 是 Observation不直接让 Worker 修改 Redis 活动池状态,也不创建
PostgreSQL Proxy 明细。异步上报队列必须有界;队列满时丢弃低价值样本并计 PostgreSQL Proxy 明细。Gateway 将每次尝试写入进程内有界队列,当前默认容量为
指标,不能反压 Gateway 热路径。 `65536` 个事件,单批最多为 `min(512, maxRuntimeCounters)`;一次 ReportOutcomes RPC
最多合并 16 个已就绪批次。队列满时只递增本地丢弃计数,不反压 Gateway 热路径。
同一 session 内Controller 仅接受递增 `sequence`;相同序列且摘要一致视为幂等
重放,相同序列且摘要不同返回 `AlreadyExists`,较小序列返回 `Aborted`。Gateway 在
未收到确认时保留并重发完全相同的批次。暂态传输错误在当前 session 内按有上限的
退避重试;会话栅栏错误交由 session supervisor 重建 session。栅栏随 session 替换
或过期清理,因此长期在线 Worker 不会因独立 Outcome TTL 接受旧序列。
## 6. Checker 任务 ## 6. Checker 任务

View File

@ -190,6 +190,11 @@ Gateway 连接 Controller 时使用独立启动参数而非 `controlPlane.listen
模板保持 `controlPlane.enabled: false`,环境 Overlay 挂载 mTLS 证书并启用后才可启动 模板保持 `controlPlane.enabled: false`,环境 Overlay 挂载 mTLS 证书并启用后才可启动
Gateway。 Gateway。
`maxRuntimeCounters` 同时限制单个 Runtime 报告和单个 Outcome 批次的条目数。Gateway
在本地维护容量为 `65536` 的非阻塞 Outcome 队列,默认微批上限为 `512`,实际取二者中
较小值;该队列与其序列确认状态仅存在于 Gateway 进程内。Controller 的 Redis 状态只保存
当前 Worker session 的最后确认序列和摘要PostgreSQL 不保存 Outcome、代理明细或逐请求记录。
## 5. Gateway ## 5. Gateway
```yaml ```yaml

View File

@ -223,7 +223,7 @@ payload 已按配置顺序和 Admin revision/current 状态发布并覆盖 check
Proxy 原子编译为同版本 View动态 Router 只匹配该未过期 View。派发器的五种上游选择已 Proxy 原子编译为同版本 View动态 Router 只匹配该未过期 View。派发器的五种上游选择已
接入该 View并在容量耗尽时在同版本候选中回退`onUnavailable` 的 reject、wait 与 direct 接入该 View并在容量耗尽时在同版本候选中回退`onUnavailable` 的 reject、wait 与 direct
已接入 Gateway`proxy-gateway` 已装配本地 HTTP/Metrics 监听、快照就绪探针和 已接入 Gateway`proxy-gateway` 已装配本地 HTTP/Metrics 监听、快照就绪探针和
控制面重连与快照凭据分发,Outcome 上报仍未实现 控制面重连与快照凭据分发,以及有界 Outcome 上报、序列确认和精确重试
已新增公用 `domain/activitypool` 契约及并发安全内存参考实现Provider 已新增公用 `domain/activitypool` 契约及并发安全内存参考实现Provider
Reconciler 通过 `UpsertFetched` 写入带供应商 TTL 和分配安全余量的批次;已覆盖 Reconciler 通过 `UpsertFetched` 写入带供应商 TTL 和分配安全余量的批次;已覆盖
@ -261,7 +261,8 @@ Controller 多副本共享同一计数。Client 身份只以 SHA-256 摘要进
Gateway 请求热路径仍只使用本地准入,不增加 Redis/PostgreSQL 调用。 Gateway 请求热路径仍只使用本地准入,不增加 Redis/PostgreSQL 调用。
WorkerControlPlane gRPC 接收端、session 签发/心跳、Snapshot ACK 账本、基础 WorkerControlPlane gRPC 接收端、session 签发/心跳、Snapshot ACK 账本、基础
Snapshot 流、Gateway 会话客户端与快照凭据分发已完成;权威 Proxy/Routing 发布、 Snapshot 流、Gateway 会话客户端与快照凭据分发已完成;权威 Proxy/Routing 发布、
Outcome 和健康执行链仍待完成,因此 Task 10 尚未全部完成。 Outcome 上报已完成为有界队列、批次序列/摘要栅栏和确认重试;健康执行链仍待完成,
因此 Task 10 尚未全部完成。
## Task 11: Checker and Health Reducer ## Task 11: Checker and Health Reducer

View File

@ -3,6 +3,9 @@
## 1. 运行边界 ## 1. 运行边界
- Gateway 是数据面,正常请求热路径不访问 PostgreSQL、Redis 或 Provider。 - Gateway 是数据面,正常请求热路径不访问 PostgreSQL、Redis 或 Provider。
- Gateway 的 Proxy Outcome 仅进入进程内有界队列Controller 只维护当前 session
的最后确认序列和摘要,原始 Outcome 不写入 Redis 或 PostgreSQL。队列满时丢弃
观测样本,代理转发和 Redis session TTL 不受影响。
- Controller 编排 Fetch、生命周期、所有权、Snapshot 与 Extract多个副本只有 - Controller 编排 Fetch、生命周期、所有权、Snapshot 与 Extract多个副本只有
一个 Provider 逻辑 Leader。短效 Proxy 明细只存在于 Redis TTL 活动池和节点 一个 Provider 逻辑 Leader。短效 Proxy 明细只存在于 Redis TTL 活动池和节点
内存,可由 Provider 重建。 内存,可由 Provider 重建。

View File

@ -38,7 +38,7 @@
- `CAP-001 / GW 热路径边界`:本地 Dispatch 条件过滤与原子容量预留。 - `CAP-001 / GW 热路径边界`:本地 Dispatch 条件过滤与原子容量预留。
- `GW-*`HTTP 正向代理、HTTPS CONNECT、双向 Tunnel、超时、重试、保护链与 - `GW-*`HTTP 正向代理、HTTPS CONNECT、双向 Tunnel、超时、重试、保护链与
优雅停机 Handler 已实现并通过定向测试。`proxy-gateway` 已装配本地 HTTP/Metrics 优雅停机 Handler 已实现并通过定向测试。`proxy-gateway` 已装配本地 HTTP/Metrics
监听、快照就绪探针、显式控制面拨号和 Register/Watch/ACK/Runtime 会话;本地 gRPC 监听、快照就绪探针、显式控制面拨号和 Register/Watch/ACK/Runtime/Outcome 会话;本地 gRPC
集成测试验证收到有效 Snapshot 后才 Ready并可完成 direct 路由。 集成测试验证收到有效 Snapshot 后才 Ready并可完成 direct 路由。
- `PROVIDER-*`Provider HTTP Client、严格响应上限、模板解析安全边界、凭据 - `PROVIDER-*`Provider HTTP Client、严格响应上限、模板解析安全边界、凭据
引用 Store 与 Reconciler Adapter 已实现。 引用 Store 与 Reconciler Adapter 已实现。
@ -56,7 +56,8 @@
索引构建已归属 Proxy payload并以租约收紧可用期凭据材料按引用去重并与 Proxy/ 索引构建已归属 Proxy payload并以租约收紧可用期凭据材料按引用去重并与 Proxy/
Routing 一起纳入 Snapshot checksum仅在 mTLS 和 Gateway 当前内存 View 中处理。Gateway Routing 一起纳入 Snapshot checksum仅在 mTLS 和 Gateway 当前内存 View 中处理。Gateway
Routing payload 已由配置顺序和 Admin 当前状态合成并覆盖 Snapshot checksum其运行时消费、 Routing payload 已由配置顺序和 Admin 当前状态合成并覆盖 Snapshot checksum其运行时消费、
Outcome 和 Checker 尚未闭环。Snapshot 签发在 Redis 中原子匹配当前 Outcome 已实现为 Gateway 本地有界队列、微批确认重试和 Controller 的 session/sequence/
摘要 Redis 栅栏;原始事件不落 Redis 或 PostgreSQL。Checker 尚未闭环。Snapshot 签发在 Redis 中原子匹配当前
`session_id`,重注册会清除旧引用,迟到旧 Stream 不会覆盖新 session。Controller `session_id`,重注册会清除旧引用,迟到旧 Stream 不会覆盖新 session。Controller
在最近成功下发的 Snapshot `valid_until` 到达时关闭流Gateway 的公用 在最近成功下发的 Snapshot `valid_until` 到达时关闭流Gateway 的公用
`SessionSupervisor` 已实现可恢复错误的有界退避重连。Gateway 会校验并执行 Snapshot `SessionSupervisor` 已实现可恢复错误的有界退避重连。Gateway 会校验并执行 Snapshot
@ -105,7 +106,7 @@ Controller/Gateway 入口,完整 mTLS 运行时拓扑仍只有静态验证。
分布式请求额度、Distribution Client 限制和 Provider 状态丢失重建已完成。 分布式请求额度、Distribution Client 限制和 Provider 状态丢失重建已完成。
6. Worker 基础网络快照流、Proxy/Gateway Routing/凭据 Snapshot payload、Gateway Snapshot 6. Worker 基础网络快照流、Proxy/Gateway Routing/凭据 Snapshot payload、Gateway Snapshot
客户端和进程装配、同版本 Routing 编译/动态匹配、五种策略上游选择与 reject/wait/direct 已完成; 客户端和进程装配、同版本 Routing 编译/动态匹配、五种策略上游选择与 reject/wait/direct 已完成;
Outcome 上报仍待完成。Redis ownership drain/ACK/过期回收及按 Worker 的可下发索引已完成。 Outcome 上报已完成基础观测链;Redis ownership drain/ACK/过期回收及按 Worker 的可下发索引已完成。
7. Checker 调度、探测器和健康 reducer。 7. Checker 调度、探测器和健康 reducer。
8. Admin/Distribution 细粒度授权和审计查询Distribution 分布式限流已完成。 8. Admin/Distribution 细粒度授权和审计查询Distribution 分布式限流已完成。
9. 真实 Compose/Kubernetes 集成、故障演练和代表性集群负载测试。 9. 真实 Compose/Kubernetes 集成、故障演练和代表性集群负载测试。

View File

@ -7,8 +7,8 @@
| ID | 最终需求 | 来源 | 验证证据 | | ID | 最终需求 | 来源 | 验证证据 |
|---|---|---|---| |---|---|---|---|
| ARCH-001 | 数据面 Worker 与控制面 Controller 分离 | 1-70 | 包、协议和部署拓扑已分离Controller 已运行 Worker Register/Watch/ACK/Runtime gRPC并发布 Proxy/Gateway Routing/按引用去重凭据完整快照Gateway 已将它们编译为同版本动态 View并由独立进程维护控制面会话。Checker/Loadgen 构建产物待实现 | | ARCH-001 | 数据面 Worker 与控制面 Controller 分离 | 1-70 | 包、协议和部署拓扑已分离Controller 已运行 Worker Register/Watch/ACK/Runtime/Outcome gRPC并发布 Proxy/Gateway Routing/按引用去重凭据完整快照Gateway 已将它们编译为同版本动态 View并由独立进程维护控制面会话。Checker/Loadgen 构建产物待实现 |
| ARCH-002 | 热路径只做认证、本地路由和网络转发 | 1-70, 380-430 | Gateway bootstrap 集成测试验证启动期控制面会话与快照就绪HTTP 请求只走本地 Snapshot/Dispatch代表性性能剖析待完成 | | ARCH-002 | 热路径只做认证、本地路由和网络转发 | 1-70, 380-430 | Gateway bootstrap 集成测试验证启动期控制面会话与快照就绪HTTP 请求只走本地 Snapshot/DispatchOutcome 仅写入有界非阻塞本地队列,代表性性能剖析待完成 |
| ARCH-003 | Gateway、Distribution、Admin、Metrics 独立入口 | 8904-8958 | Controller 命令已装配 Distribution/Admin/Metrics 三个独立监听及联动停机Gateway 命令已装配代理与 Metrics 监听,运行时 mTLS 部署 Overlay 待完成 | | ARCH-003 | Gateway、Distribution、Admin、Metrics 独立入口 | 8904-8958 | Controller 命令已装配 Distribution/Admin/Metrics 三个独立监听及联动停机Gateway 命令已装配代理与 Metrics 监听,运行时 mTLS 部署 Overlay 待完成 |
| ARCH-004 | Controller 集中 Provider 获取与切换 | 1403-1580 | Redis Leader、动态 Provider Supervisor 与 Bootstrap 生产装配已完成Admin disable/reload 驱动取消替换,多副本按权威 HMAC 指纹和 revision 栅栏收敛并拒绝旧配置换主Routing 切换到 Drain 的编排待完成 | | ARCH-004 | Controller 集中 Provider 获取与切换 | 1403-1580 | Redis Leader、动态 Provider Supervisor 与 Bootstrap 生产装配已完成Admin disable/reload 驱动取消替换,多副本按权威 HMAC 指纹和 revision 栅栏收敛并拒绝旧配置换主Routing 切换到 Drain 的编排待完成 |
| ARCH-005 | 100k QPS 峰值使用多 Worker 集群 | 当前会话 | 未验证设计目标;待代表性集群负载报告 | | ARCH-005 | 100k QPS 峰值使用多 Worker 集群 | 当前会话 | 未验证设计目标;待代表性集群负载报告 |
@ -60,6 +60,7 @@
| GW-003 | CONNECT 建立后不得透明重放 | 221-300 | `handler_test.go` 200 前重试、200 后中继故障不重放、半关闭测试 | | GW-003 | CONNECT 建立后不得透明重放 | 221-300 | `handler_test.go` 200 前重试、200 后中继故障不重放、半关闭测试 |
| GW-004 | Client 认证可关闭,但访问控制、身份识别和限流独立 | 8112-8441 | `protection_test.go`、`bootstrap_test.go` 与入口并发上限测试 | | GW-004 | Client 认证可关闭,但访问控制、身份识别和限流独立 | 8112-8441 | `protection_test.go`、`bootstrap_test.go` 与入口并发上限测试 |
| GW-005 | 防私网、回环、链路本地、保留/元数据地址、任意 CONNECT 端口和 DNS Rebinding | 8904-8931 | `policy/target_test.go` 解析前后校验、端口白名单、混合 DNS 结果与已验证 IP 绑定测试 | | GW-005 | 防私网、回环、链路本地、保留/元数据地址、任意 CONNECT 端口和 DNS Rebinding | 8904-8931 | `policy/target_test.go` 解析前后校验、端口白名单、混合 DNS 结果与已验证 IP 绑定测试 |
| GW-006 | Proxy Outcome 不反压转发并按会话序列去重 | 当前会话 | `gateway/outcome` 队列溢出与微批测试、`controlplane/outcome_reporter_test.go` 精确重试测试、Worker gRPC 和 Memory/Redis 共享栅栏契约 |
## Distribution ## Distribution

View File

@ -46,7 +46,7 @@
- Distribution 原子提取后 Gateway 不再分配同一 Proxy。 - Distribution 原子提取后 Gateway 不再分配同一 Proxy。
- 配置热更新失败保留旧 Revision成功后新请求使用新 Snapshot。 - 配置热更新失败保留旧 Revision成功后新请求使用新 Snapshot。
## 3. 必测的 11 类场景 ## 3. 必测的 12 类场景
1. **并发容量**1000 协程争用同一 Proxy始终满足 1. **并发容量**1000 协程争用同一 Proxy始终满足
`active + reserved <= effectiveMaxConcurrency` `active + reserved <= effectiveMaxConcurrency`
@ -68,6 +68,8 @@
unowned旧 Snapshot 不可再分配。 unowned旧 Snapshot 不可再分配。
11. **控制面故障**Redis、PostgreSQL、Controller、Checker 与 Provider 分别 11. **控制面故障**Redis、PostgreSQL、Controller、Checker 与 Provider 分别
失效时,行为与 Runbook 一致Gateway 热路径不被同步依赖拖垮。 失效时,行为与 Runbook 一致Gateway 热路径不被同步依赖拖垮。
12. **Outcome 栅栏**:队列写入不阻塞转发;同一 session 的相同序列/摘要可重放,
相同序列的不同摘要冲突较小序列拒绝Session 替换后旧 Worker 结果拒绝。
## 4. 测试命令 ## 4. 测试命令

View File

@ -26,6 +26,7 @@ type keyspace struct {
workerSnapshotExpiry string workerSnapshotExpiry string
workerRuntime string workerRuntime string
workerRuntimeExpiry string workerRuntimeExpiry string
workerOutcomes string
} }
func newKeyspace(namespace string) keyspace { func newKeyspace(namespace string) keyspace {
@ -48,6 +49,7 @@ func newKeyspace(namespace string) keyspace {
workerSnapshotExpiry: prefix + ":worker-snapshot-expiry", workerSnapshotExpiry: prefix + ":worker-snapshot-expiry",
workerRuntime: prefix + ":worker-runtime", workerRuntime: prefix + ":worker-runtime",
workerRuntimeExpiry: prefix + ":worker-runtime-expiry", workerRuntimeExpiry: prefix + ":worker-runtime-expiry",
workerOutcomes: prefix + ":worker-outcomes",
} }
} }

View File

@ -20,6 +20,7 @@ const (
runtimeRecordSnapshot = "record_snapshot" runtimeRecordSnapshot = "record_snapshot"
runtimeAcknowledge = "acknowledge_snapshot" runtimeAcknowledge = "acknowledge_snapshot"
runtimeReplaceReport = "replace_report" runtimeReplaceReport = "replace_report"
runtimeRecordOutcomes = "record_outcomes"
runtimeRead = "read" runtimeRead = "read"
) )
@ -84,6 +85,13 @@ type runtimeReportWire struct {
Counters []runtimeCounterWire `json:"counters"` Counters []runtimeCounterWire `json:"counters"`
} }
type runtimeOutcomeReferenceWire struct {
Version int `json:"version"`
WorkerID string `json:"workerId"`
SessionID string `json:"sessionId"`
Sequence string `json:"sequence"`
}
type runtimeOwnedProxyWire struct { type runtimeOwnedProxyWire struct {
ProxyID string `json:"proxyId"` ProxyID string `json:"proxyId"`
WorkerID string `json:"workerId"` WorkerID string `json:"workerId"`
@ -103,6 +111,7 @@ var (
_ workerruntime.SessionWriter = (*Adapter)(nil) _ workerruntime.SessionWriter = (*Adapter)(nil)
_ workerruntime.ReportWriter = (*Adapter)(nil) _ workerruntime.ReportWriter = (*Adapter)(nil)
_ workerruntime.RuntimeReader = (*Adapter)(nil) _ workerruntime.RuntimeReader = (*Adapter)(nil)
_ workerruntime.OutcomeWriter = (*Adapter)(nil)
) )
func (a *Adapter) CurrentOwnershipEpoch(ctx context.Context) (uint64, error) { func (a *Adapter) CurrentOwnershipEpoch(ctx context.Context) (uint64, error) {
@ -317,6 +326,48 @@ func (a *Adapter) ReplaceRuntime(ctx context.Context, report workerruntime.Repor
} }
} }
// RecordOutcomes persists only the monotonically increasing batch fence. The
// Gateway event payload is deliberately excluded from Redis to keep the data
// plane hot path bounded and free of per-request storage writes.
func (a *Adapter) RecordOutcomes(ctx context.Context, reference workerruntime.OutcomeReference) (uint64, error) {
if err := validateRuntimeCall(ctx, a); err != nil {
return 0, err
}
normalized, err := workerruntime.NormalizeOutcomeReference(reference)
if err != nil {
return 0, workerruntime.ErrInvalidOutcome
}
payload, err := json.Marshal(runtimeOutcomeReferenceWire{
Version: runtimeWireVersion, WorkerID: normalized.WorkerID, SessionID: normalized.SessionID,
Sequence: strconv.FormatUint(normalized.Sequence, 10),
})
if err != nil {
return 0, workerruntime.ErrInvalidOutcome
}
reply, err := a.runRuntime(ctx, runtimeRecordOutcomes, 0, payload, hex.EncodeToString(normalized.Digest[:]))
if err != nil {
return 0, err
}
switch reply.Status {
case scriptOK:
accepted, parseErr := strconv.ParseUint(reply.Record, 10, 64)
if parseErr != nil || accepted == 0 {
return 0, invalidScriptReply("invalid worker outcome reply")
}
return accepted, nil
case scriptInvalid:
return 0, workerruntime.ErrInvalidOutcome
case scriptStale:
return 0, workerruntime.ErrStaleOutcome
case scriptConflict:
return 0, workerruntime.ErrConflictingOutcome
case scriptUnavailable:
return 0, workerruntime.ErrStaleSession
default:
return 0, invalidScriptReply("unexpected worker outcome reply")
}
}
func (a *Adapter) ReadRuntime(ctx context.Context, proxies []workerruntime.OwnedProxy) ([]workerruntime.Snapshot, error) { func (a *Adapter) ReadRuntime(ctx context.Context, proxies []workerruntime.OwnedProxy) ([]workerruntime.Snapshot, error) {
if err := validateRuntimeCall(ctx, a); err != nil { if err := validateRuntimeCall(ctx, a); err != nil {
return nil, err return nil, err
@ -403,6 +454,7 @@ func (a *Adapter) runRuntime(
a.keys.workerSessions, a.keys.workerSessionExpiry, a.keys.workerSessions, a.keys.workerSessionExpiry,
a.keys.workerSnapshots, a.keys.workerSnapshotExpiry, a.keys.workerSnapshots, a.keys.workerSnapshotExpiry,
a.keys.workerRuntime, a.keys.workerRuntimeExpiry, a.keys.owners, a.keys.epoch, a.keys.workerRuntime, a.keys.workerRuntimeExpiry, a.keys.owners, a.keys.epoch,
a.keys.workerOutcomes,
}, operation, ttlMS, a.options.CleanupLimit, string(payload), digest) }, operation, ttlMS, a.options.CleanupLimit, string(payload), digest)
if err != nil { if err != nil {
return runtimeScriptReply{}, err return runtimeScriptReply{}, err

View File

@ -20,7 +20,7 @@ func TestRedisWorkerControlStoreContract(t *testing.T) {
t.Fatalf("Assign(): %v", err) t.Fatalf("Assign(): %v", err)
} }
return contracttest.Fixture{ return contracttest.Fixture{
Store: fixture.Adapter, Reader: fixture.Adapter, TTL: 100 * time.Millisecond, Store: fixture.Adapter, Reader: fixture.Adapter, Outcomes: fixture.Adapter, TTL: 100 * time.Millisecond,
Advance: time.Sleep, Advance: time.Sleep,
} }
}) })

View File

@ -6,6 +6,7 @@ local runtime_key = KEYS[5]
local runtime_expiry_key = KEYS[6] local runtime_expiry_key = KEYS[6]
local owners_key = KEYS[7] local owners_key = KEYS[7]
local epoch_key = KEYS[8] local epoch_key = KEYS[8]
local outcomes_key = KEYS[9]
local operation = ARGV[1] local operation = ARGV[1]
local ttl_ms = tonumber(ARGV[2]) local ttl_ms = tonumber(ARGV[2])
@ -60,6 +61,7 @@ local function cleanup(now)
for _, worker_id in ipairs(expired_sessions) do for _, worker_id in ipairs(expired_sessions) do
redis.call('HDEL', sessions_key, worker_id) redis.call('HDEL', sessions_key, worker_id)
redis.call('HDEL', runtime_key, worker_id) redis.call('HDEL', runtime_key, worker_id)
redis.call('HDEL', outcomes_key, worker_id)
redis.call('ZREM', session_expiry_key, worker_id) redis.call('ZREM', session_expiry_key, worker_id)
redis.call('ZREM', runtime_expiry_key, worker_id) redis.call('ZREM', runtime_expiry_key, worker_id)
end end
@ -106,6 +108,13 @@ local function valid_reference(value)
string.match(value.checksum, '^[0-9a-f]+$') ~= nil string.match(value.checksum, '^[0-9a-f]+$') ~= nil
end end
local function valid_outcome(value)
return value and value.version == 1 and type(value.workerId) == 'string' and value.workerId ~= '' and
type(value.sessionId) == 'string' and value.sessionId ~= '' and valid_uint(value.sequence) and
type(value.digest) == 'string' and string.len(value.digest) == 64 and
string.match(value.digest, '^[0-9a-f]+$') ~= nil
end
local function compare_reference(left, right) local function compare_reference(left, right)
local epoch_order = compare_uint(left.ownershipEpoch, right.ownershipEpoch) local epoch_order = compare_uint(left.ownershipEpoch, right.ownershipEpoch)
if epoch_order ~= 0 then if epoch_order ~= 0 then
@ -148,6 +157,7 @@ if operation == 'open_session' then
end end
redis.call('HDEL', runtime_key, session.workerId) redis.call('HDEL', runtime_key, session.workerId)
redis.call('ZREM', runtime_expiry_key, session.workerId) redis.call('ZREM', runtime_expiry_key, session.workerId)
redis.call('HDEL', outcomes_key, session.workerId)
redis.call('HDEL', snapshots_key, session.workerId) redis.call('HDEL', snapshots_key, session.workerId)
redis.call('ZREM', snapshot_expiry_key, session.workerId) redis.call('ZREM', snapshot_expiry_key, session.workerId)
session.expiresAtMs = now + ttl_ms session.expiresAtMs = now + ttl_ms
@ -301,10 +311,12 @@ if operation == 'replace_session' then
if epoch_order > 0 or version_order > 0 then if epoch_order > 0 or version_order > 0 then
redis.call('HDEL', runtime_key, session.workerId) redis.call('HDEL', runtime_key, session.workerId)
redis.call('ZREM', runtime_expiry_key, session.workerId) redis.call('ZREM', runtime_expiry_key, session.workerId)
redis.call('HDEL', outcomes_key, session.workerId)
end end
else else
redis.call('HDEL', runtime_key, session.workerId) redis.call('HDEL', runtime_key, session.workerId)
redis.call('ZREM', runtime_expiry_key, session.workerId) redis.call('ZREM', runtime_expiry_key, session.workerId)
redis.call('HDEL', outcomes_key, session.workerId)
end end
session.expiresAtMs = now + ttl_ms session.expiresAtMs = now + ttl_ms
redis.call('HSET', sessions_key, session.workerId, cjson.encode(session)) redis.call('HSET', sessions_key, session.workerId, cjson.encode(session))
@ -312,6 +324,39 @@ if operation == 'replace_session' then
return reply('ok') return reply('ok')
end end
if operation == 'record_outcomes' then
if type(digest) ~= 'string' or string.len(digest) ~= 64 or
string.match(digest, '^[0-9a-f]+$') == nil then
return reply('invalid')
end
local reference = decode_table(payload)
if not reference or reference.version ~= 1 or type(reference.workerId) ~= 'string' or reference.workerId == '' or
type(reference.sessionId) ~= 'string' or reference.sessionId == '' or not valid_uint(reference.sequence) then
return reply('invalid')
end
local session = decode_table(redis.call('HGET', sessions_key, reference.workerId))
if not valid_session(session) or session.workerId ~= reference.workerId or session.sessionId ~= reference.sessionId or
type(session.expiresAtMs) ~= 'number' or session.expiresAtMs <= now then
return reply('unavailable')
end
local current = decode_table(redis.call('HGET', outcomes_key, reference.workerId))
if valid_outcome(current) and current.workerId == reference.workerId and current.sessionId == reference.sessionId then
local ordering = compare_uint(reference.sequence, current.sequence)
if ordering < 0 then
return reply('stale')
end
if ordering == 0 then
if current.digest == digest then
return reply('ok', nil, current.sequence)
end
return reply('conflict')
end
end
reference.digest = digest
redis.call('HSET', outcomes_key, reference.workerId, cjson.encode(reference))
return reply('ok', nil, reference.sequence)
end
if operation == 'replace_report' then if operation == 'replace_report' then
if not ttl_ms or ttl_ms <= 0 or type(digest) ~= 'string' or digest == '' then if not ttl_ms or ttl_ms <= 0 or type(digest) ~= 'string' or digest == '' then
return reply('invalid') return reply('invalid')

View File

@ -4,9 +4,11 @@ import (
"context" "context"
"crypto/sha256" "crypto/sha256"
"errors" "errors"
"io"
"time" "time"
controlplanev1 "proxy-pool/gen/controlplane/v1" controlplanev1 "proxy-pool/gen/controlplane/v1"
"proxy-pool/internal/domain/outcome"
"proxy-pool/internal/domain/workerruntime" "proxy-pool/internal/domain/workerruntime"
"google.golang.org/grpc" "google.golang.org/grpc"
@ -197,6 +199,84 @@ func (handler *GRPCHandler) ReportRuntime(ctx context.Context, request *controlp
}, nil }, nil
} }
func (handler *GRPCHandler) ReportOutcomes(stream grpc.ClientStreamingServer[controlplanev1.OutcomeBatch, controlplanev1.ReportOutcomesResponse]) error {
if stream == nil || handler == nil || handler.service == nil || handler.identity == nil {
return grpcError(ErrInvalidCommand)
}
var workerID, sessionID string
var accepted uint64
batches := 0
for {
request, err := stream.Recv()
if errors.Is(err, io.EOF) {
if workerID == "" {
return grpcError(ErrInvalidCommand)
}
return stream.SendAndClose(&controlplanev1.ReportOutcomesResponse{AcceptedThroughSequence: accepted})
}
if err != nil {
return err
}
if batches == outcome.MaxBatchesPerStream {
return grpcError(ErrInvalidCommand)
}
batch, err := decodeOutcomeBatch(request)
if err != nil {
return grpcError(err)
}
if workerID == "" {
if err := handler.authorize(stream.Context(), batch.WorkerID); err != nil {
return err
}
workerID, sessionID = batch.WorkerID, batch.SessionID
} else if batch.WorkerID != workerID || batch.SessionID != sessionID {
return grpcError(ErrInvalidCommand)
}
accepted, err = handler.service.ReportOutcomes(stream.Context(), batch)
if err != nil {
return grpcError(err)
}
batches++
}
}
func decodeOutcomeBatch(request *controlplanev1.OutcomeBatch) (outcome.Batch, error) {
if request == nil {
return outcome.Batch{}, ErrInvalidCommand
}
events := make([]outcome.Event, len(request.GetOutcomes()))
for index, item := range request.GetOutcomes() {
if item == nil || item.GetLatency() == nil || item.GetLatency().CheckValid() != nil ||
item.GetObservedAt() == nil || item.GetObservedAt().CheckValid() != nil {
return outcome.Batch{}, ErrInvalidCommand
}
stage, ok := decodeOutcomeStage(item.GetStage())
if !ok {
return outcome.Batch{}, ErrInvalidCommand
}
events[index] = outcome.Event{
ProxyID: item.GetProxyId(), RoutingName: item.GetRoutingName(), Stage: stage, Success: item.GetSuccess(),
ErrorClass: outcome.ErrorClass(item.GetErrorClass()), Latency: item.GetLatency().AsDuration(), ObservedAt: item.GetObservedAt().AsTime(),
}
}
return outcome.Batch{WorkerID: request.GetWorkerId(), SessionID: request.GetSessionId(), Sequence: request.GetSequence(), Events: events}, nil
}
func decodeOutcomeStage(stage controlplanev1.OutcomeStage) (outcome.Stage, bool) {
switch stage {
case controlplanev1.OutcomeStage_OUTCOME_STAGE_DIAL:
return outcome.StageDial, true
case controlplanev1.OutcomeStage_OUTCOME_STAGE_PROXY_HANDSHAKE:
return outcome.StageProxyHandshake, true
case controlplanev1.OutcomeStage_OUTCOME_STAGE_RESPONSE_HEADERS:
return outcome.StageResponseHeaders, true
case controlplanev1.OutcomeStage_OUTCOME_STAGE_TUNNEL:
return outcome.StageTunnel, true
default:
return outcome.StageUnspecified, false
}
}
func (handler *GRPCHandler) authorize(ctx context.Context, workerID string) error { func (handler *GRPCHandler) authorize(ctx context.Context, workerID string) error {
if err := handler.identity.Authorize(ctx, workerID); err != nil { if err := handler.identity.Authorize(ctx, workerID); err != nil {
return status.Error(codes.PermissionDenied, "worker identity is not authorized") return status.Error(codes.PermissionDenied, "worker identity is not authorized")
@ -210,7 +290,8 @@ func grpcError(err error) error {
return status.Error(codes.Canceled, "worker control request canceled") return status.Error(codes.Canceled, "worker control request canceled")
case errors.Is(err, context.DeadlineExceeded): case errors.Is(err, context.DeadlineExceeded):
return status.Error(codes.DeadlineExceeded, "worker control request deadline exceeded") return status.Error(codes.DeadlineExceeded, "worker control request deadline exceeded")
case errors.Is(err, ErrInvalidCommand), errors.Is(err, workerruntime.ErrInvalidReport), case errors.Is(err, ErrInvalidCommand), errors.Is(err, outcome.ErrInvalidBatch), errors.Is(err, workerruntime.ErrInvalidReport),
errors.Is(err, workerruntime.ErrInvalidOutcome),
errors.Is(err, workerruntime.ErrInvalidAcknowledgement), errors.Is(err, workerruntime.ErrInvalidSnapshotReference): errors.Is(err, workerruntime.ErrInvalidAcknowledgement), errors.Is(err, workerruntime.ErrInvalidSnapshotReference):
return status.Error(codes.InvalidArgument, "invalid worker control request") return status.Error(codes.InvalidArgument, "invalid worker control request")
case errors.Is(err, ErrProtocolVersion): case errors.Is(err, ErrProtocolVersion):
@ -225,6 +306,10 @@ func grpcError(err error) error {
return status.Error(codes.Aborted, "worker runtime sequence is stale") return status.Error(codes.Aborted, "worker runtime sequence is stale")
case errors.Is(err, workerruntime.ErrConflictingReport): case errors.Is(err, workerruntime.ErrConflictingReport):
return status.Error(codes.AlreadyExists, "worker runtime sequence conflicts") return status.Error(codes.AlreadyExists, "worker runtime sequence conflicts")
case errors.Is(err, workerruntime.ErrStaleOutcome):
return status.Error(codes.Aborted, "worker outcome sequence is stale")
case errors.Is(err, workerruntime.ErrConflictingOutcome):
return status.Error(codes.AlreadyExists, "worker outcome sequence conflicts")
default: default:
return status.Error(codes.Unavailable, "worker control plane unavailable") return status.Error(codes.Unavailable, "worker control plane unavailable")
} }

View File

@ -8,12 +8,15 @@ import (
"time" "time"
controlplanev1 "proxy-pool/gen/controlplane/v1" controlplanev1 "proxy-pool/gen/controlplane/v1"
"proxy-pool/internal/domain/outcome"
"proxy-pool/internal/domain/workerruntime" "proxy-pool/internal/domain/workerruntime"
"google.golang.org/grpc" "google.golang.org/grpc"
"google.golang.org/grpc/codes" "google.golang.org/grpc/codes"
"google.golang.org/grpc/status" "google.golang.org/grpc/status"
"google.golang.org/grpc/test/bufconn" "google.golang.org/grpc/test/bufconn"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/durationpb"
"google.golang.org/protobuf/types/known/timestamppb" "google.golang.org/protobuf/types/known/timestamppb"
) )
@ -45,7 +48,7 @@ func TestGRPCHandlerMapsWorkerRequests(t *testing.T) {
} }
} }
func TestGRPCHandlerMapsErrorsAndLeavesStreamsUnimplemented(t *testing.T) { func TestGRPCHandlerMapsErrorsAndLeavesSnapshotStreamUnimplemented(t *testing.T) {
service := &grpcServiceStub{registerErr: ErrProtocolVersion} service := &grpcServiceStub{registerErr: ErrProtocolVersion}
client, cleanup := grpcWorkerClient(t, service, allowIdentity{}) client, cleanup := grpcWorkerClient(t, service, allowIdentity{})
defer cleanup() defer cleanup()
@ -58,11 +61,88 @@ func TestGRPCHandlerMapsErrorsAndLeavesStreamsUnimplemented(t *testing.T) {
if err != nil || status.Code(streamErr) != codes.Unimplemented { if err != nil || status.Code(streamErr) != codes.Unimplemented {
t.Fatalf("WatchSnapshots() = %v, %v", err, streamErr) t.Fatalf("WatchSnapshots() = %v, %v", err, streamErr)
} }
outcomes, err := client.ReportOutcomes(context.Background())
_, outcomesErr := outcomes.CloseAndRecv()
if err != nil || status.Code(outcomesErr) != codes.Unimplemented {
t.Fatalf("ReportOutcomes() = %v, %v", err, outcomesErr)
} }
func TestGRPCHandlerReportsOutcomeStream(t *testing.T) {
service := &grpcServiceStub{}
client, cleanup := grpcWorkerClient(t, service, allowIdentity{})
defer cleanup()
stream, err := client.ReportOutcomes(context.Background())
if err != nil {
t.Fatalf("ReportOutcomes() = %v", err)
}
if err := stream.Send(&controlplanev1.OutcomeBatch{
WorkerId: "worker-a", SessionId: "session-a", Sequence: 3,
Outcomes: []*controlplanev1.ProxyOutcome{{
ProxyId: "proxy-a", RoutingName: "route-a", Stage: controlplanev1.OutcomeStage_OUTCOME_STAGE_DIAL,
Success: false, ErrorClass: "timeout", Latency: durationpb.New(25 * time.Millisecond), ObservedAt: timestamppb.New(time.Now()),
}},
}); err != nil {
t.Fatalf("Send() = %v", err)
}
response, err := stream.CloseAndRecv()
if err != nil || response.GetAcceptedThroughSequence() != 3 || service.outcome.Sequence != 3 ||
service.outcome.Events[0].ErrorClass != outcome.ErrorClassTimeout {
t.Fatalf("CloseAndRecv() = %+v, %v; outcome = %+v", response, err, service.outcome)
}
}
func TestGRPCHandlerFencesOutcomesThroughWorkerService(t *testing.T) {
now := time.Date(2026, 7, 31, 12, 0, 0, 0, time.UTC)
store, err := workerruntime.NewMemoryStore(func() time.Time { return now })
if err != nil {
t.Fatalf("NewMemoryStore() = %v", err)
}
service, err := NewService(store, Options{
ProtocolVersion: 1, HeartbeatInterval: time.Second, SessionTTL: 3 * time.Second,
MaxStaleAge: time.Second, MaxRuntimeCounters: 4,
SessionID: func() (string, error) { return "session-a", nil },
})
if err != nil {
t.Fatalf("NewService() = %v", err)
}
client, cleanup := grpcWorkerClient(t, service, allowIdentity{})
defer cleanup()
registered, err := client.RegisterWorker(context.Background(), &controlplanev1.RegisterWorkerRequest{
WorkerId: "worker-a", InstanceId: "instance-a", Zone: "zone-a", SupportedProtocolVersion: 1,
})
if err != nil || registered.GetSessionId() != "session-a" {
t.Fatalf("RegisterWorker() = %+v, %v", registered, err)
}
batch := &controlplanev1.OutcomeBatch{
WorkerId: "worker-a", SessionId: "session-a", Sequence: 1,
Outcomes: []*controlplanev1.ProxyOutcome{{
ProxyId: "proxy-a", Stage: controlplanev1.OutcomeStage_OUTCOME_STAGE_DIAL, Success: true,
Latency: durationpb.New(time.Millisecond), ObservedAt: timestamppb.New(now),
}},
}
if accepted, err := reportOutcomeBatch(context.Background(), client, batch); err != nil || accepted != 1 {
t.Fatalf("ReportOutcomes(first) = %d, %v", accepted, err)
}
if accepted, err := reportOutcomeBatch(context.Background(), client, batch); err != nil || accepted != 1 {
t.Fatalf("ReportOutcomes(replay) = %d, %v", accepted, err)
}
conflicting := proto.Clone(batch).(*controlplanev1.OutcomeBatch)
conflicting.Outcomes[0].Success = false
conflicting.Outcomes[0].ErrorClass = "dial"
if _, err := reportOutcomeBatch(context.Background(), client, conflicting); status.Code(err) != codes.AlreadyExists {
t.Fatalf("ReportOutcomes(conflict) code = %s, want AlreadyExists; error=%v", status.Code(err), err)
}
}
func reportOutcomeBatch(ctx context.Context, client controlplanev1.WorkerControlPlaneClient, batch *controlplanev1.OutcomeBatch) (uint64, error) {
stream, err := client.ReportOutcomes(ctx)
if err != nil {
return 0, err
}
if err := stream.Send(batch); err != nil {
return 0, err
}
response, err := stream.CloseAndRecv()
if err != nil {
return 0, err
}
return response.GetAcceptedThroughSequence(), nil
} }
func TestGRPCHandlerStreamsIssuedFullSnapshots(t *testing.T) { func TestGRPCHandlerStreamsIssuedFullSnapshots(t *testing.T) {
@ -166,6 +246,9 @@ type grpcServiceStub struct {
acknowledgeErr error acknowledgeErr error
report workerruntime.Report report workerruntime.Report
reportErr error reportErr error
outcome outcome.Batch
outcomeAccepted uint64
outcomeErr error
issued workerruntime.SnapshotReference issued workerruntime.SnapshotReference
issuedSessionID string issuedSessionID string
validateErr error validateErr error
@ -195,6 +278,13 @@ func (stub *grpcServiceStub) ReportRuntime(_ context.Context, report workerrunti
stub.report = report stub.report = report
return RuntimeDecision{AcceptedOwnershipEpoch: 9}, stub.reportErr return RuntimeDecision{AcceptedOwnershipEpoch: 9}, stub.reportErr
} }
func (stub *grpcServiceStub) ReportOutcomes(_ context.Context, batch outcome.Batch) (uint64, error) {
stub.outcome = batch
if stub.outcomeAccepted == 0 {
stub.outcomeAccepted = batch.Sequence
}
return stub.outcomeAccepted, stub.outcomeErr
}
type allowIdentity struct{} type allowIdentity struct{}

View File

@ -8,6 +8,7 @@ import (
"errors" "errors"
"time" "time"
"proxy-pool/internal/domain/outcome"
ownershipDomain "proxy-pool/internal/domain/ownership" ownershipDomain "proxy-pool/internal/domain/ownership"
"proxy-pool/internal/domain/workerruntime" "proxy-pool/internal/domain/workerruntime"
platformCredentials "proxy-pool/internal/platform/credentials" platformCredentials "proxy-pool/internal/platform/credentials"
@ -71,6 +72,7 @@ type Service interface {
IssueSnapshot(context.Context, string, workerruntime.SnapshotReference) error IssueSnapshot(context.Context, string, workerruntime.SnapshotReference) error
Acknowledge(context.Context, SnapshotAcknowledgement) error Acknowledge(context.Context, SnapshotAcknowledgement) error
ReportRuntime(context.Context, workerruntime.Report) (RuntimeDecision, error) ReportRuntime(context.Context, workerruntime.Report) (RuntimeDecision, error)
ReportOutcomes(context.Context, outcome.Batch) (uint64, error)
} }
func (service *service) CurrentOwnershipEpoch(ctx context.Context) (uint64, error) { func (service *service) CurrentOwnershipEpoch(ctx context.Context) (uint64, error) {
@ -119,6 +121,7 @@ func (service *service) IssueSnapshot(ctx context.Context, sessionID string, ref
type service struct { type service struct {
store workerruntime.ControlStore store workerruntime.ControlStore
outcomes workerruntime.OutcomeWriter
options Options options Options
snapshots SnapshotSource snapshots SnapshotSource
} }
@ -133,6 +136,7 @@ func NewService(store workerruntime.ControlStore, options Options) (Service, err
options.SessionID = randomSessionID options.SessionID = randomSessionID
} }
result := &service{store: store, options: options} result := &service{store: store, options: options}
result.outcomes, _ = store.(workerruntime.OutcomeWriter)
if options.SnapshotReader != nil { if options.SnapshotReader != nil {
var routing []RoutingSource var routing []RoutingSource
if options.RoutingSource != nil { if options.RoutingSource != nil {
@ -252,6 +256,31 @@ func (service *service) ReportRuntime(ctx context.Context, report workerruntime.
return RuntimeDecision{AcceptedOwnershipEpoch: epoch}, nil return RuntimeDecision{AcceptedOwnershipEpoch: epoch}, nil
} }
// ReportOutcomes records only a batch digest and sequence fence. Raw Gateway
// observations are intentionally not persisted by the control plane.
func (service *service) ReportOutcomes(ctx context.Context, batch outcome.Batch) (uint64, error) {
if ctx == nil {
return 0, ErrInvalidCommand
}
if err := ctx.Err(); err != nil {
return 0, err
}
normalized, digest, err := outcome.NormalizeBatch(batch, service.options.MaxRuntimeCounters)
if err != nil {
return 0, errors.Join(ErrInvalidCommand, err)
}
if service.outcomes == nil {
return 0, ErrUnavailable
}
accepted, err := service.outcomes.RecordOutcomes(ctx, workerruntime.OutcomeReference{
WorkerID: normalized.WorkerID, SessionID: normalized.SessionID, Sequence: normalized.Sequence, Digest: digest,
})
if err != nil {
return 0, classifyStoreError(err)
}
return accepted, nil
}
func randomSessionID() (string, error) { func randomSessionID() (string, error) {
var value [16]byte var value [16]byte
if _, err := rand.Read(value[:]); err != nil { if _, err := rand.Read(value[:]); err != nil {
@ -266,8 +295,10 @@ func classifyStoreError(err error) error {
} }
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) || if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) ||
errors.Is(err, workerruntime.ErrInvalidSession) || errors.Is(err, workerruntime.ErrInvalidReport) || errors.Is(err, workerruntime.ErrInvalidSession) || errors.Is(err, workerruntime.ErrInvalidReport) ||
errors.Is(err, workerruntime.ErrInvalidOutcome) ||
errors.Is(err, workerruntime.ErrInvalidAcknowledgement) || errors.Is(err, workerruntime.ErrStaleSession) || errors.Is(err, workerruntime.ErrInvalidAcknowledgement) || errors.Is(err, workerruntime.ErrStaleSession) ||
errors.Is(err, workerruntime.ErrStaleReport) || errors.Is(err, workerruntime.ErrConflictingReport) || errors.Is(err, workerruntime.ErrStaleReport) || errors.Is(err, workerruntime.ErrConflictingReport) ||
errors.Is(err, workerruntime.ErrStaleOutcome) || errors.Is(err, workerruntime.ErrConflictingOutcome) ||
errors.Is(err, workerruntime.ErrSnapshotMismatch) || errors.Is(err, workerruntime.ErrStaleAcknowledgement) { errors.Is(err, workerruntime.ErrSnapshotMismatch) || errors.Is(err, workerruntime.ErrStaleAcknowledgement) {
return err return err
} }

View File

@ -7,6 +7,7 @@ import (
"testing" "testing"
"time" "time"
"proxy-pool/internal/domain/outcome"
"proxy-pool/internal/domain/workerruntime" "proxy-pool/internal/domain/workerruntime"
) )
@ -51,6 +52,13 @@ func TestServiceRegistersAcknowledgesAndReportsRuntime(t *testing.T) {
if err != nil || decision.RequireFullSnapshot || decision.AcceptedOwnershipEpoch != registered.OwnershipEpoch { if err != nil || decision.RequireFullSnapshot || decision.AcceptedOwnershipEpoch != registered.OwnershipEpoch {
t.Fatalf("ReportRuntime() = %+v, %v", decision, err) t.Fatalf("ReportRuntime() = %+v, %v", decision, err)
} }
accepted, err := service.ReportOutcomes(context.Background(), outcome.Batch{
WorkerID: "worker-a", SessionID: registered.SessionID, Sequence: 1,
Events: []outcome.Event{{ProxyID: "proxy-a", Stage: outcome.StageDial, Success: true, ObservedAt: now}},
})
if err != nil || accepted != 1 {
t.Fatalf("ReportOutcomes() = %d, %v", accepted, err)
}
decision, err = service.ReportRuntime(context.Background(), workerruntime.Report{ decision, err = service.ReportRuntime(context.Background(), workerruntime.Report{
WorkerID: "worker-a", SessionID: registered.SessionID, Sequence: 2, WorkerID: "worker-a", SessionID: registered.SessionID, Sequence: 2,
SnapshotVersion: 8, OwnershipEpoch: registered.OwnershipEpoch, ObservedAt: now, SnapshotVersion: 8, OwnershipEpoch: registered.OwnershipEpoch, ObservedAt: now,

View File

@ -0,0 +1,110 @@
package outcome
import (
"crypto/sha256"
"encoding/json"
"errors"
"regexp"
"time"
)
const (
maximumIdentifierBytes = 128
MaxBatchesPerStream = 16
)
var (
ErrInvalidBatch = errors.New("invalid proxy outcome batch")
identifier = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$`)
)
// Stage identifies the furthest proxy-processing boundary reached by a
// request. It is deliberately independent of a particular transport error.
type Stage uint8
const (
StageUnspecified Stage = iota
StageDial
StageProxyHandshake
StageResponseHeaders
StageTunnel
)
// ErrorClass is a bounded category suitable for aggregation and metrics.
// It must never contain raw connection, target, or credential details.
type ErrorClass string
const (
ErrorClassDial ErrorClass = "dial"
ErrorClassTimeout ErrorClass = "timeout"
ErrorClassProxyResponse ErrorClass = "proxy_response"
ErrorClassHandshake ErrorClass = "handshake"
ErrorClassRelay ErrorClass = "relay"
ErrorClassCanceled ErrorClass = "canceled"
ErrorClassInternal ErrorClass = "internal"
)
type Event struct {
ProxyID string
RoutingName string
Stage Stage
Success bool
ErrorClass ErrorClass
Latency time.Duration
ObservedAt time.Time
}
type Batch struct {
WorkerID string
SessionID string
Sequence uint64
Events []Event
}
// NormalizeBatch validates one bounded report, deep-copies it, and returns a
// digest used by the Controller's durable idempotency fence.
func NormalizeBatch(batch Batch, maximumEvents int) (Batch, [sha256.Size]byte, error) {
if !validIdentifier(batch.WorkerID) || !validIdentifier(batch.SessionID) || batch.Sequence == 0 ||
maximumEvents <= 0 || len(batch.Events) == 0 || len(batch.Events) > maximumEvents {
return Batch{}, [sha256.Size]byte{}, ErrInvalidBatch
}
normalized := batch
normalized.Events = append([]Event(nil), batch.Events...)
for index := range normalized.Events {
event := &normalized.Events[index]
if !validIdentifier(event.ProxyID) || (event.RoutingName != "" && !validIdentifier(event.RoutingName)) ||
!validStage(event.Stage) || event.Latency < 0 || event.ObservedAt.IsZero() ||
(event.Success && event.ErrorClass != "") || (!event.Success && !validErrorClass(event.ErrorClass)) {
return Batch{}, [sha256.Size]byte{}, ErrInvalidBatch
}
event.ObservedAt = event.ObservedAt.UTC()
}
payload, err := json.Marshal(normalized)
if err != nil {
return Batch{}, [sha256.Size]byte{}, ErrInvalidBatch
}
return normalized, sha256.Sum256(payload), nil
}
func validIdentifier(value string) bool {
return len(value) <= maximumIdentifierBytes && identifier.MatchString(value)
}
func validStage(stage Stage) bool {
switch stage {
case StageDial, StageProxyHandshake, StageResponseHeaders, StageTunnel:
return true
default:
return false
}
}
func validErrorClass(class ErrorClass) bool {
switch class {
case ErrorClassDial, ErrorClassTimeout, ErrorClassProxyResponse, ErrorClassHandshake,
ErrorClassRelay, ErrorClassCanceled, ErrorClassInternal:
return true
default:
return false
}
}

View File

@ -0,0 +1,52 @@
package outcome
import (
"errors"
"testing"
"time"
)
func TestNormalizeBatchCanonicalizesAndDigestsOutcomes(t *testing.T) {
observedAt := time.Date(2026, 7, 31, 12, 0, 0, 0, time.FixedZone("UTC+8", 8*60*60))
normalized, digest, err := NormalizeBatch(Batch{
WorkerID: "worker-a", SessionID: "session-a", Sequence: 7,
Events: []Event{{
ProxyID: "proxy-a", RoutingName: "gateway-default", Stage: StageResponseHeaders,
Success: false, ErrorClass: ErrorClassTimeout, Latency: 25 * time.Millisecond, ObservedAt: observedAt,
}},
}, 4)
if err != nil {
t.Fatalf("NormalizeBatch(): %v", err)
}
if normalized.Events[0].ObservedAt.Location() != time.UTC || digest == [32]byte{} {
t.Fatalf("normalized outcome = %+v, digest = %x", normalized.Events[0], digest)
}
_, repeated, err := NormalizeBatch(normalized, 4)
if err != nil || digest != repeated {
t.Fatalf("repeated NormalizeBatch() digest = %x, %v; want %x", repeated, err, digest)
}
}
func TestNormalizeBatchRejectsUnboundedOrUnsafeInput(t *testing.T) {
now := time.Now()
valid := Event{ProxyID: "proxy-a", Stage: StageDial, Success: false, ErrorClass: ErrorClassDial, ObservedAt: now}
tests := []struct {
name string
batch Batch
limit int
}{
{name: "missing worker", batch: Batch{SessionID: "session-a", Sequence: 1, Events: []Event{valid}}, limit: 1},
{name: "zero sequence", batch: Batch{WorkerID: "worker-a", SessionID: "session-a", Events: []Event{valid}}, limit: 1},
{name: "empty batch", batch: Batch{WorkerID: "worker-a", SessionID: "session-a", Sequence: 1}, limit: 1},
{name: "exceeds limit", batch: Batch{WorkerID: "worker-a", SessionID: "session-a", Sequence: 1, Events: []Event{valid, valid}}, limit: 1},
{name: "success with error", batch: Batch{WorkerID: "worker-a", SessionID: "session-a", Sequence: 1, Events: []Event{{ProxyID: "proxy-a", Stage: StageDial, Success: true, ErrorClass: ErrorClassDial, ObservedAt: now}}}, limit: 1},
{name: "unknown error class", batch: Batch{WorkerID: "worker-a", SessionID: "session-a", Sequence: 1, Events: []Event{{ProxyID: "proxy-a", Stage: StageDial, ErrorClass: "raw network failure", ObservedAt: now}}}, limit: 1},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
if _, _, err := NormalizeBatch(test.batch, test.limit); !errors.Is(err, ErrInvalidBatch) {
t.Fatalf("NormalizeBatch() error = %v, want ErrInvalidBatch", err)
}
})
}
}

View File

@ -17,7 +17,7 @@ func TestMemoryStoreContract(t *testing.T) {
t.Fatalf("NewMemoryStore(): %v", err) t.Fatalf("NewMemoryStore(): %v", err)
} }
return contracttest.Fixture{ return contracttest.Fixture{
Store: store, Reader: store, TTL: time.Minute, Store: store, Reader: store, Outcomes: store, TTL: time.Minute,
Advance: func(duration time.Duration) { now = now.Add(duration) }, Advance: func(duration time.Duration) { now = now.Add(duration) },
} }
}) })

View File

@ -13,6 +13,7 @@ import (
type Fixture struct { type Fixture struct {
Store workerruntime.ControlStore Store workerruntime.ControlStore
Reader workerruntime.RuntimeReader Reader workerruntime.RuntimeReader
Outcomes workerruntime.OutcomeWriter
TTL time.Duration TTL time.Duration
Advance func(time.Duration) Advance func(time.Duration)
} }
@ -25,6 +26,8 @@ func Run(t *testing.T, factory Factory) {
t.Run("acknowledged runtime lifecycle", func(t *testing.T) { runLifecycle(t, newFixture(t, factory)) }) t.Run("acknowledged runtime lifecycle", func(t *testing.T) { runLifecycle(t, newFixture(t, factory)) })
t.Run("negative acknowledgement fences runtime", func(t *testing.T) { runNegativeAck(t, newFixture(t, factory)) }) t.Run("negative acknowledgement fences runtime", func(t *testing.T) { runNegativeAck(t, newFixture(t, factory)) })
t.Run("session replacement fences issued snapshots", func(t *testing.T) { runSessionReplacementFence(t, newFixture(t, factory)) }) t.Run("session replacement fences issued snapshots", func(t *testing.T) { runSessionReplacementFence(t, newFixture(t, factory)) })
t.Run("outcome sequence fences retries", func(t *testing.T) { runOutcomeFence(t, newFixture(t, factory)) })
t.Run("outcome invalid call is a store error", func(t *testing.T) { runOutcomeInvalidCall(t, newFixture(t, factory)) })
} }
func runLifecycle(t *testing.T, fixture Fixture) { func runLifecycle(t *testing.T, fixture Fixture) {
@ -117,10 +120,57 @@ func runSessionReplacementFence(t *testing.T, fixture Fixture) {
} }
} }
func runOutcomeFence(t *testing.T, fixture Fixture) {
t.Helper()
ctx := context.Background()
open(t, fixture.Store, fixture.TTL)
first := workerruntime.OutcomeReference{
WorkerID: "worker-a", SessionID: "session-a", Sequence: 1,
Digest: sha256.Sum256([]byte("outcome-1")),
}
accepted, err := fixture.Outcomes.RecordOutcomes(ctx, first)
if err != nil || accepted != first.Sequence {
t.Fatalf("RecordOutcomes(first) = %d, %v; want 1, nil", accepted, err)
}
accepted, err = fixture.Outcomes.RecordOutcomes(ctx, first)
if err != nil || accepted != first.Sequence {
t.Fatalf("RecordOutcomes(replay) = %d, %v; want 1, nil", accepted, err)
}
conflicting := first
conflicting.Digest = sha256.Sum256([]byte("outcome-conflict"))
if _, err := fixture.Outcomes.RecordOutcomes(ctx, conflicting); !errors.Is(err, workerruntime.ErrConflictingOutcome) {
t.Fatalf("RecordOutcomes(conflict) = %v, want ErrConflictingOutcome", err)
}
next := first
next.Sequence = 2
next.Digest = sha256.Sum256([]byte("outcome-2"))
if accepted, err := fixture.Outcomes.RecordOutcomes(ctx, next); err != nil || accepted != next.Sequence {
t.Fatalf("RecordOutcomes(next) = %d, %v; want 2, nil", accepted, err)
}
if _, err := fixture.Outcomes.RecordOutcomes(ctx, first); !errors.Is(err, workerruntime.ErrStaleOutcome) {
t.Fatalf("RecordOutcomes(stale) = %v, want ErrStaleOutcome", err)
}
if err := fixture.Store.OpenSession(ctx, workerruntime.Session{
WorkerID: "worker-a", InstanceID: "instance-b", SessionID: "session-b", Zone: "zone-a", ProtocolVersion: 1,
}, fixture.TTL); err != nil {
t.Fatalf("OpenSession(replacement): %v", err)
}
if _, err := fixture.Outcomes.RecordOutcomes(ctx, first); !errors.Is(err, workerruntime.ErrStaleSession) {
t.Fatalf("RecordOutcomes(old session) = %v, want ErrStaleSession", err)
}
}
func runOutcomeInvalidCall(t *testing.T, fixture Fixture) {
t.Helper()
if _, err := fixture.Outcomes.RecordOutcomes(nil, workerruntime.OutcomeReference{}); !errors.Is(err, workerruntime.ErrInvalidStore) {
t.Fatalf("RecordOutcomes(nil) = %v, want ErrInvalidStore", err)
}
}
func newFixture(t *testing.T, factory Factory) Fixture { func newFixture(t *testing.T, factory Factory) Fixture {
t.Helper() t.Helper()
fixture := factory(t) fixture := factory(t)
if fixture.Store == nil || fixture.Reader == nil || fixture.TTL <= 0 || fixture.Advance == nil { if fixture.Store == nil || fixture.Reader == nil || fixture.Outcomes == nil || fixture.TTL <= 0 || fixture.Advance == nil {
t.Fatal("contract fixture is incomplete") t.Fatal("contract fixture is incomplete")
} }
return fixture return fixture

View File

@ -14,6 +14,7 @@ type MemoryStore struct {
sessions map[string]memorySession sessions map[string]memorySession
references map[string]memoryReference references map[string]memoryReference
reports map[string]memoryReport reports map[string]memoryReport
outcomes map[string]memoryOutcome
} }
type memorySession struct { type memorySession struct {
@ -33,11 +34,16 @@ type memoryReport struct {
counters map[string]Counter counters map[string]Counter
} }
type memoryOutcome struct {
value OutcomeReference
}
var ( var (
_ ControlStore = (*MemoryStore)(nil) _ ControlStore = (*MemoryStore)(nil)
_ SessionWriter = (*MemoryStore)(nil) _ SessionWriter = (*MemoryStore)(nil)
_ ReportWriter = (*MemoryStore)(nil) _ ReportWriter = (*MemoryStore)(nil)
_ RuntimeReader = (*MemoryStore)(nil) _ RuntimeReader = (*MemoryStore)(nil)
_ OutcomeWriter = (*MemoryStore)(nil)
) )
func NewMemoryStore(now func() time.Time) (*MemoryStore, error) { func NewMemoryStore(now func() time.Time) (*MemoryStore, error) {
@ -46,7 +52,7 @@ func NewMemoryStore(now func() time.Time) (*MemoryStore, error) {
} }
return &MemoryStore{ return &MemoryStore{
now: now, epoch: 1, sessions: make(map[string]memorySession), now: now, epoch: 1, sessions: make(map[string]memorySession),
references: make(map[string]memoryReference), reports: make(map[string]memoryReport), references: make(map[string]memoryReference), reports: make(map[string]memoryReport), outcomes: make(map[string]memoryOutcome),
}, nil }, nil
} }
@ -85,6 +91,7 @@ func (store *MemoryStore) OpenSession(ctx context.Context, session Session, ttl
defer store.mu.Unlock() defer store.mu.Unlock()
delete(store.reports, normalized.WorkerID) delete(store.reports, normalized.WorkerID)
delete(store.references, normalized.WorkerID) delete(store.references, normalized.WorkerID)
delete(store.outcomes, normalized.WorkerID)
store.sessions[normalized.WorkerID] = memorySession{value: normalized, expiresAt: now.Add(ttl)} store.sessions[normalized.WorkerID] = memorySession{value: normalized, expiresAt: now.Add(ttl)}
return nil return nil
} }
@ -246,12 +253,49 @@ func (store *MemoryStore) ReplaceSession(ctx context.Context, session Session, t
ackAdvanced := exists && !identityChanged && !expired && legacySessionAfter(session, current.value) ackAdvanced := exists && !identityChanged && !expired && legacySessionAfter(session, current.value)
if identityChanged || expired || ackAdvanced { if identityChanged || expired || ackAdvanced {
delete(store.reports, session.WorkerID) delete(store.reports, session.WorkerID)
delete(store.outcomes, session.WorkerID)
} }
session.RuntimeEnabled = true session.RuntimeEnabled = true
store.sessions[session.WorkerID] = memorySession{value: session, expiresAt: now.Add(ttl)} store.sessions[session.WorkerID] = memorySession{value: session, expiresAt: now.Add(ttl)}
return nil return nil
} }
func (store *MemoryStore) RecordOutcomes(ctx context.Context, reference OutcomeReference) (uint64, error) {
if ctx == nil || store == nil {
return 0, ErrInvalidStore
}
if err := ctx.Err(); err != nil {
return 0, err
}
normalized, err := NormalizeOutcomeReference(reference)
if err != nil {
return 0, err
}
now, err := store.currentTime()
if err != nil {
return 0, err
}
store.mu.Lock()
defer store.mu.Unlock()
session, exists := store.sessions[normalized.WorkerID]
if !exists || !session.expiresAt.After(now) || session.value.SessionID != normalized.SessionID {
return 0, ErrStaleSession
}
current, exists := store.outcomes[normalized.WorkerID]
if exists && current.value.SessionID == normalized.SessionID {
switch {
case normalized.Sequence < current.value.Sequence:
return 0, ErrStaleOutcome
case normalized.Sequence == current.value.Sequence && normalized.Digest != current.value.Digest:
return 0, ErrConflictingOutcome
case normalized.Sequence == current.value.Sequence:
return current.value.Sequence, nil
}
}
store.outcomes[normalized.WorkerID] = memoryOutcome{value: normalized}
return normalized.Sequence, nil
}
func (store *MemoryStore) ReplaceRuntime(ctx context.Context, report Report, ttl time.Duration) error { func (store *MemoryStore) ReplaceRuntime(ctx context.Context, report Report, ttl time.Duration) error {
if ctx == nil || store == nil || ttl <= 0 { if ctx == nil || store == nil || ttl <= 0 {
return ErrInvalidReport return ErrInvalidReport

View File

@ -0,0 +1,51 @@
package workerruntime
import (
"context"
"crypto/sha256"
"errors"
"testing"
"time"
)
func TestMemoryStoreFencesOutcomeSequenceBySessionAndDigest(t *testing.T) {
now := time.Date(2026, 7, 31, 12, 0, 0, 0, time.UTC)
store := newRuntimeStore(t, &now)
openControlSession(t, store, "session-a", time.Minute)
first := OutcomeReference{
WorkerID: "worker-a", SessionID: "session-a", Sequence: 1, Digest: sha256.Sum256([]byte("batch-1")),
}
accepted, err := store.RecordOutcomes(context.Background(), first)
if err != nil || accepted != 1 {
t.Fatalf("RecordOutcomes(first) = %d, %v; want 1, nil", accepted, err)
}
accepted, err = store.RecordOutcomes(context.Background(), first)
if err != nil || accepted != 1 {
t.Fatalf("RecordOutcomes(replay) = %d, %v; want 1, nil", accepted, err)
}
conflicting := first
conflicting.Digest = sha256.Sum256([]byte("different"))
if _, err := store.RecordOutcomes(context.Background(), conflicting); !errors.Is(err, ErrConflictingOutcome) {
t.Fatalf("RecordOutcomes(conflicting replay) error = %v, want ErrConflictingOutcome", err)
}
if _, err := store.RecordOutcomes(context.Background(), OutcomeReference{
WorkerID: "worker-a", SessionID: "session-a", Sequence: 0, Digest: first.Digest,
}); !errors.Is(err, ErrInvalidOutcome) {
t.Fatalf("RecordOutcomes(zero sequence) error = %v, want ErrInvalidOutcome", err)
}
if _, err := store.RecordOutcomes(context.Background(), OutcomeReference{
WorkerID: "worker-a", SessionID: "session-a", Sequence: 2, Digest: sha256.Sum256([]byte("batch-2")),
}); err != nil {
t.Fatalf("RecordOutcomes(next) error = %v", err)
}
if _, err := store.RecordOutcomes(context.Background(), OutcomeReference{
WorkerID: "worker-a", SessionID: "session-a", Sequence: 1, Digest: first.Digest,
}); !errors.Is(err, ErrStaleOutcome) {
t.Fatalf("RecordOutcomes(stale) error = %v, want ErrStaleOutcome", err)
}
openControlSession(t, store, "session-b", time.Minute)
if _, err := store.RecordOutcomes(context.Background(), first); !errors.Is(err, ErrStaleSession) {
t.Fatalf("RecordOutcomes(old session) error = %v, want ErrStaleSession", err)
}
}

View File

@ -21,6 +21,9 @@ var (
ErrStaleSnapshotReference = errors.New("stale worker snapshot reference") ErrStaleSnapshotReference = errors.New("stale worker snapshot reference")
ErrConflictingSnapshotReference = errors.New("conflicting worker snapshot reference") ErrConflictingSnapshotReference = errors.New("conflicting worker snapshot reference")
ErrStaleAcknowledgement = errors.New("stale worker snapshot acknowledgement") ErrStaleAcknowledgement = errors.New("stale worker snapshot acknowledgement")
ErrInvalidOutcome = errors.New("invalid worker outcome reference")
ErrStaleOutcome = errors.New("stale worker outcome sequence")
ErrConflictingOutcome = errors.New("conflicting worker outcome sequence")
) )
type Session struct { type Session struct {
@ -84,6 +87,15 @@ type Snapshot struct {
Fresh bool Fresh bool
} }
// OutcomeReference is the durable deduplication fence for one Gateway Outcome
// batch. The event payload itself remains outside the runtime store.
type OutcomeReference struct {
WorkerID string
SessionID string
Sequence uint64
Digest [sha256.Size]byte
}
type SessionWriter interface { type SessionWriter interface {
ReplaceSession(context.Context, Session, time.Duration) error ReplaceSession(context.Context, Session, time.Duration) error
} }
@ -97,6 +109,10 @@ type ControlStore interface {
ReplaceRuntime(context.Context, Report, time.Duration) error ReplaceRuntime(context.Context, Report, time.Duration) error
} }
type OutcomeWriter interface {
RecordOutcomes(context.Context, OutcomeReference) (uint64, error)
}
type ReportWriter interface { type ReportWriter interface {
ReplaceRuntime(context.Context, Report, time.Duration) error ReplaceRuntime(context.Context, Report, time.Duration) error
} }

View File

@ -82,6 +82,14 @@ func NormalizeAcknowledgement(acknowledgement SnapshotAcknowledgement) (Snapshot
return acknowledgement, nil return acknowledgement, nil
} }
func NormalizeOutcomeReference(reference OutcomeReference) (OutcomeReference, error) {
if !ValidIdentifier(reference.WorkerID) || !ValidIdentifier(reference.SessionID) || reference.Sequence == 0 ||
checksumIsZero(reference.Digest) {
return OutcomeReference{}, ErrInvalidOutcome
}
return reference, nil
}
// NormalizeReport returns the canonical sparse replacement and its digest. // NormalizeReport returns the canonical sparse replacement and its digest.
func NormalizeReport(report Report) (Report, [sha256.Size]byte, error) { func NormalizeReport(report Report) (Report, [sha256.Size]byte, error) {
if !ValidIdentifier(report.WorkerID) || !ValidIdentifier(report.SessionID) || report.Sequence == 0 || if !ValidIdentifier(report.WorkerID) || !ValidIdentifier(report.SessionID) || report.Sequence == 0 ||

View File

@ -27,6 +27,7 @@ import (
"proxy-pool/internal/domain/workerruntime" "proxy-pool/internal/domain/workerruntime"
"proxy-pool/internal/gateway/controlplane" "proxy-pool/internal/gateway/controlplane"
"proxy-pool/internal/gateway/dispatch" "proxy-pool/internal/gateway/dispatch"
gatewayOutcome "proxy-pool/internal/gateway/outcome"
"proxy-pool/internal/gateway/server" "proxy-pool/internal/gateway/server"
"proxy-pool/internal/gateway/snapshot" "proxy-pool/internal/gateway/snapshot"
"proxy-pool/internal/gateway/transport" "proxy-pool/internal/gateway/transport"
@ -45,6 +46,8 @@ const (
defaultReconnectInitialDelay = time.Second defaultReconnectInitialDelay = time.Second
defaultReconnectMaxDelay = 30 * time.Second defaultReconnectMaxDelay = 30 * time.Second
defaultReconnectJitter = 20 defaultReconnectJitter = 20
defaultOutcomeQueueCapacity = 65_536
defaultOutcomeBatchSize = 512
) )
// Options provides process-local settings. ControlPlaneAddress is deliberately // Options provides process-local settings. ControlPlaneAddress is deliberately
@ -147,6 +150,13 @@ func newRuntime(ctx context.Context, configuration *config.Config, options Optio
return nil, fmt.Errorf("build gateway target policy: %w", err) return nil, fmt.Errorf("build gateway target policy: %w", err)
} }
proxyTransport := transport.New(transport.Config{}, snapshotCredentialResolver{store: store}) proxyTransport := transport.New(transport.Config{}, snapshotCredentialResolver{store: store})
outcomes, err := gatewayOutcome.NewQueue(gatewayOutcome.QueueOptions{
Capacity: defaultOutcomeQueueCapacity, MaxBatch: min(defaultOutcomeBatchSize, configuration.ControlPlane.MaxRuntimeCounters),
})
if err != nil {
proxyTransport.CloseIdleConnections()
return nil, fmt.Errorf("build gateway outcome queue: %w", err)
}
handler, err := server.New(server.ConfigFromListener(configuration.Gateway), server.Dependencies{ handler, err := server.New(server.ConfigFromListener(configuration.Gateway), server.Dependencies{
Auth: protection.Auth, Auth: protection.Auth,
Access: protection.Access, Access: protection.Access,
@ -155,6 +165,7 @@ func newRuntime(ctx context.Context, configuration *config.Config, options Optio
Router: server.NewSnapshotRouter(store), Router: server.NewSnapshotRouter(store),
Dispatcher: dispatch.New(store), Dispatcher: dispatch.New(store),
Transport: proxyTransport, Transport: proxyTransport,
Outcomes: outcomes,
}) })
if err != nil { if err != nil {
proxyTransport.CloseIdleConnections() proxyTransport.CloseIdleConnections()
@ -198,7 +209,12 @@ func newRuntime(ctx context.Context, configuration *config.Config, options Optio
if err != nil { if err != nil {
return nil, fmt.Errorf("build snapshot watcher: %w", err) return nil, fmt.Errorf("build snapshot watcher: %w", err)
} }
session, err := controlplane.NewSessionRunner(reporter, watcher) outcomeReporter, err := controlplane.NewOutcomeReporter(controlplane.NewGeneratedOutcomeRPCClient(client), outcomes,
controlplane.OutcomeReporterOptions{WorkerID: options.WorkerID})
if err != nil {
return nil, fmt.Errorf("build outcome reporter: %w", err)
}
session, err := controlplane.NewSessionRunner(reporter, watcher, outcomeReporter)
if err != nil { if err != nil {
return nil, fmt.Errorf("build control plane session: %w", err) return nil, fmt.Errorf("build control plane session: %w", err)
} }

View File

@ -0,0 +1,254 @@
package controlplane
import (
"context"
"errors"
"fmt"
"sync"
"time"
controlplanev1 "proxy-pool/gen/controlplane/v1"
domain "proxy-pool/internal/domain/outcome"
"proxy-pool/internal/domain/workerruntime"
gatewayOutcome "proxy-pool/internal/gateway/outcome"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"google.golang.org/protobuf/types/known/durationpb"
"google.golang.org/protobuf/types/known/timestamppb"
)
var (
ErrInvalidOutcomeReporter = errors.New("invalid gateway outcome reporter")
ErrOutcomeUnconfirmed = errors.New("controller did not confirm gateway outcome batch")
)
const (
outcomeRetryInitialDelay = 100 * time.Millisecond
outcomeRetryMaxDelay = 5 * time.Second
)
type OutcomeStream interface {
Send(*controlplanev1.OutcomeBatch) error
CloseAndRecv() (*controlplanev1.ReportOutcomesResponse, error)
}
type OutcomeRPCClient interface {
ReportOutcomes(context.Context) (OutcomeStream, error)
}
type OutcomeReporterOptions struct {
WorkerID string
MaxBatchesPerReportRPC int
}
type OutcomeReporter struct {
client OutcomeRPCClient
queue *gatewayOutcome.Queue
options OutcomeReporterOptions
mu sync.Mutex
sessionID string
sequence uint64
pending []domain.Batch
}
func NewOutcomeReporter(client OutcomeRPCClient, queue *gatewayOutcome.Queue, options OutcomeReporterOptions) (*OutcomeReporter, error) {
if client == nil || queue == nil || !workerruntime.ValidIdentifier(options.WorkerID) {
return nil, ErrInvalidOutcomeReporter
}
if options.MaxBatchesPerReportRPC <= 0 {
options.MaxBatchesPerReportRPC = domain.MaxBatchesPerStream
}
if options.MaxBatchesPerReportRPC > domain.MaxBatchesPerStream {
return nil, ErrInvalidOutcomeReporter
}
return &OutcomeReporter{client: client, queue: queue, options: options}, nil
}
// RunRegistered continuously batches local observations for a single
// Controller session. Transient delivery failures retain the exact pending
// batches and retry them in the same session with bounded backoff.
func (reporter *OutcomeReporter) RunRegistered(ctx context.Context, registration Registration) error {
if reporter == nil || ctx == nil || !workerruntime.ValidIdentifier(registration.SessionID) {
return ErrInvalidOutcomeReporter
}
reporter.mu.Lock()
reporter.sessionID = registration.SessionID
reporter.sequence = 0
reporter.pending = nil
reporter.mu.Unlock()
retryDelay := outcomeRetryInitialDelay
for {
err := reporter.Report(ctx, registration.SessionID)
if err == nil {
retryDelay = outcomeRetryInitialDelay
continue
}
if ctx.Err() != nil {
return ctx.Err()
}
if terminalOutcomeError(err) {
return err
}
if err := waitOutcomeRetry(ctx, retryDelay); err != nil {
return err
}
retryDelay = nextOutcomeRetryDelay(retryDelay)
}
}
func terminalOutcomeError(err error) bool {
if errors.Is(err, ErrInvalidOutcomeReporter) || errors.Is(err, ErrNotRegistered) {
return true
}
switch status.Code(err) {
case codes.Aborted, codes.AlreadyExists, codes.FailedPrecondition,
codes.InvalidArgument, codes.PermissionDenied, codes.Unauthenticated:
return true
default:
return false
}
}
func waitOutcomeRetry(ctx context.Context, delay time.Duration) error {
timer := time.NewTimer(delay)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
return nil
}
}
func nextOutcomeRetryDelay(delay time.Duration) time.Duration {
if delay >= outcomeRetryMaxDelay/2 {
return outcomeRetryMaxDelay
}
return delay * 2
}
func (reporter *OutcomeReporter) Report(ctx context.Context, sessionID string) error {
if reporter == nil || ctx == nil || !workerruntime.ValidIdentifier(sessionID) {
return ErrInvalidOutcomeReporter
}
if err := ctx.Err(); err != nil {
return err
}
reporter.mu.Lock()
defer reporter.mu.Unlock()
if reporter.sessionID != sessionID {
return ErrNotRegistered
}
if err := reporter.fillPending(ctx); err != nil {
return err
}
streamCtx, cancel := context.WithCancel(ctx)
defer cancel()
stream, err := reporter.client.ReportOutcomes(streamCtx)
if err != nil {
return err
}
for _, batch := range reporter.pending {
request, requestErr := outcomeRequest(batch)
if requestErr != nil {
return requestErr
}
if sendErr := stream.Send(request); sendErr != nil {
return sendErr
}
}
response, err := stream.CloseAndRecv()
if err != nil {
return err
}
last := reporter.pending[len(reporter.pending)-1].Sequence
if response == nil || response.GetAcceptedThroughSequence() != last {
return ErrOutcomeUnconfirmed
}
reporter.sequence = last
reporter.pending = nil
return nil
}
func (reporter *OutcomeReporter) fillPending(ctx context.Context) error {
if len(reporter.pending) == 0 {
events, err := reporter.queue.Next(ctx)
if err != nil {
return err
}
if err := reporter.append(events); err != nil {
return err
}
}
for len(reporter.pending) < reporter.options.MaxBatchesPerReportRPC {
events, ok := reporter.queue.TryNext()
if !ok {
return nil
}
if err := reporter.append(events); err != nil {
return err
}
}
return nil
}
func (reporter *OutcomeReporter) append(events []domain.Event) error {
batch, _, err := domain.NormalizeBatch(domain.Batch{
WorkerID: reporter.options.WorkerID, SessionID: reporter.sessionID,
Sequence: reporter.sequence + uint64(len(reporter.pending)) + 1, Events: events,
}, reporter.queue.MaxBatch())
if err != nil {
return errors.Join(ErrInvalidOutcomeReporter, err)
}
reporter.pending = append(reporter.pending, batch)
return nil
}
func outcomeRequest(batch domain.Batch) (*controlplanev1.OutcomeBatch, error) {
proxies := make([]*controlplanev1.ProxyOutcome, len(batch.Events))
for index, event := range batch.Events {
stage, ok := outcomeStage(event.Stage)
if !ok {
return nil, fmt.Errorf("%w: unknown outcome stage", ErrInvalidOutcomeReporter)
}
proxies[index] = &controlplanev1.ProxyOutcome{
ProxyId: event.ProxyID, RoutingName: event.RoutingName, Stage: stage, Success: event.Success,
ErrorClass: string(event.ErrorClass), Latency: durationpb.New(event.Latency), ObservedAt: timestamppb.New(event.ObservedAt),
}
}
return &controlplanev1.OutcomeBatch{
WorkerId: batch.WorkerID, SessionId: batch.SessionID, Sequence: batch.Sequence, Outcomes: proxies,
}, nil
}
func outcomeStage(stage domain.Stage) (controlplanev1.OutcomeStage, bool) {
switch stage {
case domain.StageDial:
return controlplanev1.OutcomeStage_OUTCOME_STAGE_DIAL, true
case domain.StageProxyHandshake:
return controlplanev1.OutcomeStage_OUTCOME_STAGE_PROXY_HANDSHAKE, true
case domain.StageResponseHeaders:
return controlplanev1.OutcomeStage_OUTCOME_STAGE_RESPONSE_HEADERS, true
case domain.StageTunnel:
return controlplanev1.OutcomeStage_OUTCOME_STAGE_TUNNEL, true
default:
return controlplanev1.OutcomeStage_OUTCOME_STAGE_UNSPECIFIED, false
}
}
type generatedOutcomeRPCClient struct {
client controlplanev1.WorkerControlPlaneClient
}
func NewGeneratedOutcomeRPCClient(client controlplanev1.WorkerControlPlaneClient) OutcomeRPCClient {
return generatedOutcomeRPCClient{client: client}
}
func (client generatedOutcomeRPCClient) ReportOutcomes(ctx context.Context) (OutcomeStream, error) {
if client.client == nil {
return nil, ErrInvalidOutcomeReporter
}
return client.client.ReportOutcomes(ctx)
}

View File

@ -0,0 +1,175 @@
package controlplane
import (
"context"
"errors"
"sync"
"testing"
"time"
controlplanev1 "proxy-pool/gen/controlplane/v1"
domain "proxy-pool/internal/domain/outcome"
gatewayOutcome "proxy-pool/internal/gateway/outcome"
)
func TestOutcomeReporterRetriesExactPendingBatchAfterTransportFailure(t *testing.T) {
queue, err := gatewayOutcome.NewQueue(gatewayOutcome.QueueOptions{Capacity: 4, MaxBatch: 2})
if err != nil {
t.Fatalf("NewQueue() = %v", err)
}
queue.Record(domain.Event{ProxyID: "proxy-a", Stage: domain.StageDial, Success: true, ObservedAt: time.Now()})
client := &outcomeClientStub{closeErr: errors.New("connection dropped")}
reporter, err := NewOutcomeReporter(client, queue, OutcomeReporterOptions{WorkerID: "worker-a"})
if err != nil {
t.Fatalf("NewOutcomeReporter() = %v", err)
}
registration := Registration{SessionID: "session-a"}
reporter.sessionID = registration.SessionID
if err := reporter.Report(context.Background(), registration.SessionID); !errors.Is(err, client.closeErr) {
t.Fatalf("Report(first) = %v, want transport failure", err)
}
client.closeErr = nil
if err := reporter.Report(context.Background(), registration.SessionID); err != nil {
t.Fatalf("Report(retry) = %v", err)
}
if len(client.batches) != 2 || client.batches[0].GetSequence() != 1 || client.batches[1].GetSequence() != 1 ||
client.batches[0].GetOutcomes()[0].GetProxyId() != "proxy-a" {
t.Fatalf("sent batches = %+v", client.batches)
}
}
func TestOutcomeReporterCombinesAvailableBatchesAndRequiresConfirmation(t *testing.T) {
queue, err := gatewayOutcome.NewQueue(gatewayOutcome.QueueOptions{Capacity: 8, MaxBatch: 1})
if err != nil {
t.Fatalf("NewQueue() = %v", err)
}
for _, proxyID := range []string{"proxy-a", "proxy-b"} {
queue.Record(domain.Event{ProxyID: proxyID, Stage: domain.StageDial, Success: true, ObservedAt: time.Now()})
}
client := &outcomeClientStub{}
reporter, err := NewOutcomeReporter(client, queue, OutcomeReporterOptions{WorkerID: "worker-a", MaxBatchesPerReportRPC: 2})
if err != nil {
t.Fatalf("NewOutcomeReporter() = %v", err)
}
reporter.sessionID = "session-a"
if err := reporter.Report(context.Background(), "session-a"); err != nil {
t.Fatalf("Report() = %v", err)
}
if len(client.batches) != 2 || client.batches[0].GetSequence() != 1 || client.batches[1].GetSequence() != 2 {
t.Fatalf("sent batches = %+v", client.batches)
}
}
func TestOutcomeReporterRetriesTransientFailureWithinRegisteredSession(t *testing.T) {
queue, err := gatewayOutcome.NewQueue(gatewayOutcome.QueueOptions{Capacity: 4, MaxBatch: 2})
if err != nil {
t.Fatalf("NewQueue() = %v", err)
}
queue.Record(domain.Event{ProxyID: "proxy-a", Stage: domain.StageDial, Success: true, ObservedAt: time.Now()})
client := &retryingOutcomeClient{failures: 1, confirmed: make(chan struct{}, 1)}
reporter, err := NewOutcomeReporter(client, queue, OutcomeReporterOptions{WorkerID: "worker-a"})
if err != nil {
t.Fatalf("NewOutcomeReporter() = %v", err)
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
done := make(chan error, 1)
go func() { done <- reporter.RunRegistered(ctx, Registration{SessionID: "session-a"}) }()
select {
case <-client.confirmed:
case <-time.After(time.Second):
t.Fatal("outcome batch was not confirmed after transient failure")
}
if batches := client.Batches(); len(batches) != 2 || batches[0].GetSequence() != 1 ||
batches[1].GetSequence() != 1 || batches[0].GetOutcomes()[0].GetProxyId() != "proxy-a" {
t.Fatalf("sent batches = %+v, want exact retry in the same session", batches)
}
cancel()
if err := <-done; !errors.Is(err, context.Canceled) {
t.Fatalf("RunRegistered() = %v, want context canceled", err)
}
}
func TestOutcomeReporterRejectsUnboundedStreamOptions(t *testing.T) {
queue, err := gatewayOutcome.NewQueue(gatewayOutcome.QueueOptions{Capacity: 1, MaxBatch: 1})
if err != nil {
t.Fatalf("NewQueue() = %v", err)
}
if _, err := NewOutcomeReporter(&outcomeClientStub{}, queue, OutcomeReporterOptions{
WorkerID: "worker-a", MaxBatchesPerReportRPC: domain.MaxBatchesPerStream + 1,
}); !errors.Is(err, ErrInvalidOutcomeReporter) {
t.Fatalf("NewOutcomeReporter() error = %v, want ErrInvalidOutcomeReporter", err)
}
}
type outcomeClientStub struct {
batches []*controlplanev1.OutcomeBatch
closeErr error
accepted uint64
}
func (client *outcomeClientStub) ReportOutcomes(context.Context) (OutcomeStream, error) {
return &outcomeStreamStub{client: client}, nil
}
type outcomeStreamStub struct{ client *outcomeClientStub }
func (stream *outcomeStreamStub) Send(batch *controlplanev1.OutcomeBatch) error {
stream.client.batches = append(stream.client.batches, batch)
return nil
}
func (stream *outcomeStreamStub) CloseAndRecv() (*controlplanev1.ReportOutcomesResponse, error) {
if stream.client.closeErr != nil {
return nil, stream.client.closeErr
}
accepted := stream.client.accepted
if accepted == 0 && len(stream.client.batches) > 0 {
accepted = stream.client.batches[len(stream.client.batches)-1].GetSequence()
}
return &controlplanev1.ReportOutcomesResponse{AcceptedThroughSequence: accepted}, nil
}
type retryingOutcomeClient struct {
mu sync.Mutex
batches []*controlplanev1.OutcomeBatch
failures int
confirmed chan struct{}
}
func (client *retryingOutcomeClient) ReportOutcomes(context.Context) (OutcomeStream, error) {
return &retryingOutcomeStream{client: client}, nil
}
func (client *retryingOutcomeClient) Batches() []*controlplanev1.OutcomeBatch {
client.mu.Lock()
defer client.mu.Unlock()
return append([]*controlplanev1.OutcomeBatch(nil), client.batches...)
}
type retryingOutcomeStream struct{ client *retryingOutcomeClient }
func (stream *retryingOutcomeStream) Send(batch *controlplanev1.OutcomeBatch) error {
stream.client.mu.Lock()
stream.client.batches = append(stream.client.batches, batch)
stream.client.mu.Unlock()
return nil
}
func (stream *retryingOutcomeStream) CloseAndRecv() (*controlplanev1.ReportOutcomesResponse, error) {
stream.client.mu.Lock()
if stream.client.failures > 0 {
stream.client.failures--
stream.client.mu.Unlock()
return nil, errors.New("temporary delivery failure")
}
accepted := stream.client.batches[len(stream.client.batches)-1].GetSequence()
confirmed := stream.client.confirmed
stream.client.mu.Unlock()
select {
case confirmed <- struct{}{}:
default:
}
return &controlplanev1.ReportOutcomesResponse{AcceptedThroughSequence: accepted}, nil
}

View File

@ -10,13 +10,18 @@ var ErrSnapshotStreamClosed = errors.New("worker snapshot stream closed")
type SessionRunner struct { type SessionRunner struct {
reporter *RuntimeReporter reporter *RuntimeReporter
watcher *SnapshotWatcher watcher *SnapshotWatcher
outcomes *OutcomeReporter
} }
func NewSessionRunner(reporter *RuntimeReporter, watcher *SnapshotWatcher) (*SessionRunner, error) { func NewSessionRunner(reporter *RuntimeReporter, watcher *SnapshotWatcher, outcomes ...*OutcomeReporter) (*SessionRunner, error) {
if reporter == nil || watcher == nil { if reporter == nil || watcher == nil || len(outcomes) > 1 || (len(outcomes) == 1 && outcomes[0] == nil) {
return nil, ErrInvalidOptions return nil, ErrInvalidOptions
} }
return &SessionRunner{reporter: reporter, watcher: watcher}, nil runner := &SessionRunner{reporter: reporter, watcher: watcher}
if len(outcomes) == 1 {
runner.outcomes = outcomes[0]
}
return runner, nil
} }
// Run registers exactly once, then runs snapshot intake and Runtime reporting // Run registers exactly once, then runs snapshot intake and Runtime reporting
@ -32,21 +37,36 @@ func (runner *SessionRunner) Run(ctx context.Context) error {
} }
groupCtx, cancel := context.WithCancel(ctx) groupCtx, cancel := context.WithCancel(ctx)
defer cancel() defer cancel()
results := make(chan error, 2) runners := []func() error{
go func() { results <- runner.watcher.Watch(groupCtx, registration.SessionID) }() func() error { return runner.watcher.Watch(groupCtx, registration.SessionID) },
go func() { results <- runner.reporter.RunRegistered(groupCtx) }() func() error { return runner.reporter.RunRegistered(groupCtx) },
}
if runner.outcomes != nil {
runners = append(runners, func() error { return runner.outcomes.RunRegistered(groupCtx, registration) })
}
results := make(chan error, len(runners))
for _, run := range runners {
go func(run func() error) { results <- run() }(run)
}
first := <-results first := <-results
cancel() cancel()
second := <-results others := make([]error, 0, len(runners)-1)
for range len(runners) - 1 {
others = append(others, <-results)
}
if ctx.Err() != nil { if ctx.Err() != nil {
return ctx.Err() return ctx.Err()
} }
if first == nil { if first == nil {
return ErrSnapshotStreamClosed return ErrSnapshotStreamClosed
} }
if errors.Is(first, context.Canceled) && second != nil { if errors.Is(first, context.Canceled) {
return second for _, other := range others {
if other != nil && !errors.Is(other, context.Canceled) {
return other
}
}
} }
return first return first
} }

View File

@ -0,0 +1,99 @@
// Package outcome provides the Gateway's bounded, non-blocking observation
// buffer. It deliberately has no dependency on Controller or storage code.
package outcome
import (
"context"
"errors"
"sync/atomic"
domain "proxy-pool/internal/domain/outcome"
)
var ErrInvalidQueueOptions = errors.New("invalid gateway outcome queue options")
type QueueOptions struct {
Capacity int
MaxBatch int
}
// Queue accepts observations from the request path without blocking. Events
// beyond Capacity are intentionally dropped because outcome telemetry must not
// delay proxy forwarding or consume unbounded process memory.
type Queue struct {
events chan domain.Event
maxBatch int
dropped atomic.Uint64
}
func NewQueue(options QueueOptions) (*Queue, error) {
if options.Capacity <= 0 || options.MaxBatch <= 0 || options.MaxBatch > options.Capacity {
return nil, ErrInvalidQueueOptions
}
return &Queue{events: make(chan domain.Event, options.Capacity), maxBatch: options.MaxBatch}, nil
}
func (queue *Queue) Record(event domain.Event) {
if queue == nil {
return
}
select {
case queue.events <- event:
default:
queue.dropped.Add(1)
}
}
func (queue *Queue) Dropped() uint64 {
if queue == nil {
return 0
}
return queue.dropped.Load()
}
func (queue *Queue) MaxBatch() int {
if queue == nil {
return 0
}
return queue.maxBatch
}
// Next waits for one event then drains an immediately available bounded batch.
func (queue *Queue) Next(ctx context.Context) ([]domain.Event, error) {
if queue == nil || ctx == nil {
return nil, ErrInvalidQueueOptions
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case first := <-queue.events:
return queue.drain(first), nil
}
}
// TryNext drains one batch only when work is already buffered.
func (queue *Queue) TryNext() ([]domain.Event, bool) {
if queue == nil {
return nil, false
}
select {
case first := <-queue.events:
return queue.drain(first), true
default:
return nil, false
}
}
func (queue *Queue) drain(first domain.Event) []domain.Event {
batch := make([]domain.Event, 0, queue.maxBatch)
batch = append(batch, first)
for len(batch) < queue.maxBatch {
select {
case event := <-queue.events:
batch = append(batch, event)
default:
return batch
}
}
return batch
}

View File

@ -0,0 +1,41 @@
package outcome
import (
"context"
"errors"
"testing"
domain "proxy-pool/internal/domain/outcome"
)
func TestQueueBatchesWithoutBlockingAndCountsOverflow(t *testing.T) {
queue, err := NewQueue(QueueOptions{Capacity: 2, MaxBatch: 2})
if err != nil {
t.Fatalf("NewQueue() = %v", err)
}
queue.Record(domain.Event{ProxyID: "proxy-a"})
queue.Record(domain.Event{ProxyID: "proxy-b"})
queue.Record(domain.Event{ProxyID: "proxy-c"})
if queue.Dropped() != 1 {
t.Fatalf("Dropped() = %d, want 1", queue.Dropped())
}
batch, err := queue.Next(context.Background())
if err != nil || len(batch) != 2 || batch[0].ProxyID != "proxy-a" || batch[1].ProxyID != "proxy-b" {
t.Fatalf("Next() = %+v, %v", batch, err)
}
if batch, ok := queue.TryNext(); ok || batch != nil {
t.Fatalf("TryNext() = %+v, %t; want nil, false", batch, ok)
}
}
func TestQueueHonorsCanceledContext(t *testing.T) {
queue, err := NewQueue(QueueOptions{Capacity: 1, MaxBatch: 1})
if err != nil {
t.Fatalf("NewQueue() = %v", err)
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if _, err := queue.Next(ctx); !errors.Is(err, context.Canceled) {
t.Fatalf("Next(canceled) = %v, want context canceled", err)
}
}

View File

@ -14,6 +14,7 @@ import (
"sync/atomic" "sync/atomic"
"time" "time"
outcomeDomain "proxy-pool/internal/domain/outcome"
proxyDomain "proxy-pool/internal/domain/proxy" proxyDomain "proxy-pool/internal/domain/proxy"
"proxy-pool/internal/domain/routing" "proxy-pool/internal/domain/routing"
"proxy-pool/internal/gateway/dispatch" "proxy-pool/internal/gateway/dispatch"
@ -78,6 +79,12 @@ type DirectTransport interface {
OpenDirectTunnel(context.Context, string) (net.Conn, error) OpenDirectTunnel(context.Context, string) (net.Conn, error)
} }
// OutcomeRecorder accepts best-effort proxy observations. Implementations must
// return immediately; request forwarding never waits for telemetry delivery.
type OutcomeRecorder interface {
Record(outcomeDomain.Event)
}
type waitingDispatcher interface { type waitingDispatcher interface {
AcquireWait(context.Context, dispatch.Request, time.Duration) (*dispatch.Lease, error) AcquireWait(context.Context, dispatch.Request, time.Duration) (*dispatch.Lease, error)
} }
@ -90,6 +97,7 @@ type Dependencies struct {
Router Router Router Router
Dispatcher Dispatcher Dispatcher Dispatcher
Transport ProxyTransport Transport ProxyTransport
Outcomes OutcomeRecorder
} }
type Handler struct { type Handler struct {
@ -99,6 +107,7 @@ type Handler struct {
router Router router Router
dispatcher Dispatcher dispatcher Dispatcher
transport ProxyTransport transport ProxyTransport
outcomes OutcomeRecorder
buffers sync.Pool buffers sync.Pool
inFlight chan struct{} inFlight chan struct{}
forceClose atomic.Bool forceClose atomic.Bool
@ -136,6 +145,7 @@ func New(config Config, dependencies Dependencies) (*Handler, error) {
router: dependencies.Router, router: dependencies.Router,
dispatcher: dependencies.Dispatcher, dispatcher: dependencies.Dispatcher,
transport: dependencies.Transport, transport: dependencies.Transport,
outcomes: dependencies.Outcomes,
tunnels: make(map[*activeTunnel]struct{}), tunnels: make(map[*activeTunnel]struct{}),
shutdownDone: make(chan struct{}), shutdownDone: make(chan struct{}),
} }
@ -230,14 +240,16 @@ func (handler *Handler) connect(
lastErr = directErr lastErr = directErr
break break
} }
handler.serveTunnel(writer, request, nil, upstream) handler.serveTunnel(writer, request, nil, upstream, "", "")
return return
} }
break break
} }
started := time.Now().UTC()
upstream, err := handler.transport.OpenTunnel(request.Context(), lease.Proxy, target.DialAddress()) upstream, err := handler.transport.OpenTunnel(request.Context(), lease.Proxy, target.DialAddress())
if err != nil { if err != nil {
handler.recordOutcome(lease.Proxy.ID, route.RoutingName, outcomeDomain.StageProxyHandshake, false, err, started)
finishLease(lease, false) finishLease(lease, false)
excluded[lease.Proxy.ID] = struct{}{} excluded[lease.Proxy.ID] = struct{}{}
var responseError *transportDomain.ProxyResponseError var responseError *transportDomain.ProxyResponseError
@ -259,7 +271,8 @@ func (handler *Handler) connect(
lastErr = err lastErr = err
break break
} }
handler.serveTunnel(writer, request, func() { finishLease(lease, true) }, upstream) handler.recordOutcome(lease.Proxy.ID, route.RoutingName, outcomeDomain.StageProxyHandshake, true, nil, started)
handler.serveTunnel(writer, request, func() { finishLease(lease, true) }, upstream, lease.Proxy.ID, route.RoutingName)
return return
} }
@ -279,6 +292,8 @@ func (handler *Handler) serveTunnel(
request *http.Request, request *http.Request,
finish func(), finish func(),
upstream net.Conn, upstream net.Conn,
proxyID string,
routingName string,
) { ) {
if finish != nil { if finish != nil {
defer finish() defer finish()
@ -287,11 +302,13 @@ func (handler *Handler) serveTunnel(
hijacker, ok := writer.(http.Hijacker) hijacker, ok := writer.(http.Hijacker)
if !ok { if !ok {
handler.recordOutcome(proxyID, routingName, outcomeDomain.StageTunnel, false, errors.New("gateway response does not support connection hijacking"), time.Now().UTC())
http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
return return
} }
client, readWriter, err := hijacker.Hijack() client, readWriter, err := hijacker.Hijack()
if err != nil { if err != nil {
handler.recordOutcome(proxyID, routingName, outcomeDomain.StageTunnel, false, err, time.Now().UTC())
http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) http.Error(writer, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
return return
} }
@ -302,14 +319,18 @@ func (handler *Handler) serveTunnel(
} }
defer handler.unregisterTunnel(tunnel) defer handler.unregisterTunnel(tunnel)
started := time.Now().UTC()
if _, err := readWriter.WriteString("HTTP/1.1 200 Connection Established\r\n\r\n"); err != nil { if _, err := readWriter.WriteString("HTTP/1.1 200 Connection Established\r\n\r\n"); err != nil {
handler.recordOutcome(proxyID, routingName, outcomeDomain.StageTunnel, false, err, started)
return return
} }
if err := readWriter.Flush(); err != nil { if err := readWriter.Flush(); err != nil {
handler.recordOutcome(proxyID, routingName, outcomeDomain.StageTunnel, false, err, started)
return return
} }
bufferedClient := &bufferedClientConn{Conn: client, reader: readWriter.Reader} bufferedClient := &bufferedClientConn{Conn: client, reader: readWriter.Reader}
_ = handler.transport.Relay(request.Context(), bufferedClient, upstream) err = handler.transport.Relay(request.Context(), bufferedClient, upstream)
handler.recordOutcome(proxyID, routingName, outcomeDomain.StageTunnel, err == nil, err, started)
} }
func (handler *Handler) Shutdown(ctx context.Context) error { func (handler *Handler) Shutdown(ctx context.Context) error {
@ -444,8 +465,10 @@ func (handler *Handler) forwardHTTP(
committed.Store(true) committed.Store(true)
return nil return nil
} }
started := time.Now().UTC()
response, err := handler.transport.RoundTrip(request.Context(), lease.Proxy, attemptRequest, commit) response, err := handler.transport.RoundTrip(request.Context(), lease.Proxy, attemptRequest, commit)
if err != nil { if err != nil {
handler.recordOutcome(lease.Proxy.ID, route.RoutingName, outcomeDomain.StageDial, false, err, started)
finishLease(lease, committed.Load()) finishLease(lease, committed.Load())
excluded[lease.Proxy.ID] = struct{}{} excluded[lease.Proxy.ID] = struct{}{}
lastErr = err lastErr = err
@ -455,10 +478,17 @@ func (handler *Handler) forwardHTTP(
if err := commit(); err != nil { if err := commit(); err != nil {
_ = response.Body.Close() _ = response.Body.Close()
finishLease(lease, false) finishLease(lease, false)
handler.recordOutcome(lease.Proxy.ID, route.RoutingName, outcomeDomain.StageResponseHeaders, false, err, started)
lastErr = err lastErr = err
break break
} }
} }
if response.StatusCode == http.StatusProxyAuthRequired {
handler.recordOutcome(lease.Proxy.ID, route.RoutingName, outcomeDomain.StageResponseHeaders, false,
&transportDomain.ProxyResponseError{StatusCode: response.StatusCode}, started)
} else {
handler.recordOutcome(lease.Proxy.ID, route.RoutingName, outcomeDomain.StageResponseHeaders, true, nil, started)
}
handler.writeResponse(writer, response) handler.writeResponse(writer, response)
finishLease(lease, true) finishLease(lease, true)
return return
@ -467,6 +497,61 @@ func (handler *Handler) forwardHTTP(
writeGatewayError(writer, fmt.Errorf("forward gateway request: %w", lastErr)) writeGatewayError(writer, fmt.Errorf("forward gateway request: %w", lastErr))
} }
func (handler *Handler) recordOutcome(
proxyID, routingName string,
stage outcomeDomain.Stage,
success bool,
err error,
started time.Time,
) {
if handler == nil || handler.outcomes == nil || proxyID == "" {
return
}
observedAt := time.Now().UTC()
if started.IsZero() {
started = observedAt
}
latency := observedAt.Sub(started)
if latency < 0 {
latency = 0
}
event := outcomeDomain.Event{
ProxyID: proxyID, RoutingName: routingName, Stage: stage, Success: success,
Latency: latency, ObservedAt: observedAt,
}
if !success {
event.ErrorClass = classifyOutcomeError(stage, err)
}
handler.outcomes.Record(event)
}
func classifyOutcomeError(stage outcomeDomain.Stage, err error) outcomeDomain.ErrorClass {
switch {
case errors.Is(err, context.Canceled):
return outcomeDomain.ErrorClassCanceled
case errors.Is(err, context.DeadlineExceeded):
return outcomeDomain.ErrorClassTimeout
}
var responseError *transportDomain.ProxyResponseError
if errors.As(err, &responseError) {
return outcomeDomain.ErrorClassProxyResponse
}
var networkError net.Error
if errors.As(err, &networkError) && networkError.Timeout() {
return outcomeDomain.ErrorClassTimeout
}
switch stage {
case outcomeDomain.StageDial:
return outcomeDomain.ErrorClassDial
case outcomeDomain.StageProxyHandshake:
return outcomeDomain.ErrorClassHandshake
case outcomeDomain.StageTunnel:
return outcomeDomain.ErrorClassRelay
default:
return outcomeDomain.ErrorClassInternal
}
}
func (handler *Handler) acquireRoute(ctx context.Context, route dispatch.Request) (*dispatch.Lease, error) { func (handler *Handler) acquireRoute(ctx context.Context, route dispatch.Request) (*dispatch.Lease, error) {
lease, err := handler.dispatcher.Acquire(route) lease, err := handler.dispatcher.Acquire(route)
if !errors.Is(err, dispatch.ErrNoCandidate) || route.OnUnavailable != routing.OnUnavailableWait || route.WaitTimeout <= 0 { if !errors.Is(err, dispatch.ErrNoCandidate) || route.OnUnavailable != routing.OnUnavailableWait || route.WaitTimeout <= 0 {

View File

@ -15,6 +15,7 @@ import (
"testing" "testing"
"time" "time"
outcomeDomain "proxy-pool/internal/domain/outcome"
proxyDomain "proxy-pool/internal/domain/proxy" proxyDomain "proxy-pool/internal/domain/proxy"
"proxy-pool/internal/domain/routing" "proxy-pool/internal/domain/routing"
"proxy-pool/internal/gateway/dispatch" "proxy-pool/internal/gateway/dispatch"
@ -103,6 +104,50 @@ func TestHandlerRetriesGETWithAnotherProxyBeforeResponseCommit(t *testing.T) {
assertNoLeakedCapacity(t, view) assertNoLeakedCapacity(t, view)
} }
func TestHandlerRecordsOutcomeForEachProxyAttempt(t *testing.T) {
dispatcher, _ := dispatcherWithProxies(t, "proxy-a", "proxy-b")
transport := &fakeTransport{roundTrip: func(
_ context.Context,
selected proxyDomain.Proxy,
_ *http.Request,
commit ...func() error,
) (*http.Response, error) {
if selected.ID == "proxy-a" {
return nil, errors.New("dial failed")
}
if err := commit[0](); err != nil {
return nil, err
}
return &http.Response{StatusCode: http.StatusNoContent, Header: make(http.Header), Body: http.NoBody}, nil
}}
handler := newTestHandler(t, Config{MaxAttempts: 2, RetryMethods: []string{http.MethodGet}}, dispatcher, transport)
recorder := &outcomeRecorder{}
handler.outcomes = recorder
response := httptest.NewRecorder()
handler.ServeHTTP(response, httptest.NewRequest(http.MethodGet, "http://example.test/resource", nil))
events := recorder.Events()
if response.Code != http.StatusNoContent || len(events) != 2 {
t.Fatalf("status=%d events=%+v", response.Code, events)
}
if events[0].ProxyID != "proxy-a" || events[0].Stage != outcomeDomain.StageDial || events[0].Success ||
events[0].ErrorClass != outcomeDomain.ErrorClassDial || events[1].ProxyID != "proxy-b" ||
events[1].Stage != outcomeDomain.StageResponseHeaders || !events[1].Success {
t.Fatalf("outcomes = %+v", events)
}
}
func TestHandlerClampsNegativeOutcomeLatency(t *testing.T) {
recorder := &outcomeRecorder{}
handler := &Handler{outcomes: recorder}
handler.recordOutcome("proxy-a", "route-a", outcomeDomain.StageDial, true, nil, time.Now().Add(time.Second))
events := recorder.Events()
if len(events) != 1 || events[0].Latency != 0 {
t.Fatalf("outcomes = %+v, want zero latency", events)
}
}
func TestHandlerDoesNotRetryPOST(t *testing.T) { func TestHandlerDoesNotRetryPOST(t *testing.T) {
t.Parallel() t.Parallel()
@ -149,6 +194,8 @@ func TestHandlerReturns407WithoutRetry(t *testing.T) {
}, nil }, nil
}} }}
handler := newTestHandler(t, Config{MaxAttempts: 2, RetryMethods: []string{http.MethodGet}}, dispatcher, transport) handler := newTestHandler(t, Config{MaxAttempts: 2, RetryMethods: []string{http.MethodGet}}, dispatcher, transport)
recorder := &outcomeRecorder{}
handler.outcomes = recorder
response := httptest.NewRecorder() response := httptest.NewRecorder()
handler.ServeHTTP(response, httptest.NewRequest(http.MethodGet, "http://example.test/resource", nil)) handler.ServeHTTP(response, httptest.NewRequest(http.MethodGet, "http://example.test/resource", nil))
@ -159,6 +206,11 @@ func TestHandlerReturns407WithoutRetry(t *testing.T) {
if len(transport.attempts()) != 1 { if len(transport.attempts()) != 1 {
t.Fatalf("attempts = %v, want no retry for 407", transport.attempts()) t.Fatalf("attempts = %v, want no retry for 407", transport.attempts())
} }
events := recorder.Events()
if len(events) != 1 || events[0].Success || events[0].Stage != outcomeDomain.StageResponseHeaders ||
events[0].ErrorClass != outcomeDomain.ErrorClassProxyResponse {
t.Fatalf("outcomes = %+v, want failed proxy response", events)
}
assertNoLeakedCapacity(t, view) assertNoLeakedCapacity(t, view)
} }
@ -496,6 +548,23 @@ type fakeTransport struct {
relay func(context.Context, net.Conn, net.Conn) error relay func(context.Context, net.Conn, net.Conn) error
} }
type outcomeRecorder struct {
mu sync.Mutex
events []outcomeDomain.Event
}
func (recorder *outcomeRecorder) Record(event outcomeDomain.Event) {
recorder.mu.Lock()
recorder.events = append(recorder.events, event)
recorder.mu.Unlock()
}
func (recorder *outcomeRecorder) Events() []outcomeDomain.Event {
recorder.mu.Lock()
defer recorder.mu.Unlock()
return append([]outcomeDomain.Event(nil), recorder.events...)
}
type waitRecordingDispatcher struct{ waitTimeout time.Duration } type waitRecordingDispatcher struct{ waitTimeout time.Duration }
func (dispatcher *waitRecordingDispatcher) Acquire(dispatch.Request) (*dispatch.Lease, error) { func (dispatcher *waitRecordingDispatcher) Acquire(dispatch.Request) (*dispatch.Lease, error) {