feat: schedule egress health checks
This commit is contained in:
parent
1939225f11
commit
035cede836
15
README.md
15
README.md
@ -88,10 +88,10 @@ flowchart LR
|
|||||||
提取与限流、Controller 的 Admin/Distribution/Metrics 监听,以及 PostgreSQL
|
提取与限流、Controller 的 Admin/Distribution/Metrics 监听,以及 PostgreSQL
|
||||||
管理状态;WorkerControlPlane 的 Register、Snapshot ACK、Runtime 心跳接收和
|
管理状态;WorkerControlPlane 的 Register、Snapshot ACK、Runtime 心跳接收和
|
||||||
Redis 会话栅栏,以及 Gateway Outcome 上报的有界队列、序列确认与重试;
|
Redis 会话栅栏,以及 Gateway Outcome 上报的有界队列、序列确认与重试;
|
||||||
Controller 的 Redis 共享 BASIC 检查任务、按上游的有界调度、HTTP/HTTPS/SOCKS5
|
Controller 的 Redis 共享 BASIC/EGRESS 检查任务、按上游的有界轮转调度、HTTP/HTTPS/SOCKS5
|
||||||
Checker 探测和
|
Checker 探测和
|
||||||
Observation 状态归并。
|
Observation 状态归并。
|
||||||
- **部分完成**:EGRESS 与 TARGET 的任务编排和配置建模,Docker Compose/Kubernetes
|
- **部分完成**:EGRESS 的出口身份响应解析、TARGET 的任务编排和配置建模,Docker Compose/Kubernetes
|
||||||
运行时 mTLS Overlay。
|
运行时 mTLS Overlay。
|
||||||
- **待完成**:CONNECT 长连接/Extract 压测场景、故障演练和代表性集群压测。
|
- **待完成**:CONNECT 长连接/Extract 压测场景、故障演练和代表性集群压测。
|
||||||
|
|
||||||
@ -169,9 +169,10 @@ Checker 的参数也可通过 `PROXY_POOL_CONTROL_PLANE_ADDRESS`、
|
|||||||
`PROXY_POOL_CHECKER_ID`、`PROXY_POOL_CHECKER_INSTANCE_ID` 与
|
`PROXY_POOL_CHECKER_ID`、`PROXY_POOL_CHECKER_INSTANCE_ID` 与
|
||||||
`PROXY_POOL_CHECKER_MAX_IN_FLIGHT` 提供。它不会访问 Redis/PostgreSQL;生产
|
`PROXY_POOL_CHECKER_MAX_IN_FLIGHT` 提供。它不会访问 Redis/PostgreSQL;生产
|
||||||
Controller 在启用控制面时装配 Redis 共享任务 broker,并按启用的 Upstream 调度
|
Controller 在启用控制面时装配 Redis 共享任务 broker,并按启用的 Upstream 调度
|
||||||
HTTP/HTTPS/SOCKS5 BASIC 检查。调度监督器每轮读取已发布配置,因此 reload 后的上游启停、
|
HTTP/HTTPS/SOCKS5 BASIC 检查,并按每个 `check.urls` 创建 EGRESS 任务。调度监督器每轮读取
|
||||||
检查间隔、抖动、超时、重试次数和 `maxInFlight` 会在下一轮生效;新启用的上游无需
|
已发布配置,因此 reload 后的上游启停、检查间隔、抖动、超时、重试次数、`maxInFlight` 和
|
||||||
重启 Controller。EGRESS 与 TARGET 尚未进入生产调度。
|
EGRESS URL 都会在下一轮生效;BASIC 与 EGRESS 以有界轮转组共享上游并发上限。新启用的上游
|
||||||
|
无需重启 Controller。TARGET 尚未进入生产调度,EGRESS 的出口身份响应解析也尚未配置化。
|
||||||
|
|
||||||
初版 HTTP 容量工具可按固定请求数或固定时长运行,并将 HTTPS 目标经 Gateway 的请求
|
初版 HTTP 容量工具可按固定请求数或固定时长运行,并将 HTTPS 目标经 Gateway 的请求
|
||||||
交给标准 HTTP Transport 建立 CONNECT:
|
交给标准 HTTP Transport 建立 CONNECT:
|
||||||
@ -227,8 +228,8 @@ go run ./cmd/proxy-loadgen `
|
|||||||
|
|
||||||
- **P0 - Worker 控制面闭环**:Worker session、Snapshot ledger、ACK、运行态接收、
|
- **P0 - Worker 控制面闭环**:Worker session、Snapshot ledger、ACK、运行态接收、
|
||||||
ownership 索引,以及 Gateway 快照客户端。
|
ownership 索引,以及 Gateway 快照客户端。
|
||||||
- **P0 - Checker 健康链**:BASIC 的共享调度、实际探测、Observation reducer 和
|
- **P0 - Checker 健康链**:BASIC/EGRESS 的共享调度、实际探测、Observation reducer 和
|
||||||
`FETCHED -> AVAILABLE / SUSPECT / UNHEALTHY` 状态链已完成;继续补齐 EGRESS、
|
`FETCHED -> AVAILABLE / SUSPECT / UNHEALTHY` 状态链已完成;继续补齐 EGRESS 身份解析、
|
||||||
TARGET。
|
TARGET。
|
||||||
- **P1 - Gateway 与 Routing**:Gateway 进程、快照凭据分发、五种 Routing 策略与
|
- **P1 - Gateway 与 Routing**:Gateway 进程、快照凭据分发、五种 Routing 策略与
|
||||||
`onUnavailable` 已接入;动态容量调整和 Drain 闭环待完成。
|
`onUnavailable` 已接入;动态容量调整和 Drain 闭环待完成。
|
||||||
|
|||||||
@ -261,8 +261,8 @@ Controller 多副本共享同一计数。Client 身份只以 SHA-256 摘要进
|
|||||||
Gateway 请求热路径仍只使用本地准入,不增加 Redis/PostgreSQL 调用。
|
Gateway 请求热路径仍只使用本地准入,不增加 Redis/PostgreSQL 调用。
|
||||||
WorkerControlPlane gRPC 接收端、session 签发/心跳、Snapshot ACK 账本、基础
|
WorkerControlPlane gRPC 接收端、session 签发/心跳、Snapshot ACK 账本、基础
|
||||||
Snapshot 流、Gateway 会话客户端与快照凭据分发已完成;权威 Proxy/Routing 发布、
|
Snapshot 流、Gateway 会话客户端与快照凭据分发已完成;权威 Proxy/Routing 发布、
|
||||||
Outcome 上报已完成为有界队列、批次序列/摘要栅栏和确认重试;健康 BASIC 执行链已
|
Outcome 上报已完成为有界队列、批次序列/摘要栅栏和确认重试;健康 BASIC/EGRESS 执行链已
|
||||||
完成,EGRESS/TARGET 编排和 REMOVE 生命周期仍待完成,因此 Task 10 尚未全部完成。
|
完成,EGRESS 身份解析、TARGET 编排和 REMOVE 生命周期仍待完成,因此 Task 10 尚未全部完成。
|
||||||
|
|
||||||
## Task 11: Checker and Health Reducer
|
## Task 11: Checker and Health Reducer
|
||||||
|
|
||||||
@ -284,14 +284,16 @@ Checker Observation 上报 RPC 已复用既有控制面监听接入 Controller
|
|||||||
领取者加不可预测 lease token 的栅栏和完成后重放;任务凭据仅由认证流在执行期下发。
|
领取者加不可预测 lease token 的栅栏和完成后重放;任务凭据仅由认证流在执行期下发。
|
||||||
Redis 共享 due-index/租约持久化、每 Upstream 的跨副本 in-flight 限制和生产 broker 已完成;
|
Redis 共享 due-index/租约持久化、每 Upstream 的跨副本 in-flight 限制和生产 broker 已完成;
|
||||||
`proxy-checker` 独立进程、固定大小 worker-pool、任务期重试/微批上报和 HTTP/HTTPS/SOCKS5
|
`proxy-checker` 独立进程、固定大小 worker-pool、任务期重试/微批上报和 HTTP/HTTPS/SOCKS5
|
||||||
BASIC 探测器已完成并有测试。TARGET 探测器具备任务执行能力,但尚无生产任务调度;
|
BASIC/EGRESS 探测器已完成并有测试。Redis 已按固定 EGRESS due-index 保存任务执行 URL,
|
||||||
EGRESS 已具备任务 URL 传输、HTTP/HTTPS 探测和全局事实回传契约,但出口身份响应解析、
|
与 BASIC 独立引用并通过上游共享并发限制;配置化监督器以有界轮转组调度每个 `check.urls`。
|
||||||
EGRESS/TARGET 多维任务索引及部署运行态仍未实现,
|
TARGET 探测器具备任务执行能力,但尚无生产任务调度;EGRESS 的出口身份响应解析与部署运行态
|
||||||
|
仍未实现,
|
||||||
因此本任务保持未完成。
|
因此本任务保持未完成。
|
||||||
|
|
||||||
补充进度(2026-08-02):BASIC 调度已改为配置驱动监督器。它每轮读取已发布快照并复用
|
补充进度(2026-08-02):BASIC 调度已改为配置驱动监督器。它每轮读取已发布快照并复用
|
||||||
有界派发逻辑,所以 reload 后已启用上游的策略变更、停用,以及新启用上游都无需重启
|
有界派发逻辑,所以 reload 后已启用上游的策略变更、停用,以及新启用上游都无需重启
|
||||||
Controller 即可生效;Redis 任务存储仍仅承载 BASIC,未扩展 EGRESS/TARGET 的多维索引。
|
Controller 即可生效。Redis 任务存储现已扩展 BASIC/EGRESS 的独立有界索引;TARGET 的多维
|
||||||
|
索引仍待实现。
|
||||||
|
|
||||||
补充进度(2026-08-02):已新增 `proxy-loadgen` HTTP 场景。固定请求数和固定时长两种
|
补充进度(2026-08-02):已新增 `proxy-loadgen` HTTP 场景。固定请求数和固定时长两种
|
||||||
模式均通过固定 worker 数与有界派发通道执行,可选 QPS 限速;报告使用固定大小延迟直方图,
|
模式均通过固定 worker 数与有界派发通道执行,可选 QPS 限速;报告使用固定大小延迟直方图,
|
||||||
|
|||||||
@ -23,9 +23,9 @@
|
|||||||
|
|
||||||
`cmd/proxy-controller` 已完成配置单次加载、PostgreSQL 迁移、Redis 活动池、
|
`cmd/proxy-controller` 已完成配置单次加载、PostgreSQL 迁移、Redis 活动池、
|
||||||
Distribution/Admin/Metrics 独立监听和有界停机装配。Provider 自动补池、分布式
|
Distribution/Admin/Metrics 独立监听和有界停机装配。Provider 自动补池、分布式
|
||||||
配额、动态重载和 Admin 低基数统计已装配;Controller 已装配 Redis BASIC 任务 broker,
|
配额、动态重载和 Admin 低基数统计已装配;Controller 已装配 Redis BASIC/EGRESS 任务 broker,
|
||||||
`proxy-checker` 可执行 HTTP/HTTPS/SOCKS5 BASIC 探测。`proxy-loadgen` 已提供有界 HTTP
|
`proxy-checker` 可执行 HTTP/HTTPS/SOCKS5 BASIC/EGRESS 探测。`proxy-loadgen` 已提供有界 HTTP
|
||||||
请求场景;EGRESS/TARGET 调度、CONNECT 长连接/Extract 压测与完整 mTLS 环境 Overlay 仍属于
|
请求场景;EGRESS 身份解析、TARGET 调度、CONNECT 长连接/Extract 压测与完整 mTLS 环境 Overlay 仍属于
|
||||||
`implementation-plan.md` 后续任务。
|
`implementation-plan.md` 后续任务。
|
||||||
因此 Compose/Kubernetes 资产当前仍用于评审网络、资源、探针和依赖关系,不能
|
因此 Compose/Kubernetes 资产当前仍用于评审网络、资源、探针和依赖关系,不能
|
||||||
视为完整可运行拓扑。
|
视为完整可运行拓扑。
|
||||||
|
|||||||
@ -62,8 +62,8 @@ Outcome 已实现为 Gateway 本地有界队列、微批确认重试和 Controll
|
|||||||
记录归并,不改写 Proxy 全局状态。Controller 公用 Reducer 已作为 Observation 的唯一状态
|
记录归并,不改写 Proxy 全局状态。Controller 公用 Reducer 已作为 Observation 的唯一状态
|
||||||
归并边界,Checker Observation RPC 已在同一控制面监听以独立 SPIFFE 身份接入;有界任务领取、
|
归并边界,Checker Observation RPC 已在同一控制面监听以独立 SPIFFE 身份接入;有界任务领取、
|
||||||
租约归属与任务期凭据传输已由通用契约和 gRPC 往返测试覆盖。Redis 共享 due-index、
|
租约归属与任务期凭据传输已由通用契约和 gRPC 往返测试覆盖。Redis 共享 due-index、
|
||||||
任务/租约持久化、按上游的 in-flight 限制以及独立 Checker 的 HTTP/HTTPS/SOCKS5 BASIC 执行
|
任务/租约持久化、按上游的 in-flight 限制以及独立 Checker 的 HTTP/HTTPS/SOCKS5 BASIC/EGRESS 执行
|
||||||
进程已经闭环;EGRESS、TARGET 的多维任务索引与生产调度尚未实现。Snapshot 签发在 Redis 中原子匹配当前
|
进程已经闭环;EGRESS 已具备多维任务索引与生产调度,出口身份响应解析和 TARGET 的多维索引仍未实现。Snapshot 签发在 Redis 中原子匹配当前
|
||||||
`session_id`,重注册会清除旧引用,迟到旧 Stream 不会覆盖新 session。Controller
|
`session_id`,重注册会清除旧引用,迟到旧 Stream 不会覆盖新 session。Controller
|
||||||
在最近成功下发的 Snapshot `valid_until` 到达时关闭流;Gateway 的公用
|
在最近成功下发的 Snapshot `valid_until` 到达时关闭流;Gateway 的公用
|
||||||
`SessionSupervisor` 已实现可恢复错误的有界退避重连。Gateway 会校验并执行 Snapshot
|
`SessionSupervisor` 已实现可恢复错误的有界退避重连。Gateway 会校验并执行 Snapshot
|
||||||
|
|||||||
@ -7,7 +7,7 @@
|
|||||||
|
|
||||||
| ID | 最终需求 | 来源 | 验证证据 |
|
| ID | 最终需求 | 来源 | 验证证据 |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| ARCH-001 | 数据面 Worker 与控制面 Controller 分离 | 1-70 | 包、协议和部署拓扑已分离;Controller 已运行 Worker Register/Watch/ACK/Runtime/Outcome 与 Checker Observation gRPC,并发布 Proxy/Gateway Routing/按引用去重凭据完整快照;Checker 任务流已具备有界领取、租约栅栏和任务期凭据契约。Gateway 已将快照编译为同版本动态 View,并由独立进程维护控制面会话。`proxy-checker` 与 Redis BASIC 共享任务运行态、`proxy-loadgen` 有界 HTTP 场景均已实现;EGRESS/TARGET 生产编排待实现 |
|
| ARCH-001 | 数据面 Worker 与控制面 Controller 分离 | 1-70 | 包、协议和部署拓扑已分离;Controller 已运行 Worker Register/Watch/ACK/Runtime/Outcome 与 Checker Observation gRPC,并发布 Proxy/Gateway Routing/按引用去重凭据完整快照;Checker 任务流已具备有界领取、租约栅栏和任务期凭据契约。Gateway 已将快照编译为同版本动态 View,并由独立进程维护控制面会话。`proxy-checker` 与 Redis BASIC/EGRESS 共享任务运行态、`proxy-loadgen` 有界 HTTP 场景均已实现;TARGET 生产编排待实现 |
|
||||||
| ARCH-002 | 热路径只做认证、本地路由和网络转发 | 1-70, 380-430 | Gateway bootstrap 集成测试验证启动期控制面会话与快照就绪,HTTP 请求只走本地 Snapshot/Dispatch;Outcome 仅写入有界非阻塞本地队列,代表性性能剖析待完成 |
|
| ARCH-002 | 热路径只做认证、本地路由和网络转发 | 1-70, 380-430 | Gateway bootstrap 集成测试验证启动期控制面会话与快照就绪,HTTP 请求只走本地 Snapshot/Dispatch;Outcome 仅写入有界非阻塞本地队列,代表性性能剖析待完成 |
|
||||||
| ARCH-003 | Gateway、Distribution、Admin、Metrics 独立入口 | 8904-8958 | Controller 命令已装配 Distribution/Admin/Metrics 三个独立监听及联动停机;Gateway 命令已装配代理与 Metrics 监听,运行时 mTLS 部署 Overlay 待完成 |
|
| ARCH-003 | Gateway、Distribution、Admin、Metrics 独立入口 | 8904-8958 | Controller 命令已装配 Distribution/Admin/Metrics 三个独立监听及联动停机;Gateway 命令已装配代理与 Metrics 监听,运行时 mTLS 部署 Overlay 待完成 |
|
||||||
| ARCH-004 | Controller 集中 Provider 获取与切换 | 1403-1580 | Redis Leader、动态 Provider Supervisor 与 Bootstrap 生产装配已完成;Admin disable/reload 驱动取消替换,多副本按权威 HMAC 指纹和 revision 栅栏收敛并拒绝旧配置换主;Routing 切换到 Drain 的编排待完成 |
|
| ARCH-004 | Controller 集中 Provider 获取与切换 | 1403-1580 | Redis Leader、动态 Provider Supervisor 与 Bootstrap 生产装配已完成;Admin disable/reload 驱动取消替换,多副本按权威 HMAC 指纹和 revision 栅栏收敛并拒绝旧配置换主;Routing 切换到 Drain 的编排待完成 |
|
||||||
@ -80,7 +80,7 @@
|
|||||||
| ID | 最终需求 | 来源 | 验证证据 |
|
| ID | 最终需求 | 来源 | 验证证据 |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| HEALTH-001 | 全局健康与 Routing/目标健康分离 | 221-270, 8679-8708 | `domain/health` 已将 BASIC/EGRESS 全局 Reducer 与 TARGET Profile Reducer 分离;TARGET 在 Memory 和 Redis 独立、随代理 TTL 归并,不改写 Proxy 全局状态;Routing 消费待实现 |
|
| HEALTH-001 | 全局健康与 Routing/目标健康分离 | 221-270, 8679-8708 | `domain/health` 已将 BASIC/EGRESS 全局 Reducer 与 TARGET Profile Reducer 分离;TARGET 在 Memory 和 Redis 独立、随代理 TTL 归并,不改写 Proxy 全局状态;Routing 消费待实现 |
|
||||||
| HEALTH-002 | 健康调度有 jitter、maxInFlight 和分级频率 | 8679-8736 | 配置有效合并、URL 校验、稳定抖动/优先级 Planner、有界 Scheduler tick、Redis due-index、跨副本 in-flight 原子限制,以及 HTTP/HTTPS/SOCKS5 BASIC 生产执行器已完成;EGRESS/TARGET 多维任务索引与调度待实现 |
|
| HEALTH-002 | 健康调度有 jitter、maxInFlight 和分级频率 | 8679-8736 | 配置有效合并、URL 校验、稳定抖动/优先级 Planner、有界 Scheduler tick、Redis BASIC/EGRESS due-index、跨副本 in-flight 原子限制,以及 HTTP/HTTPS/SOCKS5 BASIC/EGRESS 生产执行器已完成;TARGET 多维任务索引与调度待实现 |
|
||||||
| HEALTH-003 | 失败分级 SUSPECT -> UNHEALTHY -> REMOVE | 8679-8736 | Controller 公用 Reducer 已通过 Memory/Redis 活动池原子提交全局连续失败、精确重放和成功恢复;BASIC 任务调度已完成,REMOVE 编排待实现 |
|
| HEALTH-003 | 失败分级 SUSPECT -> UNHEALTHY -> REMOVE | 8679-8736 | Controller 公用 Reducer 已通过 Memory/Redis 活动池原子提交全局连续失败、精确重放和成功恢复;BASIC 任务调度已完成,REMOVE 编排待实现 |
|
||||||
| SEC-001 | API 认证与 Proxy 认证分离,Secret 统一脱敏 | 7528-8111, 8904-8945 | Config 脱敏、Provider Store -> SecretRef -> Gateway Resolver 跨包测试与格式化泄漏回归测试 |
|
| SEC-001 | API 认证与 Proxy 认证分离,Secret 统一脱敏 | 7528-8111, 8904-8945 | Config 脱敏、Provider Store -> SecretRef -> Gateway Resolver 跨包测试与格式化泄漏回归测试 |
|
||||||
| SEC-002 | 非回环监听无保护时严格模式启动失败 | 8112-8441 | 配置校验测试 |
|
| SEC-002 | 非回环监听无保护时严格模式启动失败 | 8112-8441 | 配置校验测试 |
|
||||||
|
|||||||
@ -21,6 +21,8 @@ var _ healthDomain.TaskBroker = (*Adapter)(nil)
|
|||||||
type healthTaskRecord struct {
|
type healthTaskRecord struct {
|
||||||
Version int `json:"version"`
|
Version int `json:"version"`
|
||||||
TaskID string `json:"taskId"`
|
TaskID string `json:"taskId"`
|
||||||
|
Reference string `json:"reference"`
|
||||||
|
DueMember string `json:"dueMember"`
|
||||||
ProxyID string `json:"proxyId"`
|
ProxyID string `json:"proxyId"`
|
||||||
UpstreamID string `json:"upstreamId"`
|
UpstreamID string `json:"upstreamId"`
|
||||||
UpstreamTasksKey string `json:"upstreamTasksKey"`
|
UpstreamTasksKey string `json:"upstreamTasksKey"`
|
||||||
@ -49,6 +51,8 @@ type healthTaskRequest struct {
|
|||||||
LeaseTTLMS int64 `json:"leaseTTLMS,omitempty"`
|
LeaseTTLMS int64 `json:"leaseTTLMS,omitempty"`
|
||||||
Levels []healthDomain.Level `json:"levels,omitempty"`
|
Levels []healthDomain.Level `json:"levels,omitempty"`
|
||||||
UpstreamID string `json:"upstreamId,omitempty"`
|
UpstreamID string `json:"upstreamId,omitempty"`
|
||||||
|
Level string `json:"level,omitempty"`
|
||||||
|
TargetURL string `json:"targetUrl,omitempty"`
|
||||||
UpstreamTasksKey string `json:"upstreamTasksKey,omitempty"`
|
UpstreamTasksKey string `json:"upstreamTasksKey,omitempty"`
|
||||||
ScanLimit int `json:"scanLimit,omitempty"`
|
ScanLimit int `json:"scanLimit,omitempty"`
|
||||||
Tokens []string `json:"tokens,omitempty"`
|
Tokens []string `json:"tokens,omitempty"`
|
||||||
@ -84,22 +88,39 @@ func (a *Adapter) InFlight(ctx context.Context, now time.Time) (int, error) {
|
|||||||
// DueCandidates returns a bounded BASIC due batch. Stale references are
|
// DueCandidates returns a bounded BASIC due batch. Stale references are
|
||||||
// discarded inside the Lua script before they can reach the Controller.
|
// discarded inside the Lua script before they can reach the Controller.
|
||||||
func (a *Adapter) DueCandidates(ctx context.Context, now time.Time, limit int) ([]healthDomain.Candidate, error) {
|
func (a *Adapter) DueCandidates(ctx context.Context, now time.Time, limit int) ([]healthDomain.Candidate, error) {
|
||||||
return a.dueCandidates(ctx, "", now, limit)
|
return a.dueCandidates(ctx, "", healthDomain.LevelBasic, "", now, limit)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (a *Adapter) DueCandidatesForUpstream(ctx context.Context, upstreamID string, now time.Time, limit int) ([]healthDomain.Candidate, error) {
|
func (a *Adapter) DueCandidatesForUpstream(ctx context.Context, upstreamID string, now time.Time, limit int) ([]healthDomain.Candidate, error) {
|
||||||
if !validHealthTaskIdentifier(upstreamID) {
|
if !validHealthTaskIdentifier(upstreamID) {
|
||||||
return nil, healthDomain.ErrInvalidTaskBroker
|
return nil, healthDomain.ErrInvalidTaskBroker
|
||||||
}
|
}
|
||||||
return a.dueCandidates(ctx, upstreamID, now, limit)
|
return a.dueCandidates(ctx, upstreamID, healthDomain.LevelBasic, "", now, limit)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (a *Adapter) dueCandidates(ctx context.Context, upstreamID string, now time.Time, limit int) ([]healthDomain.Candidate, error) {
|
// DueEgressCandidatesForUpstream lazily initializes bounded EGRESS references
|
||||||
|
// from live BASIC references. The independent reference lets BASIC and EGRESS
|
||||||
|
// checks for one proxy be leased at the same time.
|
||||||
|
func (a *Adapter) DueEgressCandidatesForUpstream(ctx context.Context, upstreamID, targetURL string, now time.Time, limit int) ([]healthDomain.Candidate, error) {
|
||||||
|
if !validHealthTaskIdentifier(upstreamID) {
|
||||||
|
return nil, healthDomain.ErrInvalidTaskBroker
|
||||||
|
}
|
||||||
|
normalizedTarget, err := healthDomain.NormalizeEgressTarget(targetURL)
|
||||||
|
if err != nil || normalizedTarget != targetURL {
|
||||||
|
return nil, healthDomain.ErrInvalidTaskBroker
|
||||||
|
}
|
||||||
|
return a.dueCandidates(ctx, upstreamID, healthDomain.LevelEgress, targetURL, now, limit)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *Adapter) dueCandidates(ctx context.Context, upstreamID string, level healthDomain.Level, targetURL string, now time.Time, limit int) ([]healthDomain.Candidate, error) {
|
||||||
if a == nil || ctx == nil || now.IsZero() || limit <= 0 || limit > a.options.MaxCheckTasks {
|
if a == nil || ctx == nil || now.IsZero() || limit <= 0 || limit > a.options.MaxCheckTasks {
|
||||||
return nil, healthDomain.ErrInvalidTaskBroker
|
return nil, healthDomain.ErrInvalidTaskBroker
|
||||||
}
|
}
|
||||||
|
if level != healthDomain.LevelBasic && level != healthDomain.LevelEgress {
|
||||||
|
return nil, healthDomain.ErrInvalidTaskBroker
|
||||||
|
}
|
||||||
reply, err := a.runHealthTaskScript(ctx, "due", now, healthTaskRequest{
|
reply, err := a.runHealthTaskScript(ctx, "due", now, healthTaskRequest{
|
||||||
Limit: limit, ScanLimit: a.options.MaxCandidateScan, UpstreamID: upstreamID,
|
Limit: limit, ScanLimit: a.options.MaxCandidateScan, UpstreamID: upstreamID, Level: string(level), TargetURL: targetURL,
|
||||||
}, "runtime")
|
}, "runtime")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@ -119,11 +140,18 @@ func (a *Adapter) dueCandidates(ctx context.Context, upstreamID string, now time
|
|||||||
default:
|
default:
|
||||||
return nil, invalidScriptReply("health task due reply contains invalid state")
|
return nil, invalidScriptReply("health task due reply contains invalid state")
|
||||||
}
|
}
|
||||||
if !validHealthTaskIdentifier(candidate.ProxyID) || !validHealthTaskIdentifier(candidate.UpstreamID) || candidate.DueAtMS <= 0 {
|
if !validHealthTaskIdentifier(candidate.ProxyID) || !validHealthTaskIdentifier(candidate.UpstreamID) || candidate.DueAtMS <= 0 ||
|
||||||
|
healthDomain.Level(candidate.Level) != level {
|
||||||
return nil, invalidScriptReply("health task due reply contains invalid candidate")
|
return nil, invalidScriptReply("health task due reply contains invalid candidate")
|
||||||
}
|
}
|
||||||
|
if level == healthDomain.LevelEgress {
|
||||||
|
normalizedTarget, err := healthDomain.NormalizeEgressTarget(candidate.TargetURL)
|
||||||
|
if err != nil || normalizedTarget != targetURL {
|
||||||
|
return nil, invalidScriptReply("health task due reply contains invalid EGRESS target")
|
||||||
|
}
|
||||||
|
}
|
||||||
result = append(result, healthDomain.Candidate{
|
result = append(result, healthDomain.Candidate{
|
||||||
ProxyID: candidate.ProxyID, UpstreamID: candidate.UpstreamID, State: state, Level: healthDomain.LevelBasic,
|
ProxyID: candidate.ProxyID, UpstreamID: candidate.UpstreamID, State: state, Level: level, TargetURL: candidate.TargetURL,
|
||||||
DueAt: time.UnixMilli(candidate.DueAtMS).UTC(),
|
DueAt: time.UnixMilli(candidate.DueAtMS).UTC(),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@ -168,10 +196,12 @@ func (a *Adapter) Offer(ctx context.Context, plans []healthDomain.PlannedTask) (
|
|||||||
return 0, err
|
return 0, err
|
||||||
}
|
}
|
||||||
tasks[index] = healthTaskRecord{
|
tasks[index] = healthTaskRecord{
|
||||||
Version: healthTaskRecordVersion, TaskID: healthDomain.TaskIDFor(plan), ProxyID: plan.Candidate.ProxyID,
|
Version: healthTaskRecordVersion, TaskID: healthDomain.TaskIDFor(plan), Reference: redisHealthTaskReference(plan.Candidate),
|
||||||
|
DueMember: redisHealthDueMember(plan.Candidate), ProxyID: plan.Candidate.ProxyID,
|
||||||
UpstreamID: plan.Candidate.UpstreamID, UpstreamTasksKey: a.keys.upstreamTasks(plan.Candidate.UpstreamID),
|
UpstreamID: plan.Candidate.UpstreamID, UpstreamTasksKey: a.keys.upstreamTasks(plan.Candidate.UpstreamID),
|
||||||
Level: string(plan.Candidate.Level), Priority: int(plan.Priority), DeadlineMS: plan.Deadline.UnixMilli(),
|
Level: string(plan.Candidate.Level), Priority: int(plan.Priority), DeadlineMS: plan.Deadline.UnixMilli(),
|
||||||
NextDueMS: plan.NextDue.UnixMilli(), Attempts: plan.Attempts, MaxInFlight: plan.MaxInFlight, State: "QUEUED",
|
NextDueMS: plan.NextDue.UnixMilli(), Attempts: plan.Attempts, MaxInFlight: plan.MaxInFlight, State: "QUEUED",
|
||||||
|
RoutingName: plan.Candidate.RoutingName, TargetURL: plan.Candidate.TargetURL,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
reply, err := a.runHealthTaskScript(ctx, "offer", now, healthTaskRequest{Limit: a.options.MaxCheckTasks, Tasks: tasks}, "runtime")
|
reply, err := a.runHealthTaskScript(ctx, "offer", now, healthTaskRequest{Limit: a.options.MaxCheckTasks, Tasks: tasks}, "runtime")
|
||||||
@ -227,13 +257,14 @@ func (a *Adapter) Claim(ctx context.Context, claim healthDomain.TaskClaim) ([]he
|
|||||||
return nil, invalidScriptReply("health task claim reply contains invalid record")
|
return nil, invalidScriptReply("health task claim reply contains invalid record")
|
||||||
}
|
}
|
||||||
if task.State != "LEASED" || task.LeaseCheckerID != claim.CheckerID || task.LeaseToken == "" ||
|
if task.State != "LEASED" || task.LeaseCheckerID != claim.CheckerID || task.LeaseToken == "" ||
|
||||||
task.Level != string(healthDomain.LevelBasic) || task.ProxyID != record.ID || task.DeadlineMS <= now.UnixMilli() {
|
task.ProxyID != record.ID || task.DeadlineMS <= now.UnixMilli() || !validRedisTaskExecution(task) {
|
||||||
return nil, invalidScriptReply("health task claim reply violates lease contract")
|
return nil, invalidScriptReply("health task claim reply violates lease contract")
|
||||||
}
|
}
|
||||||
result = append(result, healthDomain.LeasedTask{
|
result = append(result, healthDomain.LeasedTask{
|
||||||
TaskID: task.TaskID, LeaseToken: task.LeaseToken, ProxyID: task.ProxyID,
|
TaskID: task.TaskID, LeaseToken: task.LeaseToken, ProxyID: task.ProxyID,
|
||||||
Protocol: proxyDomain.Scheme(record.Scheme), Host: record.Host, Port: uint16(record.Port),
|
Protocol: proxyDomain.Scheme(record.Scheme), Host: record.Host, Port: uint16(record.Port),
|
||||||
Username: record.Username, Password: record.Password, Level: healthDomain.Level(task.Level),
|
Username: record.Username, Password: record.Password, Level: healthDomain.Level(task.Level),
|
||||||
|
RoutingName: task.RoutingName, TargetURL: task.TargetURL,
|
||||||
Deadline: time.UnixMilli(task.DeadlineMS).UTC(), Attempts: task.Attempts,
|
Deadline: time.UnixMilli(task.DeadlineMS).UTC(), Attempts: task.Attempts,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@ -321,7 +352,7 @@ func (a *Adapter) runHealthTaskScript(
|
|||||||
result, err := runScript(ctx, a.client, healthTasksScript, []string{
|
result, err := runScript(ctx, a.client, healthTasksScript, []string{
|
||||||
a.keys.records, a.keys.expiry, a.keys.healthDue, a.keys.healthQueued, a.keys.healthLeases,
|
a.keys.records, a.keys.expiry, a.keys.healthDue, a.keys.healthQueued, a.keys.healthLeases,
|
||||||
a.keys.healthTasks, a.keys.healthTaskExpiry, a.keys.healthRefTask, a.keys.checkerLeases(checkerID),
|
a.keys.healthTasks, a.keys.healthTaskExpiry, a.keys.healthRefTask, a.keys.checkerLeases(checkerID),
|
||||||
a.keys.stateInventory,
|
a.keys.stateInventory, a.keys.healthEgressDue,
|
||||||
}, operation, now.UTC().UnixMilli(), string(payload))
|
}, operation, now.UTC().UnixMilli(), string(payload))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return healthTaskScriptReply{}, err
|
return healthTaskScriptReply{}, err
|
||||||
@ -334,12 +365,24 @@ func (a *Adapter) runHealthTaskScript(
|
|||||||
}
|
}
|
||||||
|
|
||||||
func validateRedisPlannedTask(task healthDomain.PlannedTask, now time.Time) error {
|
func validateRedisPlannedTask(task healthDomain.PlannedTask, now time.Time) error {
|
||||||
if !validHealthTaskIdentifier(task.Candidate.ProxyID) || !validHealthTaskIdentifier(task.Candidate.UpstreamID) || task.Candidate.Level != healthDomain.LevelBasic ||
|
if !validHealthTaskIdentifier(task.Candidate.ProxyID) || !validHealthTaskIdentifier(task.Candidate.UpstreamID) || task.Candidate.DueAt.IsZero() ||
|
||||||
task.Candidate.RoutingName != "" || task.Candidate.TargetURL != "" || task.Candidate.DueAt.IsZero() ||
|
|
||||||
task.Deadline.IsZero() || !task.Deadline.After(now) || task.NextDue.IsZero() || !task.NextDue.After(now) ||
|
task.Deadline.IsZero() || !task.Deadline.After(now) || task.NextDue.IsZero() || !task.NextDue.After(now) ||
|
||||||
task.Attempts <= 0 || task.MaxInFlight <= 0 {
|
task.Attempts <= 0 || task.MaxInFlight <= 0 {
|
||||||
return healthDomain.ErrInvalidLeasedTask
|
return healthDomain.ErrInvalidLeasedTask
|
||||||
}
|
}
|
||||||
|
switch task.Candidate.Level {
|
||||||
|
case healthDomain.LevelBasic:
|
||||||
|
if task.Candidate.RoutingName != "" || task.Candidate.TargetURL != "" {
|
||||||
|
return healthDomain.ErrInvalidLeasedTask
|
||||||
|
}
|
||||||
|
case healthDomain.LevelEgress:
|
||||||
|
normalizedTarget, err := healthDomain.NormalizeEgressTarget(task.Candidate.TargetURL)
|
||||||
|
if task.Candidate.RoutingName != "" || err != nil || normalizedTarget != task.Candidate.TargetURL {
|
||||||
|
return healthDomain.ErrInvalidLeasedTask
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
return healthDomain.ErrInvalidLeasedTask
|
||||||
|
}
|
||||||
switch task.Candidate.State {
|
switch task.Candidate.State {
|
||||||
case proxyDomain.StateFetched, proxyDomain.StateAvailable, proxyDomain.StateSuspect, proxyDomain.StateUnhealthy:
|
case proxyDomain.StateFetched, proxyDomain.StateAvailable, proxyDomain.StateSuspect, proxyDomain.StateUnhealthy:
|
||||||
return nil
|
return nil
|
||||||
@ -373,9 +416,17 @@ func decodeHealthTaskRecord(payload string) (healthTaskRecord, error) {
|
|||||||
if err := decodeJSON(payload, &record); err != nil {
|
if err := decodeJSON(payload, &record); err != nil {
|
||||||
return healthTaskRecord{}, err
|
return healthTaskRecord{}, err
|
||||||
}
|
}
|
||||||
|
if record.Level == string(healthDomain.LevelBasic) {
|
||||||
|
if record.Reference == "" {
|
||||||
|
record.Reference = record.ProxyID
|
||||||
|
}
|
||||||
|
if record.DueMember == "" {
|
||||||
|
record.DueMember = record.ProxyID
|
||||||
|
}
|
||||||
|
}
|
||||||
if record.Version != healthTaskRecordVersion || !validHealthTaskIdentifier(record.TaskID) ||
|
if record.Version != healthTaskRecordVersion || !validHealthTaskIdentifier(record.TaskID) ||
|
||||||
!validHealthTaskIdentifier(record.ProxyID) || !validHealthTaskIdentifier(record.UpstreamID) || record.UpstreamTasksKey == "" ||
|
!validHealthTaskIdentifier(record.ProxyID) || !validHealthTaskIdentifier(record.UpstreamID) || record.UpstreamTasksKey == "" ||
|
||||||
record.Level != string(healthDomain.LevelBasic) ||
|
!validHealthTaskReference(record.Reference) || !validHealthTaskReference(record.DueMember) ||
|
||||||
record.Priority < int(healthDomain.PriorityFetched) || record.Priority > int(healthDomain.PriorityAvailable) ||
|
record.Priority < int(healthDomain.PriorityFetched) || record.Priority > int(healthDomain.PriorityAvailable) ||
|
||||||
record.DeadlineMS <= 0 || record.NextDueMS <= 0 || record.Attempts <= 0 || record.MaxInFlight <= 0 ||
|
record.DeadlineMS <= 0 || record.NextDueMS <= 0 || record.Attempts <= 0 || record.MaxInFlight <= 0 ||
|
||||||
(record.State != "QUEUED" && record.State != "LEASED" && record.State != "DONE") {
|
(record.State != "QUEUED" && record.State != "LEASED" && record.State != "DONE") {
|
||||||
@ -384,6 +435,36 @@ func decodeHealthTaskRecord(payload string) (healthTaskRecord, error) {
|
|||||||
return record, nil
|
return record, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func redisHealthTaskReference(candidate healthDomain.Candidate) string {
|
||||||
|
if candidate.Level == healthDomain.LevelBasic {
|
||||||
|
return candidate.ProxyID
|
||||||
|
}
|
||||||
|
return healthDomain.CandidateIdentity(candidate)
|
||||||
|
}
|
||||||
|
|
||||||
|
func redisHealthDueMember(candidate healthDomain.Candidate) string {
|
||||||
|
if candidate.Level == healthDomain.LevelBasic {
|
||||||
|
return candidate.ProxyID
|
||||||
|
}
|
||||||
|
return candidate.TargetURL + "\x00" + candidate.ProxyID
|
||||||
|
}
|
||||||
|
|
||||||
|
func validRedisTaskExecution(task healthTaskRecord) bool {
|
||||||
|
switch healthDomain.Level(task.Level) {
|
||||||
|
case healthDomain.LevelBasic:
|
||||||
|
return task.RoutingName == "" && task.TargetURL == ""
|
||||||
|
case healthDomain.LevelEgress:
|
||||||
|
targetURL, err := healthDomain.NormalizeEgressTarget(task.TargetURL)
|
||||||
|
return task.RoutingName == "" && err == nil && targetURL == task.TargetURL
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validHealthTaskReference(value string) bool {
|
||||||
|
return value != "" && len(value) <= 2048 && strings.TrimSpace(value) == value && !strings.ContainsRune(value, '\x7f')
|
||||||
|
}
|
||||||
|
|
||||||
func newRedisLeaseToken() (string, error) {
|
func newRedisLeaseToken() (string, error) {
|
||||||
var entropy [24]byte
|
var entropy [24]byte
|
||||||
if _, err := rand.Read(entropy[:]); err != nil {
|
if _, err := rand.Read(entropy[:]); err != nil {
|
||||||
|
|||||||
@ -123,3 +123,59 @@ func TestRedisHealthTaskSweepReleasesLeaseAndUpstreamCapacity(t *testing.T) {
|
|||||||
t.Fatalf("AuthorizeObservation(after sweep) error = %v, want ErrTaskNotFound", err)
|
t.Fatalf("AuthorizeObservation(after sweep) error = %v, want ErrTaskNotFound", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRedisHealthTasksLeaseAndRescheduleEgressChecks(t *testing.T) {
|
||||||
|
fixture := newRedisTestFixture(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
now := time.Now().UTC()
|
||||||
|
const targetURL = "https://egress.example/identity"
|
||||||
|
if _, err := fixture.Adapter.UpsertFetched(ctx, "provider-a", activitypool.FetchedBatch{
|
||||||
|
ObservedAt: now, ConfiguredTTL: time.Minute, MaxSize: 1,
|
||||||
|
Proxies: []proxyDomain.Proxy{testProxy("proxy-a", "192.0.2.10")},
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("UpsertFetched(): %v", err)
|
||||||
|
}
|
||||||
|
candidates, err := fixture.Adapter.DueEgressCandidatesForUpstream(ctx, "provider-a", targetURL, now, 1)
|
||||||
|
if err != nil || len(candidates) != 1 || candidates[0].Level != healthDomain.LevelEgress || candidates[0].TargetURL != targetURL {
|
||||||
|
t.Fatalf("DueEgressCandidatesForUpstream() = (%+v, %v)", candidates, err)
|
||||||
|
}
|
||||||
|
planner, err := controllerHealth.NewPlanner(controllerHealth.SchedulePolicy{
|
||||||
|
Interval: 30 * time.Second, MaxInFlight: 2, Timeout: 5 * time.Second, MaxAttempts: 1,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewPlanner(): %v", err)
|
||||||
|
}
|
||||||
|
plans, err := planner.Plan(now, 0, 1, candidates)
|
||||||
|
if err != nil || len(plans) != 1 {
|
||||||
|
t.Fatalf("Plan() = (%+v, %v)", plans, err)
|
||||||
|
}
|
||||||
|
if offered, err := fixture.Adapter.Offer(ctx, plans); err != nil || offered != 1 {
|
||||||
|
t.Fatalf("Offer() = (%d, %v)", offered, err)
|
||||||
|
}
|
||||||
|
claimed, err := fixture.Adapter.Claim(ctx, healthDomain.TaskClaim{
|
||||||
|
CheckerID: "checker-a", InstanceID: "instance-a", MaxInFlight: 1,
|
||||||
|
SupportedLevels: []healthDomain.Level{healthDomain.LevelEgress},
|
||||||
|
})
|
||||||
|
if err != nil || len(claimed) != 1 || claimed[0].Level != healthDomain.LevelEgress || claimed[0].TargetURL != targetURL {
|
||||||
|
t.Fatalf("Claim() = (%+v, %v)", claimed, err)
|
||||||
|
}
|
||||||
|
observation := healthDomain.Observation{
|
||||||
|
TaskID: claimed[0].TaskID, ProxyID: claimed[0].ProxyID, Level: healthDomain.LevelEgress,
|
||||||
|
Success: true, Latency: time.Millisecond, ObservedAt: time.Now().UTC(),
|
||||||
|
}
|
||||||
|
if err := fixture.Adapter.AuthorizeObservation(ctx, "checker-a", claimed[0].LeaseToken, observation, time.Now().UTC()); err != nil {
|
||||||
|
t.Fatalf("AuthorizeObservation() = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := fixture.Adapter.ApplyGlobalObservation(ctx, activitypool.GlobalHealthCommand{
|
||||||
|
Observation: observation, MaxConsecutiveFailures: 2,
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("ApplyGlobalObservation() = %v", err)
|
||||||
|
}
|
||||||
|
if err := fixture.Adapter.CompleteObservation(ctx, "checker-a", claimed[0].LeaseToken, observation, time.Now().UTC()); err != nil {
|
||||||
|
t.Fatalf("CompleteObservation() = %v", err)
|
||||||
|
}
|
||||||
|
next, err := fixture.Adapter.DueEgressCandidatesForUpstream(ctx, "provider-a", targetURL, plans[0].NextDue, 1)
|
||||||
|
if err != nil || len(next) != 1 || next[0].ProxyID != "proxy-a" || next[0].TargetURL != targetURL {
|
||||||
|
t.Fatalf("DueEgressCandidatesForUpstream(next) = (%+v, %v)", next, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@ -28,6 +28,7 @@ type keyspace struct {
|
|||||||
workerRuntimeExpiry string
|
workerRuntimeExpiry string
|
||||||
workerOutcomes string
|
workerOutcomes string
|
||||||
healthDue string
|
healthDue string
|
||||||
|
healthEgressDue string
|
||||||
healthQueued string
|
healthQueued string
|
||||||
healthLeases string
|
healthLeases string
|
||||||
healthTasks string
|
healthTasks string
|
||||||
@ -57,6 +58,7 @@ func newKeyspace(namespace string) keyspace {
|
|||||||
workerRuntimeExpiry: prefix + ":worker-runtime-expiry",
|
workerRuntimeExpiry: prefix + ":worker-runtime-expiry",
|
||||||
workerOutcomes: prefix + ":worker-outcomes",
|
workerOutcomes: prefix + ":worker-outcomes",
|
||||||
healthDue: prefix + ":health-due",
|
healthDue: prefix + ":health-due",
|
||||||
|
healthEgressDue: prefix + ":health-egress-due",
|
||||||
healthQueued: prefix + ":health-queued",
|
healthQueued: prefix + ":health-queued",
|
||||||
healthLeases: prefix + ":health-leases",
|
healthLeases: prefix + ":health-leases",
|
||||||
healthTasks: prefix + ":health-tasks",
|
healthTasks: prefix + ":health-tasks",
|
||||||
|
|||||||
@ -55,6 +55,8 @@ type healthTaskCandidateWire struct {
|
|||||||
ProxyID string `json:"proxyId"`
|
ProxyID string `json:"proxyId"`
|
||||||
UpstreamID string `json:"upstreamId"`
|
UpstreamID string `json:"upstreamId"`
|
||||||
State string `json:"state"`
|
State string `json:"state"`
|
||||||
|
Level string `json:"level"`
|
||||||
|
TargetURL string `json:"targetUrl,omitempty"`
|
||||||
DueAtMS int64 `json:"dueAtMs"`
|
DueAtMS int64 `json:"dueAtMs"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@ -8,6 +8,7 @@ local task_expiry_key = KEYS[7]
|
|||||||
local ref_task_key = KEYS[8]
|
local ref_task_key = KEYS[8]
|
||||||
local checker_leases_key = KEYS[9]
|
local checker_leases_key = KEYS[9]
|
||||||
local state_inventory_key = KEYS[10]
|
local state_inventory_key = KEYS[10]
|
||||||
|
local egress_due_key = KEYS[11]
|
||||||
|
|
||||||
local operation = ARGV[1]
|
local operation = ARGV[1]
|
||||||
local now_ms = tonumber(ARGV[2])
|
local now_ms = tonumber(ARGV[2])
|
||||||
@ -68,14 +69,20 @@ local function remove_task(task_id, requeue)
|
|||||||
if type(task.checkerLeaseKey) == 'string' and task.checkerLeaseKey ~= '' then
|
if type(task.checkerLeaseKey) == 'string' and task.checkerLeaseKey ~= '' then
|
||||||
redis.call('ZREM', task.checkerLeaseKey, task_id)
|
redis.call('ZREM', task.checkerLeaseKey, task_id)
|
||||||
end
|
end
|
||||||
if type(task.proxyId) == 'string' and redis.call('HGET', ref_task_key, task.proxyId) == task_id then
|
local reference = task.reference or task.proxyId
|
||||||
redis.call('HDEL', ref_task_key, task.proxyId)
|
if type(reference) == 'string' and redis.call('HGET', ref_task_key, reference) == task_id then
|
||||||
|
redis.call('HDEL', ref_task_key, reference)
|
||||||
end
|
end
|
||||||
if requeue and type(task.proxyId) == 'string' then
|
local due_member = task.dueMember or task.proxyId
|
||||||
|
if requeue and type(task.proxyId) == 'string' and type(due_member) == 'string' then
|
||||||
local _, record = live_record(task.proxyId)
|
local _, record = live_record(task.proxyId)
|
||||||
if record then
|
if record then
|
||||||
redis.call('ZADD', due_key, now_ms, task.proxyId)
|
local task_due_key = due_key
|
||||||
touch(due_key, tonumber(record.expiresAtMs))
|
if task.level == 'EGRESS' then
|
||||||
|
task_due_key = egress_due_key
|
||||||
|
end
|
||||||
|
redis.call('ZADD', task_due_key, now_ms, due_member)
|
||||||
|
touch(task_due_key, tonumber(record.expiresAtMs))
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
@ -144,8 +151,13 @@ local function increment_state(upstream, state)
|
|||||||
end
|
end
|
||||||
|
|
||||||
local function matches_task(task, fact)
|
local function matches_task(task, fact)
|
||||||
return task and task.taskId == fact.taskId and task.proxyId == fact.proxyId and task.level == fact.level and
|
if not task or task.taskId ~= fact.taskId or task.proxyId ~= fact.proxyId or task.level ~= fact.level then
|
||||||
(task.routingName or '') == (fact.routingName or '') and (task.targetUrl or '') == (fact.targetUrl or '')
|
return false
|
||||||
|
end
|
||||||
|
if task.level == 'EGRESS' then
|
||||||
|
return (fact.routingName or '') == '' and (fact.targetUrl or '') == ''
|
||||||
|
end
|
||||||
|
return (task.routingName or '') == (fact.routingName or '') and (task.targetUrl or '') == (fact.targetUrl or '')
|
||||||
end
|
end
|
||||||
|
|
||||||
local function authorize(fact)
|
local function authorize(fact)
|
||||||
@ -198,22 +210,72 @@ if operation == 'due' then
|
|||||||
if not scan_limit or scan_limit < limit then
|
if not scan_limit or scan_limit < limit then
|
||||||
return finish({status = 'invalid'})
|
return finish({status = 'invalid'})
|
||||||
end
|
end
|
||||||
local ids = redis.call('ZRANGEBYSCORE', due_key, '-inf', now_ms, 'LIMIT', 0, scan_limit)
|
local level = payload.level or 'BASIC'
|
||||||
for _, proxy_id in ipairs(ids) do
|
if level ~= 'BASIC' and level ~= 'EGRESS' then
|
||||||
local _, record = live_record(proxy_id)
|
return finish({status = 'invalid'})
|
||||||
if not record or not valid_state(record.state) then
|
end
|
||||||
redis.call('ZREM', due_key, proxy_id)
|
if level == 'EGRESS' and (type(payload.targetUrl) ~= 'string' or payload.targetUrl == '') then
|
||||||
elseif type(payload.upstreamId) == 'string' and payload.upstreamId ~= '' and record.sourceUpstream ~= payload.upstreamId then
|
return finish({status = 'invalid'})
|
||||||
-- The ref remains due for the scheduler that owns this upstream.
|
end
|
||||||
else
|
local result_members = {}
|
||||||
result[#result + 1] = {
|
local function append_candidate(proxy_id, record, member)
|
||||||
proxyId = proxy_id, upstreamId = record.sourceUpstream, state = record.state, dueAtMs = now_ms
|
if result_members[member] then
|
||||||
}
|
return
|
||||||
if #result >= limit then
|
end
|
||||||
break
|
result_members[member] = true
|
||||||
|
result[#result + 1] = {
|
||||||
|
proxyId = proxy_id, upstreamId = record.sourceUpstream, state = record.state, level = level,
|
||||||
|
targetUrl = level == 'EGRESS' and payload.targetUrl or '', dueAtMs = now_ms
|
||||||
|
}
|
||||||
|
end
|
||||||
|
if level == 'EGRESS' then
|
||||||
|
local basic_ids = redis.call('ZRANGEBYSCORE', due_key, '-inf', now_ms, 'LIMIT', 0, scan_limit)
|
||||||
|
for _, proxy_id in ipairs(basic_ids) do
|
||||||
|
local _, record = live_record(proxy_id)
|
||||||
|
if record and valid_state(record.state) and
|
||||||
|
(type(payload.upstreamId) ~= 'string' or payload.upstreamId == '' or record.sourceUpstream == payload.upstreamId) then
|
||||||
|
local member = payload.targetUrl .. '\0' .. proxy_id
|
||||||
|
local reference = proxy_id .. '\0EGRESS\0\0' .. payload.targetUrl
|
||||||
|
if not redis.call('HGET', ref_task_key, reference) then
|
||||||
|
redis.call('ZADD', egress_due_key, now_ms, member)
|
||||||
|
touch(egress_due_key, tonumber(record.expiresAtMs))
|
||||||
|
end
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
local source_due_key = due_key
|
||||||
|
if level == 'EGRESS' then
|
||||||
|
source_due_key = egress_due_key
|
||||||
|
end
|
||||||
|
local ids = redis.call('ZRANGEBYSCORE', source_due_key, '-inf', now_ms, 'LIMIT', 0, scan_limit)
|
||||||
|
for _, member in ipairs(ids) do
|
||||||
|
local proxy_id = member
|
||||||
|
local matching_target = true
|
||||||
|
if level == 'EGRESS' then
|
||||||
|
local separator = string.find(member, '\0', 1, true)
|
||||||
|
if not separator or string.sub(member, 1, separator - 1) ~= payload.targetUrl then
|
||||||
|
matching_target = false
|
||||||
|
else
|
||||||
|
proxy_id = string.sub(member, separator + 1)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
if matching_target then
|
||||||
|
local _, record = live_record(proxy_id)
|
||||||
|
local reference = proxy_id .. '\0EGRESS\0\0' .. payload.targetUrl
|
||||||
|
if not record or not valid_state(record.state) then
|
||||||
|
redis.call('ZREM', source_due_key, member)
|
||||||
|
elseif level == 'EGRESS' and redis.call('HGET', ref_task_key, reference) then
|
||||||
|
redis.call('ZREM', source_due_key, member)
|
||||||
|
elseif type(payload.upstreamId) == 'string' and payload.upstreamId ~= '' and record.sourceUpstream ~= payload.upstreamId then
|
||||||
|
-- The ref remains due for the scheduler that owns this upstream.
|
||||||
|
else
|
||||||
|
append_candidate(proxy_id, record, member)
|
||||||
|
if #result >= limit then
|
||||||
|
break
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
local encoded_candidates = cjson.encode(result)
|
local encoded_candidates = cjson.encode(result)
|
||||||
if #result == 0 then
|
if #result == 0 then
|
||||||
encoded_candidates = '[]'
|
encoded_candidates = '[]'
|
||||||
@ -227,29 +289,40 @@ if operation == 'offer' then
|
|||||||
end
|
end
|
||||||
local offered = 0
|
local offered = 0
|
||||||
for _, task in ipairs(payload.tasks) do
|
for _, task in ipairs(payload.tasks) do
|
||||||
if type(task) ~= 'table' or task.version ~= 1 or task.level ~= 'BASIC' or task.state ~= 'QUEUED' or
|
if type(task) ~= 'table' or task.version ~= 1 or (task.level ~= 'BASIC' and task.level ~= 'EGRESS') or task.state ~= 'QUEUED' or
|
||||||
type(task.taskId) ~= 'string' or type(task.proxyId) ~= 'string' or tonumber(task.deadlineMs or 0) <= now_ms or
|
type(task.taskId) ~= 'string' or type(task.proxyId) ~= 'string' or tonumber(task.deadlineMs or 0) <= now_ms or
|
||||||
type(task.upstreamId) ~= 'string' or task.upstreamId == '' or type(task.upstreamTasksKey) ~= 'string' or
|
type(task.upstreamId) ~= 'string' or task.upstreamId == '' or type(task.upstreamTasksKey) ~= 'string' or
|
||||||
task.upstreamTasksKey == '' or tonumber(task.nextDueMs or 0) <= now_ms or tonumber(task.attempts or 0) <= 0 or
|
task.upstreamTasksKey == '' or type(task.reference) ~= 'string' or task.reference == '' or
|
||||||
|
type(task.dueMember) ~= 'string' or task.dueMember == '' or tonumber(task.nextDueMs or 0) <= now_ms or tonumber(task.attempts or 0) <= 0 or
|
||||||
tonumber(task.maxInFlight or 0) <= 0 then
|
tonumber(task.maxInFlight or 0) <= 0 then
|
||||||
return finish({status = 'invalid'})
|
return finish({status = 'invalid'})
|
||||||
end
|
end
|
||||||
local score = redis.call('ZSCORE', due_key, task.proxyId)
|
if task.level == 'BASIC' and ((task.routingName or '') ~= '' or (task.targetUrl or '') ~= '') then
|
||||||
local current = redis.call('HGET', ref_task_key, task.proxyId)
|
return finish({status = 'invalid'})
|
||||||
|
end
|
||||||
|
if task.level == 'EGRESS' and ((task.routingName or '') ~= '' or type(task.targetUrl) ~= 'string' or task.targetUrl == '') then
|
||||||
|
return finish({status = 'invalid'})
|
||||||
|
end
|
||||||
|
local task_due_key = due_key
|
||||||
|
if task.level == 'EGRESS' then
|
||||||
|
task_due_key = egress_due_key
|
||||||
|
end
|
||||||
|
local score = redis.call('ZSCORE', task_due_key, task.dueMember)
|
||||||
|
local current = redis.call('HGET', ref_task_key, task.reference)
|
||||||
local raw, record = live_record(task.proxyId)
|
local raw, record = live_record(task.proxyId)
|
||||||
local active = redis.call('ZCARD', task.upstreamTasksKey)
|
local active = redis.call('ZCARD', task.upstreamTasksKey)
|
||||||
if score and tonumber(score) <= now_ms and not current and raw and record.sourceUpstream == task.upstreamId and
|
if score and tonumber(score) <= now_ms and not current and raw and record.sourceUpstream == task.upstreamId and
|
||||||
active < tonumber(task.maxInFlight) and valid_state(record.state) then
|
active < tonumber(task.maxInFlight) and valid_state(record.state) then
|
||||||
if record.state == 'FETCHED' or record.state == 'UNHEALTHY' then
|
if task.level == 'BASIC' and (record.state == 'FETCHED' or record.state == 'UNHEALTHY') then
|
||||||
decrement_state(record.sourceUpstream, record.state)
|
decrement_state(record.sourceUpstream, record.state)
|
||||||
increment_state(record.sourceUpstream, 'CHECKING')
|
increment_state(record.sourceUpstream, 'CHECKING')
|
||||||
record.state = 'CHECKING'
|
record.state = 'CHECKING'
|
||||||
raw = cjson.encode(record)
|
raw = cjson.encode(record)
|
||||||
redis.call('HSET', records_key, task.proxyId, raw)
|
redis.call('HSET', records_key, task.proxyId, raw)
|
||||||
end
|
end
|
||||||
redis.call('ZREM', due_key, task.proxyId)
|
redis.call('ZREM', task_due_key, task.dueMember)
|
||||||
redis.call('HSET', tasks_key, task.taskId, cjson.encode(task))
|
redis.call('HSET', tasks_key, task.taskId, cjson.encode(task))
|
||||||
redis.call('HSET', ref_task_key, task.proxyId, task.taskId)
|
redis.call('HSET', ref_task_key, task.reference, task.taskId)
|
||||||
redis.call('ZADD', queued_key, tonumber(task.priority), task.taskId)
|
redis.call('ZADD', queued_key, tonumber(task.priority), task.taskId)
|
||||||
redis.call('ZADD', task_expiry_key, tonumber(task.deadlineMs), task.taskId)
|
redis.call('ZADD', task_expiry_key, tonumber(task.deadlineMs), task.taskId)
|
||||||
redis.call('ZADD', task.upstreamTasksKey, tonumber(task.deadlineMs), task.taskId)
|
redis.call('ZADD', task.upstreamTasksKey, tonumber(task.deadlineMs), task.taskId)
|
||||||
@ -344,8 +417,9 @@ if operation == 'authorize' or operation == 'complete' then
|
|||||||
end
|
end
|
||||||
task.state = 'DONE'
|
task.state = 'DONE'
|
||||||
redis.call('HSET', tasks_key, task.taskId, cjson.encode(task))
|
redis.call('HSET', tasks_key, task.taskId, cjson.encode(task))
|
||||||
if redis.call('HGET', ref_task_key, task.proxyId) == task.taskId then
|
local reference = task.reference or task.proxyId
|
||||||
redis.call('HDEL', ref_task_key, task.proxyId)
|
if redis.call('HGET', ref_task_key, reference) == task.taskId then
|
||||||
|
redis.call('HDEL', ref_task_key, reference)
|
||||||
end
|
end
|
||||||
local _, record = live_record(task.proxyId)
|
local _, record = live_record(task.proxyId)
|
||||||
if record then
|
if record then
|
||||||
@ -353,8 +427,12 @@ if operation == 'authorize' or operation == 'complete' then
|
|||||||
if next_due_ms <= now_ms then
|
if next_due_ms <= now_ms then
|
||||||
next_due_ms = now_ms
|
next_due_ms = now_ms
|
||||||
end
|
end
|
||||||
redis.call('ZADD', due_key, next_due_ms, task.proxyId)
|
local task_due_key = due_key
|
||||||
touch(due_key, tonumber(record.expiresAtMs))
|
if task.level == 'EGRESS' then
|
||||||
|
task_due_key = egress_due_key
|
||||||
|
end
|
||||||
|
redis.call('ZADD', task_due_key, next_due_ms, task.dueMember or task.proxyId)
|
||||||
|
touch(task_due_key, tonumber(record.expiresAtMs))
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
return finish({status = 'ok'})
|
return finish({status = 'ok'})
|
||||||
|
|||||||
@ -5,6 +5,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"sort"
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"proxy-pool/internal/config"
|
"proxy-pool/internal/config"
|
||||||
@ -30,6 +31,14 @@ type UpstreamTaskSource interface {
|
|||||||
DueCandidatesForUpstream(context.Context, string, time.Time, int) ([]Candidate, error)
|
DueCandidatesForUpstream(context.Context, string, time.Time, int) ([]Candidate, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// EgressUpstreamTaskSource supplies bounded per-upstream candidates for one
|
||||||
|
// configured egress probe URL. Its optional nature keeps BASIC-only stores
|
||||||
|
// source compatible during rollout.
|
||||||
|
type EgressUpstreamTaskSource interface {
|
||||||
|
UpstreamTaskSource
|
||||||
|
DueEgressCandidatesForUpstream(context.Context, string, string, time.Time, int) ([]Candidate, error)
|
||||||
|
}
|
||||||
|
|
||||||
type upstreamDueSource struct {
|
type upstreamDueSource struct {
|
||||||
source UpstreamTaskSource
|
source UpstreamTaskSource
|
||||||
upstreamID string
|
upstreamID string
|
||||||
@ -50,6 +59,27 @@ func (source upstreamDueSource) DueCandidates(ctx context.Context, now time.Time
|
|||||||
return source.source.DueCandidatesForUpstream(ctx, source.upstreamID, now, limit)
|
return source.source.DueCandidatesForUpstream(ctx, source.upstreamID, now, limit)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type egressUpstreamDueSource struct {
|
||||||
|
source EgressUpstreamTaskSource
|
||||||
|
upstreamID string
|
||||||
|
targetURL string
|
||||||
|
}
|
||||||
|
|
||||||
|
func newEgressUpstreamDueSource(source EgressUpstreamTaskSource, upstreamID, targetURL string) (DueSource, error) {
|
||||||
|
if nilInterface(source) || upstreamID == "" || targetURL == "" {
|
||||||
|
return nil, ErrInvalidSchedulerRunner
|
||||||
|
}
|
||||||
|
return egressUpstreamDueSource{source: source, upstreamID: upstreamID, targetURL: targetURL}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (source egressUpstreamDueSource) InFlight(ctx context.Context, now time.Time) (int, error) {
|
||||||
|
return source.source.InFlightForUpstream(ctx, source.upstreamID, now)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (source egressUpstreamDueSource) DueCandidates(ctx context.Context, now time.Time, limit int) ([]Candidate, error) {
|
||||||
|
return source.source.DueEgressCandidatesForUpstream(ctx, source.upstreamID, source.targetURL, now, limit)
|
||||||
|
}
|
||||||
|
|
||||||
// TaskSink atomically offers an already bounded batch to the shared leased
|
// TaskSink atomically offers an already bounded batch to the shared leased
|
||||||
// task store. It must leave unaccepted candidates eligible for a later tick.
|
// task store. It must leave unaccepted candidates eligible for a later tick.
|
||||||
type TaskSink interface {
|
type TaskSink interface {
|
||||||
@ -102,6 +132,7 @@ type ConfiguredSchedulerSupervisor struct {
|
|||||||
source UpstreamTaskSource
|
source UpstreamTaskSource
|
||||||
sink TaskSink
|
sink TaskSink
|
||||||
options SchedulerRunnerOptions
|
options SchedulerRunnerOptions
|
||||||
|
cursor atomic.Uint64
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewConfiguredSchedulerRunner(
|
func NewConfiguredSchedulerRunner(
|
||||||
@ -188,20 +219,58 @@ func (supervisor *ConfiguredSchedulerSupervisor) Tick(ctx context.Context) (Tick
|
|||||||
if !configured {
|
if !configured {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
source, err := NewUpstreamDueSource(supervisor.source, name)
|
basicSource, err := NewUpstreamDueSource(supervisor.source, name)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return TickResult{}, err
|
return TickResult{}, err
|
||||||
}
|
}
|
||||||
tick, err := tickWithPlanner(ctx, planner, source, supervisor.sink, schedulerOptionsForPlanner(supervisor.options, planner))
|
check := config.EffectiveCheck(configuration.Defaults.Check, configuration.Upstreams[name].Check)
|
||||||
|
groups, err := supervisor.upstreamDueGroups(basicSource, name, check.URLs)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return TickResult{}, err
|
return TickResult{}, err
|
||||||
}
|
}
|
||||||
result.Planned += tick.Planned
|
groupOptions := schedulerOptionsForPlanner(supervisor.options, planner)
|
||||||
result.Offered += tick.Offered
|
groupOptions.BatchSize = schedulerGroupBatchSize(groupOptions.BatchSize, len(groups))
|
||||||
|
start := int(supervisor.cursor.Add(1)-1) % len(groups)
|
||||||
|
for offset := range groups {
|
||||||
|
source := groups[(start+offset)%len(groups)]
|
||||||
|
tick, err := tickWithPlanner(ctx, planner, source, supervisor.sink, groupOptions)
|
||||||
|
if err != nil {
|
||||||
|
return TickResult{}, err
|
||||||
|
}
|
||||||
|
result.Planned += tick.Planned
|
||||||
|
result.Offered += tick.Offered
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return result, nil
|
return result, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (supervisor *ConfiguredSchedulerSupervisor) upstreamDueGroups(basic DueSource, upstreamID string, urls []string) ([]DueSource, error) {
|
||||||
|
groups := []DueSource{basic}
|
||||||
|
egress, supported := supervisor.source.(EgressUpstreamTaskSource)
|
||||||
|
if !supported || len(urls) == 0 {
|
||||||
|
return groups, nil
|
||||||
|
}
|
||||||
|
for _, targetURL := range urls {
|
||||||
|
source, err := newEgressUpstreamDueSource(egress, upstreamID, targetURL)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
groups = append(groups, source)
|
||||||
|
}
|
||||||
|
return groups, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func schedulerGroupBatchSize(batchSize, groups int) int {
|
||||||
|
if batchSize <= 0 || groups <= 0 {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
result := (batchSize + groups - 1) / groups
|
||||||
|
if result < 1 {
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
func tickWithPlanner(
|
func tickWithPlanner(
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
planner *Planner,
|
planner *Planner,
|
||||||
|
|||||||
@ -164,12 +164,44 @@ func TestConfiguredSchedulerSupervisorDiscoversNewEnabledUpstream(t *testing.T)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestConfiguredSchedulerSupervisorSchedulesBoundedEgressGroups(t *testing.T) {
|
||||||
|
now := time.Date(2026, 8, 2, 11, 0, 0, 0, time.UTC)
|
||||||
|
configuration := &config.Config{
|
||||||
|
Defaults: config.Defaults{Check: config.Check{
|
||||||
|
Interval: config.Duration(time.Minute), MaxInFlight: 6, Timeout: config.Duration(time.Second), MaxAttempts: 1,
|
||||||
|
URLs: []string{"https://egress-one.example/identity", "https://egress-two.example/identity"},
|
||||||
|
}},
|
||||||
|
Upstreams: map[string]config.Upstream{"provider-a": {Enabled: true}},
|
||||||
|
}
|
||||||
|
source := &upstreamTaskSourceStub{sources: map[string]*dueSourceStub{
|
||||||
|
"provider-a": {candidates: []Candidate{{
|
||||||
|
ProxyID: "basic", State: proxyDomain.StateFetched, Level: healthDomain.LevelBasic, DueAt: now,
|
||||||
|
}}, egressCandidates: map[string][]Candidate{
|
||||||
|
"https://egress-one.example/identity": {{ProxyID: "egress-one", State: proxyDomain.StateFetched, Level: healthDomain.LevelEgress, TargetURL: "https://egress-one.example/identity", DueAt: now}},
|
||||||
|
"https://egress-two.example/identity": {{ProxyID: "egress-two", State: proxyDomain.StateFetched, Level: healthDomain.LevelEgress, TargetURL: "https://egress-two.example/identity", DueAt: now}},
|
||||||
|
}},
|
||||||
|
}}
|
||||||
|
sink := &taskSinkStub{}
|
||||||
|
supervisor, err := NewConfiguredSchedulerSupervisor(&configurationSourceStub{configuration: configuration}, source, sink,
|
||||||
|
SchedulerRunnerOptions{PollInterval: time.Second, BatchSize: 6, Now: func() time.Time { return now }})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewConfiguredSchedulerSupervisor() = %v", err)
|
||||||
|
}
|
||||||
|
result, err := supervisor.Tick(context.Background())
|
||||||
|
if err != nil || result != (TickResult{Planned: 3, Offered: 3}) || len(sink.tasks) != 1 ||
|
||||||
|
sink.tasks[0].Candidate.Level != healthDomain.LevelEgress || source.sources["provider-a"].egressCalls != 2 {
|
||||||
|
t.Fatalf("Tick() = (%+v, %v); source=%+v sink=%+v", result, err, source, sink.tasks)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
type dueSourceStub struct {
|
type dueSourceStub struct {
|
||||||
inFlight int
|
inFlight int
|
||||||
candidates []Candidate
|
candidates []Candidate
|
||||||
limit int
|
limit int
|
||||||
inFlightCalls int
|
inFlightCalls int
|
||||||
dueCalls int
|
dueCalls int
|
||||||
|
egressCandidates map[string][]Candidate
|
||||||
|
egressCalls int
|
||||||
}
|
}
|
||||||
|
|
||||||
func (source *dueSourceStub) InFlight(context.Context, time.Time) (int, error) {
|
func (source *dueSourceStub) InFlight(context.Context, time.Time) (int, error) {
|
||||||
@ -183,6 +215,12 @@ func (source *dueSourceStub) DueCandidates(_ context.Context, _ time.Time, limit
|
|||||||
return source.candidates, nil
|
return source.candidates, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (source *dueSourceStub) DueEgressCandidates(_ context.Context, targetURL string, _ time.Time, limit int) ([]Candidate, error) {
|
||||||
|
source.egressCalls++
|
||||||
|
source.limit = limit
|
||||||
|
return source.egressCandidates[targetURL], nil
|
||||||
|
}
|
||||||
|
|
||||||
type upstreamTaskSourceStub struct {
|
type upstreamTaskSourceStub struct {
|
||||||
sources map[string]*dueSourceStub
|
sources map[string]*dueSourceStub
|
||||||
}
|
}
|
||||||
@ -203,6 +241,14 @@ func (source *upstreamTaskSourceStub) DueCandidatesForUpstream(ctx context.Conte
|
|||||||
return item.DueCandidates(ctx, now, limit)
|
return item.DueCandidates(ctx, now, limit)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (source *upstreamTaskSourceStub) DueEgressCandidatesForUpstream(ctx context.Context, upstreamID, targetURL string, now time.Time, limit int) ([]Candidate, error) {
|
||||||
|
item, exists := source.sources[upstreamID]
|
||||||
|
if !exists {
|
||||||
|
return nil, errors.New("missing upstream source")
|
||||||
|
}
|
||||||
|
return item.DueEgressCandidates(ctx, targetURL, now, limit)
|
||||||
|
}
|
||||||
|
|
||||||
type taskSinkStub struct {
|
type taskSinkStub struct {
|
||||||
tasks []PlannedTask
|
tasks []PlannedTask
|
||||||
offer int
|
offer int
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user