Compare commits
2 Commits
b8f5104167
...
c3b5b25597
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c3b5b25597 | ||
|
|
e54fc84a81 |
@ -154,7 +154,7 @@ Redis Adapter 应通过单个 Lua 脚本、Redis Function 或等价的原子原
|
||||
- 两个并发成功响应的 Proxy ID 集合交集为空。
|
||||
- 原子操作失败时整个批次不返回,也不得留下部分移除结果。
|
||||
- `allOrNothing` 不足时零个条目退出活动池。
|
||||
- Redis 活动池不可用时返回 503,不以内存副本冒充成功。
|
||||
- Redis 活动池或分布式限流不可用时返回 503,不以内存副本冒充成功。
|
||||
- PostgreSQL 不可用不阻断提取;需要 PostgreSQL 的 Admin 管理写入单独降级。
|
||||
|
||||
## 7. 幂等
|
||||
@ -175,7 +175,9 @@ TTL 到期或 Redis 数据丢失后不再保证旧 Key 去重,系统不回退
|
||||
|
||||
- 直连请求使用来源 IP 形成匿名 Client。
|
||||
- 只有来源属于 `trustedProxies` 时才接受转发头。
|
||||
- 全局和每 Client 限流在查询库存前执行。
|
||||
- 全局和每 Client 限流在查询库存前执行;本地监听器先做早期拒绝,Redis 使用
|
||||
服务端时间和单个 Lua 原子操作执行跨 Controller 副本的权威额度。
|
||||
- Client 身份进入 Redis 前转换为定长摘要,窗口切换时原子删除上一窗口字段。
|
||||
- 过滤条件、数量、请求体和 Header 都有长度/数量上限。
|
||||
|
||||
## 9. 错误模型
|
||||
@ -190,7 +192,7 @@ TTL 到期或 Redis 数据丢失后不再保证旧 Key 去重,系统不回退
|
||||
- `415`:请求体不是 `application/json`。
|
||||
- `422`:数量、枚举或过滤组合违反业务约束。
|
||||
- `429`:全局或 Client 速率限制,响应 `Retry-After`。
|
||||
- `503`:Redis 活动池不可用、原子提取不可执行或服务正在排空。
|
||||
- `503`:Redis 活动池/分布式限流不可用、原子提取不可执行或服务正在排空。
|
||||
- `500`:未分类的内部错误;响应不包含底层错误文本。
|
||||
|
||||
错误响应不得包含 Provider Secret、Proxy 凭据、SQL 或内部拓扑。
|
||||
|
||||
@ -104,6 +104,10 @@ limits:
|
||||
`trustedProxies` 只决定何时接受 `Forwarded` 或 `X-Forwarded-For`,不能替代
|
||||
`allowCIDRs`。来自非可信代理的转发头必须忽略。
|
||||
|
||||
`requestsPerMinute` 和 `requestsPerMinutePerClient` 为非负整数,最大值为
|
||||
`2^53-1`;Distribution 的非零额度由 Redis Lua 计数,因此配置校验统一限制在
|
||||
Lua 可精确表示的整数范围内。
|
||||
|
||||
### 3.1 认证模式
|
||||
|
||||
- `none`:无身份认证,访问控制与限流仍生效。
|
||||
|
||||
@ -174,8 +174,9 @@ Upstream、`endBehavior` 默认 `stop`,并覆盖列表末端停止;disabled
|
||||
candidate eligibility, Gateway reserve, ownership, removal, and short-lived idempotency.
|
||||
- [x] Implement Redis Worker ownership, drain/ACK, expiry reclaim, inventory and bounded
|
||||
sweep primitives with a monotonic global epoch.
|
||||
- [ ] Implement Redis Provider leader, distributed rate, Client limit and Worker
|
||||
heartbeat; wire automatic Provider inventory rebuild after Redis loss.
|
||||
- [x] Implement Redis Provider leader, distributed request quota, Client limit and
|
||||
automatic Provider inventory rebuild after Redis state loss.
|
||||
- [ ] Implement the Worker heartbeat receiving path and session lifecycle.
|
||||
- [x] Keep Provider output in Redis TTL activity state and node memory only; keep the
|
||||
Gateway request path on immutable local snapshots with no Redis/PostgreSQL calls.
|
||||
- [x] Expose Distribution extraction/status and Admin status/enable/disable/switch/reload
|
||||
@ -236,9 +237,13 @@ Redis Provider Permit 现已把 requestInterval、maxInFlight 与 maxTotal 放
|
||||
启动后动态扩容。Supervisor 以 PostgreSQL 权威 HMAC 指纹和 revision 栅栏协调
|
||||
多副本 reload:管理库瞬断沿用 last-known 状态,本地共享源落后时停止旧 Provider,
|
||||
源匹配并预检后自动替换,迟到旧 revision 不覆盖新配置。
|
||||
Distribution 现通过公用 `admission.Admitter` 接入独立 `redisadmission` Adapter;
|
||||
全局和单 Client 分钟额度使用 Redis 服务端时间并在单个 Lua 原子边界内检查、递增,
|
||||
Controller 多副本共享同一计数。Client 身份只以 SHA-256 摘要进入 Redis,窗口切换
|
||||
原子回收历史字段;Redis 异常 fail-closed 并返回 503,真实额度耗尽返回 429。
|
||||
Gateway 请求热路径仍只使用本地准入,不增加 Redis/PostgreSQL 调用。
|
||||
WorkerControlPlane gRPC 接收端、session 签发/心跳、Snapshot ACK
|
||||
账本、Client 分布式限流和健康执行链仍待完成,因此本轮不勾选 Task 10 的组合
|
||||
验收项。
|
||||
账本和健康执行链仍待完成,因此 Task 10 尚未全部完成。
|
||||
|
||||
## Task 11: Checker and Health Reducer
|
||||
|
||||
|
||||
@ -240,7 +240,7 @@ Outbox 发布器必须以稳定 consumer ID 有界领取;发布成功后原子
|
||||
回查 PostgreSQL 或 Provider。
|
||||
2. Distribution 立即失败关闭并返回 503,禁止本地内存提取或 PostgreSQL 兜底。
|
||||
3. Controller 停止活动池写入和需要分布式互斥的工作,防止多个 Fetch Leader;
|
||||
本地限流不能声称满足全局额度。
|
||||
Distribution 的 Redis 权威限流同时失败关闭,本地早期限流不冒充跨副本额度。
|
||||
4. 恢复后确认 Leader 唯一和租约 epoch 单调,由 Provider 重新 Fetch 并构建 TTL
|
||||
活动池,再恢复 Distribution。Redis 整体丢失会终止原活动池代次的排他状态和
|
||||
短期幂等窗口;高可用、持久化、监控和告警必须明确并降低该风险。
|
||||
|
||||
@ -84,11 +84,11 @@ CI 已配置 Linux race job。PostgreSQL 18 和 Redis 8.2 的隔离 Adapter fixt
|
||||
4. Controller 的 PostgreSQL 连接池、迁移和 pgx Adapter 启动装配已完成;
|
||||
公用 bootstrap 已通过 PostgreSQL 18 + Redis 8.2 双存储集成,Controller
|
||||
三监听器与探针集成已完成;可选聚合指标和完整容器进程部署验证仍待实现。
|
||||
5. Redis Provider Leader、分布式速率与 Client 限制、Worker 心跳和自动重建;
|
||||
TTL 活动池、原子提取和 Worker ownership 已完成。
|
||||
5. Worker heartbeat gRPC 接收路径;Redis Provider Leader、分布式请求额度、
|
||||
Distribution Client 限制和 Provider 状态丢失重建已完成。
|
||||
6. Worker 网络快照流;Redis ownership drain/ACK/过期回收已完成。
|
||||
7. Checker 调度、探测器和健康 reducer。
|
||||
8. Admin/Distribution 细粒度授权、分布式限流和审计查询。
|
||||
8. Admin/Distribution 细粒度授权和审计查询;Distribution 分布式限流已完成。
|
||||
9. 真实 Compose/Kubernetes 集成、故障演练和代表性集群负载测试。
|
||||
10. 将五种 Routing 策略和 `onUnavailable` 接入 Gateway/Distribution 运行链,
|
||||
补齐 Sequential 持久化恢复、跨实例 CAS 和 disabled candidate 语义。
|
||||
|
||||
@ -72,7 +72,7 @@
|
||||
| DIST-005 | 返回 expiresAt 与 remainingTtlSeconds | 9334-9360 | `extraction/service_test.go` |
|
||||
| DIST-006 | 提取前校验 minRemainingTTL 与 maxHealthCheckAge | 9334-9369 | 过滤测试 |
|
||||
| DIST-007 | reserveForGateway 防止 Extract 清空共享池 | 9281-9333 | 共享池测试 |
|
||||
| DIST-008 | 提取认证可关闭,关闭后仍有来源识别与全局限制 | 8112-8441 | 来源身份准入与 `FixedWindow` 并发测试 |
|
||||
| DIST-008 | 提取认证可关闭,关闭后仍有来源识别与全局限制 | 8112-8441 | 来源身份准入、`FixedWindow` 单元测试及 `redisadmission` 双实例/并发集成测试 |
|
||||
|
||||
## 健康、安全、运维与测试
|
||||
|
||||
|
||||
@ -30,7 +30,8 @@
|
||||
原子操作;并发与主从切换下不得部分提交。
|
||||
- PostgreSQL 配置版本、Upstream/Routing 管理状态、Admin 审计与 outbox 的事务
|
||||
更新及幂等重放;测试库断言不包含 Proxy 明细或逐次提取记录。
|
||||
- Redis TTL 活动池、Leader 租约、限流、短期幂等窗口和失联恢复。
|
||||
- Redis TTL 活动池、Leader 租约、Provider 请求额度、Distribution Client
|
||||
跨副本限流、短期幂等窗口和失联恢复。
|
||||
- Snapshot/Delta/ACK/Report 的版本与校验和兼容性。
|
||||
- OpenAPI 错误模型、认证矩阵、批量 fulfillment。
|
||||
- OpenAPI 本地引用闭合、operationId 唯一、响应集合和 security scheme 引用。
|
||||
|
||||
@ -22,7 +22,11 @@ func TestReadStateInventoryFailsClosedWhileExpiredCleanupIsBacklogged(t *testing
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
now := time.Date(2026, 7, 30, 12, 0, 0, 0, time.UTC)
|
||||
redisTime, err := fixture.Client.Time(context.Background()).Result()
|
||||
if err != nil {
|
||||
t.Fatalf("Redis TIME error = %v", err)
|
||||
}
|
||||
now := redisTime.UTC().Add(time.Minute)
|
||||
_, err = bounded.UpsertFetched(context.Background(), "provider-a", activitypool.FetchedBatch{
|
||||
ObservedAt: now, ConfiguredTTL: time.Second, MaxSize: 10,
|
||||
Proxies: []proxyDomain.Proxy{
|
||||
|
||||
141
internal/adapters/redisadmission/adapter.go
Normal file
141
internal/adapters/redisadmission/adapter.go
Normal file
@ -0,0 +1,141 @@
|
||||
package redisadmission
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
_ "embed"
|
||||
"encoding/binary"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"reflect"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
|
||||
"proxy-pool/internal/platform/admission"
|
||||
)
|
||||
|
||||
const MaximumLuaInteger int64 = 1<<53 - 1
|
||||
|
||||
var (
|
||||
ErrInvalidOptions = errors.New("invalid Redis admission options")
|
||||
namespacePattern = regexp.MustCompile(`^[A-Za-z0-9._-]+$`)
|
||||
redisAdmissionKey = "pp:{admission}:"
|
||||
redisAdmissionTail = ":window"
|
||||
)
|
||||
|
||||
//go:embed scripts/fixed_window.lua
|
||||
var fixedWindowSource string
|
||||
|
||||
var fixedWindowScript = redis.NewScript(fixedWindowSource)
|
||||
|
||||
type Options struct {
|
||||
Namespace string
|
||||
Window time.Duration
|
||||
Global int64
|
||||
PerKey int64
|
||||
}
|
||||
|
||||
type Adapter struct {
|
||||
client redis.Scripter
|
||||
key string
|
||||
windowMillis int64
|
||||
global int64
|
||||
perKey int64
|
||||
}
|
||||
|
||||
var _ admission.Admitter = (*Adapter)(nil)
|
||||
|
||||
func New(client redis.Scripter, options Options) (*Adapter, error) {
|
||||
if nilInterface(client) || options.Namespace != strings.TrimSpace(options.Namespace) ||
|
||||
!namespacePattern.MatchString(options.Namespace) || options.Window <= 0 ||
|
||||
options.Window%time.Millisecond != 0 || options.Global < 0 || options.PerKey < 0 ||
|
||||
(options.Global == 0 && options.PerKey == 0) || options.Global > MaximumLuaInteger ||
|
||||
options.PerKey > MaximumLuaInteger {
|
||||
return nil, ErrInvalidOptions
|
||||
}
|
||||
windowMillis := options.Window.Milliseconds()
|
||||
if windowMillis <= 0 || windowMillis > MaximumLuaInteger {
|
||||
return nil, ErrInvalidOptions
|
||||
}
|
||||
return &Adapter{
|
||||
client: client,
|
||||
key: redisAdmissionKey + options.Namespace + redisAdmissionTail,
|
||||
windowMillis: windowMillis,
|
||||
global: options.Global,
|
||||
perKey: options.PerKey,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (adapter *Adapter) Admit(ctx context.Context, identity string) error {
|
||||
if ctx == nil || identity == "" {
|
||||
return admission.ErrInvalidIdentity
|
||||
}
|
||||
if adapter == nil || nilInterface(adapter.client) || adapter.key == "" || adapter.windowMillis <= 0 ||
|
||||
adapter.windowMillis > MaximumLuaInteger || adapter.global < 0 || adapter.perKey < 0 ||
|
||||
(adapter.global == 0 && adapter.perKey == 0) || adapter.global > MaximumLuaInteger ||
|
||||
adapter.perKey > MaximumLuaInteger {
|
||||
return admission.ErrUnavailable
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
result, err := fixedWindowScript.Run(ctx, adapter.client, []string{adapter.key},
|
||||
strconv.FormatInt(adapter.windowMillis, 10),
|
||||
strconv.FormatInt(adapter.global, 10),
|
||||
strconv.FormatInt(adapter.perKey, 10),
|
||||
hashedIdentityField(identity),
|
||||
).Result()
|
||||
if err != nil {
|
||||
if ctxErr := ctx.Err(); ctxErr != nil {
|
||||
return ctxErr
|
||||
}
|
||||
return admission.ErrUnavailable
|
||||
}
|
||||
|
||||
var status string
|
||||
switch value := result.(type) {
|
||||
case string:
|
||||
status = value
|
||||
case []byte:
|
||||
status = string(value)
|
||||
default:
|
||||
return admission.ErrUnavailable
|
||||
}
|
||||
switch status {
|
||||
case "ok":
|
||||
return nil
|
||||
case "global":
|
||||
return admission.ErrGlobalLimit
|
||||
case "per_key":
|
||||
return admission.ErrPerKeyLimit
|
||||
default:
|
||||
return admission.ErrUnavailable
|
||||
}
|
||||
}
|
||||
|
||||
func hashedIdentityField(identity string) string {
|
||||
var size [8]byte
|
||||
binary.BigEndian.PutUint64(size[:], uint64(len(identity)))
|
||||
digest := sha256.New()
|
||||
_, _ = digest.Write(size[:])
|
||||
_, _ = digest.Write([]byte(identity))
|
||||
return "client:" + hex.EncodeToString(digest.Sum(nil))
|
||||
}
|
||||
|
||||
func nilInterface(value any) bool {
|
||||
if value == nil {
|
||||
return true
|
||||
}
|
||||
reflected := reflect.ValueOf(value)
|
||||
switch reflected.Kind() {
|
||||
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
|
||||
return reflected.IsNil()
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
223
internal/adapters/redisadmission/adapter_integration_test.go
Normal file
223
internal/adapters/redisadmission/adapter_integration_test.go
Normal file
@ -0,0 +1,223 @@
|
||||
//go:build integration
|
||||
|
||||
package redisadmission
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
|
||||
"proxy-pool/internal/platform/admission"
|
||||
)
|
||||
|
||||
var integrationNamespaceSequence atomic.Uint64
|
||||
|
||||
func TestAdaptersShareGlobalAndPerKeyLimits(t *testing.T) {
|
||||
fixture := newRedisFixture(t)
|
||||
first := fixture.adapter(t, fixture.namespace, time.Minute, 3, 2)
|
||||
second := fixture.adapter(t, fixture.namespace, time.Minute, 3, 2)
|
||||
|
||||
if err := first.Admit(t.Context(), "client-a"); err != nil {
|
||||
t.Fatalf("first Admit(client-a): %v", err)
|
||||
}
|
||||
if err := second.Admit(t.Context(), "client-a"); err != nil {
|
||||
t.Fatalf("second Admit(client-a): %v", err)
|
||||
}
|
||||
if err := first.Admit(t.Context(), "client-a"); !errors.Is(err, admission.ErrPerKeyLimit) {
|
||||
t.Fatalf("shared per-key limit error = %v, want ErrPerKeyLimit", err)
|
||||
}
|
||||
if err := second.Admit(t.Context(), "client-b"); err != nil {
|
||||
t.Fatalf("second Admit(client-b): %v", err)
|
||||
}
|
||||
if err := first.Admit(t.Context(), "client-c"); !errors.Is(err, admission.ErrGlobalLimit) {
|
||||
t.Fatalf("shared global limit error = %v, want ErrGlobalLimit", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConcurrentAdmissionsAreExactAcrossAdapters(t *testing.T) {
|
||||
fixture := newRedisFixture(t)
|
||||
tests := []struct {
|
||||
name string
|
||||
global int64
|
||||
perKey int64
|
||||
identity func(int) string
|
||||
want int64
|
||||
wantReject error
|
||||
}{
|
||||
{
|
||||
name: "global", global: 37, perKey: 1000, want: 37, wantReject: admission.ErrGlobalLimit,
|
||||
identity: func(index int) string { return fmt.Sprintf("client-%d", index) },
|
||||
},
|
||||
{
|
||||
name: "per-key", global: 1000, perKey: 23, want: 23, wantReject: admission.ErrPerKeyLimit,
|
||||
identity: func(int) string { return "shared-client" },
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
namespace := fixture.namespace + "-" + tt.name
|
||||
first := fixture.adapter(t, namespace, time.Minute, tt.global, tt.perKey)
|
||||
second := fixture.adapter(t, namespace, time.Minute, tt.global, tt.perKey)
|
||||
var accepted atomic.Int64
|
||||
var wrong atomic.Int64
|
||||
var wait sync.WaitGroup
|
||||
for index := range 200 {
|
||||
wait.Add(1)
|
||||
go func() {
|
||||
defer wait.Done()
|
||||
adapter := first
|
||||
if index%2 == 1 {
|
||||
adapter = second
|
||||
}
|
||||
err := adapter.Admit(t.Context(), tt.identity(index))
|
||||
switch {
|
||||
case err == nil:
|
||||
accepted.Add(1)
|
||||
case !errors.Is(err, tt.wantReject):
|
||||
wrong.Add(1)
|
||||
}
|
||||
}()
|
||||
}
|
||||
wait.Wait()
|
||||
if got := accepted.Load(); got != tt.want {
|
||||
t.Fatalf("accepted = %d, want %d", got, tt.want)
|
||||
}
|
||||
if got := wrong.Load(); got != 0 {
|
||||
t.Fatalf("unexpected rejection count = %d", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestWindowRolloverRemovesOldClientFields(t *testing.T) {
|
||||
fixture := newRedisFixture(t)
|
||||
const window = 100 * time.Millisecond
|
||||
adapter := fixture.adapter(t, fixture.namespace, window, 100, 10)
|
||||
if err := adapter.Admit(t.Context(), "old-client"); err != nil {
|
||||
t.Fatalf("Admit(old-client): %v", err)
|
||||
}
|
||||
if err := fixture.client.Persist(t.Context(), adapter.key).Err(); err != nil {
|
||||
t.Fatalf("PERSIST admission key: %v", err)
|
||||
}
|
||||
storedWindow, err := fixture.client.HGet(t.Context(), adapter.key, "window").Int64()
|
||||
if err != nil {
|
||||
t.Fatalf("read stored window: %v", err)
|
||||
}
|
||||
waitForRedisWindow(t, fixture.client, window, storedWindow)
|
||||
|
||||
if err := adapter.Admit(t.Context(), "new-client"); err != nil {
|
||||
t.Fatalf("Admit(new-client): %v", err)
|
||||
}
|
||||
oldField := hashedIdentityField("old-client")
|
||||
if exists, err := fixture.client.HExists(t.Context(), adapter.key, oldField).Result(); err != nil || exists {
|
||||
t.Fatalf("old client field exists = %v, error = %v", exists, err)
|
||||
}
|
||||
newField := hashedIdentityField("new-client")
|
||||
if exists, err := fixture.client.HExists(t.Context(), adapter.key, newField).Result(); err != nil || !exists {
|
||||
t.Fatalf("new client field exists = %v, error = %v", exists, err)
|
||||
}
|
||||
if length, err := fixture.client.HLen(t.Context(), adapter.key).Result(); err != nil || length != 3 {
|
||||
t.Fatalf("current window hash length = %d, error = %v, want 3", length, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNamespacesAreIsolated(t *testing.T) {
|
||||
fixture := newRedisFixture(t)
|
||||
first := fixture.adapter(t, fixture.namespace+"-a", time.Minute, 1, 1)
|
||||
second := fixture.adapter(t, fixture.namespace+"-b", time.Minute, 1, 1)
|
||||
|
||||
if err := first.Admit(t.Context(), "same-client"); err != nil {
|
||||
t.Fatalf("first namespace Admit(): %v", err)
|
||||
}
|
||||
if err := second.Admit(t.Context(), "same-client"); err != nil {
|
||||
t.Fatalf("second namespace Admit(): %v", err)
|
||||
}
|
||||
if err := first.Admit(t.Context(), "same-client"); !errors.Is(err, admission.ErrGlobalLimit) {
|
||||
t.Fatalf("first namespace second Admit() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
type redisFixture struct {
|
||||
client *redis.Client
|
||||
namespace string
|
||||
}
|
||||
|
||||
func newRedisFixture(t *testing.T) redisFixture {
|
||||
t.Helper()
|
||||
redisURL := os.Getenv("PROXY_POOL_TEST_REDIS_URL")
|
||||
if redisURL == "" {
|
||||
t.Skip("PROXY_POOL_TEST_REDIS_URL is not set")
|
||||
}
|
||||
options, err := redis.ParseURL(redisURL)
|
||||
if err != nil {
|
||||
t.Fatalf("parse PROXY_POOL_TEST_REDIS_URL: %v", err)
|
||||
}
|
||||
client := redis.NewClient(options)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
if err := client.Ping(ctx).Err(); err != nil {
|
||||
_ = client.Close()
|
||||
t.Fatalf("ping Redis: %v", err)
|
||||
}
|
||||
namespace := fmt.Sprintf("admission-it-%d-%d-%d", os.Getpid(), time.Now().UnixNano(), integrationNamespaceSequence.Add(1))
|
||||
t.Cleanup(func() {
|
||||
cleanupCtx, cleanupCancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cleanupCancel()
|
||||
pattern := redisAdmissionKey + namespace + "*"
|
||||
var cursor uint64
|
||||
for {
|
||||
keys, next, scanErr := client.Scan(cleanupCtx, cursor, pattern, 128).Result()
|
||||
if scanErr != nil {
|
||||
t.Errorf("scan Redis admission keys: %v", scanErr)
|
||||
break
|
||||
}
|
||||
if len(keys) > 0 {
|
||||
if unlinkErr := client.Unlink(cleanupCtx, keys...).Err(); unlinkErr != nil {
|
||||
t.Errorf("remove Redis admission keys: %v", unlinkErr)
|
||||
break
|
||||
}
|
||||
}
|
||||
cursor = next
|
||||
if cursor == 0 {
|
||||
break
|
||||
}
|
||||
}
|
||||
_ = client.Close()
|
||||
})
|
||||
return redisFixture{client: client, namespace: namespace}
|
||||
}
|
||||
|
||||
func (fixture redisFixture) adapter(t *testing.T, namespace string, window time.Duration, global, perKey int64) *Adapter {
|
||||
t.Helper()
|
||||
return mustAdapter(t, fixture.client, Options{
|
||||
Namespace: namespace,
|
||||
Window: window,
|
||||
Global: global,
|
||||
PerKey: perKey,
|
||||
})
|
||||
}
|
||||
|
||||
func waitForRedisWindow(t *testing.T, client *redis.Client, window time.Duration, previous int64) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(2 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
redisTime, err := client.Time(t.Context()).Result()
|
||||
if err != nil {
|
||||
t.Fatalf("Redis TIME: %v", err)
|
||||
}
|
||||
windowID := redisTime.UnixMilli() / window.Milliseconds()
|
||||
if windowID != previous {
|
||||
return
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
t.Fatal("timed out waiting for Redis window rollover")
|
||||
}
|
||||
225
internal/adapters/redisadmission/adapter_test.go
Normal file
225
internal/adapters/redisadmission/adapter_test.go
Normal file
@ -0,0 +1,225 @@
|
||||
package redisadmission
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/binary"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
|
||||
"proxy-pool/internal/platform/admission"
|
||||
)
|
||||
|
||||
type scriptCall struct {
|
||||
keys []string
|
||||
args []any
|
||||
}
|
||||
|
||||
type scriptClient struct {
|
||||
redis.Scripter
|
||||
result any
|
||||
err error
|
||||
calls []scriptCall
|
||||
}
|
||||
|
||||
func (client *scriptClient) EvalSha(_ context.Context, _ string, keys []string, args ...any) *redis.Cmd {
|
||||
client.calls = append(client.calls, scriptCall{keys: append([]string(nil), keys...), args: append([]any(nil), args...)})
|
||||
return redis.NewCmdResult(client.result, client.err)
|
||||
}
|
||||
|
||||
func TestNewRejectsInvalidDependenciesAndOptions(t *testing.T) {
|
||||
t.Parallel()
|
||||
validClient := &scriptClient{result: "ok"}
|
||||
valid := Options{Namespace: "listener-a", Window: time.Minute, Global: 10, PerKey: 2}
|
||||
var typedNilClient *redis.Client
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
client redis.Scripter
|
||||
options Options
|
||||
}{
|
||||
{name: "nil client", options: valid},
|
||||
{name: "typed nil client", client: typedNilClient, options: valid},
|
||||
{name: "empty namespace", client: validClient, options: withNamespace(valid, "")},
|
||||
{name: "leading namespace whitespace", client: validClient, options: withNamespace(valid, " listener-a")},
|
||||
{name: "trailing namespace whitespace", client: validClient, options: withNamespace(valid, "listener-a ")},
|
||||
{name: "cluster tag in namespace", client: validClient, options: withNamespace(valid, "listener{a}")},
|
||||
{name: "separator in namespace", client: validClient, options: withNamespace(valid, "listener:a")},
|
||||
{name: "zero window", client: validClient, options: withWindow(valid, 0)},
|
||||
{name: "sub-millisecond window", client: validClient, options: withWindow(valid, time.Microsecond)},
|
||||
{name: "fractional millisecond window", client: validClient, options: withWindow(valid, time.Millisecond+time.Microsecond)},
|
||||
{name: "negative global", client: validClient, options: withGlobal(valid, -1)},
|
||||
{name: "negative per-key", client: validClient, options: withPerKey(valid, -1)},
|
||||
{name: "zero limits", client: validClient, options: withLimits(valid, 0, 0)},
|
||||
{name: "global exceeds Lua integer", client: validClient, options: withGlobal(valid, MaximumLuaInteger+1)},
|
||||
{name: "per-key exceeds Lua integer", client: validClient, options: withPerKey(valid, MaximumLuaInteger+1)},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
if adapter, err := New(tt.client, tt.options); !errors.Is(err, ErrInvalidOptions) || adapter != nil {
|
||||
t.Fatalf("New() = (%v, %v), want nil adapter and ErrInvalidOptions", adapter, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdmitMapsScriptStatusAndPreservesInputErrors(t *testing.T) {
|
||||
t.Parallel()
|
||||
tests := []struct {
|
||||
name string
|
||||
status any
|
||||
want error
|
||||
}{
|
||||
{name: "accepted", status: "ok"},
|
||||
{name: "global limit", status: "global", want: admission.ErrGlobalLimit},
|
||||
{name: "per-key limit", status: "per_key", want: admission.ErrPerKeyLimit},
|
||||
{name: "unexpected status", status: "unknown", want: admission.ErrUnavailable},
|
||||
{name: "unexpected reply type", status: int64(1), want: admission.ErrUnavailable},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
client := &scriptClient{result: tt.status}
|
||||
adapter := mustAdapter(t, client, Options{Namespace: "listener-a", Window: time.Minute, Global: 10, PerKey: 2})
|
||||
err := adapter.Admit(context.Background(), "client-a")
|
||||
if !errors.Is(err, tt.want) {
|
||||
t.Fatalf("Admit() error = %v, want %v", err, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
adapter := mustAdapter(t, &scriptClient{result: "ok"}, Options{Namespace: "listener-a", Window: time.Minute, Global: 1})
|
||||
if err := adapter.Admit(nil, "client-a"); !errors.Is(err, admission.ErrInvalidIdentity) {
|
||||
t.Fatalf("Admit(nil context) error = %v, want ErrInvalidIdentity", err)
|
||||
}
|
||||
if err := adapter.Admit(context.Background(), ""); !errors.Is(err, admission.ErrInvalidIdentity) {
|
||||
t.Fatalf("Admit(empty identity) error = %v, want ErrInvalidIdentity", err)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if err := adapter.Admit(ctx, "client-a"); !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("Admit(canceled context) error = %v, want context.Canceled", err)
|
||||
}
|
||||
var nilAdapter *Adapter
|
||||
if err := nilAdapter.Admit(context.Background(), "client-a"); !errors.Is(err, admission.ErrUnavailable) {
|
||||
t.Fatalf("nil Adapter.Admit() error = %v, want ErrUnavailable", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdmitUsesHashedIdentityAndExactIntegerArguments(t *testing.T) {
|
||||
t.Parallel()
|
||||
client := &scriptClient{result: "ok"}
|
||||
adapter := mustAdapter(t, client, Options{
|
||||
Namespace: "listener-a", Window: 1500 * time.Millisecond,
|
||||
Global: MaximumLuaInteger, PerKey: MaximumLuaInteger - 1,
|
||||
})
|
||||
const identity = "private-client@example.test"
|
||||
if err := adapter.Admit(context.Background(), identity); err != nil {
|
||||
t.Fatalf("Admit(): %v", err)
|
||||
}
|
||||
if len(client.calls) != 1 {
|
||||
t.Fatalf("script calls = %d, want 1", len(client.calls))
|
||||
}
|
||||
call := client.calls[0]
|
||||
if len(call.keys) != 1 || call.keys[0] != "pp:{admission}:listener-a:window" {
|
||||
t.Fatalf("script keys = %q", call.keys)
|
||||
}
|
||||
wantArgs := []string{"1500", fmt.Sprint(MaximumLuaInteger), fmt.Sprint(MaximumLuaInteger - 1), clientField(identity)}
|
||||
if got := stringify(call.args); fmt.Sprint(got) != fmt.Sprint(wantArgs) {
|
||||
t.Fatalf("script args = %q, want %q", got, wantArgs)
|
||||
}
|
||||
serialized := fmt.Sprint(call.keys, call.args)
|
||||
if strings.Contains(serialized, identity) {
|
||||
t.Fatalf("Redis input leaks raw identity: %s", serialized)
|
||||
}
|
||||
if got := clientField(identity); len(got) != len("client:")+sha256.Size*2 || !strings.HasPrefix(got, "client:") {
|
||||
t.Fatalf("client field = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdmitFailsClosedWithoutLeakingIdentity(t *testing.T) {
|
||||
t.Parallel()
|
||||
const identity = "sensitive-client"
|
||||
client := &scriptClient{err: errors.New("backend failure: " + identity)}
|
||||
adapter := mustAdapter(t, client, Options{Namespace: "listener-a", Window: time.Minute, Global: 1})
|
||||
err := adapter.Admit(context.Background(), identity)
|
||||
if !errors.Is(err, admission.ErrUnavailable) {
|
||||
t.Fatalf("Admit() error = %v, want ErrUnavailable", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), identity) {
|
||||
t.Fatalf("Admit() error leaks identity: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdmitFailsClosedForUninitializedAdapter(t *testing.T) {
|
||||
t.Parallel()
|
||||
var typedNilClient *redis.Client
|
||||
tests := []*Adapter{
|
||||
{},
|
||||
{client: typedNilClient, key: "pp:{admission}:test:window", windowMillis: 60_000, global: 1},
|
||||
}
|
||||
for _, adapter := range tests {
|
||||
if err := adapter.Admit(context.Background(), "client-a"); !errors.Is(err, admission.ErrUnavailable) {
|
||||
t.Fatalf("Admit() error = %v, want ErrUnavailable", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func clientField(identity string) string {
|
||||
var size [8]byte
|
||||
binary.BigEndian.PutUint64(size[:], uint64(len(identity)))
|
||||
digest := sha256.New()
|
||||
_, _ = digest.Write(size[:])
|
||||
_, _ = digest.Write([]byte(identity))
|
||||
return fmt.Sprintf("client:%x", digest.Sum(nil))
|
||||
}
|
||||
|
||||
func stringify(values []any) []string {
|
||||
result := make([]string, len(values))
|
||||
for index, value := range values {
|
||||
result[index] = fmt.Sprint(value)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func mustAdapter(t *testing.T, client redis.Scripter, options Options) *Adapter {
|
||||
t.Helper()
|
||||
adapter, err := New(client, options)
|
||||
if err != nil {
|
||||
t.Fatalf("New(): %v", err)
|
||||
}
|
||||
return adapter
|
||||
}
|
||||
|
||||
func withNamespace(options Options, namespace string) Options {
|
||||
options.Namespace = namespace
|
||||
return options
|
||||
}
|
||||
|
||||
func withWindow(options Options, window time.Duration) Options {
|
||||
options.Window = window
|
||||
return options
|
||||
}
|
||||
|
||||
func withGlobal(options Options, limit int64) Options {
|
||||
options.Global = limit
|
||||
return options
|
||||
}
|
||||
|
||||
func withPerKey(options Options, limit int64) Options {
|
||||
options.PerKey = limit
|
||||
return options
|
||||
}
|
||||
|
||||
func withLimits(options Options, global, perKey int64) Options {
|
||||
options.Global = global
|
||||
options.PerKey = perKey
|
||||
return options
|
||||
}
|
||||
39
internal/adapters/redisadmission/scripts/fixed_window.lua
Normal file
39
internal/adapters/redisadmission/scripts/fixed_window.lua
Normal file
@ -0,0 +1,39 @@
|
||||
local key = KEYS[1]
|
||||
local window_ms = tonumber(ARGV[1])
|
||||
local global_limit = tonumber(ARGV[2])
|
||||
local per_key_limit = tonumber(ARGV[3])
|
||||
local client_field = ARGV[4]
|
||||
|
||||
local redis_time = redis.call("TIME")
|
||||
local now_ms = tonumber(redis_time[1]) * 1000 + math.floor(tonumber(redis_time[2]) / 1000)
|
||||
local window_id = math.floor(now_ms / window_ms)
|
||||
local stored_window = redis.call("HGET", key, "window")
|
||||
|
||||
if not stored_window or tonumber(stored_window) ~= window_id then
|
||||
redis.call("DEL", key)
|
||||
redis.call("HSET", key, "window", window_id)
|
||||
end
|
||||
|
||||
if global_limit > 0 then
|
||||
local global_used = tonumber(redis.call("HGET", key, "global") or "0")
|
||||
if global_used >= global_limit then
|
||||
return "global"
|
||||
end
|
||||
end
|
||||
|
||||
if per_key_limit > 0 then
|
||||
local client_used = tonumber(redis.call("HGET", key, client_field) or "0")
|
||||
if client_used >= per_key_limit then
|
||||
return "per_key"
|
||||
end
|
||||
end
|
||||
|
||||
if global_limit > 0 then
|
||||
redis.call("HINCRBY", key, "global", 1)
|
||||
end
|
||||
if per_key_limit > 0 then
|
||||
redis.call("HINCRBY", key, client_field, 1)
|
||||
end
|
||||
|
||||
redis.call("PEXPIREAT", key, (window_id + 1) * window_ms)
|
||||
return "ok"
|
||||
@ -428,6 +428,13 @@ func TestValidateRejectsInvalidConfigurationMatrix(t *testing.T) {
|
||||
},
|
||||
want: "requestsPerMinute",
|
||||
},
|
||||
{
|
||||
name: "listener request limit exceeds exact counter range",
|
||||
mutate: func(cfg *Config) {
|
||||
cfg.Distribution.Limits.RequestsPerMinutePerClient = int(MaximumExactCounter) + 1
|
||||
},
|
||||
want: "requestsPerMinutePerClient",
|
||||
},
|
||||
{
|
||||
name: "invalid client identification mode",
|
||||
mutate: func(cfg *Config) {
|
||||
|
||||
@ -122,6 +122,9 @@ func validateListener(name string, listener Listener, security Security) error {
|
||||
if limit.value < 0 {
|
||||
return fmt.Errorf("validate %s limits.%s: must be non-negative", name, limit.name)
|
||||
}
|
||||
if int64(limit.value) > MaximumExactCounter {
|
||||
return fmt.Errorf("validate %s limits.%s: exceeds exact counter range", name, limit.name)
|
||||
}
|
||||
}
|
||||
host, err := validateListenAddress(name, listener.Listen)
|
||||
if err != nil {
|
||||
|
||||
@ -55,6 +55,7 @@ type ports struct {
|
||||
activity activityStore
|
||||
readiness distribution.ReadinessChecker
|
||||
metricsReadiness platformMetrics.ReadinessChecker
|
||||
admission admission.Admitter
|
||||
coordinator provider.Coordinator
|
||||
credentials credentials.Store
|
||||
providerResults provider.ResultRecorder
|
||||
@ -145,10 +146,10 @@ func run(ctx context.Context, options Options, infrastructure infrastructure, fa
|
||||
|
||||
dependencies := controllerRuntime.Dependencies{}
|
||||
if loaded.Value.Distribution.Enabled {
|
||||
if nilInterface(opened.activity) || nilInterface(opened.readiness) {
|
||||
if nilInterface(opened.activity) || nilInterface(opened.readiness) || nilInterface(opened.admission) {
|
||||
return errors.Join(ErrStartup, ErrInvalidOptions)
|
||||
}
|
||||
service, serviceErr := extraction.NewService(opened.activity, extractionPolicy(loaded.Value), admission.AllowAll{}, options.Now)
|
||||
service, serviceErr := extraction.NewService(opened.activity, extractionPolicy(loaded.Value), opened.admission, options.Now)
|
||||
if serviceErr != nil {
|
||||
return fmt.Errorf("%w: build extraction service: %w", ErrStartup, serviceErr)
|
||||
}
|
||||
|
||||
@ -16,6 +16,7 @@ import (
|
||||
"proxy-pool/internal/domain/adminstate"
|
||||
extractionDomain "proxy-pool/internal/domain/extraction"
|
||||
"proxy-pool/internal/domain/upstream"
|
||||
"proxy-pool/internal/platform/admission"
|
||||
"proxy-pool/internal/platform/credentials"
|
||||
)
|
||||
|
||||
@ -33,6 +34,7 @@ func TestRunLoadsOneSnapshotCommitsItAndClosesInfrastructure(t *testing.T) {
|
||||
closeErr := errors.New("close failed")
|
||||
infrastructure := &stubInfrastructure{ports: ports{
|
||||
state: state, activity: activity, readiness: readyStub{}, metricsReadiness: readyStub{},
|
||||
admission: admission.AllowAll{},
|
||||
coordinator: coordinatorStub{}, credentials: credentialStore,
|
||||
close: func() error { return closeErr },
|
||||
}}
|
||||
@ -113,6 +115,33 @@ func TestRunRejectsMissingAdminFingerprintKeyBeforeOpeningInfrastructure(t *test
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunRejectsMissingDistributionAdmissionDependency(t *testing.T) {
|
||||
t.Parallel()
|
||||
credentialStore, err := credentials.NewMemoryStore(10)
|
||||
if err != nil {
|
||||
t.Fatalf("NewMemoryStore(): %v", err)
|
||||
}
|
||||
infrastructure := &stubInfrastructure{ports: ports{
|
||||
state: adminstate.NewMemoryStore(), activity: &stubActivityStore{},
|
||||
readiness: readyStub{}, metricsReadiness: readyStub{},
|
||||
coordinator: coordinatorStub{}, credentials: credentialStore,
|
||||
close: func() error { return nil },
|
||||
}}
|
||||
factory := &recordingRuntimeFactory{runner: runnerStub{err: errors.New("runtime should not start")}}
|
||||
err = run(context.Background(), Options{
|
||||
ConfigPath: "controller.yaml",
|
||||
Resolver: &memoryResolver{files: map[string][]byte{"controller.yaml": []byte(bootstrapTestConfig)}},
|
||||
Now: time.Now,
|
||||
FingerprintKey: bootstrapTestFingerprintKey,
|
||||
}, infrastructure, factory)
|
||||
if !errors.Is(err, ErrStartup) || !errors.Is(err, ErrInvalidOptions) {
|
||||
t.Fatalf("run() error = %v, want ErrStartup and ErrInvalidOptions", err)
|
||||
}
|
||||
if factory.configuration != nil {
|
||||
t.Fatal("runtime factory called without admission dependency")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunSupportsProviderOnlyConfigurationWithoutHTTPRuntime(t *testing.T) {
|
||||
source := strings.ReplaceAll(bootstrapTestConfig, "distribution:\n enabled: true", "distribution:\n enabled: false")
|
||||
source = strings.ReplaceAll(source, "admin:\n enabled: true", "admin:\n enabled: false")
|
||||
@ -151,6 +180,7 @@ func TestRunAdminDisableStopsActiveProviderRuntime(t *testing.T) {
|
||||
stopped := make(chan string, 2)
|
||||
infrastructure := &stubInfrastructure{ports: ports{
|
||||
state: state, activity: &stubActivityStore{}, readiness: readyStub{}, metricsReadiness: readyStub{},
|
||||
admission: admission.AllowAll{},
|
||||
coordinator: coordinatorFunc(func(ctx context.Context, upstreamID string) error {
|
||||
started <- upstreamID
|
||||
<-ctx.Done()
|
||||
|
||||
@ -13,9 +13,11 @@ import (
|
||||
|
||||
"proxy-pool/internal/adapters/postgresadmin"
|
||||
"proxy-pool/internal/adapters/redisactivity"
|
||||
"proxy-pool/internal/adapters/redisadmission"
|
||||
"proxy-pool/internal/adapters/redisprovider"
|
||||
"proxy-pool/internal/config"
|
||||
controllerProvider "proxy-pool/internal/controller/provider"
|
||||
"proxy-pool/internal/platform/admission"
|
||||
"proxy-pool/internal/platform/credentials"
|
||||
platformMetrics "proxy-pool/internal/platform/metrics"
|
||||
)
|
||||
@ -131,6 +133,16 @@ func (infrastructure *productionInfrastructure) Open(
|
||||
opened.activity = adapter
|
||||
opened.readiness = redisReadiness{client: redisClient}
|
||||
opened.credentials = credentialStore
|
||||
if configuration.Distribution.Enabled {
|
||||
opened.admission, err = newDistributionAdmitter(
|
||||
redisClient,
|
||||
namespace,
|
||||
configuration.Distribution.Limits,
|
||||
)
|
||||
if err != nil {
|
||||
return ports{}, err
|
||||
}
|
||||
}
|
||||
if providersEnabled {
|
||||
stats, statsErr := controllerProvider.NewStatsRecorder(config.MaximumUpstreams)
|
||||
if statsErr != nil {
|
||||
@ -161,6 +173,22 @@ func (infrastructure *productionInfrastructure) Open(
|
||||
return opened, nil
|
||||
}
|
||||
|
||||
func newDistributionAdmitter(
|
||||
client redis.Scripter,
|
||||
namespace string,
|
||||
limits config.Limits,
|
||||
) (admission.Admitter, error) {
|
||||
if limits.RequestsPerMinute == 0 && limits.RequestsPerMinutePerClient == 0 {
|
||||
return admission.AllowAll{}, nil
|
||||
}
|
||||
return redisadmission.New(client, redisadmission.Options{
|
||||
Namespace: namespace,
|
||||
Window: time.Minute,
|
||||
Global: int64(limits.RequestsPerMinute),
|
||||
PerKey: int64(limits.RequestsPerMinutePerClient),
|
||||
})
|
||||
}
|
||||
|
||||
func resolveRedisNamespace(configured string) (string, error) {
|
||||
if strings.TrimSpace(configured) != configured {
|
||||
return "", ErrInvalidOptions
|
||||
|
||||
@ -3,13 +3,34 @@ package bootstrap
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
|
||||
"proxy-pool/internal/config"
|
||||
"proxy-pool/internal/platform/admission"
|
||||
)
|
||||
|
||||
type recordingAdmissionScripter struct {
|
||||
redis.Scripter
|
||||
keys []string
|
||||
args []any
|
||||
}
|
||||
|
||||
func (client *recordingAdmissionScripter) EvalSha(
|
||||
_ context.Context,
|
||||
_ string,
|
||||
keys []string,
|
||||
args ...any,
|
||||
) *redis.Cmd {
|
||||
client.keys = append([]string(nil), keys...)
|
||||
client.args = append([]any(nil), args...)
|
||||
return redis.NewCmdResult("ok", nil)
|
||||
}
|
||||
|
||||
func TestProductionInfrastructureRejectsInvalidStorageWithoutLeakingURLs(t *testing.T) {
|
||||
t.Parallel()
|
||||
postgresSecret := "postgres-secret"
|
||||
@ -31,6 +52,41 @@ func TestProductionInfrastructureRejectsInvalidStorageWithoutLeakingURLs(t *test
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewDistributionAdmitterPassesConfiguredLimits(t *testing.T) {
|
||||
t.Parallel()
|
||||
client := &recordingAdmissionScripter{}
|
||||
limiter, err := newDistributionAdmitter(client, "controller-a", config.Limits{
|
||||
RequestsPerMinute: 321,
|
||||
RequestsPerMinutePerClient: 17,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("newDistributionAdmitter(): %v", err)
|
||||
}
|
||||
if err := limiter.Admit(context.Background(), "client-a"); err != nil {
|
||||
t.Fatalf("Admit(): %v", err)
|
||||
}
|
||||
if len(client.keys) != 1 || len(client.args) != 4 {
|
||||
t.Fatalf("Redis admission call = keys:%v args:%v", client.keys, client.args)
|
||||
}
|
||||
if got := fmt.Sprintf("%s|%v|%v|%v", client.keys[0], client.args[0], client.args[1], client.args[2]); got != "pp:{admission}:controller-a:window|60000|321|17" {
|
||||
t.Fatalf("Redis admission inputs = %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewDistributionAdmitterAllowsAllWhenQuotasAreDisabled(t *testing.T) {
|
||||
t.Parallel()
|
||||
limiter, err := newDistributionAdmitter(nil, "controller-a", config.Limits{})
|
||||
if err != nil {
|
||||
t.Fatalf("newDistributionAdmitter(): %v", err)
|
||||
}
|
||||
if _, ok := limiter.(admission.AllowAll); !ok {
|
||||
t.Fatalf("limiter type = %T, want admission.AllowAll", limiter)
|
||||
}
|
||||
if err := limiter.Admit(context.Background(), "client-a"); err != nil {
|
||||
t.Fatalf("AllowAll.Admit(): %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSelectMetricsReadinessPreservesDistributionWhenAdminStoreFails(t *testing.T) {
|
||||
t.Parallel()
|
||||
adminCalls := &atomic.Int64{}
|
||||
|
||||
@ -7,6 +7,7 @@ import (
|
||||
"time"
|
||||
|
||||
domain "proxy-pool/internal/domain/extraction"
|
||||
"proxy-pool/internal/platform/admission"
|
||||
)
|
||||
|
||||
var (
|
||||
@ -70,16 +71,12 @@ type Response struct {
|
||||
type Service struct {
|
||||
store domain.Store
|
||||
policy Policy
|
||||
admission Admission
|
||||
admission admission.Admitter
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
type Admission interface {
|
||||
Admit(context.Context, string) error
|
||||
}
|
||||
|
||||
func NewService(store domain.Store, policy Policy, admission Admission, now func() time.Time) (*Service, error) {
|
||||
if admission == nil {
|
||||
func NewService(store domain.Store, policy Policy, admitter admission.Admitter, now func() time.Time) (*Service, error) {
|
||||
if admitter == nil {
|
||||
return nil, fmt.Errorf("%w: admission is required", ErrInvalidServicePolicy)
|
||||
}
|
||||
if store == nil {
|
||||
@ -95,7 +92,7 @@ func NewService(store domain.Store, policy Policy, admission Admission, now func
|
||||
if now == nil {
|
||||
now = time.Now
|
||||
}
|
||||
return &Service{store: store, policy: policy, admission: admission, now: now}, nil
|
||||
return &Service{store: store, policy: policy, admission: admitter, now: now}, nil
|
||||
}
|
||||
|
||||
func (s *Service) Extract(ctx context.Context, request Request) (Response, error) {
|
||||
@ -114,6 +111,9 @@ func (s *Service) Extract(ctx context.Context, request Request) (Response, error
|
||||
return response, ErrInvalidFulfillment
|
||||
}
|
||||
if err := s.admission.Admit(ctx, admissionKey(request)); err != nil {
|
||||
if errors.Is(err, admission.ErrUnavailable) {
|
||||
return response, errors.Join(ErrUnavailable, err)
|
||||
}
|
||||
return response, errors.Join(ErrAdmissionRejected, err)
|
||||
}
|
||||
|
||||
|
||||
@ -9,6 +9,7 @@ import (
|
||||
"proxy-pool/internal/domain/activitypool"
|
||||
domain "proxy-pool/internal/domain/extraction"
|
||||
proxyDomain "proxy-pool/internal/domain/proxy"
|
||||
platformAdmission "proxy-pool/internal/platform/admission"
|
||||
)
|
||||
|
||||
func TestServiceAppliesPolicyAndBuildsResponse(t *testing.T) {
|
||||
@ -122,6 +123,30 @@ func TestServiceAppliesAdmissionBeforeStoreUsingStableIdentity(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestServiceMapsUnavailableAdmissionToServiceUnavailable(t *testing.T) {
|
||||
store := &recordingStore{}
|
||||
admitter := &recordingAdmission{err: platformAdmission.ErrUnavailable}
|
||||
service, err := NewService(store, Policy{
|
||||
MaxCountPerRequest: 1,
|
||||
DefaultFulfillment: domain.Partial,
|
||||
}, admitter, time.Now)
|
||||
if err != nil {
|
||||
t.Fatalf("NewService(): %v", err)
|
||||
}
|
||||
|
||||
_, err = service.Extract(context.Background(), Request{
|
||||
RequestID: "req-1",
|
||||
ClientID: "client-1",
|
||||
Count: 1,
|
||||
})
|
||||
if !errors.Is(err, ErrUnavailable) || errors.Is(err, ErrAdmissionRejected) {
|
||||
t.Fatalf("Extract() error = %v, want only ErrUnavailable", err)
|
||||
}
|
||||
if store.calls != 0 {
|
||||
t.Fatalf("store calls = %d, want 0", store.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestServiceUsesSourceIdentityForEphemeralIdempotency(t *testing.T) {
|
||||
store := &recordingStore{}
|
||||
service, err := NewService(store, Policy{
|
||||
|
||||
9
internal/platform/admission/admitter.go
Normal file
9
internal/platform/admission/admitter.go
Normal file
@ -0,0 +1,9 @@
|
||||
package admission
|
||||
|
||||
import "context"
|
||||
|
||||
// Admitter is the shared admission-control contract used by controller
|
||||
// services and infrastructure adapters.
|
||||
type Admitter interface {
|
||||
Admit(context.Context, string) error
|
||||
}
|
||||
@ -15,6 +15,9 @@ func TestAllowAllPreservesContextAndIdentityValidation(t *testing.T) {
|
||||
if err := admission.Admit(context.Background(), ""); !errors.Is(err, ErrInvalidIdentity) {
|
||||
t.Fatalf("Admit(empty identity) error = %v", err)
|
||||
}
|
||||
if err := admission.Admit(nil, "client-a"); !errors.Is(err, ErrInvalidIdentity) {
|
||||
t.Fatalf("Admit(nil context) error = %v", err)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if err := admission.Admit(ctx, "client-a"); !errors.Is(err, context.Canceled) {
|
||||
|
||||
@ -12,6 +12,7 @@ var (
|
||||
ErrInvalidIdentity = errors.New("invalid admission identity")
|
||||
ErrGlobalLimit = errors.New("global admission limit exceeded")
|
||||
ErrPerKeyLimit = errors.New("per-key admission limit exceeded")
|
||||
ErrUnavailable = errors.New("admission control unavailable")
|
||||
)
|
||||
|
||||
type FixedWindowConfig struct {
|
||||
@ -54,12 +55,12 @@ func NewFixedWindow(config FixedWindowConfig) (*FixedWindow, error) {
|
||||
}
|
||||
|
||||
func (l *FixedWindow) Admit(ctx context.Context, key string) error {
|
||||
if ctx == nil || l == nil || key == "" {
|
||||
return ErrInvalidIdentity
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
if l == nil || key == "" {
|
||||
return ErrInvalidIdentity
|
||||
}
|
||||
|
||||
l.mu.Lock()
|
||||
defer l.mu.Unlock()
|
||||
|
||||
@ -74,3 +74,27 @@ func TestFixedWindowResetsAndIsConcurrencySafe(t *testing.T) {
|
||||
t.Fatalf("Admit(after reset): %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFixedWindowRejectsNilContextAndEmptyIdentity(t *testing.T) {
|
||||
t.Parallel()
|
||||
limiter, err := NewFixedWindow(FixedWindowConfig{
|
||||
Window: time.Minute,
|
||||
Global: 1,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewFixedWindow(): %v", err)
|
||||
}
|
||||
|
||||
if err := limiter.Admit(nil, "client-a"); !errors.Is(err, ErrInvalidIdentity) {
|
||||
t.Fatalf("Admit(nil context) error = %v, want ErrInvalidIdentity", err)
|
||||
}
|
||||
if err := limiter.Admit(context.Background(), ""); !errors.Is(err, ErrInvalidIdentity) {
|
||||
t.Fatalf("Admit(empty identity) error = %v, want ErrInvalidIdentity", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdmitterImplementations(t *testing.T) {
|
||||
t.Parallel()
|
||||
var _ Admitter = (*FixedWindow)(nil)
|
||||
var _ Admitter = AllowAll{}
|
||||
}
|
||||
|
||||
@ -14,7 +14,7 @@ try {
|
||||
$env:PROXY_POOL_TEST_REDIS_URL = "redis://127.0.0.1:16379/15"
|
||||
Push-Location $repositoryRoot
|
||||
try {
|
||||
go test -count=1 -tags=integration -timeout 60s ./internal/adapters/redisactivity/... ./internal/adapters/redisprovider/...
|
||||
go test -count=1 -tags=integration -timeout 60s ./internal/adapters/redisactivity/... ./internal/adapters/redisadmission/... ./internal/adapters/redisprovider/...
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "Redis integration tests failed with exit code $LASTEXITCODE"
|
||||
}
|
||||
|
||||
Loading…
Reference in New Issue
Block a user