Compare commits

..

No commits in common. "c3b5b2559737ef1fd578cd99bc91c87f1b360629" and "b8f51041672089616f2c5b4cc5e56f36e8125786" have entirely different histories.

25 changed files with 28 additions and 859 deletions

View File

@ -154,7 +154,7 @@ Redis Adapter 应通过单个 Lua 脚本、Redis Function 或等价的原子原
- 两个并发成功响应的 Proxy ID 集合交集为空。
- 原子操作失败时整个批次不返回,也不得留下部分移除结果。
- `allOrNothing` 不足时零个条目退出活动池。
- Redis 活动池或分布式限流不可用时返回 503不以内存副本冒充成功。
- Redis 活动池不可用时返回 503不以内存副本冒充成功。
- PostgreSQL 不可用不阻断提取;需要 PostgreSQL 的 Admin 管理写入单独降级。
## 7. 幂等
@ -175,9 +175,7 @@ TTL 到期或 Redis 数据丢失后不再保证旧 Key 去重,系统不回退
- 直连请求使用来源 IP 形成匿名 Client。
- 只有来源属于 `trustedProxies` 时才接受转发头。
- 全局和每 Client 限流在查询库存前执行本地监听器先做早期拒绝Redis 使用
服务端时间和单个 Lua 原子操作执行跨 Controller 副本的权威额度。
- Client 身份进入 Redis 前转换为定长摘要,窗口切换时原子删除上一窗口字段。
- 全局和每 Client 限流在查询库存前执行。
- 过滤条件、数量、请求体和 Header 都有长度/数量上限。
## 9. 错误模型
@ -192,7 +190,7 @@ TTL 到期或 Redis 数据丢失后不再保证旧 Key 去重,系统不回退
- `415`:请求体不是 `application/json`
- `422`:数量、枚举或过滤组合违反业务约束。
- `429`:全局或 Client 速率限制,响应 `Retry-After`
- `503`Redis 活动池/分布式限流不可用、原子提取不可执行或服务正在排空。
- `503`Redis 活动池不可用、原子提取不可执行或服务正在排空。
- `500`:未分类的内部错误;响应不包含底层错误文本。
错误响应不得包含 Provider Secret、Proxy 凭据、SQL 或内部拓扑。

View File

@ -104,10 +104,6 @@ limits:
`trustedProxies` 只决定何时接受 `Forwarded``X-Forwarded-For`,不能替代
`allowCIDRs`。来自非可信代理的转发头必须忽略。
`requestsPerMinute``requestsPerMinutePerClient` 为非负整数,最大值为
`2^53-1`Distribution 的非零额度由 Redis Lua 计数,因此配置校验统一限制在
Lua 可精确表示的整数范围内。
### 3.1 认证模式
- `none`:无身份认证,访问控制与限流仍生效。

View File

@ -174,9 +174,8 @@ Upstream、`endBehavior` 默认 `stop`并覆盖列表末端停止disabled
candidate eligibility, Gateway reserve, ownership, removal, and short-lived idempotency.
- [x] Implement Redis Worker ownership, drain/ACK, expiry reclaim, inventory and bounded
sweep primitives with a monotonic global epoch.
- [x] Implement Redis Provider leader, distributed request quota, Client limit and
automatic Provider inventory rebuild after Redis state loss.
- [ ] Implement the Worker heartbeat receiving path and session lifecycle.
- [ ] Implement Redis Provider leader, distributed rate, Client limit and Worker
heartbeat; wire automatic Provider inventory rebuild after Redis loss.
- [x] Keep Provider output in Redis TTL activity state and node memory only; keep the
Gateway request path on immutable local snapshots with no Redis/PostgreSQL calls.
- [x] Expose Distribution extraction/status and Admin status/enable/disable/switch/reload
@ -237,13 +236,9 @@ Redis Provider Permit 现已把 requestInterval、maxInFlight 与 maxTotal 放
启动后动态扩容。Supervisor 以 PostgreSQL 权威 HMAC 指纹和 revision 栅栏协调
多副本 reload管理库瞬断沿用 last-known 状态,本地共享源落后时停止旧 Provider
源匹配并预检后自动替换,迟到旧 revision 不覆盖新配置。
Distribution 现通过公用 `admission.Admitter` 接入独立 `redisadmission` Adapter
全局和单 Client 分钟额度使用 Redis 服务端时间并在单个 Lua 原子边界内检查、递增,
Controller 多副本共享同一计数。Client 身份只以 SHA-256 摘要进入 Redis窗口切换
原子回收历史字段Redis 异常 fail-closed 并返回 503真实额度耗尽返回 429。
Gateway 请求热路径仍只使用本地准入,不增加 Redis/PostgreSQL 调用。
WorkerControlPlane gRPC 接收端、session 签发/心跳、Snapshot ACK
账本和健康执行链仍待完成,因此 Task 10 尚未全部完成。
账本、Client 分布式限流和健康执行链仍待完成,因此本轮不勾选 Task 10 的组合
验收项。
## Task 11: Checker and Health Reducer

View File

@ -240,7 +240,7 @@ Outbox 发布器必须以稳定 consumer ID 有界领取;发布成功后原子
回查 PostgreSQL 或 Provider。
2. Distribution 立即失败关闭并返回 503禁止本地内存提取或 PostgreSQL 兜底。
3. Controller 停止活动池写入和需要分布式互斥的工作,防止多个 Fetch Leader
Distribution 的 Redis 权威限流同时失败关闭,本地早期限流不冒充跨副本额度。
本地限流不能声称满足全局额度。
4. 恢复后确认 Leader 唯一和租约 epoch 单调,由 Provider 重新 Fetch 并构建 TTL
活动池,再恢复 Distribution。Redis 整体丢失会终止原活动池代次的排他状态和
短期幂等窗口;高可用、持久化、监控和告警必须明确并降低该风险。

View File

@ -84,11 +84,11 @@ CI 已配置 Linux race job。PostgreSQL 18 和 Redis 8.2 的隔离 Adapter fixt
4. Controller 的 PostgreSQL 连接池、迁移和 pgx Adapter 启动装配已完成;
公用 bootstrap 已通过 PostgreSQL 18 + Redis 8.2 双存储集成Controller
三监听器与探针集成已完成;可选聚合指标和完整容器进程部署验证仍待实现。
5. Worker heartbeat gRPC 接收路径Redis Provider Leader、分布式请求额度、
Distribution Client 限制和 Provider 状态丢失重建已完成。
5. Redis Provider Leader、分布式速率与 Client 限制、Worker 心跳和自动重建;
TTL 活动池、原子提取和 Worker ownership 已完成。
6. Worker 网络快照流Redis ownership drain/ACK/过期回收已完成。
7. Checker 调度、探测器和健康 reducer。
8. Admin/Distribution 细粒度授权和审计查询Distribution 分布式限流已完成
8. Admin/Distribution 细粒度授权、分布式限流和审计查询
9. 真实 Compose/Kubernetes 集成、故障演练和代表性集群负载测试。
10. 将五种 Routing 策略和 `onUnavailable` 接入 Gateway/Distribution 运行链,
补齐 Sequential 持久化恢复、跨实例 CAS 和 disabled candidate 语义。

View File

@ -72,7 +72,7 @@
| DIST-005 | 返回 expiresAt 与 remainingTtlSeconds | 9334-9360 | `extraction/service_test.go` |
| DIST-006 | 提取前校验 minRemainingTTL 与 maxHealthCheckAge | 9334-9369 | 过滤测试 |
| DIST-007 | reserveForGateway 防止 Extract 清空共享池 | 9281-9333 | 共享池测试 |
| DIST-008 | 提取认证可关闭,关闭后仍有来源识别与全局限制 | 8112-8441 | 来源身份准入、`FixedWindow` 单元测试及 `redisadmission` 双实例/并发集成测试 |
| DIST-008 | 提取认证可关闭,关闭后仍有来源识别与全局限制 | 8112-8441 | 来源身份准入`FixedWindow` 并发测试 |
## 健康、安全、运维与测试

View File

@ -30,8 +30,7 @@
原子操作;并发与主从切换下不得部分提交。
- PostgreSQL 配置版本、Upstream/Routing 管理状态、Admin 审计与 outbox 的事务
更新及幂等重放;测试库断言不包含 Proxy 明细或逐次提取记录。
- Redis TTL 活动池、Leader 租约、Provider 请求额度、Distribution Client
跨副本限流、短期幂等窗口和失联恢复。
- Redis TTL 活动池、Leader 租约、限流、短期幂等窗口和失联恢复。
- Snapshot/Delta/ACK/Report 的版本与校验和兼容性。
- OpenAPI 错误模型、认证矩阵、批量 fulfillment。
- OpenAPI 本地引用闭合、operationId 唯一、响应集合和 security scheme 引用。

View File

@ -22,11 +22,7 @@ func TestReadStateInventoryFailsClosedWhileExpiredCleanupIsBacklogged(t *testing
if err != nil {
t.Fatalf("New() error = %v", err)
}
redisTime, err := fixture.Client.Time(context.Background()).Result()
if err != nil {
t.Fatalf("Redis TIME error = %v", err)
}
now := redisTime.UTC().Add(time.Minute)
now := time.Date(2026, 7, 30, 12, 0, 0, 0, time.UTC)
_, err = bounded.UpsertFetched(context.Background(), "provider-a", activitypool.FetchedBatch{
ObservedAt: now, ConfiguredTTL: time.Second, MaxSize: 10,
Proxies: []proxyDomain.Proxy{

View File

@ -1,141 +0,0 @@
package redisadmission
import (
"context"
"crypto/sha256"
_ "embed"
"encoding/binary"
"encoding/hex"
"errors"
"reflect"
"regexp"
"strconv"
"strings"
"time"
"github.com/redis/go-redis/v9"
"proxy-pool/internal/platform/admission"
)
const MaximumLuaInteger int64 = 1<<53 - 1
var (
ErrInvalidOptions = errors.New("invalid Redis admission options")
namespacePattern = regexp.MustCompile(`^[A-Za-z0-9._-]+$`)
redisAdmissionKey = "pp:{admission}:"
redisAdmissionTail = ":window"
)
//go:embed scripts/fixed_window.lua
var fixedWindowSource string
var fixedWindowScript = redis.NewScript(fixedWindowSource)
type Options struct {
Namespace string
Window time.Duration
Global int64
PerKey int64
}
type Adapter struct {
client redis.Scripter
key string
windowMillis int64
global int64
perKey int64
}
var _ admission.Admitter = (*Adapter)(nil)
func New(client redis.Scripter, options Options) (*Adapter, error) {
if nilInterface(client) || options.Namespace != strings.TrimSpace(options.Namespace) ||
!namespacePattern.MatchString(options.Namespace) || options.Window <= 0 ||
options.Window%time.Millisecond != 0 || options.Global < 0 || options.PerKey < 0 ||
(options.Global == 0 && options.PerKey == 0) || options.Global > MaximumLuaInteger ||
options.PerKey > MaximumLuaInteger {
return nil, ErrInvalidOptions
}
windowMillis := options.Window.Milliseconds()
if windowMillis <= 0 || windowMillis > MaximumLuaInteger {
return nil, ErrInvalidOptions
}
return &Adapter{
client: client,
key: redisAdmissionKey + options.Namespace + redisAdmissionTail,
windowMillis: windowMillis,
global: options.Global,
perKey: options.PerKey,
}, nil
}
func (adapter *Adapter) Admit(ctx context.Context, identity string) error {
if ctx == nil || identity == "" {
return admission.ErrInvalidIdentity
}
if adapter == nil || nilInterface(adapter.client) || adapter.key == "" || adapter.windowMillis <= 0 ||
adapter.windowMillis > MaximumLuaInteger || adapter.global < 0 || adapter.perKey < 0 ||
(adapter.global == 0 && adapter.perKey == 0) || adapter.global > MaximumLuaInteger ||
adapter.perKey > MaximumLuaInteger {
return admission.ErrUnavailable
}
if err := ctx.Err(); err != nil {
return err
}
result, err := fixedWindowScript.Run(ctx, adapter.client, []string{adapter.key},
strconv.FormatInt(adapter.windowMillis, 10),
strconv.FormatInt(adapter.global, 10),
strconv.FormatInt(adapter.perKey, 10),
hashedIdentityField(identity),
).Result()
if err != nil {
if ctxErr := ctx.Err(); ctxErr != nil {
return ctxErr
}
return admission.ErrUnavailable
}
var status string
switch value := result.(type) {
case string:
status = value
case []byte:
status = string(value)
default:
return admission.ErrUnavailable
}
switch status {
case "ok":
return nil
case "global":
return admission.ErrGlobalLimit
case "per_key":
return admission.ErrPerKeyLimit
default:
return admission.ErrUnavailable
}
}
func hashedIdentityField(identity string) string {
var size [8]byte
binary.BigEndian.PutUint64(size[:], uint64(len(identity)))
digest := sha256.New()
_, _ = digest.Write(size[:])
_, _ = digest.Write([]byte(identity))
return "client:" + hex.EncodeToString(digest.Sum(nil))
}
func nilInterface(value any) bool {
if value == nil {
return true
}
reflected := reflect.ValueOf(value)
switch reflected.Kind() {
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
return reflected.IsNil()
default:
return false
}
}

View File

@ -1,223 +0,0 @@
//go:build integration
package redisadmission
import (
"context"
"errors"
"fmt"
"os"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/redis/go-redis/v9"
"proxy-pool/internal/platform/admission"
)
var integrationNamespaceSequence atomic.Uint64
func TestAdaptersShareGlobalAndPerKeyLimits(t *testing.T) {
fixture := newRedisFixture(t)
first := fixture.adapter(t, fixture.namespace, time.Minute, 3, 2)
second := fixture.adapter(t, fixture.namespace, time.Minute, 3, 2)
if err := first.Admit(t.Context(), "client-a"); err != nil {
t.Fatalf("first Admit(client-a): %v", err)
}
if err := second.Admit(t.Context(), "client-a"); err != nil {
t.Fatalf("second Admit(client-a): %v", err)
}
if err := first.Admit(t.Context(), "client-a"); !errors.Is(err, admission.ErrPerKeyLimit) {
t.Fatalf("shared per-key limit error = %v, want ErrPerKeyLimit", err)
}
if err := second.Admit(t.Context(), "client-b"); err != nil {
t.Fatalf("second Admit(client-b): %v", err)
}
if err := first.Admit(t.Context(), "client-c"); !errors.Is(err, admission.ErrGlobalLimit) {
t.Fatalf("shared global limit error = %v, want ErrGlobalLimit", err)
}
}
func TestConcurrentAdmissionsAreExactAcrossAdapters(t *testing.T) {
fixture := newRedisFixture(t)
tests := []struct {
name string
global int64
perKey int64
identity func(int) string
want int64
wantReject error
}{
{
name: "global", global: 37, perKey: 1000, want: 37, wantReject: admission.ErrGlobalLimit,
identity: func(index int) string { return fmt.Sprintf("client-%d", index) },
},
{
name: "per-key", global: 1000, perKey: 23, want: 23, wantReject: admission.ErrPerKeyLimit,
identity: func(int) string { return "shared-client" },
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
namespace := fixture.namespace + "-" + tt.name
first := fixture.adapter(t, namespace, time.Minute, tt.global, tt.perKey)
second := fixture.adapter(t, namespace, time.Minute, tt.global, tt.perKey)
var accepted atomic.Int64
var wrong atomic.Int64
var wait sync.WaitGroup
for index := range 200 {
wait.Add(1)
go func() {
defer wait.Done()
adapter := first
if index%2 == 1 {
adapter = second
}
err := adapter.Admit(t.Context(), tt.identity(index))
switch {
case err == nil:
accepted.Add(1)
case !errors.Is(err, tt.wantReject):
wrong.Add(1)
}
}()
}
wait.Wait()
if got := accepted.Load(); got != tt.want {
t.Fatalf("accepted = %d, want %d", got, tt.want)
}
if got := wrong.Load(); got != 0 {
t.Fatalf("unexpected rejection count = %d", got)
}
})
}
}
func TestWindowRolloverRemovesOldClientFields(t *testing.T) {
fixture := newRedisFixture(t)
const window = 100 * time.Millisecond
adapter := fixture.adapter(t, fixture.namespace, window, 100, 10)
if err := adapter.Admit(t.Context(), "old-client"); err != nil {
t.Fatalf("Admit(old-client): %v", err)
}
if err := fixture.client.Persist(t.Context(), adapter.key).Err(); err != nil {
t.Fatalf("PERSIST admission key: %v", err)
}
storedWindow, err := fixture.client.HGet(t.Context(), adapter.key, "window").Int64()
if err != nil {
t.Fatalf("read stored window: %v", err)
}
waitForRedisWindow(t, fixture.client, window, storedWindow)
if err := adapter.Admit(t.Context(), "new-client"); err != nil {
t.Fatalf("Admit(new-client): %v", err)
}
oldField := hashedIdentityField("old-client")
if exists, err := fixture.client.HExists(t.Context(), adapter.key, oldField).Result(); err != nil || exists {
t.Fatalf("old client field exists = %v, error = %v", exists, err)
}
newField := hashedIdentityField("new-client")
if exists, err := fixture.client.HExists(t.Context(), adapter.key, newField).Result(); err != nil || !exists {
t.Fatalf("new client field exists = %v, error = %v", exists, err)
}
if length, err := fixture.client.HLen(t.Context(), adapter.key).Result(); err != nil || length != 3 {
t.Fatalf("current window hash length = %d, error = %v, want 3", length, err)
}
}
func TestNamespacesAreIsolated(t *testing.T) {
fixture := newRedisFixture(t)
first := fixture.adapter(t, fixture.namespace+"-a", time.Minute, 1, 1)
second := fixture.adapter(t, fixture.namespace+"-b", time.Minute, 1, 1)
if err := first.Admit(t.Context(), "same-client"); err != nil {
t.Fatalf("first namespace Admit(): %v", err)
}
if err := second.Admit(t.Context(), "same-client"); err != nil {
t.Fatalf("second namespace Admit(): %v", err)
}
if err := first.Admit(t.Context(), "same-client"); !errors.Is(err, admission.ErrGlobalLimit) {
t.Fatalf("first namespace second Admit() error = %v", err)
}
}
type redisFixture struct {
client *redis.Client
namespace string
}
func newRedisFixture(t *testing.T) redisFixture {
t.Helper()
redisURL := os.Getenv("PROXY_POOL_TEST_REDIS_URL")
if redisURL == "" {
t.Skip("PROXY_POOL_TEST_REDIS_URL is not set")
}
options, err := redis.ParseURL(redisURL)
if err != nil {
t.Fatalf("parse PROXY_POOL_TEST_REDIS_URL: %v", err)
}
client := redis.NewClient(options)
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := client.Ping(ctx).Err(); err != nil {
_ = client.Close()
t.Fatalf("ping Redis: %v", err)
}
namespace := fmt.Sprintf("admission-it-%d-%d-%d", os.Getpid(), time.Now().UnixNano(), integrationNamespaceSequence.Add(1))
t.Cleanup(func() {
cleanupCtx, cleanupCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cleanupCancel()
pattern := redisAdmissionKey + namespace + "*"
var cursor uint64
for {
keys, next, scanErr := client.Scan(cleanupCtx, cursor, pattern, 128).Result()
if scanErr != nil {
t.Errorf("scan Redis admission keys: %v", scanErr)
break
}
if len(keys) > 0 {
if unlinkErr := client.Unlink(cleanupCtx, keys...).Err(); unlinkErr != nil {
t.Errorf("remove Redis admission keys: %v", unlinkErr)
break
}
}
cursor = next
if cursor == 0 {
break
}
}
_ = client.Close()
})
return redisFixture{client: client, namespace: namespace}
}
func (fixture redisFixture) adapter(t *testing.T, namespace string, window time.Duration, global, perKey int64) *Adapter {
t.Helper()
return mustAdapter(t, fixture.client, Options{
Namespace: namespace,
Window: window,
Global: global,
PerKey: perKey,
})
}
func waitForRedisWindow(t *testing.T, client *redis.Client, window time.Duration, previous int64) {
t.Helper()
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
redisTime, err := client.Time(t.Context()).Result()
if err != nil {
t.Fatalf("Redis TIME: %v", err)
}
windowID := redisTime.UnixMilli() / window.Milliseconds()
if windowID != previous {
return
}
time.Sleep(5 * time.Millisecond)
}
t.Fatal("timed out waiting for Redis window rollover")
}

View File

@ -1,225 +0,0 @@
package redisadmission
import (
"context"
"crypto/sha256"
"encoding/binary"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/redis/go-redis/v9"
"proxy-pool/internal/platform/admission"
)
type scriptCall struct {
keys []string
args []any
}
type scriptClient struct {
redis.Scripter
result any
err error
calls []scriptCall
}
func (client *scriptClient) EvalSha(_ context.Context, _ string, keys []string, args ...any) *redis.Cmd {
client.calls = append(client.calls, scriptCall{keys: append([]string(nil), keys...), args: append([]any(nil), args...)})
return redis.NewCmdResult(client.result, client.err)
}
func TestNewRejectsInvalidDependenciesAndOptions(t *testing.T) {
t.Parallel()
validClient := &scriptClient{result: "ok"}
valid := Options{Namespace: "listener-a", Window: time.Minute, Global: 10, PerKey: 2}
var typedNilClient *redis.Client
tests := []struct {
name string
client redis.Scripter
options Options
}{
{name: "nil client", options: valid},
{name: "typed nil client", client: typedNilClient, options: valid},
{name: "empty namespace", client: validClient, options: withNamespace(valid, "")},
{name: "leading namespace whitespace", client: validClient, options: withNamespace(valid, " listener-a")},
{name: "trailing namespace whitespace", client: validClient, options: withNamespace(valid, "listener-a ")},
{name: "cluster tag in namespace", client: validClient, options: withNamespace(valid, "listener{a}")},
{name: "separator in namespace", client: validClient, options: withNamespace(valid, "listener:a")},
{name: "zero window", client: validClient, options: withWindow(valid, 0)},
{name: "sub-millisecond window", client: validClient, options: withWindow(valid, time.Microsecond)},
{name: "fractional millisecond window", client: validClient, options: withWindow(valid, time.Millisecond+time.Microsecond)},
{name: "negative global", client: validClient, options: withGlobal(valid, -1)},
{name: "negative per-key", client: validClient, options: withPerKey(valid, -1)},
{name: "zero limits", client: validClient, options: withLimits(valid, 0, 0)},
{name: "global exceeds Lua integer", client: validClient, options: withGlobal(valid, MaximumLuaInteger+1)},
{name: "per-key exceeds Lua integer", client: validClient, options: withPerKey(valid, MaximumLuaInteger+1)},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
if adapter, err := New(tt.client, tt.options); !errors.Is(err, ErrInvalidOptions) || adapter != nil {
t.Fatalf("New() = (%v, %v), want nil adapter and ErrInvalidOptions", adapter, err)
}
})
}
}
func TestAdmitMapsScriptStatusAndPreservesInputErrors(t *testing.T) {
t.Parallel()
tests := []struct {
name string
status any
want error
}{
{name: "accepted", status: "ok"},
{name: "global limit", status: "global", want: admission.ErrGlobalLimit},
{name: "per-key limit", status: "per_key", want: admission.ErrPerKeyLimit},
{name: "unexpected status", status: "unknown", want: admission.ErrUnavailable},
{name: "unexpected reply type", status: int64(1), want: admission.ErrUnavailable},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
client := &scriptClient{result: tt.status}
adapter := mustAdapter(t, client, Options{Namespace: "listener-a", Window: time.Minute, Global: 10, PerKey: 2})
err := adapter.Admit(context.Background(), "client-a")
if !errors.Is(err, tt.want) {
t.Fatalf("Admit() error = %v, want %v", err, tt.want)
}
})
}
adapter := mustAdapter(t, &scriptClient{result: "ok"}, Options{Namespace: "listener-a", Window: time.Minute, Global: 1})
if err := adapter.Admit(nil, "client-a"); !errors.Is(err, admission.ErrInvalidIdentity) {
t.Fatalf("Admit(nil context) error = %v, want ErrInvalidIdentity", err)
}
if err := adapter.Admit(context.Background(), ""); !errors.Is(err, admission.ErrInvalidIdentity) {
t.Fatalf("Admit(empty identity) error = %v, want ErrInvalidIdentity", err)
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if err := adapter.Admit(ctx, "client-a"); !errors.Is(err, context.Canceled) {
t.Fatalf("Admit(canceled context) error = %v, want context.Canceled", err)
}
var nilAdapter *Adapter
if err := nilAdapter.Admit(context.Background(), "client-a"); !errors.Is(err, admission.ErrUnavailable) {
t.Fatalf("nil Adapter.Admit() error = %v, want ErrUnavailable", err)
}
}
func TestAdmitUsesHashedIdentityAndExactIntegerArguments(t *testing.T) {
t.Parallel()
client := &scriptClient{result: "ok"}
adapter := mustAdapter(t, client, Options{
Namespace: "listener-a", Window: 1500 * time.Millisecond,
Global: MaximumLuaInteger, PerKey: MaximumLuaInteger - 1,
})
const identity = "private-client@example.test"
if err := adapter.Admit(context.Background(), identity); err != nil {
t.Fatalf("Admit(): %v", err)
}
if len(client.calls) != 1 {
t.Fatalf("script calls = %d, want 1", len(client.calls))
}
call := client.calls[0]
if len(call.keys) != 1 || call.keys[0] != "pp:{admission}:listener-a:window" {
t.Fatalf("script keys = %q", call.keys)
}
wantArgs := []string{"1500", fmt.Sprint(MaximumLuaInteger), fmt.Sprint(MaximumLuaInteger - 1), clientField(identity)}
if got := stringify(call.args); fmt.Sprint(got) != fmt.Sprint(wantArgs) {
t.Fatalf("script args = %q, want %q", got, wantArgs)
}
serialized := fmt.Sprint(call.keys, call.args)
if strings.Contains(serialized, identity) {
t.Fatalf("Redis input leaks raw identity: %s", serialized)
}
if got := clientField(identity); len(got) != len("client:")+sha256.Size*2 || !strings.HasPrefix(got, "client:") {
t.Fatalf("client field = %q", got)
}
}
func TestAdmitFailsClosedWithoutLeakingIdentity(t *testing.T) {
t.Parallel()
const identity = "sensitive-client"
client := &scriptClient{err: errors.New("backend failure: " + identity)}
adapter := mustAdapter(t, client, Options{Namespace: "listener-a", Window: time.Minute, Global: 1})
err := adapter.Admit(context.Background(), identity)
if !errors.Is(err, admission.ErrUnavailable) {
t.Fatalf("Admit() error = %v, want ErrUnavailable", err)
}
if strings.Contains(err.Error(), identity) {
t.Fatalf("Admit() error leaks identity: %v", err)
}
}
func TestAdmitFailsClosedForUninitializedAdapter(t *testing.T) {
t.Parallel()
var typedNilClient *redis.Client
tests := []*Adapter{
{},
{client: typedNilClient, key: "pp:{admission}:test:window", windowMillis: 60_000, global: 1},
}
for _, adapter := range tests {
if err := adapter.Admit(context.Background(), "client-a"); !errors.Is(err, admission.ErrUnavailable) {
t.Fatalf("Admit() error = %v, want ErrUnavailable", err)
}
}
}
func clientField(identity string) string {
var size [8]byte
binary.BigEndian.PutUint64(size[:], uint64(len(identity)))
digest := sha256.New()
_, _ = digest.Write(size[:])
_, _ = digest.Write([]byte(identity))
return fmt.Sprintf("client:%x", digest.Sum(nil))
}
func stringify(values []any) []string {
result := make([]string, len(values))
for index, value := range values {
result[index] = fmt.Sprint(value)
}
return result
}
func mustAdapter(t *testing.T, client redis.Scripter, options Options) *Adapter {
t.Helper()
adapter, err := New(client, options)
if err != nil {
t.Fatalf("New(): %v", err)
}
return adapter
}
func withNamespace(options Options, namespace string) Options {
options.Namespace = namespace
return options
}
func withWindow(options Options, window time.Duration) Options {
options.Window = window
return options
}
func withGlobal(options Options, limit int64) Options {
options.Global = limit
return options
}
func withPerKey(options Options, limit int64) Options {
options.PerKey = limit
return options
}
func withLimits(options Options, global, perKey int64) Options {
options.Global = global
options.PerKey = perKey
return options
}

View File

@ -1,39 +0,0 @@
local key = KEYS[1]
local window_ms = tonumber(ARGV[1])
local global_limit = tonumber(ARGV[2])
local per_key_limit = tonumber(ARGV[3])
local client_field = ARGV[4]
local redis_time = redis.call("TIME")
local now_ms = tonumber(redis_time[1]) * 1000 + math.floor(tonumber(redis_time[2]) / 1000)
local window_id = math.floor(now_ms / window_ms)
local stored_window = redis.call("HGET", key, "window")
if not stored_window or tonumber(stored_window) ~= window_id then
redis.call("DEL", key)
redis.call("HSET", key, "window", window_id)
end
if global_limit > 0 then
local global_used = tonumber(redis.call("HGET", key, "global") or "0")
if global_used >= global_limit then
return "global"
end
end
if per_key_limit > 0 then
local client_used = tonumber(redis.call("HGET", key, client_field) or "0")
if client_used >= per_key_limit then
return "per_key"
end
end
if global_limit > 0 then
redis.call("HINCRBY", key, "global", 1)
end
if per_key_limit > 0 then
redis.call("HINCRBY", key, client_field, 1)
end
redis.call("PEXPIREAT", key, (window_id + 1) * window_ms)
return "ok"

View File

@ -428,13 +428,6 @@ func TestValidateRejectsInvalidConfigurationMatrix(t *testing.T) {
},
want: "requestsPerMinute",
},
{
name: "listener request limit exceeds exact counter range",
mutate: func(cfg *Config) {
cfg.Distribution.Limits.RequestsPerMinutePerClient = int(MaximumExactCounter) + 1
},
want: "requestsPerMinutePerClient",
},
{
name: "invalid client identification mode",
mutate: func(cfg *Config) {

View File

@ -122,9 +122,6 @@ func validateListener(name string, listener Listener, security Security) error {
if limit.value < 0 {
return fmt.Errorf("validate %s limits.%s: must be non-negative", name, limit.name)
}
if int64(limit.value) > MaximumExactCounter {
return fmt.Errorf("validate %s limits.%s: exceeds exact counter range", name, limit.name)
}
}
host, err := validateListenAddress(name, listener.Listen)
if err != nil {

View File

@ -55,7 +55,6 @@ type ports struct {
activity activityStore
readiness distribution.ReadinessChecker
metricsReadiness platformMetrics.ReadinessChecker
admission admission.Admitter
coordinator provider.Coordinator
credentials credentials.Store
providerResults provider.ResultRecorder
@ -146,10 +145,10 @@ func run(ctx context.Context, options Options, infrastructure infrastructure, fa
dependencies := controllerRuntime.Dependencies{}
if loaded.Value.Distribution.Enabled {
if nilInterface(opened.activity) || nilInterface(opened.readiness) || nilInterface(opened.admission) {
if nilInterface(opened.activity) || nilInterface(opened.readiness) {
return errors.Join(ErrStartup, ErrInvalidOptions)
}
service, serviceErr := extraction.NewService(opened.activity, extractionPolicy(loaded.Value), opened.admission, options.Now)
service, serviceErr := extraction.NewService(opened.activity, extractionPolicy(loaded.Value), admission.AllowAll{}, options.Now)
if serviceErr != nil {
return fmt.Errorf("%w: build extraction service: %w", ErrStartup, serviceErr)
}

View File

@ -16,7 +16,6 @@ import (
"proxy-pool/internal/domain/adminstate"
extractionDomain "proxy-pool/internal/domain/extraction"
"proxy-pool/internal/domain/upstream"
"proxy-pool/internal/platform/admission"
"proxy-pool/internal/platform/credentials"
)
@ -34,7 +33,6 @@ func TestRunLoadsOneSnapshotCommitsItAndClosesInfrastructure(t *testing.T) {
closeErr := errors.New("close failed")
infrastructure := &stubInfrastructure{ports: ports{
state: state, activity: activity, readiness: readyStub{}, metricsReadiness: readyStub{},
admission: admission.AllowAll{},
coordinator: coordinatorStub{}, credentials: credentialStore,
close: func() error { return closeErr },
}}
@ -115,33 +113,6 @@ func TestRunRejectsMissingAdminFingerprintKeyBeforeOpeningInfrastructure(t *test
}
}
func TestRunRejectsMissingDistributionAdmissionDependency(t *testing.T) {
t.Parallel()
credentialStore, err := credentials.NewMemoryStore(10)
if err != nil {
t.Fatalf("NewMemoryStore(): %v", err)
}
infrastructure := &stubInfrastructure{ports: ports{
state: adminstate.NewMemoryStore(), activity: &stubActivityStore{},
readiness: readyStub{}, metricsReadiness: readyStub{},
coordinator: coordinatorStub{}, credentials: credentialStore,
close: func() error { return nil },
}}
factory := &recordingRuntimeFactory{runner: runnerStub{err: errors.New("runtime should not start")}}
err = run(context.Background(), Options{
ConfigPath: "controller.yaml",
Resolver: &memoryResolver{files: map[string][]byte{"controller.yaml": []byte(bootstrapTestConfig)}},
Now: time.Now,
FingerprintKey: bootstrapTestFingerprintKey,
}, infrastructure, factory)
if !errors.Is(err, ErrStartup) || !errors.Is(err, ErrInvalidOptions) {
t.Fatalf("run() error = %v, want ErrStartup and ErrInvalidOptions", err)
}
if factory.configuration != nil {
t.Fatal("runtime factory called without admission dependency")
}
}
func TestRunSupportsProviderOnlyConfigurationWithoutHTTPRuntime(t *testing.T) {
source := strings.ReplaceAll(bootstrapTestConfig, "distribution:\n enabled: true", "distribution:\n enabled: false")
source = strings.ReplaceAll(source, "admin:\n enabled: true", "admin:\n enabled: false")
@ -180,7 +151,6 @@ func TestRunAdminDisableStopsActiveProviderRuntime(t *testing.T) {
stopped := make(chan string, 2)
infrastructure := &stubInfrastructure{ports: ports{
state: state, activity: &stubActivityStore{}, readiness: readyStub{}, metricsReadiness: readyStub{},
admission: admission.AllowAll{},
coordinator: coordinatorFunc(func(ctx context.Context, upstreamID string) error {
started <- upstreamID
<-ctx.Done()

View File

@ -13,11 +13,9 @@ import (
"proxy-pool/internal/adapters/postgresadmin"
"proxy-pool/internal/adapters/redisactivity"
"proxy-pool/internal/adapters/redisadmission"
"proxy-pool/internal/adapters/redisprovider"
"proxy-pool/internal/config"
controllerProvider "proxy-pool/internal/controller/provider"
"proxy-pool/internal/platform/admission"
"proxy-pool/internal/platform/credentials"
platformMetrics "proxy-pool/internal/platform/metrics"
)
@ -133,16 +131,6 @@ func (infrastructure *productionInfrastructure) Open(
opened.activity = adapter
opened.readiness = redisReadiness{client: redisClient}
opened.credentials = credentialStore
if configuration.Distribution.Enabled {
opened.admission, err = newDistributionAdmitter(
redisClient,
namespace,
configuration.Distribution.Limits,
)
if err != nil {
return ports{}, err
}
}
if providersEnabled {
stats, statsErr := controllerProvider.NewStatsRecorder(config.MaximumUpstreams)
if statsErr != nil {
@ -173,22 +161,6 @@ func (infrastructure *productionInfrastructure) Open(
return opened, nil
}
func newDistributionAdmitter(
client redis.Scripter,
namespace string,
limits config.Limits,
) (admission.Admitter, error) {
if limits.RequestsPerMinute == 0 && limits.RequestsPerMinutePerClient == 0 {
return admission.AllowAll{}, nil
}
return redisadmission.New(client, redisadmission.Options{
Namespace: namespace,
Window: time.Minute,
Global: int64(limits.RequestsPerMinute),
PerKey: int64(limits.RequestsPerMinutePerClient),
})
}
func resolveRedisNamespace(configured string) (string, error) {
if strings.TrimSpace(configured) != configured {
return "", ErrInvalidOptions

View File

@ -3,34 +3,13 @@ package bootstrap
import (
"context"
"errors"
"fmt"
"strings"
"sync/atomic"
"testing"
"github.com/redis/go-redis/v9"
"proxy-pool/internal/config"
"proxy-pool/internal/platform/admission"
)
type recordingAdmissionScripter struct {
redis.Scripter
keys []string
args []any
}
func (client *recordingAdmissionScripter) EvalSha(
_ context.Context,
_ string,
keys []string,
args ...any,
) *redis.Cmd {
client.keys = append([]string(nil), keys...)
client.args = append([]any(nil), args...)
return redis.NewCmdResult("ok", nil)
}
func TestProductionInfrastructureRejectsInvalidStorageWithoutLeakingURLs(t *testing.T) {
t.Parallel()
postgresSecret := "postgres-secret"
@ -52,41 +31,6 @@ func TestProductionInfrastructureRejectsInvalidStorageWithoutLeakingURLs(t *test
}
}
func TestNewDistributionAdmitterPassesConfiguredLimits(t *testing.T) {
t.Parallel()
client := &recordingAdmissionScripter{}
limiter, err := newDistributionAdmitter(client, "controller-a", config.Limits{
RequestsPerMinute: 321,
RequestsPerMinutePerClient: 17,
})
if err != nil {
t.Fatalf("newDistributionAdmitter(): %v", err)
}
if err := limiter.Admit(context.Background(), "client-a"); err != nil {
t.Fatalf("Admit(): %v", err)
}
if len(client.keys) != 1 || len(client.args) != 4 {
t.Fatalf("Redis admission call = keys:%v args:%v", client.keys, client.args)
}
if got := fmt.Sprintf("%s|%v|%v|%v", client.keys[0], client.args[0], client.args[1], client.args[2]); got != "pp:{admission}:controller-a:window|60000|321|17" {
t.Fatalf("Redis admission inputs = %q", got)
}
}
func TestNewDistributionAdmitterAllowsAllWhenQuotasAreDisabled(t *testing.T) {
t.Parallel()
limiter, err := newDistributionAdmitter(nil, "controller-a", config.Limits{})
if err != nil {
t.Fatalf("newDistributionAdmitter(): %v", err)
}
if _, ok := limiter.(admission.AllowAll); !ok {
t.Fatalf("limiter type = %T, want admission.AllowAll", limiter)
}
if err := limiter.Admit(context.Background(), "client-a"); err != nil {
t.Fatalf("AllowAll.Admit(): %v", err)
}
}
func TestSelectMetricsReadinessPreservesDistributionWhenAdminStoreFails(t *testing.T) {
t.Parallel()
adminCalls := &atomic.Int64{}

View File

@ -7,7 +7,6 @@ import (
"time"
domain "proxy-pool/internal/domain/extraction"
"proxy-pool/internal/platform/admission"
)
var (
@ -71,12 +70,16 @@ type Response struct {
type Service struct {
store domain.Store
policy Policy
admission admission.Admitter
admission Admission
now func() time.Time
}
func NewService(store domain.Store, policy Policy, admitter admission.Admitter, now func() time.Time) (*Service, error) {
if admitter == nil {
type Admission interface {
Admit(context.Context, string) error
}
func NewService(store domain.Store, policy Policy, admission Admission, now func() time.Time) (*Service, error) {
if admission == nil {
return nil, fmt.Errorf("%w: admission is required", ErrInvalidServicePolicy)
}
if store == nil {
@ -92,7 +95,7 @@ func NewService(store domain.Store, policy Policy, admitter admission.Admitter,
if now == nil {
now = time.Now
}
return &Service{store: store, policy: policy, admission: admitter, now: now}, nil
return &Service{store: store, policy: policy, admission: admission, now: now}, nil
}
func (s *Service) Extract(ctx context.Context, request Request) (Response, error) {
@ -111,9 +114,6 @@ func (s *Service) Extract(ctx context.Context, request Request) (Response, error
return response, ErrInvalidFulfillment
}
if err := s.admission.Admit(ctx, admissionKey(request)); err != nil {
if errors.Is(err, admission.ErrUnavailable) {
return response, errors.Join(ErrUnavailable, err)
}
return response, errors.Join(ErrAdmissionRejected, err)
}

View File

@ -9,7 +9,6 @@ import (
"proxy-pool/internal/domain/activitypool"
domain "proxy-pool/internal/domain/extraction"
proxyDomain "proxy-pool/internal/domain/proxy"
platformAdmission "proxy-pool/internal/platform/admission"
)
func TestServiceAppliesPolicyAndBuildsResponse(t *testing.T) {
@ -123,30 +122,6 @@ func TestServiceAppliesAdmissionBeforeStoreUsingStableIdentity(t *testing.T) {
}
}
func TestServiceMapsUnavailableAdmissionToServiceUnavailable(t *testing.T) {
store := &recordingStore{}
admitter := &recordingAdmission{err: platformAdmission.ErrUnavailable}
service, err := NewService(store, Policy{
MaxCountPerRequest: 1,
DefaultFulfillment: domain.Partial,
}, admitter, time.Now)
if err != nil {
t.Fatalf("NewService(): %v", err)
}
_, err = service.Extract(context.Background(), Request{
RequestID: "req-1",
ClientID: "client-1",
Count: 1,
})
if !errors.Is(err, ErrUnavailable) || errors.Is(err, ErrAdmissionRejected) {
t.Fatalf("Extract() error = %v, want only ErrUnavailable", err)
}
if store.calls != 0 {
t.Fatalf("store calls = %d, want 0", store.calls)
}
}
func TestServiceUsesSourceIdentityForEphemeralIdempotency(t *testing.T) {
store := &recordingStore{}
service, err := NewService(store, Policy{

View File

@ -1,9 +0,0 @@
package admission
import "context"
// Admitter is the shared admission-control contract used by controller
// services and infrastructure adapters.
type Admitter interface {
Admit(context.Context, string) error
}

View File

@ -15,9 +15,6 @@ func TestAllowAllPreservesContextAndIdentityValidation(t *testing.T) {
if err := admission.Admit(context.Background(), ""); !errors.Is(err, ErrInvalidIdentity) {
t.Fatalf("Admit(empty identity) error = %v", err)
}
if err := admission.Admit(nil, "client-a"); !errors.Is(err, ErrInvalidIdentity) {
t.Fatalf("Admit(nil context) error = %v", err)
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if err := admission.Admit(ctx, "client-a"); !errors.Is(err, context.Canceled) {

View File

@ -12,7 +12,6 @@ var (
ErrInvalidIdentity = errors.New("invalid admission identity")
ErrGlobalLimit = errors.New("global admission limit exceeded")
ErrPerKeyLimit = errors.New("per-key admission limit exceeded")
ErrUnavailable = errors.New("admission control unavailable")
)
type FixedWindowConfig struct {
@ -55,12 +54,12 @@ func NewFixedWindow(config FixedWindowConfig) (*FixedWindow, error) {
}
func (l *FixedWindow) Admit(ctx context.Context, key string) error {
if ctx == nil || l == nil || key == "" {
return ErrInvalidIdentity
}
if err := ctx.Err(); err != nil {
return err
}
if l == nil || key == "" {
return ErrInvalidIdentity
}
l.mu.Lock()
defer l.mu.Unlock()

View File

@ -74,27 +74,3 @@ func TestFixedWindowResetsAndIsConcurrencySafe(t *testing.T) {
t.Fatalf("Admit(after reset): %v", err)
}
}
func TestFixedWindowRejectsNilContextAndEmptyIdentity(t *testing.T) {
t.Parallel()
limiter, err := NewFixedWindow(FixedWindowConfig{
Window: time.Minute,
Global: 1,
})
if err != nil {
t.Fatalf("NewFixedWindow(): %v", err)
}
if err := limiter.Admit(nil, "client-a"); !errors.Is(err, ErrInvalidIdentity) {
t.Fatalf("Admit(nil context) error = %v, want ErrInvalidIdentity", err)
}
if err := limiter.Admit(context.Background(), ""); !errors.Is(err, ErrInvalidIdentity) {
t.Fatalf("Admit(empty identity) error = %v, want ErrInvalidIdentity", err)
}
}
func TestAdmitterImplementations(t *testing.T) {
t.Parallel()
var _ Admitter = (*FixedWindow)(nil)
var _ Admitter = AllowAll{}
}

View File

@ -14,7 +14,7 @@ try {
$env:PROXY_POOL_TEST_REDIS_URL = "redis://127.0.0.1:16379/15"
Push-Location $repositoryRoot
try {
go test -count=1 -tags=integration -timeout 60s ./internal/adapters/redisactivity/... ./internal/adapters/redisadmission/... ./internal/adapters/redisprovider/...
go test -count=1 -tags=integration -timeout 60s ./internal/adapters/redisactivity/... ./internal/adapters/redisprovider/...
if ($LASTEXITCODE -ne 0) {
throw "Redis integration tests failed with exit code $LASTEXITCODE"
}