feat: expose capacity metrics

This commit is contained in:
youfak 2026-08-02 14:21:18 +08:00
parent 4691b9150c
commit b24d71fb90
14 changed files with 497 additions and 12 deletions

View File

@ -57,6 +57,10 @@ Proxy Pool 用 Controller 协调这些变化,并让 Gateway 数据面只消费
`proxy_pool_controller_extraction_returned_proxies_total``result` 仅有完成、 `proxy_pool_controller_extraction_returned_proxies_total``result` 仅有完成、
部分、空、库存不足、幂等冲突、限流、不可用、无效与内部错误等固定枚举, 部分、空、库存不足、幂等冲突、限流、不可用、无效与内部错误等固定枚举,
不包含 Client、请求、过滤条件、Upstream、Proxy 或错误文本标签。 不包含 Client、请求、过滤条件、Upstream、Proxy 或错误文本标签。
- **容量指标**Controller 在既有 Provider 库存对账周期聚合托管 Proxy、可用/有效
Slot、待拉取数量和活跃上游数并暴露
`proxy_pool_controller_capacity_inventory_reads_total{result}``result` 仅有
`success``error`,不按 Upstream、Worker、Proxy、会话或地址拆分。
- **PostgreSQL 管理面**持久化配置版本、Upstream/Routing 管理状态、Admin - **PostgreSQL 管理面**持久化配置版本、Upstream/Routing 管理状态、Admin
审计与 Outbox不保存 Proxy 明细或逐次提取记录。 审计与 Outbox不保存 Proxy 明细或逐次提取记录。
- **Gateway 组件**HTTP 正向代理、HTTPS CONNECT、双向 Tunnel、重试、超时、 - **Gateway 组件**HTTP 正向代理、HTTPS CONNECT、双向 Tunnel、重试、超时、

View File

@ -526,8 +526,10 @@ Metrics 启用时 `listen` 必须是合法 `host:port`。该入口固定提供 `
`result` 固定为 accepted、rejected。Gateway 另暴露 `result` 固定为 accepted、rejected。Gateway 另暴露
`proxy_pool_gateway_outcomes_total{stage,result}` `proxy_pool_gateway_outcomes_total{stage,result}`
`proxy_pool_gateway_outcome_queue_dropped_total``stage` 固定为 DIAL、PROXY_HANDSHAKE、 `proxy_pool_gateway_outcome_queue_dropped_total``stage` 固定为 DIAL、PROXY_HANDSHAKE、
RESPONSE_HEADERS、TUNNEL`result` 固定为 success、failure。绝不包含 Proxy、IP、Checker、 RESPONSE_HEADERS、TUNNEL`result` 固定为 success、failure。Provider、Extraction、
Client、路由、目标 URL 或凭据标签。Provider、提取和容量等业务指标仍在后续实施范围。Distribution 启用时 `/readyz` 只以 Redis 活动池为 Capacity 和 Drain 业务指标也已接入,所有标签均为固定枚举;绝不包含 Proxy、IP、Checker、
Client、路由、目标 URL 或凭据标签。Capacity 从既有 Provider 库存对账循环聚合,
不进入 Gateway 热路径。Distribution 启用时 `/readyz` 只以 Redis 活动池为
服务流量门槛PostgreSQL 故障由 Admin 接口独立报告。Metrics 开关或监听地址 服务流量门槛PostgreSQL 故障由 Admin 接口独立报告。Metrics 开关或监听地址
变更需要重启 Controller。 变更需要重启 Controller。

View File

@ -206,7 +206,7 @@ Distribution/Admin 服务构造、错误合并和资源关闭。生产 Provider
组合 fixture 已验证隔离 Redis namespace 下的选主、Provider HTTP 调用、模板解析 组合 fixture 已验证隔离 Redis namespace 下的选主、Provider HTTP 调用、模板解析
和活动池写入。Controller Metrics 独立入口现已提供 `/livez`、`/readyz` 与基础 和活动池写入。Controller Metrics 独立入口现已提供 `/livez`、`/readyz` 与基础
Prometheus 运行时指标三监听器隔离已通过测试Checker、Gateway、Drain、 Prometheus 运行时指标三监听器隔离已通过测试Checker、Gateway、Drain、
Provider 与 Extraction 业务指标已接入,容量指标和密钥安全的结构化日志仍待实现。双存储 bootstrap 已通过 PostgreSQL 18 + Redis 8.2 组合 fixture覆盖 Provider、Extraction 与容量业务指标已接入,密钥安全的结构化日志仍待实现。双存储 bootstrap 已通过 PostgreSQL 18 + Redis 8.2 组合 fixture覆盖
迁移、启动配置提交、Readiness、Admin Status 和 Metrics 探针。 迁移、启动配置提交、Readiness、Admin Status 和 Metrics 探针。
WorkerControlPlane 现已接入 Controller 生命周期Register、ACK 和 Runtime WorkerControlPlane 现已接入 Controller 生命周期Register、ACK 和 Runtime
@ -355,7 +355,8 @@ Supervisor 也改为同时服从静态配置与管理态,消除两条启停消
`proxy_pool_controller_provider_valid_candidates_total` `proxy_pool_controller_provider_valid_candidates_total`
`proxy_pool_controller_provider_new_proxies_total``class` 仅允许 `valid`、`empty`、 `proxy_pool_controller_provider_new_proxies_total``class` 仅允许 `valid`、`empty`、
`duplicate_only`、`error`。Extraction 还暴露固定 `result` 的请求次数、请求数与响应 `duplicate_only`、`error`。Extraction 还暴露固定 `result` 的请求次数、请求数与响应
交付数,幂等重放按响应交付统计。容量指标与密钥安全的结构化日志仍待补齐。 交付数,幂等重放按响应交付统计。容量指标由既有 Provider 库存对账周期聚合,不进入
Gateway 热路径;仅密钥安全的结构化日志仍待补齐。
## Task 14: Documentation, Examples, and Diagrams ## Task 14: Documentation, Examples, and Diagrams

View File

@ -104,8 +104,8 @@ Controller/Gateway 入口,完整 mTLS 运行时拓扑仍只有静态验证。
响应校验场景;`proxy-checker` 的 BASIC 任务进程已经完成, 响应校验场景;`proxy-checker` 的 BASIC 任务进程已经完成,
`proxy-controller` 已完成 `proxy-controller` 已完成
Admin/Distribution/Metrics 与 PostgreSQL/Redis 启动装配,`proxy-gateway` 已完成 Admin/Distribution/Metrics 与 PostgreSQL/Redis 启动装配,`proxy-gateway` 已完成
HTTP/Metrics 与控制面 Session 装配Checker 与 Gateway Outcome 的低基数业务指标 HTTP/Metrics 与控制面 Session 装配Checker、Gateway Outcome、Drain、Provider、
已接入,但 Provider、提取和容量指标链未闭环 Extraction 与 Capacity 的低基数业务指标已闭环,密钥安全的结构化日志仍待补齐
2. Gateway 的生产连接池调优与代表性流量压测。 2. Gateway 的生产连接池调优与代表性流量压测。
3. Provider 分布式 singleflight/Leader、长期凭据回收和累计额度执行器。 3. Provider 分布式 singleflight/Leader、长期凭据回收和累计额度执行器。
4. Controller 的 PostgreSQL 连接池、迁移和 pgx Adapter 启动装配已完成; 4. Controller 的 PostgreSQL 连接池、迁移和 pgx Adapter 启动装配已完成;

View File

@ -87,5 +87,5 @@
| OPS-001 | 配置校验后构建不可变快照并原子替换 | 8959-8999 | 100k 索引、版本/epoch 与并发 Apply/Acquire 测试 | | OPS-001 | 配置校验后构建不可变快照并原子替换 | 8959-8999 | 100k 索引、版本/epoch 与并发 Apply/Acquire 测试 |
| OPS-002 | 优雅停机停止新请求/Fetch等待现有流量后超时关闭 | 8981-9000 | Provider Run 收敛与 `Handler.Shutdown` HTTP 排空、Hijacked CONNECT 超时关闭测试 | | OPS-002 | 优雅停机停止新请求/Fetch等待现有流量后超时关闭 | 8981-9000 | Provider Run 收敛与 `Handler.Shutdown` HTTP 排空、Hijacked CONNECT 超时关闭测试 |
| OPS-003 | PostgreSQL 只保存管理修订、Upstream/Routing 状态、Admin 审计与 Outbox | 当前会话 | ADR-006、`adminstate` 公用契约和六表 Schema 边界测试;真实 PostgreSQL 契约待完成 | | OPS-003 | PostgreSQL 只保存管理修订、Upstream/Routing 状态、Admin 审计与 Outbox | 当前会话 | ADR-006、`adminstate` 公用契约和六表 Schema 边界测试;真实 PostgreSQL 契约待完成 |
| OBS-001 | 指标禁止 Proxy IP、session、Client、完整 URL 高基数标签 | 9001-9029 | Controller Prometheus/探针模块已实现Checker Collector 暴露固定等级和结果Gateway Collector 暴露 `proxy_pool_gateway_outcomes_total{stage,result}``proxy_pool_gateway_outcome_queue_dropped_total`Drain Collector 暴露 `proxy_pool_controller_drain_candidates_total{reason}``proxy_pool_controller_drains_started_total{reason}`,其中 reason 固定为 unhealthy/upstream_disabledProvider Collector 暴露固定 class 的拉取次数与候选计数Extraction Collector 暴露 `proxy_pool_controller_extraction_requests_total{result}`、请求数与交付数result 固定为 complete/partial/empty/insufficient/idempotency_conflict/rate_limited/unavailable/invalid/error。注册表测试锁定标签集容量指标和密钥安全的结构化日志仍待实现 | | OBS-001 | 指标禁止 Proxy IP、session、Client、完整 URL 高基数标签 | 9001-9029 | Controller Prometheus/探针模块已实现Checker Collector 暴露固定等级和结果Gateway Collector 暴露 `proxy_pool_gateway_outcomes_total{stage,result}``proxy_pool_gateway_outcome_queue_dropped_total`Drain Collector 暴露 `proxy_pool_controller_drain_candidates_total{reason}``proxy_pool_controller_drains_started_total{reason}`,其中 reason 固定为 unhealthy/upstream_disabledProvider Collector 暴露固定 class 的拉取次数与候选计数Extraction Collector 暴露固定 result 的请求、请求数与交付数Capacity Collector 按 Controller 进程聚合库存 Gauge并只暴露固定 success/error 的库存读取结果,不把 Upstream 写成标签。注册表测试锁定标签集;密钥安全的结构化日志仍待实现 |
| TEST-001 | 覆盖对话中列出的 11 个关键并发与故障场景 | 9030-9082 | 测试清单Redis 活动池由 Memory/Redis 公用契约覆盖,跨进程故障场景仍按清单推进 | | TEST-001 | 覆盖对话中列出的 11 个关键并发与故障场景 | 9030-9082 | 测试清单Redis 活动池由 Memory/Redis 公用契约覆盖,跨进程故障场景仍按清单推进 |

View File

@ -54,12 +54,17 @@ Extract 请求使用哪些 Upstream。
- Provider 拉取结果只按固定 `class` 聚合:`valid`、`empty`、`duplicate_only`、`error`。 - Provider 拉取结果只按固定 `class` 聚合:`valid`、`empty`、`duplicate_only`、`error`。
指标分别记录拉取结果次数、合法候选数和新入池代理数,不按 Upstream、Proxy、IP、错误 指标分别记录拉取结果次数、合法候选数和新入池代理数,不按 Upstream、Proxy、IP、错误
文本或凭据拆分。 文本或凭据拆分。
- Provider 指标已完成;提取、容量指标和密钥安全的结构化日志仍待实现。 - Provider 指标已完成;密钥安全的结构化日志仍待实现。
- Extraction 只按固定 `result` 聚合请求:`complete`、`partial`、`empty`、 - Extraction 只按固定 `result` 聚合请求:`complete`、`partial`、`empty`、
`insufficient`、`idempotency_conflict`、`rate_limited`、`unavailable`、`invalid`、 `insufficient`、`idempotency_conflict`、`rate_limited`、`unavailable`、`invalid`、
`error`。计数只记录请求数量和响应交付数量;幂等重放不被误记为新的 Proxy 消费。 `error`。计数只记录请求数量和响应交付数量;幂等重放不被误记为新的 Proxy 消费。
- Extraction 指标已完成;容量指标和密钥安全的结构化日志仍待实现。 - Extraction 指标已完成;密钥安全的结构化日志仍待实现。
- 容量指标由 Provider 库存对账循环采样,聚合 managed、available、effective、pending
与 active Upstream Gauge。Upstream ID 仅保留在 Controller 进程内用于聚合替换和
任期结束清理,不输出为 Prometheus 标签;读取结果只允许 `success``error`
- 容量指标已完成;密钥安全的结构化日志仍待实现。
## 集群与性能 ## 集群与性能

View File

@ -77,6 +77,7 @@ type ports struct {
coordinator provider.Coordinator coordinator provider.Coordinator
credentials credentials.Store credentials credentials.Store
providerResults provider.ResultRecorder providerResults provider.ResultRecorder
capacityMetrics pool.CapacityObserver
workerStore workerruntime.ControlStore workerStore workerruntime.ControlStore
close func() error close func() error
} }
@ -203,6 +204,11 @@ func runWithWorkerFactory(
if registrar, ok := opened.providerResults.(provider.ResultObserverRegistrar); ok && !nilInterface(registrar) { if registrar, ok := opened.providerResults.(provider.ResultObserverRegistrar); ok && !nilInterface(registrar) {
registrar.AddResultObserver(providerCollector) registrar.AddResultObserver(providerCollector)
} }
capacityCollector, capacityCollectorErr := platformMetrics.NewCapacityCollector(prometheus.DefaultRegisterer)
if capacityCollectorErr != nil {
return fmt.Errorf("%w: build capacity metrics: %w", ErrStartup, capacityCollectorErr)
}
opened.capacityMetrics = capacityCollector
extractionCollector, extractionCollectorErr := platformMetrics.NewExtractionCollector(prometheus.DefaultRegisterer) extractionCollector, extractionCollectorErr := platformMetrics.NewExtractionCollector(prometheus.DefaultRegisterer)
if extractionCollectorErr != nil { if extractionCollectorErr != nil {
return fmt.Errorf("%w: build extraction metrics: %w", ErrStartup, extractionCollectorErr) return fmt.Errorf("%w: build extraction metrics: %w", ErrStartup, extractionCollectorErr)

View File

@ -75,6 +75,7 @@ func providerRuntimeBuilder(opened ports) (buildUpstreamRuntime, error) {
Parser: parser, Parser: parser,
Activity: opened.activity, Activity: opened.activity,
Results: results, Results: results,
Capacity: opened.capacityMetrics,
}) })
}, nil }, nil
} }

View File

@ -0,0 +1,29 @@
package pool
// CapacityReadResult is the fixed outcome of one authoritative inventory read.
type CapacityReadResult string
const (
CapacityReadSuccess CapacityReadResult = "success"
CapacityReadError CapacityReadResult = "error"
)
// CapacityObservation contains one Controller-side inventory sample. Its
// UpstreamID and SourceID are for process-local aggregation only and must not
// be exposed as Prometheus labels.
type CapacityObservation struct {
UpstreamID string
SourceID string
Result CapacityReadResult
Managed int
AvailableSlots int64
EffectiveSlots int64
PendingExpected int
}
// CapacityObserver receives cold-path Provider inventory samples. It must
// return promptly and must not issue another inventory read or persistent I/O.
type CapacityObserver interface {
ObserveCapacity(CapacityObservation)
RemoveCapacityUpstream(upstreamID, sourceID string)
}

View File

@ -6,6 +6,7 @@ import (
"fmt" "fmt"
"hash/fnv" "hash/fnv"
"reflect" "reflect"
"sync/atomic"
"time" "time"
"proxy-pool/internal/controller/pool" "proxy-pool/internal/controller/pool"
@ -30,6 +31,7 @@ type UpstreamRuntimeDependencies struct {
Parser Parser Parser Parser
Activity activitypool.Upserter Activity activitypool.Upserter
Results ResultRecorder Results ResultRecorder
Capacity pool.CapacityObserver
} }
// UpstreamRuntime owns every leader-scoped object for one Upstream. A fresh // UpstreamRuntime owns every leader-scoped object for one Upstream. A fresh
@ -38,6 +40,7 @@ type UpstreamRuntime struct {
config UpstreamRuntimeConfig config UpstreamRuntimeConfig
dependencies UpstreamRuntimeDependencies dependencies UpstreamRuntimeDependencies
sleeper Sleeper sleeper Sleeper
termSequence atomic.Uint64
} }
func NewUpstreamRuntime( func NewUpstreamRuntime(
@ -126,26 +129,33 @@ func (runtime *UpstreamRuntime) newLeaderTerm() (*upstreamLeaderTerm, error) {
} }
return &upstreamLeaderTerm{ return &upstreamLeaderTerm{
upstreamID: runtime.config.Provider.UpstreamID, upstreamID: runtime.config.Provider.UpstreamID,
sourceID: fmt.Sprintf("%p:%d", runtime, runtime.termSequence.Add(1)),
interval: runtime.config.ReconcileInterval, interval: runtime.config.ReconcileInterval,
safetyMargin: runtime.config.ReconcilePolicy.SafetyMargin, safetyMargin: runtime.config.ReconcilePolicy.SafetyMargin,
inventory: runtime.dependencies.Inventory, inventory: runtime.dependencies.Inventory,
pool: poolReconciler, pool: poolReconciler,
provider: providerReconciler, provider: providerReconciler,
capacity: runtime.dependencies.Capacity,
sleeper: runtime.sleeper, sleeper: runtime.sleeper,
}, nil }, nil
} }
type upstreamLeaderTerm struct { type upstreamLeaderTerm struct {
upstreamID string upstreamID string
sourceID string
interval time.Duration interval time.Duration
safetyMargin time.Duration safetyMargin time.Duration
inventory pool.InventoryReader inventory pool.InventoryReader
pool *pool.Reconciler pool *pool.Reconciler
provider *Reconciler provider *Reconciler
capacity pool.CapacityObserver
sleeper Sleeper sleeper Sleeper
} }
func (term *upstreamLeaderTerm) run(ctx context.Context, session LeaderSession) error { func (term *upstreamLeaderTerm) run(ctx context.Context, session LeaderSession) error {
if !nilRuntimeDependency(term.capacity) {
defer term.capacity.RemoveCapacityUpstream(term.upstreamID, term.sourceID)
}
termCtx, cancel := context.WithCancel(ctx) termCtx, cancel := context.WithCancel(ctx)
defer cancel() defer cancel()
done := make(chan error, 2) done := make(chan error, 2)
@ -183,7 +193,16 @@ func (term *upstreamLeaderTerm) reconcileInventory(ctx context.Context) error {
term.safetyMargin, term.safetyMargin,
) )
if err == nil && inventory.Managed >= 0 && inventory.AvailableSlots >= 0 { if err == nil && inventory.Managed >= 0 && inventory.AvailableSlots >= 0 {
term.pool.ReconcileSnapshot(inventory) decision := term.pool.ReconcileSnapshot(inventory)
term.observeCapacity(pool.CapacityObservation{
UpstreamID: term.upstreamID, SourceID: term.sourceID, Result: pool.CapacityReadSuccess,
Managed: inventory.Managed, AvailableSlots: inventory.AvailableSlots,
EffectiveSlots: decision.EffectiveSlots, PendingExpected: decision.PendingExpected,
})
} else {
term.observeCapacity(pool.CapacityObservation{
UpstreamID: term.upstreamID, SourceID: term.sourceID, Result: pool.CapacityReadError,
})
} }
if err := term.sleeper.Sleep(ctx, term.interval); err != nil { if err := term.sleeper.Sleep(ctx, term.interval); err != nil {
if ctx.Err() != nil { if ctx.Err() != nil {
@ -195,6 +214,13 @@ func (term *upstreamLeaderTerm) reconcileInventory(ctx context.Context) error {
return nil return nil
} }
func (term *upstreamLeaderTerm) observeCapacity(observation pool.CapacityObservation) {
if term == nil || nilRuntimeDependency(term.capacity) {
return
}
term.capacity.ObserveCapacity(observation)
}
func initialReconcileDelay(upstreamID string, interval time.Duration) time.Duration { func initialReconcileDelay(upstreamID string, interval time.Duration) time.Duration {
window := min(interval/4, 250*time.Millisecond) window := min(interval/4, 250*time.Millisecond)
if window <= 1 { if window <= 1 {

View File

@ -126,6 +126,112 @@ func TestUpstreamRuntimeFailsClosedUntilInventoryReadRecovers(t *testing.T) {
} }
} }
func TestUpstreamRuntimeReportsCapacityReadsAndClearsTermState(t *testing.T) {
observer := newCapacityObserverStub()
runtime, err := NewUpstreamRuntime(UpstreamRuntimeConfig{
Provider: runtimeProviderConfig("provider-a"),
ReconcilePolicy: runtimeReconcilePolicy(),
ReconcileInterval: 10 * time.Millisecond,
}, UpstreamRuntimeDependencies{
Coordinator: coordinatorFunc(func(
ctx context.Context,
_ string,
_ CoordinationLimits,
work func(context.Context, LeaderSession) error,
) error {
return work(ctx, unlimitedLeaderSession{})
}),
Inventory: inventoryReaderFunc(func(context.Context, string, time.Duration) (pool.InventorySnapshot, error) {
return pool.InventorySnapshot{Managed: 3, AvailableSlots: 7}, nil
}),
Adapter: adapterFunc(func(context.Context) (FetchResponse, error) { return FetchResponse{}, nil }),
Parser: parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) { return nil, nil }),
Activity: activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
return activitypool.UpsertResult{}, nil
}),
Results: resultRecorderFunc(func(Result) {}),
Capacity: observer,
})
if err != nil {
t.Fatalf("NewUpstreamRuntime() error = %v", err)
}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- runtime.Run(ctx) }()
select {
case observed := <-observer.observations:
if observed.UpstreamID != "provider-a" || observed.SourceID == "" || observed.Result != pool.CapacityReadSuccess ||
observed.Managed != 3 || observed.AvailableSlots != 7 || observed.EffectiveSlots != 7 || observed.PendingExpected != 0 {
t.Fatalf("capacity observation = %+v", observed)
}
case <-time.After(time.Second):
cancel()
<-done
t.Fatal("timed out waiting for capacity observation")
}
cancel()
if err := <-done; !errors.Is(err, context.Canceled) {
t.Fatalf("Run() error = %v, want context cancellation", err)
}
select {
case removal := <-observer.removed:
if removal.upstreamID != "provider-a" || removal.sourceID == "" {
t.Fatalf("capacity removal = %+v", removal)
}
case <-time.After(time.Second):
t.Fatal("timed out waiting for capacity cleanup")
}
}
func TestUpstreamRuntimeReportsFailedCapacityReads(t *testing.T) {
observer := newCapacityObserverStub()
runtime, err := NewUpstreamRuntime(UpstreamRuntimeConfig{
Provider: runtimeProviderConfig("provider-a"),
ReconcilePolicy: runtimeReconcilePolicy(),
ReconcileInterval: 10 * time.Millisecond,
}, UpstreamRuntimeDependencies{
Coordinator: coordinatorFunc(func(
ctx context.Context,
_ string,
_ CoordinationLimits,
work func(context.Context, LeaderSession) error,
) error {
return work(ctx, unlimitedLeaderSession{})
}),
Inventory: inventoryReaderFunc(func(context.Context, string, time.Duration) (pool.InventorySnapshot, error) {
return pool.InventorySnapshot{}, errors.New("redis unavailable")
}),
Adapter: adapterFunc(func(context.Context) (FetchResponse, error) { return FetchResponse{}, nil }),
Parser: parserFunc(func(context.Context, []byte) ([]proxyDomain.Proxy, error) { return nil, nil }),
Activity: activitySinkFunc(func(context.Context, string, activitypool.FetchedBatch) (activitypool.UpsertResult, error) {
return activitypool.UpsertResult{}, nil
}),
Results: resultRecorderFunc(func(Result) {}),
Capacity: observer,
})
if err != nil {
t.Fatalf("NewUpstreamRuntime() error = %v", err)
}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- runtime.Run(ctx) }()
select {
case observed := <-observer.observations:
if observed.UpstreamID != "provider-a" || observed.SourceID == "" || observed.Result != pool.CapacityReadError ||
observed.Managed != 0 || observed.AvailableSlots != 0 || observed.EffectiveSlots != 0 || observed.PendingExpected != 0 {
t.Fatalf("capacity observation = %+v", observed)
}
case <-time.After(time.Second):
cancel()
<-done
t.Fatal("timed out waiting for failed capacity observation")
}
cancel()
<-done
}
func TestUpstreamRuntimeDelegatesRequestIntervalOnlyToCoordinator(t *testing.T) { func TestUpstreamRuntimeDelegatesRequestIntervalOnlyToCoordinator(t *testing.T) {
providerConfig := runtimeProviderConfig("provider-a") providerConfig := runtimeProviderConfig("provider-a")
providerConfig.RequestInterval = 500 * time.Millisecond providerConfig.RequestInterval = 500 * time.Millisecond
@ -245,3 +351,28 @@ func (f inventoryReaderFunc) ReadInventory(
) (pool.InventorySnapshot, error) { ) (pool.InventorySnapshot, error) {
return f(ctx, upstreamID, safetyMargin) return f(ctx, upstreamID, safetyMargin)
} }
type capacityObserverStub struct {
observations chan pool.CapacityObservation
removed chan capacityRemoval
}
type capacityRemoval struct {
upstreamID string
sourceID string
}
func newCapacityObserverStub() *capacityObserverStub {
return &capacityObserverStub{
observations: make(chan pool.CapacityObservation, 4),
removed: make(chan capacityRemoval, 2),
}
}
func (observer *capacityObserverStub) ObserveCapacity(observation pool.CapacityObservation) {
observer.observations <- observation
}
func (observer *capacityObserverStub) RemoveCapacityUpstream(upstreamID, sourceID string) {
observer.removed <- capacityRemoval{upstreamID: upstreamID, sourceID: sourceID}
}

View File

@ -0,0 +1,160 @@
package metrics
import (
"errors"
"fmt"
"sync"
"github.com/prometheus/client_golang/prometheus"
controllerPool "proxy-pool/internal/controller/pool"
)
// CapacityCollector aggregates Controller leader-term inventory samples. An
// upstream ID is retained only in process memory for replacement and removal;
// it is never emitted as a Prometheus label.
type CapacityCollector struct {
mu sync.Mutex
byUpstream map[string]capacityState
reads *prometheus.CounterVec
managed *prometheus.Desc
available *prometheus.Desc
effective *prometheus.Desc
pending *prometheus.Desc
activeUpstreams *prometheus.Desc
}
type capacityState struct {
sourceID string
managed int
available int64
effective int64
pending int
}
var (
_ controllerPool.CapacityObserver = (*CapacityCollector)(nil)
_ prometheus.Collector = (*CapacityCollector)(nil)
)
func NewCapacityCollector(registerer prometheus.Registerer) (*CapacityCollector, error) {
if registerer == nil {
return nil, ErrInvalidDependencies
}
reads, err := registerCounterVec(registerer, prometheus.NewCounterVec(prometheus.CounterOpts{
Namespace: "proxy_pool", Subsystem: "controller", Name: "capacity_inventory_reads_total",
Help: "Number of Provider inventory reads by fixed result.",
}, []string{"result"}))
if err != nil {
return nil, err
}
candidate := &CapacityCollector{
byUpstream: map[string]capacityState{}, reads: reads,
managed: prometheus.NewDesc(
"proxy_pool_controller_capacity_managed_proxies",
"Managed non-extracted proxies from the latest successful samples of active Controller Provider terms.", nil, nil,
),
available: prometheus.NewDesc(
"proxy_pool_controller_capacity_available_slots",
"Available proxy concurrency slots from the latest successful samples of active Controller Provider terms.", nil, nil,
),
effective: prometheus.NewDesc(
"proxy_pool_controller_capacity_effective_slots",
"Available slots plus expected pending fetch capacity from the latest successful active Controller samples.", nil, nil,
),
pending: prometheus.NewDesc(
"proxy_pool_controller_capacity_pending_expected_proxies",
"Expected proxy candidates reserved by in-flight fetches in the latest successful active Controller samples.", nil, nil,
),
activeUpstreams: prometheus.NewDesc(
"proxy_pool_controller_capacity_active_upstreams",
"Number of Upstreams with a successful inventory sample in this Controller process.", nil, nil,
),
}
return registerCapacityCollector(registerer, candidate)
}
func (collector *CapacityCollector) ObserveCapacity(observation controllerPool.CapacityObservation) {
if collector == nil || collector.reads == nil || observation.UpstreamID == "" || observation.SourceID == "" {
return
}
switch observation.Result {
case controllerPool.CapacityReadError:
collector.reads.WithLabelValues(string(observation.Result)).Inc()
return
case controllerPool.CapacityReadSuccess:
if observation.Managed < 0 || observation.AvailableSlots < 0 || observation.EffectiveSlots < 0 || observation.PendingExpected < 0 {
return
}
collector.reads.WithLabelValues(string(observation.Result)).Inc()
default:
return
}
collector.mu.Lock()
collector.byUpstream[observation.UpstreamID] = capacityState{
sourceID: observation.SourceID,
managed: observation.Managed, available: observation.AvailableSlots,
effective: observation.EffectiveSlots, pending: observation.PendingExpected,
}
collector.mu.Unlock()
}
func (collector *CapacityCollector) RemoveCapacityUpstream(upstreamID, sourceID string) {
if collector == nil || upstreamID == "" || sourceID == "" {
return
}
collector.mu.Lock()
if state, exists := collector.byUpstream[upstreamID]; exists && state.sourceID == sourceID {
delete(collector.byUpstream, upstreamID)
}
collector.mu.Unlock()
}
func (collector *CapacityCollector) Describe(descriptions chan<- *prometheus.Desc) {
if collector == nil {
return
}
descriptions <- collector.managed
descriptions <- collector.available
descriptions <- collector.effective
descriptions <- collector.pending
descriptions <- collector.activeUpstreams
}
func (collector *CapacityCollector) Collect(metrics chan<- prometheus.Metric) {
if collector == nil {
return
}
collector.mu.Lock()
var managed, available, effective, pending int64
for _, state := range collector.byUpstream {
managed += int64(state.managed)
available += state.available
effective += state.effective
pending += int64(state.pending)
}
activeUpstreams := len(collector.byUpstream)
collector.mu.Unlock()
metrics <- prometheus.MustNewConstMetric(collector.managed, prometheus.GaugeValue, float64(managed))
metrics <- prometheus.MustNewConstMetric(collector.available, prometheus.GaugeValue, float64(available))
metrics <- prometheus.MustNewConstMetric(collector.effective, prometheus.GaugeValue, float64(effective))
metrics <- prometheus.MustNewConstMetric(collector.pending, prometheus.GaugeValue, float64(pending))
metrics <- prometheus.MustNewConstMetric(collector.activeUpstreams, prometheus.GaugeValue, float64(activeUpstreams))
}
func registerCapacityCollector(registerer prometheus.Registerer, candidate *CapacityCollector) (*CapacityCollector, error) {
if err := registerer.Register(candidate); err == nil {
return candidate, nil
} else {
var registered prometheus.AlreadyRegisteredError
if !errors.As(err, &registered) {
return nil, fmt.Errorf("register capacity collector: %w", err)
}
existing, ok := registered.ExistingCollector.(*CapacityCollector)
if !ok {
return nil, fmt.Errorf("register capacity collector: existing collector has unexpected type")
}
return existing, nil
}
}

View File

@ -0,0 +1,115 @@
package metrics
import (
"testing"
"github.com/prometheus/client_golang/prometheus"
controllerPool "proxy-pool/internal/controller/pool"
)
func TestCapacityCollectorAggregatesActiveUpstreamsWithoutLabels(t *testing.T) {
registry := prometheus.NewRegistry()
collector, err := NewCapacityCollector(registry)
if err != nil {
t.Fatalf("NewCapacityCollector() error = %v", err)
}
collector.ObserveCapacity(controllerPool.CapacityObservation{
UpstreamID: "provider-a", SourceID: "term-a", Result: controllerPool.CapacityReadSuccess,
Managed: 2, AvailableSlots: 5, EffectiveSlots: 7, PendingExpected: 2,
})
collector.ObserveCapacity(controllerPool.CapacityObservation{
UpstreamID: "provider-b", SourceID: "term-b", Result: controllerPool.CapacityReadSuccess,
Managed: 3, AvailableSlots: 4, EffectiveSlots: 6, PendingExpected: 1,
})
collector.ObserveCapacity(controllerPool.CapacityObservation{
UpstreamID: "provider-a", SourceID: "term-a", Result: controllerPool.CapacityReadError,
})
collector.ObserveCapacity(controllerPool.CapacityObservation{
UpstreamID: "client-a", SourceID: "term-client", Result: "invalid", Managed: 100,
})
assertMetricValue(t, registry, "proxy_pool_controller_capacity_inventory_reads_total", map[string]string{"result": "success"}, 2)
assertMetricValue(t, registry, "proxy_pool_controller_capacity_inventory_reads_total", map[string]string{"result": "error"}, 1)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_managed_proxies", 5)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_available_slots", 9)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_effective_slots", 13)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_pending_expected_proxies", 3)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_active_upstreams", 2)
collector.RemoveCapacityUpstream("provider-a", "term-a")
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_managed_proxies", 3)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_available_slots", 4)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_effective_slots", 6)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_pending_expected_proxies", 1)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_active_upstreams", 1)
}
func TestNewCapacityCollectorReusesRegisteredCollectors(t *testing.T) {
registry := prometheus.NewRegistry()
first, err := NewCapacityCollector(registry)
if err != nil {
t.Fatalf("first NewCapacityCollector() error = %v", err)
}
second, err := NewCapacityCollector(registry)
if err != nil {
t.Fatalf("second NewCapacityCollector() error = %v", err)
}
first.ObserveCapacity(controllerPool.CapacityObservation{
UpstreamID: "provider-a", SourceID: "term-a", Result: controllerPool.CapacityReadSuccess,
Managed: 1, AvailableSlots: 2, EffectiveSlots: 3,
})
second.ObserveCapacity(controllerPool.CapacityObservation{
UpstreamID: "provider-b", SourceID: "term-b", Result: controllerPool.CapacityReadSuccess,
Managed: 2, AvailableSlots: 3, EffectiveSlots: 5, PendingExpected: 1,
})
assertMetricValue(t, registry, "proxy_pool_controller_capacity_inventory_reads_total", map[string]string{"result": "success"}, 2)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_managed_proxies", 3)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_available_slots", 5)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_effective_slots", 8)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_pending_expected_proxies", 1)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_active_upstreams", 2)
}
func TestCapacityCollectorKeepsNewTermWhenOldTermStops(t *testing.T) {
registry := prometheus.NewRegistry()
collector, err := NewCapacityCollector(registry)
if err != nil {
t.Fatalf("NewCapacityCollector() error = %v", err)
}
collector.ObserveCapacity(controllerPool.CapacityObservation{
UpstreamID: "provider-a", SourceID: "old-term", Result: controllerPool.CapacityReadSuccess,
Managed: 1, AvailableSlots: 2, EffectiveSlots: 3,
})
collector.ObserveCapacity(controllerPool.CapacityObservation{
UpstreamID: "provider-a", SourceID: "new-term", Result: controllerPool.CapacityReadSuccess,
Managed: 4, AvailableSlots: 5, EffectiveSlots: 6, PendingExpected: 1,
})
collector.RemoveCapacityUpstream("provider-a", "old-term")
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_managed_proxies", 4)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_available_slots", 5)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_effective_slots", 6)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_pending_expected_proxies", 1)
assertGaugeValue(t, registry, "proxy_pool_controller_capacity_active_upstreams", 1)
}
func assertGaugeValue(t *testing.T, registry *prometheus.Registry, name string, want float64) {
t.Helper()
metrics, err := registry.Gather()
if err != nil {
t.Fatalf("Gather() = %v", err)
}
for _, family := range metrics {
if family.GetName() != name {
continue
}
for _, metric := range family.GetMetric() {
if len(metric.GetLabel()) == 0 && metric.GetGauge().GetValue() == want {
return
}
}
}
t.Fatalf("gauge %s value=%v was not found", name, want)
}

View File

@ -2,17 +2,22 @@
## 2026-08-02 ## 2026-08-02
- 容量可观测性已接入 Provider 补池对账循环:每个成功 Redis 库存读数更新 Controller
进程级 managed/available/effective/pending/active Gauge失败只增加固定
`success`/`error` 的读取计数。Upstream 仅用于进程内聚合,领导任期结束即清除,
不成为 Prometheus 标签;不进入 Gateway 10 万 QPS 容量热路径,也不增加 Redis 查询。
密钥安全的结构化日志仍待实现。
- Extraction 可观测性已接入公用 `MetricsObserver`Controller 暴露固定 `result` - Extraction 可观测性已接入公用 `MetricsObserver`Controller 暴露固定 `result`
的请求计数、请求 Proxy 总数和响应交付 Proxy 总数。事件在服务单一出口分类,覆盖 的请求计数、请求 Proxy 总数和响应交付 Proxy 总数。事件在服务单一出口分类,覆盖
complete/partial/empty/insufficient/idempotency_conflict/rate_limited/unavailable/ complete/partial/empty/insufficient/idempotency_conflict/rate_limited/unavailable/
invalid/error不带 Client、Request、Filter、Upstream、Proxy 或错误文本标签。 invalid/error不带 Client、Request、Filter、Upstream、Proxy 或错误文本标签。
幂等重放统计响应交付而不误计为新的 Redis 消费。容量指标和密钥安全结构化日志仍待实现。 幂等重放统计响应交付而不误计为新的 Redis 消费。密钥安全结构化日志仍待实现。
- Provider 拉取可观测性已接入公用 `ResultObserver`Controller 暴露 - Provider 拉取可观测性已接入公用 `ResultObserver`Controller 暴露
`proxy_pool_controller_provider_fetch_results_total{class}` `proxy_pool_controller_provider_fetch_results_total{class}`
`proxy_pool_controller_provider_valid_candidates_total` `proxy_pool_controller_provider_valid_candidates_total`
`proxy_pool_controller_provider_new_proxies_total`。`class` 仅允许 `valid`、`empty`、 `proxy_pool_controller_provider_new_proxies_total`。`class` 仅允许 `valid`、`empty`、
`duplicate_only`、`error`;指标不包含 Upstream、Proxy、IP、错误文本或凭据标签。 `duplicate_only`、`error`;指标不包含 Upstream、Proxy、IP、错误文本或凭据标签。
Provider 指标已完成,提取、容量指标和密钥安全的结构化日志仍待实现。 Provider 指标已完成密钥安全的结构化日志仍待实现。
- 已实现管理态驱动的即时完整快照刷新:`SnapshotRefreshBroker` 以每个 Worker 流一格的 - 已实现管理态驱动的即时完整快照刷新:`SnapshotRefreshBroker` 以每个 Worker 流一格的
非阻塞通道广播连续变更自动合并。Admin 成功提交 Upstream 启停、Routing 切换和配置 非阻塞通道广播连续变更自动合并。Admin 成功提交 Upstream 启停、Routing 切换和配置
发布时触发通知Worker 保留原有定时刷新作为失效保护。Routing 切换不按旧 Upstream 发布时触发通知Worker 保留原有定时刷新作为失效保护。Routing 切换不按旧 Upstream