Compare commits
2 Commits
6f5a2faea6
...
c71f5985d2
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c71f5985d2 | ||
|
|
596e4ab371 |
@ -100,6 +100,8 @@ pp:{activity}:carrier:<value> ZSET 运营商候选索引
|
|||||||
pp:{activity}:upstream:<value> ZSET 供应商候选索引
|
pp:{activity}:upstream:<value> ZSET 供应商候选索引
|
||||||
pp:{activity}:owners HASH proxyID -> ownership assignment
|
pp:{activity}:owners HASH proxyID -> ownership assignment
|
||||||
pp:{activity}:owner-expiry ZSET proxyID -> ownership expiry milliseconds
|
pp:{activity}:owner-expiry ZSET proxyID -> ownership expiry milliseconds
|
||||||
|
pp:{activity}:drain-tickets HASH proxyID -> pending drain ticket
|
||||||
|
pp:{activity}:worker-draining:<digest> ZSET 单 Worker 待绑定 Drain Ticket
|
||||||
pp:{activity}:epoch STRING ownership 全局递增代次
|
pp:{activity}:epoch STRING ownership 全局递增代次
|
||||||
pp:{activity}:inventory HASH upstreamID -> 当前未提取库存
|
pp:{activity}:inventory HASH upstreamID -> 当前未提取库存
|
||||||
pp:{activity}:worker-sessions HASH workerID -> 当前 Worker session
|
pp:{activity}:worker-sessions HASH workerID -> 当前 Worker session
|
||||||
@ -180,6 +182,8 @@ Assign、Renew、BeginDrain 和 AcknowledgeDrain 分别使用有界小脚本,
|
|||||||
- Assign 只接受 AVAILABLE、无 owner 且未到 `usableUntil` 的 Proxy。
|
- Assign 只接受 AVAILABLE、无 owner 且未到 `usableUntil` 的 Proxy。
|
||||||
- Renew 必须匹配 worker、epoch,并把 lease 截断到 `usableUntil`。
|
- Renew 必须匹配 worker、epoch,并把 lease 截断到 `usableUntil`。
|
||||||
- BeginDrain 对同一 assignment 幂等。
|
- BeginDrain 对同一 assignment 幂等。
|
||||||
|
- 首次 BeginDrain 原子写入按 Worker 索引的待绑定 Drain Ticket,并推进全局 ownership
|
||||||
|
epoch;`worker-owned` 已移除该 Proxy,下一份完整快照可据此撤销新预留。
|
||||||
- AcknowledgeDrain 仅在 Active 和 Reserved 都为零时释放所有权。
|
- AcknowledgeDrain 仅在 Active 和 Reserved 都为零时释放所有权。
|
||||||
- Assign 与 Extract 并发竞争同一 Proxy 时,只允许一个操作成功。
|
- Assign 与 Extract 并发竞争同一 Proxy 时,只允许一个操作成功。
|
||||||
- Expire 使用 `limit` 分批回收过期 assignment,禁止无界返回。
|
- Expire 使用 `limit` 分批回收过期 assignment,禁止无界返回。
|
||||||
@ -190,6 +194,10 @@ Gateway 的 `Capacity` 使用一次打包原子读取取得同一时刻的 Activ
|
|||||||
`snapshot.Store` 周期生成完整稀疏报告。当前 Snapshot 已移除但仍有活动连接的
|
`snapshot.Store` 周期生成完整稀疏报告。当前 Snapshot 已移除但仍有活动连接的
|
||||||
Proxy 继续以 `draining=true` 上报,直到 Active/Reserved 同时归零。
|
Proxy 继续以 `draining=true` 上报,直到 Active/Reserved 同时归零。
|
||||||
|
|
||||||
|
Drain Ticket 目前只记录“必须由不早于所需 epoch 的完整快照排除”的持久化前置条件。
|
||||||
|
后续会在 Runtime 替换 Lua 事务中同时验证 Snapshot ACK、Ticket 屏障和零计数,避免
|
||||||
|
拆分为读 Runtime 再释放所有权产生竞态。
|
||||||
|
|
||||||
Redis Adapter 在单个 `{activity}` 原子边界内维护 Worker session 和运行态报告:
|
Redis Adapter 在单个 `{activity}` 原子边界内维护 Worker session 和运行态报告:
|
||||||
|
|
||||||
1. 新 Worker session 替换旧 session,并隔离旧实例后续写入。
|
1. 新 Worker session 替换旧 session,并隔离旧实例后续写入。
|
||||||
|
|||||||
@ -102,6 +102,10 @@ Worker 自报的超前版本或 epoch 也必须拒绝。
|
|||||||
和停止新分配的 `usable_until`。
|
和停止新分配的 `usable_until`。
|
||||||
- 内容 `checksum`。
|
- 内容 `checksum`。
|
||||||
|
|
||||||
|
`version` 在同一 Worker 流内严格连续递增,即使 `ownership_epoch` 因分配、Drain
|
||||||
|
或所有权恢复而前进也不重置。Gateway 只接受比当前 `version` 恰好大一的完整快照,
|
||||||
|
并拒绝倒退的 epoch;因此 epoch 变化不会让仍在长连接内的刷新快照被误判为缺口。
|
||||||
|
|
||||||
`usable_until = expires_at - allocationSafetyMargin`。Worker 必须以
|
`usable_until = expires_at - allocationSafetyMargin`。Worker 必须以
|
||||||
`usable_until` 作为最后可分配时刻;达到该时间后即使尚未到 `expires_at`,
|
`usable_until` 作为最后可分配时刻;达到该时间后即使尚未到 `expires_at`,
|
||||||
也不得再为新请求选择该 Proxy。
|
也不得再为新请求选择该 Proxy。
|
||||||
@ -131,6 +135,12 @@ Delta 声明 `base_version`。Worker 只有在本地版本恰好等于 base 且
|
|||||||
3. 两个计数都归零后 Controller 清除所有权。
|
3. 两个计数都归零后 Controller 清除所有权。
|
||||||
4. 无所有权 Proxy 才能进入 Distribution 的 Redis 原子提取操作。
|
4. 无所有权 Proxy 才能进入 Distribution 的 Redis 原子提取操作。
|
||||||
|
|
||||||
|
首次 `BeginDrain` 会原子创建按 `proxy_id + worker_id + assignment_epoch` 栅栏的待绑定
|
||||||
|
Drain Ticket,并推进全局 ownership epoch,促使下一份完整 Snapshot 撤销该 Proxy。Ticket
|
||||||
|
按 Worker 有界读取,供 Controller 在确认完整 Snapshot 确实不含该 Proxy 后绑定快照屏障。
|
||||||
|
当前实现尚未把该屏障和 Runtime 报告放入同一最终确认事务,因此不会自动释放所有权;
|
||||||
|
`AcknowledgeDrain` 仍是已有的显式完成原语。
|
||||||
|
|
||||||
Worker 崩溃时必须等待所有权 epoch/有效期失效后再转移,避免双主。Proto 中
|
Worker 崩溃时必须等待所有权 epoch/有效期失效后再转移,避免双主。Proto 中
|
||||||
`ReportRuntimeResponse.revoke_proxy_ids` 是加速 Drain 的控制信号,不绕过
|
`ReportRuntimeResponse.revoke_proxy_ids` 是加速 Drain 的控制信号,不绕过
|
||||||
Snapshot 版本和 Redis 中的权威所有权状态。
|
Snapshot 版本和 Redis 中的权威所有权状态。
|
||||||
|
|||||||
12
findings.md
12
findings.md
@ -123,6 +123,18 @@ Routing 自上而下匹配,首条命中停止;支持 Gateway 与 Extract 两
|
|||||||
Gateway 使用既有 `SessionSupervisor` 退避重连。Gateway 本地 Store 对从完整 Snapshot
|
Gateway 使用既有 `SessionSupervisor` 退避重连。Gateway 本地 Store 对从完整 Snapshot
|
||||||
消失且仍有 Active/Reserved 的 Proxy 已按 draining 继续上报,但 Controller 尚未具备
|
消失且仍有 Active/Reserved 的 Proxy 已按 draining 继续上报,但 Controller 尚未具备
|
||||||
自动 `BeginDrain`/`AcknowledgeDrain` 编排或 Worker 分配器,不能视为自动 Drain 闭环。
|
自动 `BeginDrain`/`AcknowledgeDrain` 编排或 Worker 分配器,不能视为自动 Drain 闭环。
|
||||||
|
- Snapshot 的 `version` 是同一 Worker 流的连续序列,`ownership_epoch` 是独立且只能前进
|
||||||
|
的权威栅栏;epoch 变化不重置 version。此前 Gateway 错把 epoch 前进要求为 version=1,
|
||||||
|
与 Controller 的 `lastAppliedVersion+1` 生成规则冲突,现已用连续版本规则统一。
|
||||||
|
|
||||||
|
## Drain Ticket(2026-08-02)
|
||||||
|
|
||||||
|
- 原有 `BeginDrain` 只会把 Proxy 从 Worker 下发索引移除,无法让后续步骤区分“已发起
|
||||||
|
撤销”与“Gateway 已收到排除该 Proxy 的完整快照”。现已增加按 Worker 有界读取的持久化
|
||||||
|
Ticket,并以 RequiredSnapshotEpoch 强制下一份权威快照至少跨过 Drain 操作。
|
||||||
|
- Ticket 不能单独成为释放依据。最终清理必须把 Ticket 屏障、当前 session ACK 和完整
|
||||||
|
Runtime 替换中的零计数置于同一 Redis Lua 原子边界;`ReadRuntime` 后再调用
|
||||||
|
`AcknowledgeDrain` 会保留竞争窗口。
|
||||||
|
|
||||||
## Git 同步事实(2026-07-29)
|
## Git 同步事实(2026-07-29)
|
||||||
|
|
||||||
|
|||||||
@ -57,6 +57,15 @@ type ownershipRecord struct {
|
|||||||
Draining bool `json:"draining"`
|
Draining bool `json:"draining"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type drainTicketRecord struct {
|
||||||
|
Version int `json:"version"`
|
||||||
|
ProxyID string `json:"proxyId"`
|
||||||
|
WorkerID string `json:"workerId"`
|
||||||
|
WorkerIndexKey string `json:"workerIndexKey"`
|
||||||
|
AssignmentEpoch uint64 `json:"assignmentEpoch"`
|
||||||
|
RequiredSnapshotEpoch uint64 `json:"requiredSnapshotEpoch"`
|
||||||
|
}
|
||||||
|
|
||||||
type idempotencyRecord struct {
|
type idempotencyRecord struct {
|
||||||
Version int
|
Version int
|
||||||
RequestDigest string
|
RequestDigest string
|
||||||
@ -240,6 +249,16 @@ func validateOwnershipRecord(record ownershipRecord) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateDrainTicketRecord(record drainTicketRecord) error {
|
||||||
|
if record.Version != recordVersion || record.ProxyID == "" || record.WorkerID == "" ||
|
||||||
|
record.AssignmentEpoch == 0 || record.RequiredSnapshotEpoch == 0 ||
|
||||||
|
record.RequiredSnapshotEpoch <= record.AssignmentEpoch || record.WorkerIndexKey == "" ||
|
||||||
|
!strings.Contains(record.WorkerIndexKey, "{activity}") {
|
||||||
|
return ErrInvalidRecord
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func validateIdempotencyRecord(record idempotencyRecord) error {
|
func validateIdempotencyRecord(record idempotencyRecord) error {
|
||||||
if record.Version != recordVersion {
|
if record.Version != recordVersion {
|
||||||
return invalidRecord("unsupported idempotency version")
|
return invalidRecord("unsupported idempotency version")
|
||||||
|
|||||||
@ -17,6 +17,7 @@ type keyspace struct {
|
|||||||
available string
|
available string
|
||||||
owners string
|
owners string
|
||||||
ownerExpiry string
|
ownerExpiry string
|
||||||
|
drainTickets string
|
||||||
epoch string
|
epoch string
|
||||||
inventory string
|
inventory string
|
||||||
stateInventory string
|
stateInventory string
|
||||||
@ -49,6 +50,7 @@ func newKeyspace(namespace string) keyspace {
|
|||||||
available: prefix + ":available",
|
available: prefix + ":available",
|
||||||
owners: prefix + ":owners",
|
owners: prefix + ":owners",
|
||||||
ownerExpiry: prefix + ":owner-expiry",
|
ownerExpiry: prefix + ":owner-expiry",
|
||||||
|
drainTickets: prefix + ":drain-tickets",
|
||||||
epoch: prefix + ":epoch",
|
epoch: prefix + ":epoch",
|
||||||
inventory: prefix + ":inventory",
|
inventory: prefix + ":inventory",
|
||||||
stateInventory: prefix + ":state-inventory",
|
stateInventory: prefix + ":state-inventory",
|
||||||
@ -122,6 +124,10 @@ func (keys keyspace) workerOwned(workerID string) string {
|
|||||||
return keys.facet("worker-owned", workerID)
|
return keys.facet("worker-owned", workerID)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (keys keyspace) workerDraining(workerID string) string {
|
||||||
|
return keys.facet("worker-draining", workerID)
|
||||||
|
}
|
||||||
|
|
||||||
func (keys keyspace) facet(name, value string) string {
|
func (keys keyspace) facet(name, value string) string {
|
||||||
return keys.prefix + ":" + name + ":" + digestToken(value)
|
return keys.prefix + ":" + name + ":" + digestToken(value)
|
||||||
}
|
}
|
||||||
|
|||||||
@ -19,6 +19,7 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
var _ ownershipDomain.Repository = (*Adapter)(nil)
|
var _ ownershipDomain.Repository = (*Adapter)(nil)
|
||||||
|
var _ ownershipDomain.DrainTicketStore = (*Adapter)(nil)
|
||||||
|
|
||||||
func (a *Adapter) Assign(
|
func (a *Adapter) Assign(
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
@ -109,6 +110,43 @@ func (a *Adapter) BeginDrain(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (a *Adapter) PendingDrains(ctx context.Context, workerID string, limit int) ([]ownershipDomain.DrainTicket, error) {
|
||||||
|
if err := validateOwnershipCall(ctx, a); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if workerID == "" || limit <= 0 {
|
||||||
|
return nil, ownershipDomain.ErrInvalidDrainTicket
|
||||||
|
}
|
||||||
|
result, err := runScript(ctx, a.client, drainTicketsScript, []string{
|
||||||
|
a.keys.drainTickets, a.keys.workerDraining(workerID), a.keys.owners,
|
||||||
|
}, workerID, limit)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
var reply drainTicketsScriptReply
|
||||||
|
if err := decodeScriptResult(result, &reply); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if reply.Status == scriptInvalid {
|
||||||
|
return nil, ownershipDomain.ErrInvalidDrainTicket
|
||||||
|
}
|
||||||
|
if reply.Status != scriptOK {
|
||||||
|
return nil, invalidScriptReply("unexpected pending drain tickets reply")
|
||||||
|
}
|
||||||
|
tickets := make([]ownershipDomain.DrainTicket, 0, len(reply.Tickets))
|
||||||
|
for _, raw := range reply.Tickets {
|
||||||
|
var record drainTicketRecord
|
||||||
|
if err := decodeJSON(raw, &record); err != nil || validateDrainTicketRecord(record) != nil {
|
||||||
|
return nil, invalidScriptReply("pending drain tickets reply contained an invalid ticket")
|
||||||
|
}
|
||||||
|
tickets = append(tickets, ownershipDomain.DrainTicket{
|
||||||
|
ProxyID: record.ProxyID, WorkerID: record.WorkerID, AssignmentEpoch: record.AssignmentEpoch,
|
||||||
|
RequiredSnapshotEpoch: record.RequiredSnapshotEpoch,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return tickets, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (a *Adapter) AcknowledgeDrain(
|
func (a *Adapter) AcknowledgeDrain(
|
||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
proxyID string,
|
proxyID string,
|
||||||
@ -226,8 +264,9 @@ func (a *Adapter) runOwnership(
|
|||||||
a.keys.inventory, a.keys.stateInventory, a.keys.owners, a.keys.ownerExpiry,
|
a.keys.inventory, a.keys.stateInventory, a.keys.owners, a.keys.ownerExpiry,
|
||||||
a.keys.epoch, operationKey, a.keys.healthDue, a.keys.healthQueued, a.keys.healthLeases,
|
a.keys.epoch, operationKey, a.keys.healthDue, a.keys.healthQueued, a.keys.healthLeases,
|
||||||
a.keys.healthTasks, a.keys.healthTaskExpiry, a.keys.healthRefTask, a.keys.healthUnhealthy,
|
a.keys.healthTasks, a.keys.healthTaskExpiry, a.keys.healthRefTask, a.keys.healthUnhealthy,
|
||||||
|
a.keys.drainTickets,
|
||||||
}, operation, operationTTLMillis(a.options.OperationTTL), a.options.CleanupLimit,
|
}, operation, operationTTLMillis(a.options.OperationTTL), a.options.CleanupLimit,
|
||||||
nowMS, proxyID, workerID, epoch, value, active, reserved, a.keys.workerOwned(workerID))
|
nowMS, proxyID, workerID, epoch, value, active, reserved, a.keys.workerOwned(workerID), a.keys.workerDraining(workerID))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ownershipScriptReply{}, err
|
return ownershipScriptReply{}, err
|
||||||
}
|
}
|
||||||
|
|||||||
@ -64,6 +64,14 @@ func TestRedisOwnershipLifecycle(t *testing.T) {
|
|||||||
t.Fatalf("BeginDrain() = %+v, %v", draining, err)
|
t.Fatalf("BeginDrain() = %+v, %v", draining, err)
|
||||||
}
|
}
|
||||||
assertRedisWorkerOwned(t, fixture, "worker-a")
|
assertRedisWorkerOwned(t, fixture, "worker-a")
|
||||||
|
pending, err := fixture.Adapter.PendingDrains(context.Background(), "worker-a", 1)
|
||||||
|
if err != nil || len(pending) != 1 || pending[0].ProxyID != "proxy-a" ||
|
||||||
|
pending[0].AssignmentEpoch != assigned.Epoch || pending[0].RequiredSnapshotEpoch <= assigned.Epoch {
|
||||||
|
t.Fatalf("PendingDrains() = %+v, %v", pending, err)
|
||||||
|
}
|
||||||
|
if tickets, err := fixture.Client.ZRange(context.Background(), fixture.Adapter.keys.workerDraining("worker-a"), 0, -1).Result(); err != nil || len(tickets) != 1 || tickets[0] != "proxy-a" {
|
||||||
|
t.Fatalf("worker-draining = %v, %v", tickets, err)
|
||||||
|
}
|
||||||
replayed, err := fixture.Adapter.BeginDrain(context.Background(), "proxy-a", "worker-a", assigned.Epoch)
|
replayed, err := fixture.Adapter.BeginDrain(context.Background(), "proxy-a", "worker-a", assigned.Epoch)
|
||||||
if err != nil || replayed != draining {
|
if err != nil || replayed != draining {
|
||||||
t.Fatalf("BeginDrain(replay) = %+v, %v", replayed, err)
|
t.Fatalf("BeginDrain(replay) = %+v, %v", replayed, err)
|
||||||
@ -88,6 +96,9 @@ func TestRedisOwnershipLifecycle(t *testing.T) {
|
|||||||
if current, ok, err := fixture.Adapter.Get(context.Background(), "proxy-a"); err != nil || ok {
|
if current, ok, err := fixture.Adapter.Get(context.Background(), "proxy-a"); err != nil || ok {
|
||||||
t.Fatalf("Get(after ACK) = %+v, %t, %v", current, ok, err)
|
t.Fatalf("Get(after ACK) = %+v, %t, %v", current, ok, err)
|
||||||
}
|
}
|
||||||
|
if pending, err := fixture.Adapter.PendingDrains(context.Background(), "worker-a", 1); err != nil || len(pending) != 0 {
|
||||||
|
t.Fatalf("PendingDrains(after ACK) = %+v, %v", pending, err)
|
||||||
|
}
|
||||||
restored, err := fixture.Adapter.Extract(context.Background(), extractionDomain.Command{
|
restored, err := fixture.Adapter.Extract(context.Background(), extractionDomain.Command{
|
||||||
RequestID: "req-restored", ClientID: "client-a", Requested: 1,
|
RequestID: "req-restored", ClientID: "client-a", Requested: 1,
|
||||||
Fulfillment: extractionDomain.Partial, Now: now.Add(40 * time.Second),
|
Fulfillment: extractionDomain.Partial, Now: now.Add(40 * time.Second),
|
||||||
|
|||||||
@ -87,6 +87,11 @@ type ownershipScriptReply struct {
|
|||||||
Record string `json:"record,omitempty"`
|
Record string `json:"record,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type drainTicketsScriptReply struct {
|
||||||
|
Status scriptStatus `json:"status"`
|
||||||
|
Tickets []string `json:"tickets"`
|
||||||
|
}
|
||||||
|
|
||||||
type maintenanceScriptReply struct {
|
type maintenanceScriptReply struct {
|
||||||
Status scriptStatus `json:"status"`
|
Status scriptStatus `json:"status"`
|
||||||
Count int `json:"count"`
|
Count int `json:"count"`
|
||||||
@ -153,6 +158,9 @@ var extractSource string
|
|||||||
//go:embed scripts/ownership.lua
|
//go:embed scripts/ownership.lua
|
||||||
var ownershipSource string
|
var ownershipSource string
|
||||||
|
|
||||||
|
//go:embed scripts/drain_tickets.lua
|
||||||
|
var drainTicketsSource string
|
||||||
|
|
||||||
//go:embed scripts/sweep.lua
|
//go:embed scripts/sweep.lua
|
||||||
var sweepSource string
|
var sweepSource string
|
||||||
|
|
||||||
@ -176,6 +184,7 @@ var (
|
|||||||
upstreamLookupScript = redis.NewScript(upstreamLookupSource)
|
upstreamLookupScript = redis.NewScript(upstreamLookupSource)
|
||||||
extractScript = redis.NewScript(extractSource)
|
extractScript = redis.NewScript(extractSource)
|
||||||
ownershipScript = redis.NewScript(ownershipSource)
|
ownershipScript = redis.NewScript(ownershipSource)
|
||||||
|
drainTicketsScript = redis.NewScript(drainTicketsSource)
|
||||||
sweepScript = redis.NewScript(sweepSource)
|
sweepScript = redis.NewScript(sweepSource)
|
||||||
statusScript = redis.NewScript(statusSource)
|
statusScript = redis.NewScript(statusSource)
|
||||||
runtimeScript = redis.NewScript(runtimeSource)
|
runtimeScript = redis.NewScript(runtimeSource)
|
||||||
|
|||||||
55
internal/adapters/redisactivity/scripts/drain_tickets.lua
Normal file
55
internal/adapters/redisactivity/scripts/drain_tickets.lua
Normal file
@ -0,0 +1,55 @@
|
|||||||
|
local tickets_key = KEYS[1]
|
||||||
|
local worker_index_key = KEYS[2]
|
||||||
|
local owners_key = KEYS[3]
|
||||||
|
|
||||||
|
local worker_id = ARGV[1]
|
||||||
|
local limit = tonumber(ARGV[2])
|
||||||
|
|
||||||
|
local function reply(status, tickets)
|
||||||
|
return cjson.encode({status = status, tickets = tickets or {}})
|
||||||
|
end
|
||||||
|
|
||||||
|
local function decode_table(raw)
|
||||||
|
if not raw then
|
||||||
|
return nil
|
||||||
|
end
|
||||||
|
local ok, value = pcall(cjson.decode, raw)
|
||||||
|
if not ok or type(value) ~= 'table' then
|
||||||
|
return nil
|
||||||
|
end
|
||||||
|
return value
|
||||||
|
end
|
||||||
|
|
||||||
|
local function valid_ticket(ticket, proxy_id)
|
||||||
|
return ticket and ticket.version == 1 and ticket.proxyId == proxy_id and
|
||||||
|
ticket.workerId == worker_id and type(ticket.workerIndexKey) == 'string' and
|
||||||
|
ticket.workerIndexKey == worker_index_key and tonumber(ticket.assignmentEpoch) and
|
||||||
|
tonumber(ticket.assignmentEpoch) > 0 and tonumber(ticket.requiredSnapshotEpoch) and
|
||||||
|
tonumber(ticket.requiredSnapshotEpoch) > tonumber(ticket.assignmentEpoch)
|
||||||
|
end
|
||||||
|
|
||||||
|
local function valid_assignment(assignment, ticket)
|
||||||
|
return assignment and assignment.version == 1 and assignment.draining == true and
|
||||||
|
assignment.proxyId == ticket.proxyId and assignment.workerId == ticket.workerId and
|
||||||
|
tonumber(assignment.epoch) == tonumber(ticket.assignmentEpoch)
|
||||||
|
end
|
||||||
|
|
||||||
|
if type(worker_id) ~= 'string' or worker_id == '' or not limit or limit <= 0 then
|
||||||
|
return reply('invalid')
|
||||||
|
end
|
||||||
|
|
||||||
|
local scan_limit = limit * 4
|
||||||
|
local ids = redis.call('ZRANGE', worker_index_key, 0, scan_limit - 1)
|
||||||
|
local tickets = {}
|
||||||
|
for _, proxy_id in ipairs(ids) do
|
||||||
|
local raw = redis.call('HGET', tickets_key, proxy_id)
|
||||||
|
local ticket = decode_table(raw)
|
||||||
|
local assignment = decode_table(redis.call('HGET', owners_key, proxy_id))
|
||||||
|
if valid_ticket(ticket, proxy_id) and valid_assignment(assignment, ticket) and #tickets < limit then
|
||||||
|
tickets[#tickets + 1] = raw
|
||||||
|
else
|
||||||
|
redis.call('HDEL', tickets_key, proxy_id)
|
||||||
|
redis.call('ZREM', worker_index_key, proxy_id)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
return reply('ok', tickets)
|
||||||
@ -16,6 +16,7 @@ local health_tasks_key = KEYS[15]
|
|||||||
local health_task_expiry_key = KEYS[16]
|
local health_task_expiry_key = KEYS[16]
|
||||||
local health_ref_task_key = KEYS[17]
|
local health_ref_task_key = KEYS[17]
|
||||||
local health_unhealthy_key = KEYS[18]
|
local health_unhealthy_key = KEYS[18]
|
||||||
|
local drain_tickets_key = KEYS[19]
|
||||||
|
|
||||||
local operation = ARGV[1]
|
local operation = ARGV[1]
|
||||||
local operation_ttl_ms = tonumber(ARGV[2])
|
local operation_ttl_ms = tonumber(ARGV[2])
|
||||||
@ -28,6 +29,7 @@ local value = tonumber(ARGV[8])
|
|||||||
local active = tonumber(ARGV[9])
|
local active = tonumber(ARGV[9])
|
||||||
local reserved = tonumber(ARGV[10])
|
local reserved = tonumber(ARGV[10])
|
||||||
local worker_index_key = ARGV[11]
|
local worker_index_key = ARGV[11]
|
||||||
|
local worker_draining_index_key = ARGV[12]
|
||||||
local mutating = operation ~= 'get'
|
local mutating = operation ~= 'get'
|
||||||
|
|
||||||
local function finish(reply)
|
local function finish(reply)
|
||||||
@ -108,6 +110,17 @@ local function remove_owned(id, record)
|
|||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
local function clear_drain(id)
|
||||||
|
local raw = redis.call('HGET', drain_tickets_key, id)
|
||||||
|
if raw then
|
||||||
|
local decoded, ticket = pcall(cjson.decode, raw)
|
||||||
|
if decoded and type(ticket) == 'table' and type(ticket.workerIndexKey) == 'string' and ticket.workerIndexKey ~= '' then
|
||||||
|
redis.call('ZREM', ticket.workerIndexKey, id)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
redis.call('HDEL', drain_tickets_key, id)
|
||||||
|
end
|
||||||
|
|
||||||
local function add_available(id, record, at_ms)
|
local function add_available(id, record, at_ms)
|
||||||
local usable_until_ms = record and tonumber(record.usableUntilMs)
|
local usable_until_ms = record and tonumber(record.usableUntilMs)
|
||||||
if not usable_until_ms or record.state ~= 'AVAILABLE' or usable_until_ms <= at_ms then
|
if not usable_until_ms or record.state ~= 'AVAILABLE' or usable_until_ms <= at_ms then
|
||||||
@ -176,6 +189,7 @@ local function remove_proxy(id)
|
|||||||
redis.call('ZREM', expiry_key, id)
|
redis.call('ZREM', expiry_key, id)
|
||||||
redis.call('HDEL', owners_key, id)
|
redis.call('HDEL', owners_key, id)
|
||||||
redis.call('ZREM', owner_expiry_key, id)
|
redis.call('ZREM', owner_expiry_key, id)
|
||||||
|
clear_drain(id)
|
||||||
redis.call('ZREM', health_unhealthy_key, id)
|
redis.call('ZREM', health_unhealthy_key, id)
|
||||||
remove_health_task(id)
|
remove_health_task(id)
|
||||||
end
|
end
|
||||||
@ -220,6 +234,7 @@ local function clear_owner(id, assignment, at_ms, restore)
|
|||||||
end
|
end
|
||||||
redis.call('HDEL', owners_key, id)
|
redis.call('HDEL', owners_key, id)
|
||||||
redis.call('ZREM', owner_expiry_key, id)
|
redis.call('ZREM', owner_expiry_key, id)
|
||||||
|
clear_drain(id)
|
||||||
if assignment and type(assignment.workerIndexKey) == 'string' and assignment.workerIndexKey ~= '' then
|
if assignment and type(assignment.workerIndexKey) == 'string' and assignment.workerIndexKey ~= '' then
|
||||||
redis.call('ZREM', assignment.workerIndexKey, id)
|
redis.call('ZREM', assignment.workerIndexKey, id)
|
||||||
end
|
end
|
||||||
@ -322,8 +337,14 @@ if operation == 'begin_drain' then
|
|||||||
return finish({status = 'stale'})
|
return finish({status = 'stale'})
|
||||||
end
|
end
|
||||||
if not current.draining then
|
if not current.draining then
|
||||||
|
if type(worker_draining_index_key) ~= 'string' or worker_draining_index_key == '' or
|
||||||
|
not string.find(worker_draining_index_key, '{activity}', 1, true) then
|
||||||
|
return finish({status = 'invalid'})
|
||||||
|
end
|
||||||
current.draining = true
|
current.draining = true
|
||||||
current.assignmentVersion = tonumber(current.assignmentVersion) + 1
|
current.assignmentVersion = tonumber(current.assignmentVersion) + 1
|
||||||
|
local drain_epoch = redis.call('INCR', epoch_key)
|
||||||
|
redis.call('PERSIST', epoch_key)
|
||||||
local encoded = cjson.encode(current)
|
local encoded = cjson.encode(current)
|
||||||
redis.call('HSET', owners_key, proxy_id, encoded)
|
redis.call('HSET', owners_key, proxy_id, encoded)
|
||||||
local record = decode_table(redis.call('HGET', records_key, proxy_id))
|
local record = decode_table(redis.call('HGET', records_key, proxy_id))
|
||||||
@ -331,6 +352,18 @@ if operation == 'begin_drain' then
|
|||||||
if type(current.workerIndexKey) == 'string' and current.workerIndexKey ~= '' then
|
if type(current.workerIndexKey) == 'string' and current.workerIndexKey ~= '' then
|
||||||
redis.call('ZREM', current.workerIndexKey, proxy_id)
|
redis.call('ZREM', current.workerIndexKey, proxy_id)
|
||||||
end
|
end
|
||||||
|
local ticket = {
|
||||||
|
version = 1,
|
||||||
|
proxyId = proxy_id,
|
||||||
|
workerId = worker_id,
|
||||||
|
workerIndexKey = worker_draining_index_key,
|
||||||
|
assignmentEpoch = current.epoch,
|
||||||
|
requiredSnapshotEpoch = drain_epoch,
|
||||||
|
}
|
||||||
|
redis.call('HSET', drain_tickets_key, proxy_id, cjson.encode(ticket))
|
||||||
|
redis.call('ZADD', worker_draining_index_key, current.expiresAtMs, proxy_id)
|
||||||
|
touch(drain_tickets_key, tonumber(current.expiresAtMs))
|
||||||
|
touch(worker_draining_index_key, tonumber(current.expiresAtMs))
|
||||||
return finish({status = 'ok', record = encoded})
|
return finish({status = 'ok', record = encoded})
|
||||||
end
|
end
|
||||||
return finish({status = 'ok', record = cjson.encode(current)})
|
return finish({status = 'ok', record = cjson.encode(current)})
|
||||||
|
|||||||
@ -22,6 +22,7 @@ type Store interface {
|
|||||||
activitypool.Maintainer
|
activitypool.Maintainer
|
||||||
extractionDomain.Store
|
extractionDomain.Store
|
||||||
ownershipDomain.Repository
|
ownershipDomain.Repository
|
||||||
|
ownershipDomain.DrainTicketStore
|
||||||
}
|
}
|
||||||
|
|
||||||
type Factory func(*testing.T) (Store, func())
|
type Factory func(*testing.T) (Store, func())
|
||||||
@ -293,6 +294,11 @@ func runOwnershipContract(t *testing.T, factory Factory) {
|
|||||||
if replayed, err := store.BeginDrain(context.Background(), "proxy-a", "worker-a", assigned.Epoch); err != nil || replayed != draining {
|
if replayed, err := store.BeginDrain(context.Background(), "proxy-a", "worker-a", assigned.Epoch); err != nil || replayed != draining {
|
||||||
t.Fatalf("BeginDrain(replay) = %+v, %v", replayed, err)
|
t.Fatalf("BeginDrain(replay) = %+v, %v", replayed, err)
|
||||||
}
|
}
|
||||||
|
pending, err := store.PendingDrains(context.Background(), "worker-a", 1)
|
||||||
|
if err != nil || len(pending) != 1 || pending[0].ProxyID != "proxy-a" ||
|
||||||
|
pending[0].AssignmentEpoch != assigned.Epoch || pending[0].RequiredSnapshotEpoch <= assigned.Epoch {
|
||||||
|
t.Fatalf("PendingDrains() = %+v, %v", pending, err)
|
||||||
|
}
|
||||||
if err := store.AcknowledgeDrain(context.Background(), "proxy-a", "worker-a", assigned.Epoch, 1, 0); !errors.Is(err, ownershipDomain.ErrDrainNotReady) {
|
if err := store.AcknowledgeDrain(context.Background(), "proxy-a", "worker-a", assigned.Epoch, 1, 0); !errors.Is(err, ownershipDomain.ErrDrainNotReady) {
|
||||||
t.Fatalf("AcknowledgeDrain(active) error = %v", err)
|
t.Fatalf("AcknowledgeDrain(active) error = %v", err)
|
||||||
}
|
}
|
||||||
@ -302,6 +308,9 @@ func runOwnershipContract(t *testing.T, factory Factory) {
|
|||||||
if current, ok, err := store.Get(context.Background(), "proxy-a"); err != nil || ok {
|
if current, ok, err := store.Get(context.Background(), "proxy-a"); err != nil || ok {
|
||||||
t.Fatalf("Get(after ACK) = %+v, %t, %v", current, ok, err)
|
t.Fatalf("Get(after ACK) = %+v, %t, %v", current, ok, err)
|
||||||
}
|
}
|
||||||
|
if pending, err := store.PendingDrains(context.Background(), "worker-a", 1); err != nil || len(pending) != 0 {
|
||||||
|
t.Fatalf("PendingDrains(after ACK) = %+v, %v", pending, err)
|
||||||
|
}
|
||||||
if result, err := store.Extract(context.Background(), extractionDomain.Command{
|
if result, err := store.Extract(context.Background(), extractionDomain.Command{
|
||||||
RequestID: "req-after-ack", ClientID: "client-a", Requested: 1,
|
RequestID: "req-after-ack", ClientID: "client-a", Requested: 1,
|
||||||
Fulfillment: extractionDomain.Partial, Now: now.Add(20 * time.Second),
|
Fulfillment: extractionDomain.Partial, Now: now.Add(20 * time.Second),
|
||||||
@ -537,6 +546,7 @@ func runCancellationContract(t *testing.T, store Store) {
|
|||||||
{name: "assign", call: func() error { _, err := store.Assign(ctx, now, "proxy-a", "worker-a", time.Minute); return err }},
|
{name: "assign", call: func() error { _, err := store.Assign(ctx, now, "proxy-a", "worker-a", time.Minute); return err }},
|
||||||
{name: "renew", call: func() error { _, err := store.Renew(ctx, now, "proxy-a", "worker-a", 1, time.Minute); return err }},
|
{name: "renew", call: func() error { _, err := store.Renew(ctx, now, "proxy-a", "worker-a", 1, time.Minute); return err }},
|
||||||
{name: "begin drain", call: func() error { _, err := store.BeginDrain(ctx, "proxy-a", "worker-a", 1); return err }},
|
{name: "begin drain", call: func() error { _, err := store.BeginDrain(ctx, "proxy-a", "worker-a", 1); return err }},
|
||||||
|
{name: "pending drains", call: func() error { _, err := store.PendingDrains(ctx, "worker-a", 1); return err }},
|
||||||
{name: "acknowledge drain", call: func() error { return store.AcknowledgeDrain(ctx, "proxy-a", "worker-a", 1, 0, 0) }},
|
{name: "acknowledge drain", call: func() error { return store.AcknowledgeDrain(ctx, "proxy-a", "worker-a", 1, 0, 0) }},
|
||||||
{name: "get", call: func() error { _, _, err := store.Get(ctx, "proxy-a"); return err }},
|
{name: "get", call: func() error { _, _, err := store.Get(ctx, "proxy-a"); return err }},
|
||||||
{name: "expire", call: func() error { _, err := store.Expire(ctx, now, 1); return err }},
|
{name: "expire", call: func() error { _, err := store.Expire(ctx, now, 1); return err }},
|
||||||
|
|||||||
@ -159,6 +159,7 @@ type MemoryPool struct {
|
|||||||
keyByID map[string]string
|
keyByID map[string]string
|
||||||
idempotent map[string]idempotencyEntry
|
idempotent map[string]idempotencyEntry
|
||||||
ownership map[string]ownershipDomain.Assignment
|
ownership map[string]ownershipDomain.Assignment
|
||||||
|
drains map[string]ownershipDomain.DrainTicket
|
||||||
targets map[targetHealthKey]healthDomain.TargetState
|
targets map[targetHealthKey]healthDomain.TargetState
|
||||||
unhealthy map[string]time.Time
|
unhealthy map[string]time.Time
|
||||||
nextEpoch uint64
|
nextEpoch uint64
|
||||||
@ -176,17 +177,18 @@ type idempotencyEntry struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
_ Upserter = (*MemoryPool)(nil)
|
_ Upserter = (*MemoryPool)(nil)
|
||||||
_ HealthStore = (*MemoryPool)(nil)
|
_ HealthStore = (*MemoryPool)(nil)
|
||||||
_ GlobalHealthStore = (*MemoryPool)(nil)
|
_ GlobalHealthStore = (*MemoryPool)(nil)
|
||||||
_ TargetHealthStore = (*MemoryPool)(nil)
|
_ TargetHealthStore = (*MemoryPool)(nil)
|
||||||
_ ProxyUpstreamReader = (*MemoryPool)(nil)
|
_ ProxyUpstreamReader = (*MemoryPool)(nil)
|
||||||
_ InventoryReader = (*MemoryPool)(nil)
|
_ InventoryReader = (*MemoryPool)(nil)
|
||||||
_ StateInventoryReader = (*MemoryPool)(nil)
|
_ StateInventoryReader = (*MemoryPool)(nil)
|
||||||
_ Maintainer = (*MemoryPool)(nil)
|
_ Maintainer = (*MemoryPool)(nil)
|
||||||
_ UnhealthyRemover = (*MemoryPool)(nil)
|
_ UnhealthyRemover = (*MemoryPool)(nil)
|
||||||
_ extractionDomain.Store = (*MemoryPool)(nil)
|
_ extractionDomain.Store = (*MemoryPool)(nil)
|
||||||
_ ownershipDomain.Repository = (*MemoryPool)(nil)
|
_ ownershipDomain.Repository = (*MemoryPool)(nil)
|
||||||
|
_ ownershipDomain.DrainTicketStore = (*MemoryPool)(nil)
|
||||||
)
|
)
|
||||||
|
|
||||||
func NewMemoryPool() *MemoryPool {
|
func NewMemoryPool() *MemoryPool {
|
||||||
@ -195,6 +197,7 @@ func NewMemoryPool() *MemoryPool {
|
|||||||
keyByID: make(map[string]string),
|
keyByID: make(map[string]string),
|
||||||
idempotent: make(map[string]idempotencyEntry),
|
idempotent: make(map[string]idempotencyEntry),
|
||||||
ownership: make(map[string]ownershipDomain.Assignment),
|
ownership: make(map[string]ownershipDomain.Assignment),
|
||||||
|
drains: make(map[string]ownershipDomain.DrainTicket),
|
||||||
targets: make(map[targetHealthKey]healthDomain.TargetState),
|
targets: make(map[targetHealthKey]healthDomain.TargetState),
|
||||||
unhealthy: make(map[string]time.Time),
|
unhealthy: make(map[string]time.Time),
|
||||||
}
|
}
|
||||||
@ -819,6 +822,7 @@ func (p *MemoryPool) Assign(ctx context.Context, now time.Time, proxyID, workerI
|
|||||||
p.setEntryByIDLocked(proxyID, entry)
|
p.setEntryByIDLocked(proxyID, entry)
|
||||||
}
|
}
|
||||||
delete(p.ownership, proxyID)
|
delete(p.ownership, proxyID)
|
||||||
|
delete(p.drains, proxyID)
|
||||||
}
|
}
|
||||||
entry, ok := p.entryByIDLocked(proxyID)
|
entry, ok := p.entryByIDLocked(proxyID)
|
||||||
if !ok || entry.State != proxyDomain.StateAvailable || entry.OwnerWorkerID != "" || !entry.UsableUntil.After(now) {
|
if !ok || entry.State != proxyDomain.StateAvailable || entry.OwnerWorkerID != "" || !entry.UsableUntil.After(now) {
|
||||||
@ -882,10 +886,49 @@ func (p *MemoryPool) BeginDrain(ctx context.Context, proxyID, workerID string, e
|
|||||||
assignment.Draining = true
|
assignment.Draining = true
|
||||||
assignment.Version++
|
assignment.Version++
|
||||||
p.ownership[proxyID] = assignment
|
p.ownership[proxyID] = assignment
|
||||||
|
p.nextEpoch++
|
||||||
|
p.drains[proxyID] = ownershipDomain.DrainTicket{
|
||||||
|
ProxyID: proxyID, WorkerID: workerID, AssignmentEpoch: assignment.Epoch,
|
||||||
|
RequiredSnapshotEpoch: p.nextEpoch,
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return assignment, nil
|
return assignment, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (p *MemoryPool) PendingDrains(ctx context.Context, workerID string, limit int) ([]ownershipDomain.DrainTicket, error) {
|
||||||
|
if err := ownershipContextError(ctx); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if p == nil || workerID == "" || limit <= 0 {
|
||||||
|
return nil, ownershipDomain.ErrInvalidDrainTicket
|
||||||
|
}
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
proxyIDs := make([]string, 0)
|
||||||
|
for proxyID, ticket := range p.drains {
|
||||||
|
assignment, exists := p.ownership[proxyID]
|
||||||
|
if !exists || !assignment.Draining || assignment.WorkerID != ticket.WorkerID || assignment.Epoch != ticket.AssignmentEpoch {
|
||||||
|
delete(p.drains, proxyID)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if ticket.WorkerID == workerID {
|
||||||
|
proxyIDs = append(proxyIDs, proxyID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Strings(proxyIDs)
|
||||||
|
if len(proxyIDs) > limit {
|
||||||
|
proxyIDs = proxyIDs[:limit]
|
||||||
|
}
|
||||||
|
tickets := make([]ownershipDomain.DrainTicket, 0, len(proxyIDs))
|
||||||
|
for _, proxyID := range proxyIDs {
|
||||||
|
tickets = append(tickets, p.drains[proxyID])
|
||||||
|
}
|
||||||
|
return tickets, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (p *MemoryPool) AcknowledgeDrain(ctx context.Context, proxyID, workerID string, epoch uint64, active, reserved int64) error {
|
func (p *MemoryPool) AcknowledgeDrain(ctx context.Context, proxyID, workerID string, epoch uint64, active, reserved int64) error {
|
||||||
if err := ownershipContextError(ctx); err != nil {
|
if err := ownershipContextError(ctx); err != nil {
|
||||||
return err
|
return err
|
||||||
@ -913,6 +956,7 @@ func (p *MemoryPool) AcknowledgeDrain(ctx context.Context, proxyID, workerID str
|
|||||||
p.setEntryByIDLocked(proxyID, entry)
|
p.setEntryByIDLocked(proxyID, entry)
|
||||||
}
|
}
|
||||||
delete(p.ownership, proxyID)
|
delete(p.ownership, proxyID)
|
||||||
|
delete(p.drains, proxyID)
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -969,6 +1013,7 @@ func (p *MemoryPool) Expire(ctx context.Context, now time.Time, limit int) ([]ow
|
|||||||
}
|
}
|
||||||
expired = append(expired, assignment)
|
expired = append(expired, assignment)
|
||||||
delete(p.ownership, proxyID)
|
delete(p.ownership, proxyID)
|
||||||
|
delete(p.drains, proxyID)
|
||||||
}
|
}
|
||||||
return expired, nil
|
return expired, nil
|
||||||
}
|
}
|
||||||
@ -1012,6 +1057,7 @@ func (p *MemoryPool) removeEntryByIDLocked(proxyID string) {
|
|||||||
|
|
||||||
func (p *MemoryPool) removeEntryLocked(key string, entry Entry) {
|
func (p *MemoryPool) removeEntryLocked(key string, entry Entry) {
|
||||||
delete(p.ownership, entry.Proxy.ID)
|
delete(p.ownership, entry.Proxy.ID)
|
||||||
|
delete(p.drains, entry.Proxy.ID)
|
||||||
delete(p.unhealthy, entry.Proxy.ID)
|
delete(p.unhealthy, entry.Proxy.ID)
|
||||||
for target := range p.targets {
|
for target := range p.targets {
|
||||||
if target.proxyID == entry.Proxy.ID {
|
if target.proxyID == entry.Proxy.ID {
|
||||||
|
|||||||
@ -13,6 +13,7 @@ var (
|
|||||||
ErrStaleAssignment = errors.New("proxy ownership assignment is stale")
|
ErrStaleAssignment = errors.New("proxy ownership assignment is stale")
|
||||||
ErrNotDraining = errors.New("proxy ownership is not draining")
|
ErrNotDraining = errors.New("proxy ownership is not draining")
|
||||||
ErrDrainNotReady = errors.New("proxy still has active or reserved runtime")
|
ErrDrainNotReady = errors.New("proxy still has active or reserved runtime")
|
||||||
|
ErrInvalidDrainTicket = errors.New("invalid drain ticket request")
|
||||||
)
|
)
|
||||||
|
|
||||||
type Assignment struct {
|
type Assignment struct {
|
||||||
@ -24,6 +25,23 @@ type Assignment struct {
|
|||||||
Draining bool
|
Draining bool
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// DrainTicket records a pending removal after an ownership assignment enters
|
||||||
|
// draining. RequiredSnapshotEpoch fences the first complete Worker snapshot
|
||||||
|
// that may prove the proxy was excluded.
|
||||||
|
type DrainTicket struct {
|
||||||
|
ProxyID string
|
||||||
|
WorkerID string
|
||||||
|
AssignmentEpoch uint64
|
||||||
|
RequiredSnapshotEpoch uint64
|
||||||
|
}
|
||||||
|
|
||||||
|
// DrainTicketStore exposes bounded, Worker-local pending drain tickets. It
|
||||||
|
// deliberately does not finalize ownership: a later runtime transaction must
|
||||||
|
// prove both an acknowledged exclusion snapshot and zero live counters.
|
||||||
|
type DrainTicketStore interface {
|
||||||
|
PendingDrains(context.Context, string, int) ([]DrainTicket, error)
|
||||||
|
}
|
||||||
|
|
||||||
// Repository is the shared authority for ownership changes. Implementations
|
// Repository is the shared authority for ownership changes. Implementations
|
||||||
// that also support extraction must serialize both operations transactionally.
|
// that also support extraction must serialize both operations transactionally.
|
||||||
type Repository interface {
|
type Repository interface {
|
||||||
|
|||||||
@ -354,11 +354,11 @@ func TestAcquireSurvivesConcurrentSnapshotApply(t *testing.T) {
|
|||||||
{epoch: 1, version: 2},
|
{epoch: 1, version: 2},
|
||||||
{epoch: 1, version: 3},
|
{epoch: 1, version: 3},
|
||||||
{epoch: 1, version: 4},
|
{epoch: 1, version: 4},
|
||||||
{epoch: 2, version: 1},
|
{epoch: 2, version: 5},
|
||||||
{epoch: 2, version: 2},
|
{epoch: 2, version: 6},
|
||||||
{epoch: 2, version: 3},
|
{epoch: 2, version: 7},
|
||||||
{epoch: 3, version: 1},
|
{epoch: 3, version: 8},
|
||||||
{epoch: 3, version: 2},
|
{epoch: 3, version: 9},
|
||||||
}
|
}
|
||||||
for _, step := range steps {
|
for _, step := range steps {
|
||||||
next := snapshot.Envelope{
|
next := snapshot.Envelope{
|
||||||
|
|||||||
@ -267,9 +267,7 @@ func (s *Store) Apply(envelope Envelope) error {
|
|||||||
switch {
|
switch {
|
||||||
case envelope.Epoch < current.Epoch:
|
case envelope.Epoch < current.Epoch:
|
||||||
return ErrResyncRequired
|
return ErrResyncRequired
|
||||||
case envelope.Epoch == current.Epoch && envelope.Version != current.Version+1:
|
case envelope.Version != current.Version+1:
|
||||||
return ErrResyncRequired
|
|
||||||
case envelope.Epoch > current.Epoch && envelope.Version != 1:
|
|
||||||
return ErrResyncRequired
|
return ErrResyncRequired
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@ -78,6 +78,12 @@ func TestStoreRejectsWrongWorkerVersionGapAndChecksum(t *testing.T) {
|
|||||||
t.Fatalf("version gap error = %v, want ErrResyncRequired", err)
|
t.Fatalf("version gap error = %v, want ErrResyncRequired", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
epochReset := base
|
||||||
|
epochReset.Epoch = 2
|
||||||
|
if err := store.Apply(epochReset); !errors.Is(err, ErrResyncRequired) {
|
||||||
|
t.Fatalf("epoch reset version error = %v, want ErrResyncRequired", err)
|
||||||
|
}
|
||||||
|
|
||||||
badChecksum := base
|
badChecksum := base
|
||||||
badChecksum.Version = 2
|
badChecksum.Version = 2
|
||||||
badChecksum.Checksum = "bad"
|
badChecksum.Checksum = "bad"
|
||||||
@ -272,7 +278,7 @@ func TestStoreApplyReusesCapacityAcrossVersionsAndEpochs(t *testing.T) {
|
|||||||
|
|
||||||
third := second
|
third := second
|
||||||
third.Epoch = 2
|
third.Epoch = 2
|
||||||
third.Version = 1
|
third.Version = 3
|
||||||
third.Checksum = Checksum(third.Proxies)
|
third.Checksum = Checksum(third.Proxies)
|
||||||
if err := store.Apply(third); err != nil {
|
if err := store.Apply(third); err != nil {
|
||||||
t.Fatalf("Apply(third): %v", err)
|
t.Fatalf("Apply(third): %v", err)
|
||||||
|
|||||||
@ -21,6 +21,13 @@
|
|||||||
Delta 的 fail-closed 校验,也不把自动 Drain/ACK 记为已完成。
|
Delta 的 fail-closed 校验,也不把自动 Drain/ACK 记为已完成。
|
||||||
- Health Scheduler 现在先建立 EGRESS 与 Routing TARGET 的独立 due 引用,再投递
|
- Health Scheduler 现在先建立 EGRESS 与 Routing TARGET 的独立 due 引用,再投递
|
||||||
BASIC 任务;在单项小批次下,短 TTL Proxy 不会因 BASIC 先出队而错过首次出口/目标探测。
|
BASIC 任务;在单项小批次下,短 TTL Proxy 不会因 BASIC 先出队而错过首次出口/目标探测。
|
||||||
|
- 修复 Snapshot epoch/version 栅栏:Worker 流的 `version` 在 ownership epoch 前进时
|
||||||
|
仍严格递增,Gateway 以连续版本加非倒退 epoch 进行完整快照原子替换,避免动态分配后的
|
||||||
|
有效刷新被错误拒绝。这是后续自动 Drain 编排的必要前提。
|
||||||
|
- Drain 首次发起现在原子推进 ownership epoch,并写入由 Proxy、Worker、原 assignment
|
||||||
|
epoch 与所需 Snapshot epoch 组成的待绑定 Ticket;Redis 使用 `drain-tickets` 与按
|
||||||
|
Worker 的 `worker-draining` 索引,ACK、过期和硬删除会清理 Ticket。完整快照排除、
|
||||||
|
ACK 与 Runtime 零计数的原子自动完成仍在后续切片,尚未标记为完成。
|
||||||
|
|
||||||
## 2026-07-30
|
## 2026-07-30
|
||||||
|
|
||||||
|
|||||||
@ -41,7 +41,9 @@
|
|||||||
通过 Controller 后台的有界回收清理无 Worker ownership 的持续异常 Proxy;拥有
|
通过 Controller 后台的有界回收清理无 Worker ownership 的持续异常 Proxy;拥有
|
||||||
Worker 的候选延后到既有 Drain/ACK 清除所有权后再处理。
|
Worker 的候选延后到既有 Drain/ACK 清除所有权后再处理。
|
||||||
15. [进行中] 收敛 Worker 发布生命周期:已完成权威 Proxy/Routing 的持续完整快照
|
15. [进行中] 收敛 Worker 发布生命周期:已完成权威 Proxy/Routing 的持续完整快照
|
||||||
刷新;后续补齐自动 Drain 编排,使配置停用、健康淘汰与 Snapshot 撤销形成可观测闭环。
|
刷新,以及 Drain 发起时的持久化 Ticket、Worker 待绑定索引和 ownership epoch 推进;
|
||||||
|
后续在完整快照排除、ACK 与 Runtime 零计数之间补齐原子自动完成编排,使配置停用、
|
||||||
|
健康淘汰与 Snapshot 撤销形成可观测闭环。
|
||||||
|
|
||||||
## 串并行关系
|
## 串并行关系
|
||||||
|
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user