!5262odin/fix-statesync-frame-timeout-flake → develop983e02c0b6

Restore the 1s state-sync test frame timeout

AI-generatedEverything on this page was written by an AI agent from the diff: the summaries, the narration, the diagrams, the flags and the glossary. It can be wrong or incomplete, so check it against the code before you rely on it.

1commits
1files
7lines changed
0lines to read
250ms → 1stest frame timeout
0 / 1,120failed subtests at 1s
78 → 94 sbifrost/observer run

Where the diff is

Read these0 files · 0 lines

none

3-minute explainer

What changed

The flake

Retry step fails since !5233: 7 of 20 failed unit-test jobs.

stall / reset / ack→export released→retry same peer✗

Where it bites

Sender's ordinary 40 s reads shrink to the frame timeout.

requestbeginheaderdatadoneframe timeout15 s budgetrequest read armed before the receiver opens leveldb

The fix

One constant back to 1 s, plus a why-comment.

1 sbefore !5233
250 msf9de2e1612
1 sthis MR

Evidence and cost

8, 2, 3 failures on develop; 0, 0 at 1 s.

82300develop (of 560)1 s

Look here first

Likelier flake window: receiver built mid-readLine 1734 starts the sender, arming its request read on the frame timeout; 1735 then opens the receiver's leveldb. Building it first, as line 1660 does, closes the gap.attestation_state_init_test.go:1735 ↗verify
unchanged in this MR · context
17261726 case "reset":
17271727 require.NoError(t, client.Reset())
17281728 }
17291729 }
17301730 waitStateSyncDeadlineHandler(t, done)
17311731 require.Zero(t, stateSyncExportCount(sender))
17321732 // The same identity can obtain a new copy after any exit.
17331733 sender.config.StateSyncTimeout = 15 * time.Second
17341734 _, retry, retryDone := startStateSyncExport(t, sender, "peer")
17351735 receiver := stateSyncDeadlineFixture(t, 0)
17361736 require.True(t, receiver.askStreamForState(retry).completed)
17371737 waitStateSyncDeadlineHandler(t, retryDone)
17381738 require.Zero(t, stateSyncExportCount(sender))
17391739 })
17401740 }
17411741 }
17421742
17431743 // A small framed reply or a blocked read. Cancellation is exercised through
17441744 // Reset, rather than pretending deadline setters interrupt a bytes.Buffer.
17451745 type startupRetryStream struct {
17461746 MockStream
17471747 data *bytes.Reader
17481748 started, stopped chan struct{}
New comment names the wrong waitThe real receiver ACKs data (state_init.go:551) before its async import; the final done (:322) uses the absolute budget, not the frame.attestation_state_init_test.go:1238 ↗verify
12311231 assert.Equal(t, 0, len(agReceiver.observedTxs), "receiver should have no txs after empty-state init")
12321232 assert.Equal(t, 0, len(agReceiver.networkFees), "receiver should have no network fees after empty-state init")
12331233 assert.Equal(t, 0, len(agReceiver.solvencies), "receiver should have no solvencies after empty-state init")
12341234 }
12351235
12361236 // stateSyncTestFrameTimeout replaces the ordinary per-frame read deadline in
12371237 // the deadline tests; their stalls, slow batches and budgets are multiples of
1238−// it. Local pipes answer in microseconds, so a quarter second is ample.
1239−const stateSyncTestFrameTimeout = 250 * time.Millisecond
1238+// it. It also bounds how long the sender waits for the receiver to process
1239+// each batch, which includes leveldb writes. 250ms was too short on a loaded
1240+// CI runner: TestStateSyncExportReleaseOnEveryExit failed its retry step in
1241+// 1 of 70 to 1 of 280 subtests with 8 copies in parallel; 1s gave 0 of 1,120.
1242+const stateSyncTestFrameTimeout = time.Second
12401243
12411244 // Successful stream cleanup may log after the handler and its test return.
12421245 type stateSyncLogBuffer struct {
12431246 mu sync.Mutex
12441247 buffer bytes.Buffer
12451248 }
12461249
12471250 func (b *stateSyncLogBuffer) Write(p []byte) (int, error) {
12481251 b.mu.Lock()
300 ms budget now ends stalls firstWith a 1 s frame, the 300 ms budget reset ends the begin-phase stall, not the frame timeout. Same as before f9de2e1612.attestation_state_init_test.go:1711 ↗careful
unchanged in this MR · context
17021702 t.Run(phase, func(t *testing.T) {
17031703 count := 1
17041704 if phase == "next header" {
17051705 count = 2
17061706 }
17071707 if phase == "empty" {
17081708 count = 0
17091709 }
17101710 sender := stateSyncDeadlineFixture(t, count)
17111711 sender.config.StateSyncTimeout = 300 * time.Millisecond
17121712 _, client, done := startStateSyncExport(t, sender, "peer")
17131713 if phase == "success" || phase == "empty" {
17141714 receiver := stateSyncDeadlineFixture(t, 0)
17151715 require.True(t, receiver.askStreamForState(client).completed)
17161716 } else {
17171717 stall := phase
17181718 if phase == "bad ack" || phase == "reset" {
17191719 stall = "begin"
17201720 }
17211721 stateSyncStallAtACK(t, client, stall)
17221722 require.Equal(t, 1, stateSyncExportCount(sender))
17231723 switch phase {
17241724 case "bad ack":
Same window in the success/empty stepThe export starts at 1712 before this receiver fixture is built; 1 s only widens the margin.attestation_state_init_test.go:1714 ↗look
unchanged in this MR · context
17051705 count = 2
17061706 }
17071707 if phase == "empty" {
17081708 count = 0
17091709 }
17101710 sender := stateSyncDeadlineFixture(t, count)
17111711 sender.config.StateSyncTimeout = 300 * time.Millisecond
17121712 _, client, done := startStateSyncExport(t, sender, "peer")
17131713 if phase == "success" || phase == "empty" {
17141714 receiver := stateSyncDeadlineFixture(t, 0)
17151715 require.True(t, receiver.askStreamForState(client).completed)
17161716 } else {
17171717 stall := phase
17181718 if phase == "bad ack" || phase == "reset" {
17191719 stall = "begin"
17201720 }
17211721 stateSyncStallAtACK(t, client, stall)
17221722 require.Equal(t, 1, stateSyncExportCount(sender))
17231723 switch phase {
17241724 case "bad ack":
17251725 require.NoError(t, p2p.WriteStreamWithBuffer([]byte("wrong"), client))
17261726 case "reset":
17271727 require.NoError(t, client.Reset())
Scaled stalls add serial runtimeThe slow-batch sleep becomes 2 s and the expiry budget at 1552 becomes 3 s; these tests run serially, so the package takes 94 s.attestation_state_init_test.go:1417 ↗look
unchanged in this MR · context
14081408 require.Len(t, header, 9)
14091409 require.Equal(t, i, binary.LittleEndian.Uint32(header[1:5]))
14101410 write([]byte(streamAckHeader))
14111411 data := read(prefixBatchData)
14121412 require.EqualValues(t, len(data)-1, binary.LittleEndian.Uint32(header[5:9]))
14131413 var batch common.QuorumState
14141414 require.NoError(t, batch.Unmarshal(data[1:]))
14151415 write([]byte(streamAckData))
14161416 if int(i)+1 == slowBatch {
14171417 time.Sleep(2 * stateSyncTestFrameTimeout)
14181418 }
14191419 require.NoError(t, receiver.processAttestationStateBatch(zerolog.Nop(), batch))
14201420 }
14211421 require.Equal(t, prefixBatchEnd, read(prefixBatchEnd))
14221422 write([]byte(p2p.StreamMsgDone))
14231423 }
14241424
14251425 func TestStateSyncSenderProcessingACKDeadlines(t *testing.T) {
14261426 stateSyncDeadlineTestSettings(t)
14271427 for _, tc := range []struct {
14281428 name string
14291429 slow int
14301430 }{

Ask an AI about this change

Chat assistantClaude, ChatGPT, … (reads a link)

Coding agentClaude Code, Cursor, … (reads the repo)

The context file is context.md. Use Copy with context for an AI that can't open links; it pastes the whole file in.

Glossary

-count / -race
go test flags: -count N reruns each test N times; -race enables the data race detector. go test flags ↗
ACK frames
ack_begin, ack_header and ack_data: short replies the receiver sends after each state-sync frame; the sender waits for each one. attestation_gossip.go ↗
Bifrost
THORNode's chain-observer and signer daemon; the observer package gossips and attests to external-chain events. docs/bifrost/how-bifrost-works.md ↗
leveldb
The embedded key-value store backing observer storage; each test fixture opens a fresh one with CompactOnInit. observer_storage_test.go ↗
State sync
A Bifrost node joining or restarting asks a peer for its attestation state, streamed in batches over libp2p. docs/bifrost/state-sync-recovery.md ↗ attestation_state_init.go ↗
stateSyncTestFrameTimeout
Test-only constant that replaces the sender's ordinary 40-second per-frame read deadline in the state-sync deadline tests. This MR sets it back to 1 s. attestation_state_init_test.go ↗
StateSyncTimeout
The sender's absolute budget for a whole export (15 s in these fixtures, 300 ms in the release test's first step). config/config.go ↗
TimeoutReadPayload
The ordinary 40-second read deadline for one p2p stream frame. The test stream swaps it for the frame timeout on the sender side. bifrost/p2p/stream_helper.go ↗