Stop submitting bids on clickhouse problems.#864
Merged
Conversation
dvush
approved these changes
Jan 21, 2026
Contributor
dvush
left a comment
There was a problem hiding this comment.
Approved for faster merging, but check out my comment
| ) -> Box<dyn MultiRelayBlockBuildingSink> { | ||
| // If submission is disabled, return a sink that throws away the blocks. | ||
| if !self.submission_policy.should_submit() { | ||
| warn!("Submission is disabled by submission_policy, throwing away the blocks"); |
Contributor
There was a problem hiding this comment.
I think this should be error and we need to have a metric for this, if its increasing we alert
Collaborator
Author
There was a problem hiding this comment.
I didn't put this as an error since the real errors are "failed to commit bundle to clickhouse from backup" and a LOT more (we are not going to miss it). I saw this as a normal behavior triggered by the other errors, but I'll make it error to play it safer.
I though about the metric but we already have initiated_submissions going to 0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 Summary
This PR introduces a trait RelaySubmissionPolicy to stop submitting on any problem.
I decided to cut the flow at the very end of the chain (I know it's inefficient) to play it super safe just in case any module is expecting new slot/block info periodically (eg: bidding services, caching stuff).
A new field was added to BuiltBlocksClickhouseConfig:
/// If set must be < disk_max_size_mb.
/// If the disk backup size is greater than this value, clickhouse will ask we stop submitting blocks.
pub disk_max_size_to_submit_bids_to_relays_mb: Option,
New metric clickhouse_disk_backup_max_size_to_submit_bids_to_relays_bytes with this value.
✅ I have completed the following steps:
make lintmake test