failed to decode block protobuf / bincode

[dancamarg0]

I started to see this error recently in some of our servers

It shows for different slots and sometimes with different error descriptions:

{"timestamp":"2026-01-26T13:47:11.213605Z","level":"ERROR","fields":{"message":"failed to decode transaction protobuf / bincode","slot":393114324,"error":"DecodeError { description: \"unexpected end group tag\", stack: [] }"},"target":"alpamayo::rpc::api_jsonrpc","line_number":3000}
{"timestamp":"2026-01-26T13:47:15.058651Z","level":"ERROR","fields":{"message":"failed to decode block protobuf / bincode","slot":393069604,"error":"DecodeError { description: \"invalid tag value: 0\", stack: [] }"},"target":"alpamayo::rpc::api_jsonrpc","line_number":876}
{"timestamp":"2026-01-26T13:47:16.099924Z","level":"ERROR","fields":{"message":"failed to decode block protobuf / bincode","slot":392215948,"error":"DecodeError { description: \"invalid wire type: Varint (expected LengthDelimited)\", stack: [(\"ConfirmedBlock\", \"block_height\")] }"},"target":"alpamayo::rpc::api_jsonrpc","line_number":876}
{"timestamp":"2026-01-26T13:47:17.187294Z","level":"ERROR","fields":{"message":"failed to decode transaction protobuf / bincode","slot":392147923,"error":"DecodeError { description: \"buffer underflow\", stack: [] }"},"target":"alpamayo::rpc::api_jsonrpc","line_number":3000}
{"timestamp":"2026-01-26T13:47:17.187567Z","level":"ERROR","fields":{"message":"failed to decode transaction protobuf / bincode","slot":392147923,"error":"DecodeError { description: \"invalid wire type value: 7\", stack: [] }"},"target":"alpamayo::rpc::api_jsonrpc","line_number":3000}

It might be a coincidence, but recently the DB in these nodes shrunk half one epoch due to insufficient disk space for max_blocks and I notice a lot of the slots affected are close to First Available Slot in alpamayo (Although I also see some slots with errors like 393055843 that are almost 3 epochs after the first available slot)

From @fanatid

probably wrong data offset;size in indexes and txs can't be decoded?

Also if possible I think we should have a prometheus metric to alert for this type of error since it we can have hidden issues because of this