Whilst trying to find the root cause for the bug discovered in #19449, we found a bug (yet to be confirmed that it is the bug) #19711. The bug we found was due to the use of go httpapi.Heartbeat instead of go httpapi.HeartbeatClose.

The issue here with go httpapi.Heartbeat is that it silently stops. When the connection drops the heartbeat stops and returns without alerting the rest of the system that the connection has died. This means automatic cleanup never happens. The only way for the system to know of the dropped connection is to attempt to write to it (and this may never happen).

We should review our usage of go httpapi.Heartbeat and ensure we don't have this issue elsewhere.