Streaming snapshot sends

[jbooth]

Hi guys, I thought it would be nice to enable streaming snapshots. As bigger projects storing more data rely on go-raft, eventually serializing the whole snapshot into a []byte could be problematic. Especially if you're dealing with disk storage and data that doesn't fit (or barely fits) in memory.

I modified the StateMachine interface to support the following 2 options (NewServer() takes the base StateMachine marker interface and switches on type).

type StateMachine interface {
}

type StateMachineBytes interface {
    StateMachine
    Save() ([]byte, error)
    Recovery([]byte) error
}

type StateMachineIo interface {
    StateMachine
    WriteSnapshot(w io.Writer) (int, error)
    RecoverSnapshot(r io.Reader) error
}

If the StateMachineBytes type is provided to the server, we wrap it with a StateMachineIoWrapper that manages things. All tests are passing, and with some TeeReader cleverness I avoid making multiple passes over the data when streaming to both a live statemachine and an on-disk snapshot.

As part of doing this, I had to break the dependency that HttpTransporter has on the remote connection being closed while deserializing in order for ioutil.ReadAll to work. So the types used by Transporter now all support being sent on a connection that doesn't close, by sending length,data. This could enable different transporter implementations, and I could make it it's own patch if the rest of this is too scary.

The bad news is, I had to modify the on-disk snapshot format. Before it was a JSON envelope containing State:[]byte, but we can't easily do that if we never have the whole state in a single []byte. So that's a complicating factor. I did preserve the checksum capability.

Between changing the snapshot format and the wire protocols, I can see this being a little hard to push upstream. But do I think the project will need to support snapshots in a streaming manner eventually. Let me know if there's anything I could do to make it easier?