It seems like frames are dropped when the central server is overloaded with the decoding and encoding of multiple streams. All the ffmpeg commands (client and server side) should be improved and validated to make sure no frames are ever dropped.