This is possibly related to the hacky way it works with a udpsink and uspsrc in between two pipelines. Alternative solutions are being looked at.