Expose Kafka metadata w/ Kafka-sourced messages

[charlie]

Apologies if this idea has been discussed and rejected before. I feel like maybe it has. Or if there's already some other way to do this, and I just couldn't see it.

Intended usage:

offset := int64(-1)
if withMetadata, ok := msg.(kafka.MessageWithMetadata); ok {
	offset = withMetadata.GetMetadata().Offset
}

[mjgarton]

I think this currently sits outside the goals, or perhaps even goes against the goals of substrate, but I'm interested in hearing more about why you want to do this in your particular use case, so that I can try think about alternate solutions. @charlie

[charlie]

To describe it narrowly, the motivation is to provide a unique identifier for each message within a topic.

In practice, the goal is to project the entire contents of a topic into an Elasticsearch index, using these identifiers as the document IDs. Note that while the contents of the messages contain IDs, these are not guaranteed to be unique, as the same message contents (i.e. the same ID) may be published to the topic more than once (by misbehaving publishers). I want the projection to reflect the existence of both messages in the topic.