NATS jetstream support

[tamalsaha]

Is it possible to publish these changes directly to a NATS jetsream?

[rueian]

Hi @tamalsaha, Apache Pulsar is currently the only implemented external sink and source.

A new sink and source can be implemented by embedding BaseSink and BaseSource.

pgcapture/pkg/sink/main.go

Lines 24 to 31 in 1a64198

	type BaseSink struct {
	CleanFn CleanFn
	cleanOnce sync.Once
	committed chan cursor.Checkpoint
	state int64
	err atomic.Value
	}

pgcapture/pkg/source/main.go

Lines 33 to 40 in 1a64198

	type BaseSource struct {
	ReadTimeout time.Duration
	state int64
	stopped chan struct{}
	err atomic.Value
	}

[rueian]

Hi @tamalsaha, we are considering expanding the support of different messaging systems. Just curious, what makes you want to use NATS?

[tamalsaha]

We are coming from the cloud native / Kubernetes world. NATs is a very lightweight (compared to Kafka) message queue with its Jetstream feature can be used to implement Outbox pattern. NATs is a top level project for CNCF https://www.cncf.io/projects/nats/ . So, NATs integration will be a very welcome addition for us.

[rueian]

Just adding an important note here: NATs has a default 8MB message limit. We probably need to enlarge the limit if we store WALs into NATs.

[Imtiaz246]

Hello @rueian, I’m interested in the progress of adding NATS jetstream as an external sink/source in pgcapture. Could you provide an update on recent developments or anticipated timelines for this feature?

[rueian]

Hi @Imtiaz24,

Thank you for your interest in the NATS integration but we don't have a timeline right now. We probably would like to have a integration for Kafka first.