Nex and databases

[hpvd]

There are several use cases where usage of different database types is the preferred way over event sourcing.
-> Are there already any plans to make it possible to also deploy and run databases in the easy Nex native way?

[hpvd]

I like to share this vid because it give many really interesting background and insights how this is done in Kubernetes in a good way (not using stateless sets...):
https://www.youtube.com/watch?v=Ny9RxM6H6Hg

[hpvd]

Could one think of 2 new "storage types" of typ db in addition to existing kv-store and the object-store brought by Jetstream??
DBs in the "nex native way".. easy to deploy and perfectly connected to nats.

For the inside-tech

• for the sql-db variant: duckdb with its quality, feature set (extensions) and speed may be a good choice...
  • https://duckdb.org/why_duckdb
  • functionality and performance over time https://duckdb.org/2024/06/26/benchmarks-over-time.html
  • https://github.com/duckdb/duckdb
• for the graph-db variant
  • kuzu https://docs.kuzudb.com/ may be the way to go
  • https://github.com/kuzudb/kuzu

[hpvd]

and maybe to get everyone more excited about the possibilities:
There are also wasm versions of both dbs to make them run anywhere...

• https://github.com/duckdb/duckdb-wasm
• https://github.com/unswdb/kuzu-wasm

[hpvd]

if anyone wonder why favouring duckdb over sqlite:

To sum up, they both share many characteristics. However, DuckDB is optimized for queries that apply aggregate calculations across large numbers of rows, whereas SQLite is optimized for fast scanning and lookup for individual rows of data. If you need to perform analytical queries, go with DuckDB otherwise, use SQLite.
https://marclamberti.com/blog/duckdb-getting-started-for-beginners/

=> this should bring the bigger step forward in comparison to having no db when going to pure route (only use things onboard of nex -> event sourcing...)

[hpvd]

as last input on this topic:
this a really interesting write up (overview) of the embedded DBs named above:
https://thedataquarry.com/posts/embedded-db-1/

[jordan-rash]

Commenting to let you know I've read all your input, thanks a ton. It will take me a while to grok all the resources you've provided, but once I do, ill come repost thoughts here.

[nickchomey]

I don't quite follow what's being proposed here, bug to the extent it is making some sort of Db API /integration for Nats/nex, I'd like to propose using https://conduit.io

It is a fantastic golang cdc/etl tool, which would allow anyone to connect to, essentially, any database. They already have nats core and jetstream connectors, so you could easily have a cdc stream from mysql, or whatever, through conduit, out to jetstream. And from there, it could be used in many ways, such as through another conduit pipeline and into a different db.

Conduit + nats is a match made in heaven

[hpvd]

regarding combining DBs and NATs, this may also be an approach:
S3 next level: offload, backup plus also data-sharing, querying? nats-io/nats-server#6478

[jordan-rash]

WRT S3, have you seen this yet? https://github.com/synadia-io/connect

[hpvd]

many thanks for pointing to this!