Feature set for Alpha MVP

[bugs181]

Features:

• Modular (Build your own DB)
• Pre-built (Use our pre-built code with configuration) - server.js
• Exposed Query to HTTP, Websockets

Modularity:

• Config files
• Schema
• Default communication adapters (Websocket, HTTP)
• ACL
• Built-In and Custom indexing algorithms (linked lists, time based, geospatial, History, etc)
• Storage

Conceptual:

• Linked lists use the prev and next iterators via an Index and expose node.prev() and node.next() API
• History index offers version control, revisions, and rollback support

Details:
JoyDB is a modular first database, this means users have the ability to roll their own database like Legos. The default server.js is for non-devs who want out of the box functionality, custom configuration, and potentially an IDE. Other modular parts include exposing the Query module to communication adapters (like HTTP server, Websockets, etc).

Gossip:

• Mesh module should allow for data splitting between super peers for enterprise federated setups. To provide authentication between these peers, we use the ACL module with whitelist.
• Storage has a shouldSave placeholder which can be over-ridden for more fine-grained control over which data to save based on factors.
• Storage is a higher-order component which exposes APIs for other modules that use it. (like shouldSave)

TODO:

• Flatten object and detect circular references, creating links between node data

[merarischroeder]

1. "modular" - for some reason I don't like this; I'm not sure why, and I have no suggestions at this stage.

2. "Config files" - I strongly expect that you will be able to partition a JoyDB forest/cluster. One set of infrastructure, but multiple "databases". One "database" should be "master", that self-describes config.

3. [New] Varying Redundancy configuration - for Collections or Databases (some sort of grouping) - should have redundancy configuration. The API could have a callback/event for local-saving/remote-saving/redundancy-saving states of a record. This could then be possibly reflected in the UI - saved offline, saved, solid - whatever language the developer chooses.

Types of configurations:

• Local Only - to store local-only collections of data about the local node. This might include the node's private key(s).
• Super Peer - requires storage to a super peer with operational redundant disks before being "happy".
• Multiple Peers - requires N peers to have a copy of the data before being "happy".

4. "Flatten object" - I like the idea of shallow collections. This requires the concept of "References". This also makes it easy to replicate the data to a relational database for use of report engines.

5. "ACL" - the ability to use an external ACL/IAM system. This includes use of JWT for Authorisation (and Role-based claims). This might leverage an existing AWS/GCP/Azure auth system to begin with. You can always build your own, but do that much later, not first.

6. [New] Cloud Functions - if you have a network of data, I would like to be able to deploy arbitrary code to arbitrary peer locations to:

• Trigger upon data event types
• Process data in bulk/batch
• Integrate with 3rd party systems

This can be used as a foundation to implement a lot of the system features.

---

I see the base framework as:

• Gun / DB
• Cloud Functions
• Infrastructure enrollment and management. Upon Gun/Db. Running as Cloud Functions.
• Authentication / Authorisation
• BLOB storage
• Extendable Admin User Interface (Web-based) - basically being a Single Page Application.

[bugs181]

"modular" - for some reason I don't like this; I'm not sure why, and I have no suggestions at this stage.

If you could clearly articulate a reason why modular isn't good, I can come up with concrete examples why this is THE solution to ANY form of technology.

One example I can clearly think of in "Big Biz" is that facebook recently moved into a modular form with their React/React Native products. With their Lean Core initiate they've had exponential growth by deprecating in-house modules, merging and moving components into the community domain, and thousands of bugs and feature requests being implemented in a stable and progressive approach by the community at large. Their main rationale was that the community who knows those components, do best at maintaining and improving them.

"Config files" - I strongly expect that you will be able to partition a JoyDB forest/cluster. One set of infrastructure, but multiple "databases". One "database" should be "master", that self-describes config.

Your expectation would be correct. Joy holds no principals or rules based on the underlying architectures used to process and share configuration. These are purely up to implementors of said modular components.

[New] Varying Redundancy configuration

Applies to previous comment.

Types of configurations

All of these can be broken up into a much easier to read configuration file. For example, the primary configuration file is just about "which" modular parts are inserted "where" into the flow. Again, Joy makes no further assumptions except for gluing these modular pieces together.

The rest of your points can easily be created with modular plugins, which are always opt-in to keep the database running fast and smooth.

[merarischroeder]

@bugs181

1. RE: Modular - I'm thinking particularly about promoting that as the Centrepoint of JoyDB. As you say, I can't think of any modern software that isn't modular. For that reason, perhaps it should be a footnote. As I say, my thinking here is incomplete. It's best to have a better suggestion when criticising. I don't have a better suggestion yet.

2. RE: "Config files" - I saw "files", so I said "db". That's two options, and you replied "yes". Are you proposing config "files", or a "master" db partition that is namespaced and used by all modules? Is there a convention for modules to store configuration?

3. [New] Varying Redundancy configuration. You said "Applies to previous comment.". So does that mean the redundant copying of data across multiple nodes is not going to be standardised as part of the core framework? How would server.js work? Are there core modules as a default?

...

6. "Cloud Functions" - this is infrastructure level stuff. Perhaps node.js will be needed to run such functions on nodes. How will "modules" be running across JoyDB if not for some kind of module runtime (presumably node.js)? Will the modules be running monalithic with the rest of the code, or will there be secure isolation?

Cheers

[bugs181]

1. I hope you can be excited for the modular aspect as much as we are! =)

2. "Config files" is a relatively loose terminology here. I don't mean to say "files on a disk". Think of "files on the cloud". When you're storing files in the cloud, it's highly unlikely they are "true" files and not just some virtual file inserted into a database somewhere. A better naming convention would be "config" as in Configuration, which can be stored and communicated anywhere.

3. Redundancy is built into the default configuration set, thanks to the principals of "mesh" networking. Super peers keep a copy, other peers that have the data can optionally keep a copy (or purge it to free up space), and the local copy of the sender. Server.js works by providing a default configuration file (yes, this time I mean actual file). From there, the user may make changes and use more modular parts if they want additional features.

...

6. Cloud functions will be extremely easy to accommodate and an excellent suggestion! There are quite a few ways to accomplish this. RPC (Remote Procedural Call), using NapaJS with a communication module hooked into it, or some other Mesh networking hooked interface. Perhaps one or all of these will be included in the first stable release.

[merarischroeder]

1a. A modular database system is certainly a good feature, but again, it's up to you to consider whether or not that should be the first feature in the list. Perhaps it's an internal software architecture principle, but not a key user feature.

1b. Here's a quick attempt at an alternative approach, one that focuses on problems:

There are so many sad databases that don't help you with offline mobile applications, forces you to find your connect your own realtime communication system, burdens you with a complex array of replication modes that need a systems engineer to get right, then need rebuilding of replication when the schema changes.

Joybase is a delightful database designed with modern needs at heart. It's lightweight and simple, yet powerful.

Get started with a free project at Joybase.io.

Features:

• Enjoy simple access and configure it all via the web browser
• Cheeky Logging and Auditing
• Carefree Security
• Curious Queries
• Surprising Functions and Triggers
• Playful Data Browser
• Unbridled Schema Designer
• Giggly Replication - Leverage popular reporting engines with replication to SQL Server, mySQL, PostGreSQL, and Oracle

1c. (I actually have a startup, that is similar in modular nature, and it has taken me a while to refine the public message to be clear WHAT it's doing for users, and avoid too much of the HOW.)

1d. I suggest that you build Joybase in a modular way, but for now, only focus on server.js and a single specific configuration. After you gain traction, and have some alternative module configurations that are highly sought after, you can demonstrate the power of the modularity. Specifics are much more powerful. "By the way, because Joybase is modular, we have 3 main configurations you'll find useful, and you can always customise your own"

2. Reducing to the sole term "Configuration" is the right move.

3. I was particularly describing the ability for the client-side to be able to let the user know when a data-redundancy policy had been accomplished. I wonder if such feedback is part of the underlying GUN system, does the publishing node get confirmation about redundancy later? If so, I'm sure that can be exposed via the Joybase API.

4. I am particularly suggesting that you DO NOT implement an RPC pattern. Rather, I recommend that you have triggered functions.

[Dletta]

@merarischroeder

Not sure GUN qualifies as running in the Cloud just yet, ;).

I think you are raising some good points for marketing. JOY is super early in development and I think it has the huge advantage of having a big user community (from the original GUN community) that is just waiting to switch to 'Normal Happy API' and 'Joyful Consistency'.

I have a feeling that it won't take very long for first adopters to jump on this bandwagon, as long as we get it rolling.

One of the things that would help the Team is to communicate directly in the Gitter Channel, that's where we are building up the MVP ideas, the API discussions and talk about how we plan to fundamentally run the roadmap etc. You are most welcome to join us there?
https://gitter.im/JoyDB/community

[bugs181]

@merarischroeder @Dletta couldn't have said it better myself.

Re: 1a
Maybe I get too "giddy" about the prospect of having a modular database. However, simply put.. because one doesn't exist yet. I've been seeking for something like Joy (Joybase?) for years in my software engineering career.

I generally don't want X or Y feature and yet I have no choice (it was decided for me). So I download the entire database bundle thats riddled with feature bloat and slow performance. Typical databases consumes tremendous amounts of resources (like memory and storage) because of this. Being modular means that the database is ready, sleek, and minimal.. and with a couple of configuration parameters, you have all of the features at your fingertips. I think THAT is a huge selling point.

Typically anyone writing software and doing database research will find this prospect certainly flavorful. It would make "me" want to use it and research more about it.

[amark]

Quick Q?

Isn't GUN suppose to be the modular one?

And JOY the big fat enterprise bundle?

You know how I love modular systems... But I'd be a bit confused what the point of JOY is if it doesn't take a very opposite approach?

Maybe reiterate what the goals / purpose is here?

[bugs181]

@amark As much as I LOVE Gun, after speaking with several devs we can come to the agreement that contributing to Gun isn't easy. I think being MORE modular than Gun's large build and confusing code base would help enable more contribution, not less. Although, while it's more modular - the default configuration with a pluggable standardized interface will help to bring the bundle together. The goal is to have an official repo with all of the outlined features and contributions from PRs, but only enabling the default base-line modules. The primary script is geared toward enterprise federated setups and with a few modifications to the configuration can enable a decentralized experience.

The goals after several discussions with what I consider to be the core Joy team:

• Modularity
• Default enterprise setup
• All modules included in official repository and enabled via configuration
• Bring your own Query (Joy intends to be query compatible with existing DBs so that users don't have to learn a new query syntax)
• Migration scripts for migrating data from existing database setups
• Query interfaces exposed via HTTP, WebSockets, etc

Like Gun, Joy has modular storage adapters. It is still being discussed, however an acronym called CUDDLER has been seen fit to expose every current database implementation. It adds three new API to the traditional CRUD approach.

CUDDLER is Create, Update, Delete, Depart, List, Event, and Read. Depart is a two way binding when a database leaves a space (like when used client-side) and/or on federated setups if a database needs to restart or has throttling overload issues. Events are what we've come to expect from real-time streaming experiences. And listing is for aggregations and other sorts of features that most users seem to have an issue with when grasping how to iterate a list in a decentralized setting.

Overview:
An overview of this set up might hopefully make things more clear.

Communication <-> Bridge <-> Modules
Query Service <-> Frame <-> Blueprints: Storage, Indexers, Schema, ACL, etc

Do note, just because I've proposed Frame and Blueprints as core concepts. Frame offers "unbuild"-like CLI utils that can build or use compliant CommonJS modules for easily converting to and from. Nothing is stopping someone from bringing their favorite CommonJS modules into the mix or exporting Blueprints to a CommonJS module to be used elsewhere.

[bugs181]

WARNING: Be prepared to read a long response, because I think it's important to explain how Frame, thus Joy works under the hood.

6. Perhaps node.js will be needed to run such functions on nodes. How will "modules" be running across JoyDB if not for some kind of module runtime (presumably node.js)?

I completely spaced on this. Sorry I didn't reply more in depth. While it's true node.js can be used for server-side federated setups, it's not the only environment that can. Frame actually already has a dynamic asynchronous module dependency loader (kind of a mouthful) which knows how to resolve things on a variety of platforms.

So long as the "loader" modules don't do anything platform specific, they will work. Even some of these are shimmed thanks to the architecture of Frame to provide for typical require() used in node.js modules (called CommonJS).

For example, as a proof of concept I have require() working in the web browser without a bundler like Browserify, Webpack, etc. The require() also works with varying different protocols for the loaders. We already have two built-in bootstrap module loaders for URLs and Files.

Example:

require('http://example.com/someModule.js') // Module on the web somewhere
// or
require('/some/local/file') // Local files for node

Do note, that even if using relative paths in the browser, they always resolve to http URLs, and vice versa for files. I fully intend to complete a more robust feature-set for the loaders, however other things should be implemented first to get a stable alpha released rolled out. How these work is directly out of the node.js architecture handbook, just with some additional features in a very small form factor.

Will the modules be running monalithic with the rest of the code, or will there be secure isolation?

If using what we call "Blueprint" type modules, then the code is very much isolated from the rest of the running code. This is generally up to how secure the "loader" is, but can easily be improved upon to varying degrees.

Blueprints use a VERY minimal yet complete API with one concept. That all types of technology can be represented as having an input and/or an output. Because of this, blueprints only need to have an in function and use an out function, both provided via the Frame architecture but are easily shimmed for any uses outside of Frame. Blueprints do have other optional functions like init which behaves similar to a class initializer, on for handling events, etc.

There is also a describe object, which can provide descriptions and in the near future type checking and sanitization for what types of data the blueprint expects and outputs. Blueprints are also safe thanks to how Frame creates a default blueprint stub to be worked on until the module is resolved.

Frame also has a built in schema to validate blueprints and not let them do anything wonky. Because of this, there is type safety for functions and expectations. An error inside one blueprint will not cause the entire application to crash, instead errors are propagated out into proper error handlers to attempt recovery (whatever that means to the application layer).

This is by far not a complete feature set for the core of Frame and has too many features to list for one Github issue. Expect a documentation on the Frame repo soon. That said, Frame minified and gzipped is only 3KB in file size. Compare this to the most popular libraries and frameworks, and prepare to be amazed.

[merarischroeder]

I second @amark request for clear project purpose and goals. The modular stuff is interesting, and clearly important, but needs to be discussed with clear direction. Perhaps we need a long list of what we think the goals and purpose is, and then prioritise.

Example Direction:

• Catering for very specific use-cases in an opinionated and simple way
• Catering for these one at a time
• Doing so in a way that reuses components
• Doing so in a way that minimizes first-load time
• Catering for "Enterprise" first with common-joy.js i) Hybrid public/private cloud VM server nodes; ii) Admin UI; iii) Auditing and Logging; iv) Replication with popular relational databases
• Catering for "Mobile App" also - if there are any differences with "Enterprise".
• Catering for "Custom" configuration; i) With a web form that allows the user to tick what configuration they would like; ii) They then download the packaged js file(s). iii) They can choose either to bundle a single js file, or dynamically load multiple js files.

Just ideas. This direction is clearly assuming a lot of things though.

1. We should be documenting the problems we're trying to solve. We need personas, target markets, end-user problems, and mapping to expected solutions.

2. We should also be "measuring". We might think a particularly default configuration adds to "bloat", but is that 1kB or 10MB of JS code?

@bugs181 FYI

[bugs181]

@merarischroeder if spending an hour to write up two relatively long github issues isn’t clarifying my position on “clear project purpose and goals”, then I’m just wasting time. I apologize but I don't have time to spend speaking about the conceptual points if I'm not going to be taken seriously. I would rather spend my time creating an implementation. If people use it, that's fantastic. If not, then I'll take a step back and re-analyze.

[merarischroeder]

@bugs181 You've certainly clarified the technology, my last response was expanding on @amark 's point.

"If people use it, that's fantastic. If not, then I'll take a step back and re-analyze." - that approach can work, but not often. You can certainly work on code, bottom-up, while I work on project management top-down.

You have a summary of product features which looks great, but without the documentation of an initial target market, and their problems, there's a risk the project's implementation can miss the mark. I'm sure you've got all that information in your head. I'm just not a mind reader. So I'm asking for this kind of information and taking the time to provide applicable examples in an attempt to be useful.

[merarischroeder]

To put it simply, this issue is about "Specification", but there are no "Requirements" documented. I propose we create an issue for "Requirements". A Gitter Chat room is not a great way to document a growing collection of "Requirements"

[bugs181]

I just like the idea that actions speak louder than words. To be fair, these aren't points just in my head. This is feedback from the Joy community via Gitter chat as @Dletta has mentioned.

During the conceptual brainstorming phase, we openly discuss implementation details, feature ideas, compatibility, etc. Currently, we're at Stage 0 awaiting progress on Frame to at least ready to consume.

On that front, I've just pushed a relatively large update which adds flow functionality. There's still more work to do to add error propagation, allow flows to speak to one another and allow the flows to be reusable from top-down. Aside from that, we need to write up unit tests and document everything. Hopefully we can make strides toward useful progress this week.

Once all of that is complete, we can begin to "mockup" the structure of Joy core and determine the best way to move forward with configuration, modules, and application logic.