Design goals for next major revision of nexus

[gammazero]

Draft issue to collect ideas to consider for next major update

Motivation

Nexus was written years ago and has accumulated features and interfaces to support an evolving standard. Now that the WAMP specification is much more mature and stable, nexus could be given a more unified and modernized code base. This would involve changes to many APIs to improve usability and consistency, and updates to the core code to improve performance and scalability. Additional interfaces for metrics could be provided as well.

This new version would make nexus a current, full-featured, and valuable provider of WAMP protocol functionality.

Goals / Ideas

Simplified APIs

• Options parameters instead of large config structures
• Cleaner invocation of user-provided functionality. Relying on closures can be awkward, and a simplified channel model can simplify writing wamp client code.

Performance

• There are some bottlenecks caused by the current approach to synchronization in the router, where messages pass through a single goroutine that accesses router data structures. This could be replaced by a different approach to making these data structures concurrent-safe to allow many concurrent goroutines to execute router code.
• More message data can remain serialized as it transits the router to minimize [de]serialization work.

Durability

• Subscriptions, procedures, and undelivered messages can be persisted in a backing store to allow router restarts without re-registering all clients and be able to resume or at least report lost messages in flight.
• Message queues can overflow to backing storage to allow larger spikes of pubsub messages in the presence of slow subscribers, without discarding messages or blocking publishers.
• Configure limits
  • queued message limits
  • max clients, subscriptions, procedures, etc.
  • configure rate limiting

Router Scalability

• Multiple router instances can route messages between each other, allowing each instance to handle a different set of clients. Additional routers can start and join to handle more clients.

Metrics interface

• Provides endpoints for metrics collection and query.
• Track per-client metrics.

Core reorganization

• Removal of deprecated and unused code.
• Cleaner router package organization.
• There is too much overlap in unit tests and integration tests, and much redundancy in unit test code. This can be simplified.
• Use of generics where applicable
• Structured logging?

Major Version Update or New Project

Such features will require breaking changes and code reorganization. This could be done as nexus v4.0.0, but may be better done as a new project/repo. Advantages of a new project:

• New repo does not need to abide by semver > 1 rules and avoids problems with /v3 /v4 inconsistency.
• New repo can be private or be marked as under construction until ready for initial release.
• Clean separation of documentation.
• Project can be renamed to something more appealing to the WAMP-proto community.

A disadvantage of a new project/repo is possible confusion about what project is the most current nexus (or whatever it is called), but that can be addressed with clear documentation in the repo README.md. Another problem is the many pieces of code and documentation in the world that refer to the nexus repo that will need to be updated.

[KSDaemon]

Hehey! @gammazero Glad to hear from you!) Great idea! I support this!

My first comments are related to the new project/repo/version:

• I don't see any reason for doing this work in a separate project. We can create another alternative main dev branch and accumulate all the work there.
• Nexus is already open-source project and if you (we) plan to keep it this way — why make some work private? Maybe some other participants will come and suggest new ideas or even help with the implementation.
• After we agree that the new version is ready for release — we can merge/replace the code and the documentation in real main branch with the new version and mark it as a new major release.
• I like semver and expect that the new version may break the existing apis.
• I also think that Nexus is already well-known in the world of WAMP so why we should rename it? I like this name.  There are other implementations whose names are not constructed from the Wamp word, like Spell, Turnpike, Minion, Bondy, Wiola.

[gammazero]

@KSDaemon Thank you for the feedback. You make good points about keeping the same repo, so I am happy to do that. Even though changing import paths to "github.com/gammazero/nexus/v4/..." seems annoying, it is there for a reason. I would also prefer to develop in public, and we may even get some help/interest if people see the development happening.

I think that the next step is to create a set of more detailed development tasks, determine how important they are, and decide how best to order them. I am planning on starting that by creating a top level v4.0 issue containing a tree of child issues, and then dividing those into child issues. Comments on these issues may be a good way for the world to suggest more general changes to the project.

[KSDaemon]

@gammazero Yeah! Good idea! I'd be happy to participate as much as I can in that!

[mkdreams]

What is the new progress? I want to use these new features. where is "github.com/gammazero/nexus/v4"?