Replace bundled command with compose

[sergerad]

Context

We want to replace the bundled command with a docker-compose deployment for local development.

Closes #1765.

Changes

• Remove the bundled command which ran all components in a single process
• Add ntx-builder as a standalone CLI subcommand
• Add docker-compose.yml with individual services: store, validator, block-producer, rpc, ntx-builder
• Add a separate genesis compose profile that wipes volumes and bootstraps a fresh network
• Update Makefile: make compose-genesis (wipe + bootstrap), make compose-up (start), make compose-down (stop), make compose-logs (follow logs)

[Mirko-von-Leipzig]

I have not yet read through this, but I don't think we can actively remove the bundled command just yet, until we are extremely certain that all things work correctly as separate components.

[SantiagoPittella]

Did not review it yet, but tried running docker-compose up and got this errors:

rpc-1 exited with code 2
ntx-builder-1 exited with code 2
block-producer-1 exited with code 2
validator-1 exited with code 2

[sergerad]

Did not review it yet, but tried running docker-compose up and got this errors:

rpc-1 exited with code 2
ntx-builder-1 exited with code 2
block-producer-1 exited with code 2
validator-1 exited with code 2

You can get the logs of the containers to determine why they exited. But you do need to run make compose-genesis first.

[sergerad]

I have not yet read through this, but I don't think we can actively remove the bundled command just yet, until we are extremely certain that all things work correctly as separate components.

My intention with this PR is to have it working against the integration test suite so it will at least be working for local dev use case. As for deployment to devnet, yea we need to make sure we merge this at a moment that makes sense in terms of our planned releases.

All seems to work but I don't know which commit to match miden-client test suite against node next. So I'm not able to verify the test suite atm.

[Mirko-von-Leipzig]

I have not yet read through this, but I don't think we can actively remove the bundled command just yet, until we are extremely certain that all things work correctly as separate components.

My intention with this PR is to have it working against the integration test suite so it will at least be working for local dev use case. As for deployment to devnet, yea we need to make sure we merge this at a moment that makes sense in terms of our planned releases.

All seems to work but I don't know which commit to match miden-client test suite against node next. So I'm not able to verify the test suite atm.

Makes sense; I think I expected there to be more work to physically decouple the components reliably at startup. But maybe there is less than I thought.

Something we also need to check on is midenup which I believe incorporates the node in some capacity. So switching from a single binary to a docker compose would likely break them. I'm unsure who to ping for that.

[sergerad]

Something we also need to check on is midenup which I believe incorporates the node in some capacity. So switching from a single binary to a docker compose would likely break them. I'm unsure who to ping for that.

https://github.com/search?q=repo:0xMiden/midenup%20bundled&type=code

We just need to replace this with a series of commands (like the ones in the readmes in this PR).

[drahnr]

Could we possibly split the deprecation of bundled from the introduction of the containerized all-in-one deployment option? That way we can actively dog-feed this with little friction while retaining bundled until we're confident to deploy it to devnet.