feat: debugger hot restart

[jpeletier]

This PR introduces hot restart so the debugger can be completely turned off and turned on back again, which is useful in server scenarios where you want to enable debugging, enter, look at what is going on and then leave things running as before.

There is a new debugger method stop() that will shut down the debugger cleanly. The debugger can then be started again with start().

In the case where the host process is running multiple Lua VMs simultaneously, the DebugMaster acquires a unique key so there may be multiple instances running simultaneously without conflict.

Additionally, worker/master channels are cleaned up when a debug session is shut down.

[actboy168]

1. lua-debug is not designed to support multiple instances. However, lua-debug supports debugging multiple lua VMs at the same time.
2. lua-debug's master is very lightweight, and when debugging is not activated, it just listens to an fd. So is it really necessary to exit it?

[jpeletier]

lua-debug is not designed to support multiple instances. However, lua-debug supports debugging multiple lua VMs at the same time.

Forgive me because I have been spending the last week wrapping my head around the design with the multiple internal lua-debug VMs, and what runs where and how VMs communicate. Considering your comment I have done another review. Allow me to describe what I have understood, in my own words. At the very least this description may help others understand this project better.

Host (debuggee) application uses own helper script to locate script/debugger.lua stored in the extension which then actually loads the appropriate luadebug.so from the extension according to the Lua version, initializes it and obtains a handle to rdebug lua library.

Host application then calls start():

require("<helper script>"):start('127.0.0.1:12306')

The above forwards the call to rdebug (rdebug_debughost.cpp). rdebug instantiates and configures an independent internal "luadebug" VM. lua_State for both VMs is cross-stored in each other VM so they can communicate.

Bootstrap code is then run in this new "internal" luadebug VM and becomes the first worker. The first worker creates a new "thread", called the master which is in fact a completely separate Lua VM in its own thread. This "master" opens a socket and listens for connections coming from the vscode extension. An scheme involving channels allows the master VM to communicate with the luadebug/worker VMs.
It is important to note that this first worker who instantiated the master should be the last to be destroyed, or the thread will hang until all other workers have exited:

lua-debug/extension/script/backend/bootstrap.lua

Line 35 in 35b2912

thread.wait(mt)

Even though the first worker created the master, its associated host (user's) Lua VM can be debugged also. All workers interact with the "other side" of rdebug called hookmgr, bound to rdebug_hookmgr.cpp

Once the master is up and running, further unrelated VMs in the same process can attach to it and be debugged by using the helper script and invoking attach() instead of start():

require("<helper script>"):attach({})

Given the above, I have reconsidered my proposed changes and now in my code I instantiate a dummy Lua VM which just calls start() to become the master. The rest of the actual game-related VMs just do attach. To shut down, we first shut down all game-related VMs and then the master one is destroyed last.

With the changes in this PR, any game-related VM can attach() at any time to activate debugging and then, using the new stop(), the debugging session uninstalls the debugging hooks restoring performance to normal, all without having to restart the whole process.

Therefore, I dropped some of the changes but kept part of the modifications that perform orderly cleanup and allow to enter or exit the debugging state, this is very useful.

Thanks,