Integration with Sleuth for distributed tracing when using Parallel workflow

[c-ankur]

Currently, when integrating flowret with Spring cloud sleuth for distributed log tracing, the traceId is not propagated when using parallel workflows. This is because new threads are created in a parallel workflow and flowret uses ThreadPoolExecutor, and this implementation is not extendable (so, we cannot use any other option such as LazyTraceExecutor or LazyTraceThreadPoolTaskExecutor)

The traceId is successfully propagated when a sequential workflow is used, because it works on the parent/caller thread.

Can flowret be made either extensible in a way that a different ThreadPoolExecutor can be used or more control can be given to the user of the library on how any new threads are created?

[deepakarora3]

Understood your issue. I have a couple of options to address this.

First is to allow the clients to create their own ExecutorService which Flowret can use. This problem I see with this approach is that Flowret will not have any control on the blocking queue used in the ExecutorService. This is an issue because I would like to retain the use of a bounded blocking queue such that the rejected item handled never gets invoked. I want to do this because in case the bounded queue is full, I would like to block rather than execute on the current thread. I have seen use cases where running on the current thread has caused problems.

Second is to allow the clients to pass in their own implementation of a ThreadFactory to Flowret. Clients could create their own thread factory and set whatever information they want in the threads created which Flowret can use. I would prefer to use this method.

Will the second option work for you? If so, I will go ahead and do the change in the next couple of days.

[c-ankur]

Oh, thanks. I will try and see if this helps.

[c-ankur]

Based on further testing, this solution hasn't worked yet.
The setting of traceId/correlationId per case is not functioning correctly when a sleuth Tracer is autowired through the ThreadFactory. The value gets set only for the first case, but the same value gets used in every subsequent flowret case. I'm thinking of some alternatives, possibly having callbacks to the parent thread that could potentially supply the correlationId dynamically. Something like this could be enabled in flowret? Or may be another approach, for example, ability to supply a LazyTraceExecutor (but still doing what you want to do with the bounded queue?)

[deepakarora3]

I understood what you are trying to accomplish and that is not going to be straightforward to do. Reason is that is that there is a pool of threads for parallel processing which is independent of the main thread. In order to do what you are wanting, you would need some kind of callback / hook to inform you when a thread from this pool is being used so that you can set the thread specific variables (most likely Spring is going to be setting thread locals internally). This is not possible at the moment. The other alternative is for me to provide a mechanism where the parallel processing threads are created from the parent thread and die out when they go out of scope. This is not so performance efficient but may solve your need if the context can be made to propagate from parent thread to the child thread. Is it being propagated?

Give me some time to think more about this and I will get back on how to solve this.

[deepakarora3]

Hi - I have release a new version (1.4.2) in which I have introduced a new way to do parallel processing. The documentation has been updated. You could look it up there - providing the explanation here too - what you need to do is to specify the value of maxThreads as 0 in the init method. When you do that, Flowret will not create the Executor Service thread pool and will instead create threads on the fly whenever needed. This means that there will not be an upper bound on the maximum number of threads that can be created / used. But, since it is the parent thread which is creating the child threads, it will be possible to propagate the context from the parent to the child thread for example using inheritable thread locals. I am not sure of how Spring does this, but if you set the trace id / correlation id in the parent thread as a inheritable thread local variable, then it will propagate to the child threads and it could solve your problem. Let me know how it goes. Thanks.

[c-ankur]

Update:
Tried, but didn't work out well using Sleuth. I think there are multiple issues here, flowret API not easily autowired is one I believe that complicates things. The creation of new thread itself changes the context that Sleuth internally uses. So, currently, relying on other workarounds in the application code using Sleuth span creation (which partly works, not ideal though).

[deepakarora3]

Sorry that did not work as I don't have much knowledge of how Spring works behind the scenes - in fact that was one of my reasons for sticking to core Java. But I am not sure what you mean when you say that Flowret API is not easily auto wired. Reason I say this is because all operations on Flowret are done on the singleton Flowret object. Hence in this sense it is as auto wirable as any other singleton object that you create in an application. The fact that changing the thread changes the context of Sleuth internally seems to be something that needs to be addressed by Sleuth as there may be other multithreading use cases that may need to run with Sleuth and may face the same problem. Sorry that I could not help you on this and I hope the workaround does it for you.