From the spark code walkthrough, the spark driver can allocate the same cores to different workers in sparkle. Since we run all the workers on the same system the core ID level distinguish does not happen in spark and the thread pool created by spark for RPC events may all bind to same core IDs and resulting in a performance bottleneck. So, can we try c-group and try?
`Additional Code-
private val threadpool: ThreadPoolExecutor = {
val availableCores =
if (numUsableCores > 0) numUsableCores else Runtime.getRuntime.availableProcessors()
val numThreads = nettyEnv.conf.getInt("spark.rpc.netty.dispatcher.numThreads",
math.max(2, availableCores))
val pool = ThreadUtils.newDaemonFixedThreadPool(numThreads, "dispatcher-event-loop")
for (i <- 0 until numThreads) {
pool.execute(new MessageLoop)
}
pool
}`