Skip to content

Conversation

@jeynmann
Copy link

@jeynmann jeynmann commented Jul 17, 2023

  1. Use wakeup feature for clients/servers' worker.
  2. Clients/servers use their own thread to progress and send/reply message.

The following table is the testing results on the customer's cluster.
(Patch means using the spark-ucx of this branch.)

mode sparkucx_version shuffle data shuffle read stage
DC_X/TCPCM master 559.4 GB 0.15 min
DC_X/TCPCM master 559.4 GB 0.12 min
DC_X/TCPCM patch 559.4 GB 0.08 min
DC_X/TCPCM patch 559.4 GB 0.08 min
TCP 559.4 GB 0.08 min
TCP 559.4 GB 0.08 min
DC_X/TCPCM master 1118.8 GB 0.2 min
DC_X/TCPCM master 1118.8 GB 0.18 min
DC_X/TCPCM patch 1118.8 GB 0.15 min
DC_X/TCPCM patch 1118.8 GB 0.13 min
TCP 1118.8 GB 0.2 min
TCP 1118.8 GB 0.2 min
DC_X/TCPCM master 2.2 TB 0.3 min
DC_X/TCPCM master 2.2 TB 0.33 min
DC_X/TCPCM patch 2.2 TB 0.3 min
DC_X/TCPCM patch 2.2 TB 0.25 mi
TCP 2.2 TB 0.48 min
TCP 2.2 TB 0.3 min
DC_X/TCPCM master 3.3 TB 0.53 min
DC_X/TCPCM master 3.3 TB 0.53 min
DC_X/TCPCM patch 3.3 TB 0.38 min
DC_X/TCPCM patch 3.3 TB 0.43 min
TCP 3.3 TB 0.58 min
TCP 3.3 TB 0.6 min
DC_X/TCPCM master 4.4 TB 1.1 min
DC_X/TCPCM master 4.4 TB 1.1 min
DC_X/TCPCM patch 4.4 TB 0.73 min
DC_X/TCPCM patch 4.4 TB 0.7 min
TCP 4.4 TB 0.77 min
TCP 4.4 TB 0.75 min
DC_X/TCPCM master 5.5 TB 1.4 min
DC_X/TCPCM master 5.5 TB 1.4 min
DC_X/TCPCM patch 5.5 TB 0.93 min
DC_X/TCPCM patch 5.5 TB 0.95 min
TCP 5.5 TB 0.98 min
TCP 5.5 TB 1.0 min
DC_X/TCPCM master 10.9 TB 2.4 min
DC_X/TCPCM master 10.9 TB 2.4 min
DC_X/TCPCM patch 10.9 TB 1.6 min
DC_X/TCPCM patch 10.9 TB 1.6 min
TCP 10.9 TB 1.8 min
TCP 10.9 TB 1.9 min

@jeynmann jeynmann changed the title Client/Server use their own thread [DNM] Client/Server use their own thread Jul 17, 2023
@jeynmann jeynmann changed the title [DNM] Client/Server use their own thread [DNM] Client/Server use their own thread and wakeup feature Jul 17, 2023
@jeynmann jeynmann changed the title [DNM] Client/Server use their own thread and wakeup feature [DNM] Client/Server wakeup feature and use their own thread Jul 17, 2023
@jeynmann jeynmann changed the title [DNM] Client/Server wakeup feature and use their own thread [DNM] Client/Server use wakeup feature and their own thread Jul 17, 2023
@jeynmann jeynmann changed the title [DNM] Client/Server use wakeup feature and their own thread Client/Server use wakeup feature and their own thread May 30, 2024
@jeynmann jeynmann force-pushed the client_server_use_own_thread branch from 53ce91c to 2d1c464 Compare June 3, 2024 08:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants