close writers during end of stage using onStageCompleted

[tuckerh99]

Improvement on when to close writers. While implementing an s3 storage engine, noticed that we want to close the writers sooner than later to ensure shuffle files are stored on s3 by the end of a stage.

[CLAassistant]

All committers have signed the CLA.

[hiboyang]

@tuckerh99, thanks for the PR! Curious whether you run your Spark production workload with shuffle files storing on S3? If so, how is the performance?

[cpd85]

@hiboyang hi we are testing this out actively right now. in the first iteration it seems the performance impact is very high due to the nature of shuffle data. Basically if we wait to merge the shuffle files until the stage is completed, then we need to add in some delay before starting the next stage while the shuffle data is uploaded to S3.

[leletan]

Hi @tuckerh99 how are things going on the later iterations? Any luck?