This repository was archived by the owner on Dec 13, 2022. It is now read-only.

This repository was archived by the owner on Dec 13, 2022. It is now read-only.

Question: Why delete new partitions data manually, but do not use spark `overwrite` mode? #2

Open

opened

jaquet/src/main/scala/com/fyber/KafkaToS3Stream.scala

Line 139 in 9788252

    
           private def deleteDestPathsOnS3(df: DataFrame, partitionsSeq: Seq[String]): Unit = {

Why you delete partitions data and then write these partitions in append mode?
Why do not use overwrite mode?

Metadata

Assignees

No one assigned

Labels

No labels

No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests