fix: update Spark plugin for compatibility with Spark 3.x and 4.x#3350

pingsutw merged 2 commits intoflyteorg:masterflyteorg/flytekit:masterfrom

kevinliao852:spark-plugin-compatibilitykevinliao852/flytekit:spark-plugin-compatibilityCopy head branch name to clipboard