Parallelize file uploads in fs cp command. (#4132)

## What changes are proposed in this pull request?

This PR improves the performance of the `databricks fs cp` command when
copying directories by parallelizing file uploads. The command uses 8
concurrent workers by default but the number can be controlled via
`--concurrency`.

Implementation details:

- **No ordering guarantee:** Files are now copied in parallel with no
guaranteed order (previously sequential).
- **Fail-fast on errors:** If any file copy fails, the context is
cancelled and remaining operations are stopped (first error is
returned).
- **Retry responsibility:** The implementation does not retry failed
operations; this remains the responsibility of the underlying `Filer`
implementation as before.

**Why `--concurrency`?** No strong preference here, it does not seem
that there is a pattern in the CLI to control concurrency in other
places. This is the flag name used in most Go tools but I'm happy to use
something else.

## How is this tested?

Added acceptance tests to exercise most code paths + unit tests to
validate that the context cancellation and propagation works properly.

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>

Author: renaudhartert-db

NEXT_CHANGELOG.md

@@ -6,6 +6,9 @@
 
 ### CLI
 
+* Improve performance of `databricks fs cp` command by parallelizing file uploads when
+  copying directories with the `--recursive` flag.
+
 ### Bundles
 
 ### Dependency updates

acceptance/cmd/fs/cp/dir-to-dir/localdir/file1.txt

@@ -0,0 +1 @@
+file1 content

acceptance/cmd/fs/cp/dir-to-dir/localdir/file2.txt

@@ -0,0 +1 @@
+file2 content

acceptance/cmd/fs/cp/dir-to-dir/out.test.toml

@@ -0,0 +1,10 @@
+
+>>> [CLI] fs cp -r localdir dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir
+localdir/file1.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt
+localdir/file2.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file2.txt
+
+>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file1.txt
+file1 content
+
+>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/uploaded-dir/file2.txt
+file2 content

acceptance/cmd/fs/cp/dir-to-dir/output.txt

@@ -0,0 +1,21 @@
+CATALOG_NAME="main"
+SCHEMA_NAME="default"
+VOLUME_NAME="fs-cp-test-${UNIQUE_NAME}"
+
+cleanup() {
+  $CLI volumes delete "${CATALOG_NAME}.${SCHEMA_NAME}.${VOLUME_NAME}" 2>/dev/null || true
+}
+trap cleanup EXIT
+
+# Create volume for testing.
+$CLI volumes create "${CATALOG_NAME}" "${SCHEMA_NAME}" "${VOLUME_NAME}" MANAGED >/dev/null
+
+# Create parent directory.
+$CLI fs mkdir dbfs:/Volumes/${CATALOG_NAME}/${SCHEMA_NAME}/${VOLUME_NAME}
+
+# Recursive directory copy (output sorted for deterministic ordering).
+trace $CLI fs cp -r localdir dbfs:/Volumes/${CATALOG_NAME}/${SCHEMA_NAME}/${VOLUME_NAME}/uploaded-dir 2>&1 | sort
+
+# Verify files were uploaded correctly.
+trace $CLI fs cat dbfs:/Volumes/${CATALOG_NAME}/${SCHEMA_NAME}/${VOLUME_NAME}/uploaded-dir/file1.txt
+trace $CLI fs cat dbfs:/Volumes/${CATALOG_NAME}/${SCHEMA_NAME}/${VOLUME_NAME}/uploaded-dir/file2.txt

acceptance/cmd/fs/cp/dir-to-dir/script

@@ -0,0 +1,3 @@
+Local = true
+Cloud = true
+RequiresUnityCatalog = true

acceptance/cmd/fs/cp/dir-to-dir/test.toml

@@ -0,0 +1 @@
+hello world!

acceptance/cmd/fs/cp/file-to-dir/local.txt

@@ -0,0 +1,6 @@
+
+>>> [CLI] fs cp local.txt dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/mydir/
+local.txt -> dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/mydir/local.txt
+
+>>> [CLI] fs cat dbfs:/Volumes/main/default/fs-cp-test-[UNIQUE_NAME]/mydir/local.txt
+hello world!