Skip to content

Parallelize file uploads in fs cp command. #4132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
renaudhartert-db merged 32 commits into from
Jan 25, 2026
Merged

Parallelize file uploads in fs cp command. #4132

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
5896ae4
Fs cp now upload in parallel
renaudhartert-db Dec 11, 2025
b933398
Improve performance
renaudhartert-db Dec 11, 2025
2fa03e1
Use error value
renaudhartert-db Dec 11, 2025
d393388
Cancel go routines
renaudhartert-db Dec 11, 2025
14cfad9
Properly propagate context
renaudhartert-db Dec 11, 2025
2a9ec1b
Linter
renaudhartert-db Dec 11, 2025
179a944
Test context cancellation
renaudhartert-db Dec 12, 2025
79dd600
Clarified rationale
renaudhartert-db Dec 12, 2025
2d1f19d
Gracefully wait for cancellation to complete
renaudhartert-db Dec 12, 2025
caaff68
Merge branch 'main' into renaud-hartert_data/fs-cp-fast
renaudhartert-db Dec 13, 2025
12dec5c
Add acceptance tests for fs cp
renaudhartert-db Dec 15, 2025
2be2174
Merge branch 'renaud-hartert_data/fs-cp-fast' of github.com-ghec:data…
renaudhartert-db Dec 15, 2025
7db079d
Merge branch 'main' into renaud-hartert_data/fs-cp-fast
renaudhartert-db Dec 16, 2025
e720670
Merge branch 'main' into renaud-hartert_data/fs-cp-fast
renaudhartert-db Dec 16, 2025
4029571
Merge branch 'main' into renaud-hartert_data/fs-cp-fast
renaudhartert-db Dec 29, 2025
6a8077b
Address feedback
renaudhartert-db Jan 8, 2026
1b0ecb2
Merge branch 'renaud-hartert_data/fs-cp-fast' of github.com-ghec:data…
renaudhartert-db Jan 8, 2026
8d0ede6
Enable cloud testing
renaudhartert-db Jan 9, 2026
4bbbe57
Fix whitespace: add trailing newlines to acceptance test files
renaudhartert-db Jan 9, 2026
1f331ff
Fix acceptance test output formatting
renaudhartert-db Jan 9, 2026
f8bf48a
Fix test server HEAD handler for volumes with file extension heuristic
renaudhartert-db Jan 9, 2026
22da2b3
Improve documentation for test server file extension heuristic
renaudhartert-db Jan 9, 2026
674fa2e
Improve test server HEAD handler with volume-specific heuristic
renaudhartert-db Jan 9, 2026
fdfc0e1
Improve test server HEAD handler for Unity Catalog Volumes
renaudhartert-db Jan 9, 2026
2dc735b
Merge remote changes, keep improved HEAD handler logic
renaudhartert-db Jan 9, 2026
ff6c248
Merge branch 'main' into renaud-hartert_data/fs-cp-fast
renaudhartert-db Jan 9, 2026
0a96e6a
Disable cloud testing for fs cp acceptance tests
renaudhartert-db Jan 9, 2026
fddf223
Simplify fs directory handlers and enable cloud tests
renaudhartert-db Jan 25, 2026
f11edd1
Merge branch 'main' into renaud-hartert_data/fs-cp-fast
renaudhartert-db Jan 25, 2026
7acce54
Register volume paths as directories in test server
renaudhartert-db Jan 25, 2026
ecf7113
Add explicit mkdir calls before fs cp in tests
renaudhartert-db Jan 25, 2026
7227b83
Fix test outputs
renaudhartert-db Jan 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions NEXT_CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@

### CLI

* Improve performance of `databricks fs cp` command by parallelizing file uploads when
copying directories with the `--recursive` flag.

### Bundles
* Pass SYSTEM_ACCESSTOKEN from env to the Terraform provider ([#4135](https://github.com/databricks/cli/pull/4135))

Expand Down
5 changes: 5 additions & 0 deletions acceptance/cmd/fs/cp/dir-to-dir/out.test.toml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions acceptance/cmd/fs/cp/dir-to-dir/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
localdir/file1.txt -> dbfs:/Volumes/main/default/data/uploaded-dir/file1.txt
localdir/file2.txt -> dbfs:/Volumes/main/default/data/uploaded-dir/file2.txt
6 changes: 6 additions & 0 deletions acceptance/cmd/fs/cp/dir-to-dir/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
mkdir -p localdir
echo -n "file1 content" > localdir/file1.txt
echo -n "file2 content" > localdir/file2.txt

# Recursive directory copy (output sorted for deterministic ordering).
$CLI fs cp -r localdir dbfs:/Volumes/main/default/data/uploaded-dir 2>&1 | sort
20 changes: 20 additions & 0 deletions acceptance/cmd/fs/cp/dir-to-dir/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
Local = true
Cloud = false
Ignore = ["localdir"]

# Recursive copy: localdir/ -> uploaded-dir/.
[[Server]]
Pattern = "PUT /api/2.0/fs/directories/Volumes/main/default/data/uploaded-dir"
Response.StatusCode = 200

[[Server]]
Pattern = "HEAD /api/2.0/fs/directories/Volumes/main/default/data/uploaded-dir"
Response.StatusCode = 200

[[Server]]
Pattern = "PUT /api/2.0/fs/files/Volumes/main/default/data/uploaded-dir/file1.txt"
Response.StatusCode = 200

[[Server]]
Pattern = "PUT /api/2.0/fs/files/Volumes/main/default/data/uploaded-dir/file2.txt"
Response.StatusCode = 200
5 changes: 5 additions & 0 deletions acceptance/cmd/fs/cp/file-to-dir/out.test.toml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions acceptance/cmd/fs/cp/file-to-dir/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@

>>> [CLI] fs cp local.txt dbfs:/Volumes/main/default/data/mydir/
local.txt -> dbfs:/Volumes/main/default/data/mydir/local.txt
4 changes: 4 additions & 0 deletions acceptance/cmd/fs/cp/file-to-dir/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
echo -n "hello world!" > local.txt

# Copy file into a directory (trailing slash indicates directory target).
trace $CLI fs cp local.txt dbfs:/Volumes/main/default/data/mydir/
12 changes: 12 additions & 0 deletions acceptance/cmd/fs/cp/file-to-dir/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
Local = true
Cloud = false
Ignore = ["local.txt"]

# Copy file into existing directory: local.txt -> mydir/local.txt.
[[Server]]
Pattern = "HEAD /api/2.0/fs/directories/Volumes/main/default/data/mydir"
Response.StatusCode = 200

[[Server]]
Pattern = "PUT /api/2.0/fs/files/Volumes/main/default/data/mydir/local.txt"
Response.StatusCode = 200
5 changes: 5 additions & 0 deletions acceptance/cmd/fs/cp/file-to-file/out.test.toml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions acceptance/cmd/fs/cp/file-to-file/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@

>>> [CLI] fs cp local.txt dbfs:/Volumes/main/default/data/uploaded.txt
local.txt -> dbfs:/Volumes/main/default/data/uploaded.txt

>>> [CLI] fs cp dbfs:/Volumes/main/default/data/remote.txt downloaded.txt
dbfs:/Volumes/main/default/data/remote.txt -> downloaded.txt
content from volume
9 changes: 9 additions & 0 deletions acceptance/cmd/fs/cp/file-to-file/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
echo -n "hello world!" > local.txt

# Upload local file to volume.
trace $CLI fs cp local.txt dbfs:/Volumes/main/default/data/uploaded.txt

# Download file from volume to local.
trace $CLI fs cp dbfs:/Volumes/main/default/data/remote.txt downloaded.txt

cat downloaded.txt
34 changes: 34 additions & 0 deletions acceptance/cmd/fs/cp/file-to-file/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
Local = true
Cloud = false
Ignore = ["local.txt", "downloaded.txt"]

# Upload: local.txt -> dbfs:/Volumes/.../uploaded.txt.
[[Server]]
Pattern = "HEAD /api/2.0/fs/directories/Volumes/main/default/data/uploaded.txt"
Response.StatusCode = 404

[[Server]]
Pattern = "HEAD /api/2.0/fs/files/Volumes/main/default/data/uploaded.txt"
Response.StatusCode = 404

[[Server]]
Pattern = "HEAD /api/2.0/fs/directories/Volumes/main/default/data"
Response.StatusCode = 200

[[Server]]
Pattern = "PUT /api/2.0/fs/files/Volumes/main/default/data/uploaded.txt"
Response.StatusCode = 200

# Download: dbfs:/Volumes/.../remote.txt -> downloaded.txt.
[[Server]]
Pattern = "HEAD /api/2.0/fs/directories/Volumes/main/default/data/remote.txt"
Response.StatusCode = 404

[[Server]]
Pattern = "HEAD /api/2.0/fs/files/Volumes/main/default/data/remote.txt"
Response.StatusCode = 200

[[Server]]
Pattern = "GET /api/2.0/fs/files/Volumes/main/default/data/remote.txt"
Response.StatusCode = 200
Response.Body = "content from volume"
5 changes: 5 additions & 0 deletions acceptance/cmd/fs/cp/input-validation/out.test.toml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 10 additions & 0 deletions acceptance/cmd/fs/cp/input-validation/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@

>>> errcode [CLI] fs cp src dst --concurrency -1
Error: --concurrency must be at least 1

Exit code: 1

>>> errcode [CLI] fs cp src dst --concurrency 0
Error: --concurrency must be at least 1

Exit code: 1
3 changes: 3 additions & 0 deletions acceptance/cmd/fs/cp/input-validation/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Invalid concurrency values should fail.
trace errcode $CLI fs cp src dst --concurrency -1
trace errcode $CLI fs cp src dst --concurrency 0
2 changes: 2 additions & 0 deletions acceptance/cmd/fs/cp/input-validation/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Local = true
Cloud = false
Loading