Skip to content

Make checksum calculation configurable #66

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rkoster merged 1 commit into from
Feb 19, 2026
Merged

Make checksum calculation configurable #66

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 42 additions & 24 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ blobstore.

Continuous integration: <https://bosh-cpi.ci.cf-app.com/pipelines/s3cli>

Releases can be found in `https://s3.amazonaws.com/bosh-s3cli-artifacts`. The Linux binaries follow the regex `s3cli-(\d+\.\d+\.\d+)-linux-amd64` and the windows binaries `s3cli-(\d+\.\d+\.\d+)-windows-amd64`.
Releases can be found in `https://s3.amazonaws.com/bosh-s3cli-artifacts`. The Linux binaries follow the regex
`s3cli-(\d+\.\d+\.\d+)-linux-amd64` and the windows binaries `s3cli-(\d+\.\d+\.\d+)-windows-amd64`.

## Installation

Expand All @@ -19,26 +20,30 @@ Given a JSON config file (`config.json`)...

``` json
{
"bucket_name": "<string> (required)",

"credentials_source": "<string> [static|env_or_profile|none]",
"access_key_id": "<string> (required if credentials_source = 'static')",
"secret_access_key": "<string> (required if credentials_source = 'static')",

"region": "<string> (optional - default: 'us-east-1')",
"host": "<string> (optional)",
"port": "<int> (optional)",

"ssl_verify_peer": "<bool> (optional - default: true)",
"use_ssl": "<bool> (optional - default: true)",
"signature_version": "<string> (optional)",
"server_side_encryption": "<string> (optional)",
"sse_kms_key_id": "<string> (optional)",
"multipart_upload": "<bool> (optional - default: true)",
"download_concurrency": <int> (optional - default: 5),
"download_part_size": <int64> (optional - default: 5242880), # 5 MB
"upload_concurrency": <int> (optional - default: 5),
"upload_part_size": <int64> (optional - default: 5242880) # 5 MB
"bucket_name": "<string> (required)",

"credentials_source": "<string> [static|env_or_profile|none]",
"access_key_id": "<string> (required if credentials_source = 'static')",
"secret_access_key": "<string> (required if credentials_source = 'static')",

"region": "<string> (optional - default: 'us-east-1')",
"host": "<string> (optional)",
"port": "<int> (optional)",

"ssl_verify_peer": "<bool> (optional - default: true)",
"use_ssl": "<bool> (optional - default: true)",
"signature_version": "<string> (optional)",
"server_side_encryption": "<string> (optional)",
"sse_kms_key_id": "<string> (optional)",
"multipart_upload": "<bool> (optional - default: true)",
"request_checksum_calculation_enabled": "<bool> (optional - default: true)",
"response_checksum_calculation_enabled": "<bool> (optional - default: true)",
"uploader_request_checksum_calculation_enabled": "<bool> (optional - default: true)"

"download_concurrency": "<int> (optional - default: 5)",
"download_part_size": "<int64> (optional - default: 5242880) # 5 MB",
"upload_concurrency": "<int> (optional - default: 5)",
"upload_part_size": "<int64> (optional - default: 5242880) # 5 MB"
}
```

Expand Down Expand Up @@ -97,8 +102,11 @@ Follow these steps to make a contribution to the project:
- Create a GitHub pull request, selecting `main` as the target branch

## Running integration tests

### Steps to run the integration tests on AWS

1. Export the following variables into your environment

```
export access_key_id=<YOUR_AWS_ACCESS_KEY>
export focus_regex="GENERAL AWS|AWS V2 REGION|AWS V4 REGION|AWS US-EAST-1"
Expand All @@ -108,37 +116,47 @@ export secret_access_key=<YOUR_SECRET_ACCESS_KEY>
export stack_name=s3cli-iam
export bucket_name=s3cli-pipeline
```

2. Setup infrastructure with `ci/tasks/setup-aws-infrastructure.sh`
3. Run the desired tests by executing one or more of the scripts `run-integration-*` in `ci/tasks` (to run `run-integration-s3-compat` see [Setup for GCP](#setup-for-GCP) or [Setup for AliCloud](#setup-for-alicloud))
3. Run the desired tests by executing one or more of the scripts `run-integration-*` in `ci/tasks` (to run
`run-integration-s3-compat` see [Setup for GCP](#setup-for-GCP) or [Setup for AliCloud](#setup-for-alicloud))
4. Teardown infrastructure with `ci/tasks/teardown-infrastructure.sh`

### Setup for GCP

1. Create a bucket in GCP
2. Create access keys
3. Navigate to **IAM & Admin > Service Accounts**.
4. Select your service account or create a new one if needed.
5. Ensure your service account has necessary permissions (like `Storage Object Creator`, `Storage Object Viewer`, `Storage Admin`) depending on what access you want.
5. Ensure your service account has necessary permissions (like `Storage Object Creator`, `Storage Object Viewer`,
`Storage Admin`) depending on what access you want.
6. Go to **Cloud Storage** and select **Settings**.
7. In the **Interoperability** section, create an HMAC key for your service account. This generates an "access key ID" and a "secret access key".
7. In the **Interoperability** section, create an HMAC key for your service account. This generates an "access key ID"
and a "secret access key".
8. Export the following variables into your environment:

```
export access_key_id=<YOUR_ACCESS_KEY>
export secret_access_key=<YOUR_SECRET_ACCESS_KEY>
export bucket_name=<YOUR_BUCKET_NAME>
export s3_endpoint_host=storage.googleapis.com
export s3_endpoint_port=443
```

4. Run `run-integration-s3-compat.sh` in `ci/tasks`

### Setup for AliCloud

1. Create bucket in AliCloud
2. Create access keys from `RAM -> User -> Create Accesskey`
3. Export the following variables into your environment:

```
export access_key_id=<YOUR_ACCESS_KEY>
export secret_access_key=<YOUR_SECRET_ACCESS_KEY>
export bucket_name=<YOUR_BUCKET_NAME>
export s3_endpoint_host="oss-<YOUR_REGION>.aliyuncs.com"
export s3_endpoint_port=443
```

4. Run `run-integration-s3-compat.sh` in `ci/tasks`
6 changes: 3 additions & 3 deletions config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ type S3Cli struct {
HostStyle bool `json:"host_style"`
SwiftAuthAccount string `json:"swift_auth_account"`
SwiftTempURLKey string `json:"swift_temp_url_key"`
RequestChecksumCalculationEnabled bool
ResponseChecksumCalculationEnabled bool
UploaderRequestChecksumCalculationEnabled bool
RequestChecksumCalculationEnabled bool `json:"request_checksum_calculation_enabled"`
ResponseChecksumCalculationEnabled bool `json:"response_checksum_calculation_enabled"`
UploaderRequestChecksumCalculationEnabled bool `json:"uploader_request_checksum_calculation_enabled"`
// Optional knobs to tune transfer performance.
// If zero, the client will apply sensible defaults (handled by the S3 client layer).
// Part size values are provided in bytes.
Expand Down
Loading