Skip to content

GCS bucket name parsing error: InvalidBucketName with single letter ('b') when using custom endpoint #158

@Its-Alex

Description

@Its-Alex

Summary

When using --list-remote=gcs (or any GCS operation) with a custom --gcs-endpoint, pg_back attempts to list the GCS bucket but the request fails with:

WARN: could not list remote file: googleapi: got HTTP response code 400 with body: 
<?xml version='1.0' encoding='UTF-8'?>
<Error>
  <Code>InvalidBucketName</Code>
  <Message>The specified bucket is not valid.</Message>
  <Details>Bucket names must be at least 3 characters in length, got 1: 'b'</Details>
</Error>

This suggests only the first character 'b' is being used as the bucket name instead of the full bucket name.

How to reproduce

  1. Configure pg_back with a valid GCS bucket and a custom endpoint:
    • --gcs-bucket=my-bucket
    • --gcs-endpoint=https://storage.googleapis.com (or another endpoint)
    • --gcs-keyfile=/path/to/your/keyfile.json
  2. Try to list the remote files with --list-remote=gcs
  3. Observe the above error.

Expected behavior

pg_back should use the complete bucket name as provided and interact with the correct GCS bucket.

Actual behavior

Only the single character 'b' is passed as bucket name, causing GCS to reject the request.

Code and analysis

The NewGCSRepo function in upload.go seems to gather the bucket name from opts.GCSBucket and set the endpoint using option.WithEndpoint(). However, the code may be parsing the endpoint or bucket name incorrectly, resulting in this issue.

See:

pg_back/upload.go

Lines 762 to 790 in 851663f

func NewGCSRepo(opts options) (*gcsRepo, error) {
r := &gcsRepo{
bucket: opts.GCSBucket,
url: opts.GCSEndPoint,
keyFile: opts.GCSCredentialsFile,
}
options := make([]option.ClientOption, 0)
if r.url != "" {
options = append(options, option.WithEndpoint(r.url))
}
if r.keyFile != "" {
options = append(options, option.WithCredentialsFile(r.keyFile))
}
client, err := storage.NewClient(context.Background(), options...)
if err != nil {
return nil, fmt.Errorf("could not create GCS client: %w", err)
}
r.client = client
return r, nil
}
func (r *gcsRepo) Close() error {
return r.client.Close()
}

Suggestions

  • Check that opts.GCSBucket is correctly propagated and used everywhere the bucket is referenced.
  • Make sure the endpoint does not contain or override the bucket name. The endpoint should only be the base API URL (e.g., https://storage.googleapis.com), not with the bucket name appended.
  • Add input checks and helpful error messages if a bucket name is too short or if a bad endpoint format is detected.
  • Consider documenting expected endpoint/bucket formats in README.

Environment

  • Version: (please specify)
  • Command/args: (see above)

Additional info

  • This bug affects listing, and possibly upload/download.
  • Setting the bucket directly in the command/config and omitting --gcs-endpoint works as expected.

Do you want me to try to make a PR?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions