-
Notifications
You must be signed in to change notification settings - Fork 58
Description
Summary
When using --list-remote=gcs (or any GCS operation) with a custom --gcs-endpoint, pg_back attempts to list the GCS bucket but the request fails with:
WARN: could not list remote file: googleapi: got HTTP response code 400 with body:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>InvalidBucketName</Code>
<Message>The specified bucket is not valid.</Message>
<Details>Bucket names must be at least 3 characters in length, got 1: 'b'</Details>
</Error>
This suggests only the first character 'b' is being used as the bucket name instead of the full bucket name.
How to reproduce
- Configure pg_back with a valid GCS bucket and a custom endpoint:
--gcs-bucket=my-bucket--gcs-endpoint=https://storage.googleapis.com(or another endpoint)--gcs-keyfile=/path/to/your/keyfile.json
- Try to list the remote files with
--list-remote=gcs - Observe the above error.
Expected behavior
pg_back should use the complete bucket name as provided and interact with the correct GCS bucket.
Actual behavior
Only the single character 'b' is passed as bucket name, causing GCS to reject the request.
Code and analysis
The NewGCSRepo function in upload.go seems to gather the bucket name from opts.GCSBucket and set the endpoint using option.WithEndpoint(). However, the code may be parsing the endpoint or bucket name incorrectly, resulting in this issue.
See:
Lines 762 to 790 in 851663f
| func NewGCSRepo(opts options) (*gcsRepo, error) { | |
| r := &gcsRepo{ | |
| bucket: opts.GCSBucket, | |
| url: opts.GCSEndPoint, | |
| keyFile: opts.GCSCredentialsFile, | |
| } | |
| options := make([]option.ClientOption, 0) | |
| if r.url != "" { | |
| options = append(options, option.WithEndpoint(r.url)) | |
| } | |
| if r.keyFile != "" { | |
| options = append(options, option.WithCredentialsFile(r.keyFile)) | |
| } | |
| client, err := storage.NewClient(context.Background(), options...) | |
| if err != nil { | |
| return nil, fmt.Errorf("could not create GCS client: %w", err) | |
| } | |
| r.client = client | |
| return r, nil | |
| } | |
| func (r *gcsRepo) Close() error { | |
| return r.client.Close() | |
| } |
Suggestions
- Check that
opts.GCSBucketis correctly propagated and used everywhere the bucket is referenced. - Make sure the endpoint does not contain or override the bucket name. The endpoint should only be the base API URL (e.g.,
https://storage.googleapis.com), not with the bucket name appended. - Add input checks and helpful error messages if a bucket name is too short or if a bad endpoint format is detected.
- Consider documenting expected endpoint/bucket formats in README.
Environment
- Version: (please specify)
- Command/args: (see above)
Additional info
- This bug affects listing, and possibly upload/download.
- Setting the bucket directly in the command/config and omitting
--gcs-endpointworks as expected.
Do you want me to try to make a PR?