Skip to content

As a User, I can provide all arguments as click options for the GCS Reader, S3 Reader, GCS Writer and BQ Writer #93

@gabrielleberanger

Description

@gabrielleberanger

WHY
The following readers/writers are using implicit config.<VAR> variables:

  • gcs_reader, via the object_storage_reader
    • config.PROJECT_ID
  • s3_reader, via the object_storage_reader
    • config.REGION_NAME, config.AWS_ACCESS_KEY_ID, config.AWS_SECRET_ACCESS_KEY
  • gcs_writer
    • config.PROJECT_ID
  • bq_writer
    • config.PROJECT_ID

These variables are not provided by the user as click options, but as environment variables. These environment variables are retrieved from the following snippet that appears in the nck/config.py module:

for key, var in os.environ.items():
    locals()[key] = var

This can be quite confusing for the user, and does not match the common NCK logic.

HOW
Explicitly implement these variables as click options

Metadata

Metadata

Labels

P22nd priorityenhancementRefactoring, factorizing, adding tests, etc.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions