Create a custom urllib handler for azure blob storage

[lorengordon]

• https://github.com/Azure/azure-sdk-for-python
• https://azure.github.io/azure-sdk-for-python/

Example of how we're doing it for S3. Looking for a comparable method for Azure...

• watchmaker/src/watchmaker/utils/urllib/request_handlers.py

  Lines 27 to 28 in dd75c75

  	class S3Handler(urllib.request.BaseHandler):
  	"""Define urllib handler for S3 objects."""

[Larz98]

Hey @lorengordon

"blob" not "blog" storage?

In the past, I got around the Azure DEBUG default in BlobServiceClient by resetting the logging basicConfig, which uses INFO if not defined:

def main():
  global logger

  args = parse_arguments()
  logging.basicConfig(format='%(levelname)s:%(asctime)s %(message)s')
  logger = logging.getLogger(__name__)
  logger.setLevel(level=getattr(logging, args.log_level.upper()))

[lorengordon]

@Larz98! I would gladly accept debug logging, if it meant we could retrieve files from azure similar to how we can retrieve them from s3. :)

• watchmaker/src/watchmaker/utils/urllib/request_handlers.py

  Lines 27 to 28 in dd75c75

  	class S3Handler(urllib.request.BaseHandler):
  	"""Define urllib handler for S3 objects."""

[ferricoxide]

This issue is pretty old (opened in 2018) and last activity was the 05/2020 comments: can it be closed/is it "OBE"?

[lorengordon]

Still valid

[Larz98]

Below are snippets from an old ci/cd script used (pre2020 py27) to pull from hdfs and upload to azure blob storage (can't share the full script, but pulled pieces from functions in the old script which could be useful here?):

Can look into download-blob instead of upload-blob.

from azure.storage.blob import BlobServiceClient
from azure.identity import ClientSecretCredential

azCreds = ClientSecretCredential(transfer_args.az_tenantId, transfer_args.az_clientId, transfer_args.az_clientKey)
blob_service_client = BlobServiceClient(account_url=transfer_args.az_blobUrl, credential=azCreds)
transfer_conn_az = blob_service_client.get_container_client(transfer_args.az_container)
transfer_conn_az.upload_blob(name=absolute_path, data=data, overwrite=True)

Also here is info with multiple different methods to download from Azure for whoever picks this up: https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-download-python