S3 device using Backblaze with ObjectLock: Error "Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters"

[moritzboth]

We are testing the s3 device with Backblaze B2 service. Backblaze uses the "AWS4" authentication mechanism.

Backblaze supports a "ObjectLock" function which locks objects for a preset amount of days. When we switch on ObjectLock, the creation of any object with amanda results in this error message in the taper log file:

Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters

Content-MD5 is, as far as I found out, optional with AWS4, but can be applied to any request. Maybe amanda should always send it. As it turns out, the code for that is already in the amanda s3 driver, but the creation of the Content-MD5 HTTP header is explicitly turned of for AWS4:

amanda/device-src/s3.c

Line 1171 in a7c8c3e

md5_hash = NULL;

We removed the line setting md5_hash to NULL. Also, we enabled the calculation of md5 sum for AWS4. Now we can upload data to Backblaze B2 with ObjectLock in place.

I don't know how / if this affects other services such as Amazon S3, but it is officially documented.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html

By the way, to get the s3 driver to work at all, we needed to apply the patch from pull request #244 .

Edit: Corrections, the description of our patch was incomplete before.

[moritzboth]

Here is the patch that fixed the issue.

--- amanda-3.5.1.orig/device-src/s3.c
+++ amanda-3.5.1/device-src/s3.c
@@ -1158,6 +1158,7 @@ authenticate_request(S3Handle *hdl,
            g_debug("canonical_hash: %s", canonical_hash);
            g_debug("strSecretKey: %s", strSecretKey);
            g_debug("signatureHex: %s", signatureHex);
+           g_debug("md5_hash: %s", md5_hash);
        }
 
        g_free(canonical_hash);
@@ -1168,7 +1169,6 @@ authenticate_request(S3Handle *hdl,
        g_free(signingKey);
        g_free(signature);
        g_free(signatureHex);
-       md5_hash = NULL;
 
     } else { /* hdl->s3_api == S3_API_S3 */
        /* Build the string to sign, per the S3 spec.
@@ -2496,8 +2496,10 @@ perform_request(S3Handle *hdl,
        } else {
            data_SHA256Hash = s3_compute_sha256_hash((unsigned char *)"", 0);
        }
-    } else if (md5_func) {
+    }
+    if (md5_func) {
         md5_hash = md5_func(read_data);
+       g_debug("md5_hash: %p", md5_hash);
         if (md5_hash) {
             md5_hash_b64 = s3_base64_encode(md5_hash);
             md5_hash_hex = s3_hex_encode(md5_hash);