Empty (zero byte) S3 file causes AMI to stop

[davidbasswwu]

I was running an AMI ingest of 10k+ records (via AJAX/browser, not queued) and when it encountered an empty file in our Minio instance (S3), the AMI ingest stopped.

User warning: Error executing "GetObject" on "http://esmero-minio:9000/path/to/file/450.html"; AWS HTTP error: Client error: GET http://esmero-minio:9000/path/to/file/450.html resulted in a 416 Requested Range Not Satisfiable response (client): 416 Requested Range Not Satisfiable (Request-ID: 188936CEBE199F2F) - InvalidRangepath/to/file/450.htmlbucket-name-here/path/to/file/450.htmlrequest-id-herehost-id-here in Aws\S3\StreamWrapper->triggerError() (line 764 of /var/www/html/vendor/aws/aws-sdk-php/src/S3/StreamWrapper.php). =>

It turns out that file (450.html) was empty, which seems to have caused the AMI set to fail. In an ideal world, it would just flag that event and keep going.
Thanks!

[DiegoPino]

Hi @davidbasswwu. Thanks for opening this issue. As we discussed via email, this exception is associated to mismatched S3 metadata v/s the actual Object (a.k.a) file in an S3 compatible bucket. The metadata says it has X bytes, the file is actually empty. That is very hard to replicate on normal operations since e.g AWS bucket will effectively do a checksum check before accepting the file as being there but other implementations of the S3 API might allow a failed S3 Object to exists.

The complexity with this is that the failure is not an PHP exception (which we intercept and deal with, e.g timeouts, missing files, local storage full, etc), but a USER ERROR notice, which sadly permeates on the AJAX interface and breaks Drupal.

Only possible solution is to create a custom error handler that converts the USER NOTICE into an exception, throws it so we can log it/continue and then restores it. That said, only way I can really test that is by "simulating" (a test that throws the exception on purpose) bc in all my local attempts via min.io or AWS I could not manage to generate that S3 Object metadata != Real S3 Object.

I will mark this as a bug but a fix might take a few weeks. Thanks!

Note: is your backend really min.io or is min.io routing a vendor specific S3? (e.g Azure/MS or custom?)