S3 input does not delete files

[keeshoekzema]

I am aware that I am using an S3-compatible solution and this is not supported. However, this is an easy fix.

Logstash information:

Please include the following information:

1. Logstash version: 8.13.4
2. Logstash installation source: docker.elastic.co/logstash/logstash:8.13.4

Description of the problem including expected versus actual behavior:

When running the S3 input-plugin, files are not deleted or backed up when using Ceph-RGW as S3 storage

Steps to reproduce:

input {
  s3 {
    access_key_id => "${ACCESS_KEY_ID}"
    secret_access_key => "${SECRET_ACCESS_KEY}"
    bucket => "test"
    additional_settings => {
      force_path_style => true
      follow_redirects => false
    }
    endpoint => "http://ceph-rgw"
    delete => true
  }
}

Provide logs (if relevant):

This gives the following output:

[2024-05-23T14:52:01,565][INFO ][logstash.inputs.s3       ][main][..] object-in-bucket-xyz is updated at 2024-05-23 13:58:33 UTC and will process in the next cycle

Which means the file is not deleted. In the next cycle, the timestamps still don't match so it again is not deleted.

When changing the log output to include both timestamps here the issue becomes clear:

object.last_modified: 2024-05-23 13:33:33.000000000 Z
log.last_modified: 2024-05-23 13:33:33.898000000 Z

An easy fix would be to change line 380 to:

if object.last_modified.floor == log.last_modified.floor

I have not tested this against S3, but i assume they give the timestamp in miliseconds while ceph-rgw does not. I totally understand it if you don't want to fix it, in that case i'll just leave this information here as a workaround for anyone having the same problem.

[lenchv]

are there any plans to deliver the fix?

The same issue happens with Minio, and patching the condition as proposed solves the issue

[v8v5v]

probably this is the related issue:
logstash-plugins/logstash-input-s3#236
I am using Minio as well.
I would be happy to see the fix as well..

[antonvigo]

The problem has been relevant for several YEARS, so many people would like to have some fix. But it seems that none of responsible contributors care.

[yongkyun]

Hi there — I believe the root cause of this issue is that the “LastModified” timestamp returned by the list-objects-v2 and head-object calls differs between AWS S3 and Cloudflare R2. Specifically:

• log is result of list-objects-v2 : ( @s3bucket.objects(:prefix => @prefix) in list_new_files function )
• object is result of head-object : ( object = @s3bucket.object(log.key) in process_log function )

In AWS S3:

• list-objects-v2 → LastModified includes milliseconds but milliseconds is always zero
• head-object → LastModified is only in seconds

In Cloudflare R2:

• list-objects-v2 → LastModified includes milliseconds
• head-object → LastModified is only in seconds

in function : def process_log(queue, log)

    object = @s3bucket.object(log.key)   ### <-- call head-object

    filename = File.join(temporary_directory, File.basename(log.key))
    if download_remote_file(object, filename)
      if process_local_log(queue, filename, object)
        if object.last_modified == log.last_modified   ### <-- object.last_modified and log.last_modified are rarely equal 
          backup_to_bucket(object)
          backup_to_dir(filename)
          delete_file_from_bucket(object)
          FileUtils.remove_entry_secure(filename, true)
          sincedb.write(log.last_modified)
        else
          @logger.info("#{log.key} is updated at #{object.last_modified} and will process in the next cycle")
        end
      end
    else
      FileUtils.remove_entry_secure(filename, true)
    end

S3 : list-objects-v2 result example

{
    "Contents": [
        {
            "Key": "2025/12/08/00/2025-12-08-00-32-30-498cca12-f4f0-4cf3-8fc0-54b9a2e73c7f",
            "LastModified": "2025-12-08T00:32:39.000Z",
            "ETag": "\"ba639f6ca1dfb081440e0dd38624dfb1\"",
            "Size": 5242948,
            "StorageClass": "STANDARD"
        }
    ]
}

S3 : head-object result example

{
    "AcceptRanges": "bytes",
    "Expiration": "expiry-date=\"Thu, 08 Jan 2026 00:00:00 GMT\", rule-id=\"a-schedule\"",
    "LastModified": "Mon, 08 Dec 2025 00:32:39 GMT",
    "ContentLength": 5242948,
    "ETag": "\"ba639f6ca1dfb081440e0dd38624dfb1\"",
    "ContentType": "application/octet-stream",
    "ServerSideEncryption": "AES256",
    "Metadata": {}
}

R2 : list-objects-v2 result example

{
    "Contents": [
        {
            "Key": "20251204/20251204T044142Z_20251204T044142Z_63860e1d.log.gz",
            "LastModified": "2025-12-04T04:42:50.902Z",
            "ETag": "\"811bfedf3a5d3471813953e99e7e6b66\"",
            "Size": 823,
            "StorageClass": "STANDARD"
        }
    ]
}

R2 : head-object result example

{
    "LastModified": "Thu, 04 Dec 2025 04:42:50 GMT",
    "ETag": "W/\"811bfedf3a5d3471813953e99e7e6b66\"",
    "ContentType": "application/json",
    "Metadata": {}
}