Buffer Overflow produce queue increases exponential.

[jvs87]

Problem

We are facing with a problem of buffer overflow, that produce queue increase and cant liberate chunks correctlly so at the end we lose data due to police of drop_oldest_chunks.

Here you can see how buffer increases and also the queue,

...

Steps to replicate

We have cofigurated 7 workers and here is the Elasticsearch configuration

  <match NUEVOES>
    @type elasticsearch
    @id elasticsearch
    validate_client_version false
#    hosts "tdo-pro-elkmaster1.cloudready.cloud.si.orange.es,tdo-pro-elkmaster2.cloudready.cloud.si.orange.es,tdo-pro-elkmaster3.cloudready.cloud.si.orange.es,tdo-pro-elkdatahot1.cloudready.cloud.si.orange.es,tdo-pro-elkdatahot2.cloudready.cloud.si.orange.es,tdo-pro-elkdatahot3.cloudready.cloud.si.orange.es,tdo-pro-elkdatahot4.cloudready.cloud.si.orange.es,tdo-pro-elkdatahot5.cloudready.cloud.si.orange.es,tdo-pro-elkdatahot6.cloudready.cloud.si.orange.es,tdo-pro-elkdatahot7.cloudready.cloud.si.orange.es,tdo-pro-elkdatawarm1.cloudready.cloud.si.orange.es"
    host "kibana.cloudready.cloud.si.orange.es"
    port 9200
    user 
    password 
    scheme https
    ssl_verify false
    ssl_version TLSv1_2
    logstash_format false
    index_name logs.${$.index_name}.%Y-%m-%d
    type_name _doc
    include_timestamp true
    reload_on_failure false
    reload_connections false
    log_es_400_reason true
    id_key _hash
    remove_keys _hash
    <buffer time, tag, $.index_name>
      @type memory
      timekey 60
      timekey_wait 5
      chunk_limit_size 100m
      queue_limit_length 1000
      flush_mode interval
      flush_interval 1s
      flush_at_shutdown true
      flush_thread_count 2
      overflow_action drop_oldest_chunk
    </buffer>
  </match>

Expected Behavior or What you need to ask

Is there any bug on our plugin version that produces that overflow or simply whe have to configurated it correctly?

Using Fluentd and ES plugin versions

• OS version
• EC2 instance
• Fluentd v0.12 or v0.14/v1.0
  • fluentd 1.10.4
• ES plugin 3.x.y/2.x.y or 1.x.y
  • elasticsearch (7.4.0)
  • elasticsearch-api (7.4.0)
  • elasticsearch-transport (7.4.0)
  • fluent-plugin-elasticsearch (4.0.8)
• ES version 8.11.1
• ES template(s) (optional)