Elasticsearch error : Content-Type version must be either version 8 or 7, but found 9

[loic-brtd]

(check apply)

• read the contribution guideline
• (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Problem

Hi, I'm using the Logging Operator Chart 6.0.3 and Fluentd receives this error when it sends logs to Elasticsearch :

2025-09-26 12:55:02 +0000 [warn]: #0 [flow:xxx:logs-microservices.flow:output:xxx:logs-microservices.output] failed to flush the buffer. retry_times=0 next_retry_time=2025-09-26 12:55:04 +0000 chunk="63fb3caff2f6d6097b699b0e499d5020" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster (ror-structured-logs-microservices-dev): [400] {\"error\":{\"root_cause\":[{\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Content-Type, Accept]\"}],\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Content-Type, Accept]\",\"caused_by\":{\"type\":\"status_exception\",\"reason\":\"Content-Type version must be either version 8 or 7, but found 9. Content-Type=application/vnd.elasticsearch+x-ndjson; compatible-with=9\"}},\"status\":400}"

with a bit of formatting:

{
  "error": {
    "root_cause": [
      {
        "type": "media_type_header_exception",
        "reason": "Invalid media-type value on headers [Content-Type, Accept]"
      }
    ],
    "type": "media_type_header_exception",
    "reason": "Invalid media-type value on headers [Content-Type, Accept]",
    "caused_by": {
      "type": "status_exception",
      "reason": "Content-Type version must be either version 8 or 7, but found 9. Content-Type=application/vnd.elasticsearch+x-ndjson; compatible-with=9"
    }
  },
  "status": 400
}

Our Elasticsearch version is 8.15.2, and in Fluent's logs I see "Detected ES 8.x", so I don't understand why this header is sent with version 9.

I tried with and without the following configuration, but i get the exact same error :

custom_headers {"Accept":"application/vnd.elasticsearch+json; compatible-with=8","Content-Type":"application/vnd.elasticsearch+x-ndjson; compatible-with=8"}

Steps to replicate

Fluentd logs (trace level), including fluentd config

2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-aliyun-oss' version '0.0.1'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-cloudwatch-logs' version '0.15.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-concat' version '2.5.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-datadog' version '0.14.4'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-detect-exceptions' version '0.0.16'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-elasticsearch' version '6.0.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-gcs' version '0.4.4'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-gelf-best' version '1.4.1'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-geoip' version '1.3.2'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-grok-parser' version '2.6.2'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-kafka' version '0.19.4'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-kinesis' version '3.5.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-label-router' version '0.5.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-lm-logs' version '1.2.5'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-logzio' version '0.2.2'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-mattermost' version '0.2.2'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-multi-format-parser' version '1.1.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-mysqlslowquery' version '0.0.9'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-newrelic' version '1.2.3'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-opensearch' version '1.1.5'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-prometheus' version '2.2.1'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-record-modifier' version '2.2.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-redis' version '0.3.5'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-remote-syslog' version '1.1'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-s3' version '1.8.3'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-splunk-hec' version '1.3.3'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-sqs' version '3.0.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-tag-normaliser' version '0.1.3'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-throttle' version '0.0.5'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-ua-parser' version '1.2.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-vmware-log-intelligence' version '2.0.8'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-vmware-loginsight' version '1.4.2'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluent-plugin-webhdfs' version '1.6.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluentd' version '1.18.0'
2025-09-26 13:55:02 +0000 [info]: fluent/log.rb:362:: gem 'fluentd' version '1.17.1'
2025-09-26 13:55:02 +0000 [trace]: fluent/log.rb:319:: registered filter plugin 'grep'
2025-09-26 13:55:02 +0000 [trace]: fluent/log.rb:319:: registered metrics plugin 'local'
2025-09-26 13:55:02 +0000 [trace]: fluent/log.rb:319:: registered filter plugin 'parser'
2025-09-26 13:55:02 +0000 [trace]: fluent/log.rb:319:: registered parser plugin 'json'
2025-09-26 13:55:03 +0000 [trace]: fluent/log.rb:319:: registered output plugin 'elasticsearch'
2025-09-26 13:55:03 +0000 [trace]: fluent/log.rb:319:: registered output plugin 'elasticsearch_data_stream'
2025-09-26 13:55:03 +0000 [trace]: fluent/log.rb:319:: registered buffer plugin 'file'
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'host x.x.x.x' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: x.x.x.x' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'host x.x.x.x' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: x.x.x.x' doesn't have tag placeholder
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have timestamp placeholder for day('%d') for timekey 600
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] Need substitution: false
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'host_placeholder x.x.x.x' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: x.x.x.x' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'host_placeholder x.x.x.x' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: x.x.x.x' doesn't have tag placeholder
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'data_stream_name_placeholder ror-structured-logs-microservices-dev' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: ror-structured-logs-microservices-dev' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:03 +0000 [debug]: [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'data_stream_name_placeholder ror-structured-logs-microservices-dev' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: ror-structured-logs-microservices-dev' doesn't have tag placeholder
2025-09-26 13:55:03 +0000 [trace]: fluent/log.rb:319:: registered output plugin 'null'
2025-09-26 13:55:03 +0000 [trace]: fluent/log.rb:319:: registered buffer plugin 'memory'
2025-09-26 13:55:03 +0000 [trace]: fluent/log.rb:319:: registered output plugin 'label_router'
2025-09-26 13:55:03 +0000 [trace]: fluent/log.rb:319:: registered input plugin 'forward'
2025-09-26 13:55:03 +0000 [info]: fluent/log.rb:362:: using configuration file: <ROOT>
  <system>
    rpc_endpoint "x.x.x.x:24444"
    log_level trace
    workers 1
    root_dir "/buffers"
  </system>
  <source>
    @type forward
    @id main_forward
    bind "x.x.x.x"
    port 24240
  </source>
  <match **>
    @type label_router
    @id main
    metrics false
    <route>
      @label "@2b0669f6af9ddff877bae45460fe9d59"
      metrics_labels {"id":"flow:xxxx:logs-microservices.flow"}
      <match>
        container_names foo,bar,baz
        namespaces xxxx
        negate false
      </match>
    </route>
  </match>
  <label @2b0669f6af9ddff877bae45460fe9d59>
    <filter **>
      @type grep
      @id flow:xxxx:logs-microservices.flow:0
      <regexp>
        key "log"
        pattern /^\{.*\}/
      </regexp>
    </filter>
    <filter **>
      @type parser
      @id flow:xxxx:logs-microservices.flow:1
      key_name "log"
      remove_key_name_field true
      reserve_data true
      <parse>
        @type "json"
      </parse>
    </filter>
    <match **>
      @type elasticsearch_data_stream
      @id flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output
      custom_headers {"Accept":"application/vnd.elasticsearch+json; compatible-with=8","Content-Type":"application/vnd.elasticsearch+x-ndjson; compatible-with=8"}
      data_stream_name "ror-structured-logs-microservices-dev"
      data_stream_template_name "ror-structured-logs-template-dev"
      data_stream_template_use_index_patterns_wildcard true
      exception_backup true
      fail_on_detecting_es_version_retry_exceed true
      fail_on_putting_template_retry_exceed true
      host "x.x.x.x"
      port 9200
      reload_connections true
      scheme http
      ssl_verify true
      utc_index true
      verify_es_version_at_startup true
      <buffer tag,time>
        @type "file"
        chunk_limit_size 16MB
        flush_at_shutdown true
        flush_interval 5s
        flush_mode interval
        flush_thread_count 8
        overflow_action drop_oldest_chunk
        path "/buffers/flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output.*.buffer"
        retry_forever true
        retry_max_interval 30s
        retry_timeout 1h
        retry_type exponential_backoff
        timekey 10m
        timekey_wait 1m
      </buffer>
    </match>
  </label>
  <label @ERROR>
    <match **>
      @type null
      @id main-fluentd-error
    </match>
  </label>
  <match **>
    @type null
    @id main-no-output
  </match>
  <label @FLUENT_LOG>
    <match fluent.*>
      @type null
      @id main-fluentd-log
    </match>
  </label>
</ROOT>
2025-09-26 13:55:03 +0000 [info]: fluent/log.rb:362:: starting fluentd-1.18.0 pid=7 ruby="3.4.4"
2025-09-26 13:55:03 +0000 [info]: fluent/log.rb:362:: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/local/bundle/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2025-09-26 13:55:03 +0000 [debug]: fluent/log.rb:341:: register /api/processes.interruptWorkers RPC handler
2025-09-26 13:55:03 +0000 [debug]: fluent/log.rb:341:: register /api/processes.killWorkers RPC handler
2025-09-26 13:55:03 +0000 [debug]: fluent/log.rb:341:: register /api/processes.flushBuffersAndKillWorkers RPC handler
2025-09-26 13:55:03 +0000 [debug]: fluent/log.rb:341:: register /api/processes.zeroDowntimeRestart RPC handler
2025-09-26 13:55:03 +0000 [debug]: fluent/log.rb:341:: register /api/plugins.flushBuffers RPC handler
2025-09-26 13:55:03 +0000 [debug]: fluent/log.rb:341:: register /api/config.reload RPC handler
2025-09-26 13:55:03 +0000 [debug]: fluent/log.rb:341:: register /api/config.dump RPC handler
2025-09-26 13:55:03 +0000 [debug]: fluent/log.rb:341:: register /api/config.gracefulReload RPC handler
2025-09-26 13:55:03 +0000 [debug]: fluent/log.rb:341:: listening RPC http server on http://x.x.x.x:24444/
2025-09-26 13:55:03 +0000 [info]: #0 fluent/log.rb:362:: init worker0 logger path=nil rotate_age=nil rotate_size=nil
2025-09-26 13:55:03 +0000 [info]: fluent/log.rb:362:: adding filter in @2b0669f6af9ddff877bae45460fe9d59 pattern="**" type="grep"
2025-09-26 13:55:03 +0000 [trace]: #0 fluent/log.rb:319:: registered filter plugin 'grep'
2025-09-26 13:55:03 +0000 [trace]: #0 fluent/log.rb:319:: registered metrics plugin 'local'
2025-09-26 13:55:03 +0000 [info]: fluent/log.rb:362:: adding filter in @2b0669f6af9ddff877bae45460fe9d59 pattern="**" type="parser"
2025-09-26 13:55:03 +0000 [trace]: #0 fluent/log.rb:319:: registered filter plugin 'parser'
2025-09-26 13:55:03 +0000 [trace]: #0 fluent/log.rb:319:: registered parser plugin 'json'
2025-09-26 13:55:03 +0000 [info]: fluent/log.rb:362:: adding match in @2b0669f6af9ddff877bae45460fe9d59 pattern="**" type="elasticsearch_data_stream"
2025-09-26 13:55:04 +0000 [trace]: #0 fluent/log.rb:319:: registered output plugin 'elasticsearch'
2025-09-26 13:55:04 +0000 [trace]: #0 fluent/log.rb:319:: registered output plugin 'elasticsearch_data_stream'
2025-09-26 13:55:04 +0000 [trace]: #0 fluent/log.rb:319:: registered buffer plugin 'file'
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'host x.x.x.x' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: x.x.x.x' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'host x.x.x.x' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host: x.x.x.x' doesn't have tag placeholder
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'index_name fluentd' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'index_name: fluentd' doesn't have tag placeholder
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'template_name ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'template_name: ' doesn't have tag placeholder
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'logstash_prefix logstash' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_prefix: logstash' doesn't have tag placeholder
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have timestamp placeholder for day('%d') for timekey 600
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'logstash_dateformat %Y.%m.%d' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'logstash_dateformat: %Y.%m.%d' doesn't have tag placeholder
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'deflector_alias ' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'deflector_alias: ' doesn't have tag placeholder
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'application_name default' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'application_name: default' doesn't have tag placeholder
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'ilm_policy_id logstash-policy' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'ilm_policy_id: logstash-policy' doesn't have tag placeholder
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] Need substitution: false
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'host_placeholder x.x.x.x' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: x.x.x.x' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'host_placeholder x.x.x.x' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'host_placeholder: x.x.x.x' doesn't have tag placeholder
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] Detected ES 8.x: This parameter has no effect.
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'data_stream_name_placeholder ror-structured-logs-microservices-dev' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: ror-structured-logs-microservices-dev' doesn't have timestamp placeholders for timekey 600
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] 'data_stream_name_placeholder ror-structured-logs-microservices-dev' is tested built-in placeholder(s) but there is no valid placeholder(s). error: Parameter 'data_stream_name_placeholder: ror-structured-logs-microservices-dev' doesn't have tag placeholder
2025-09-26 13:55:04 +0000 [info]: fluent/log.rb:362:: adding match in @FLUENT_LOG pattern="fluent.*" type="null"
2025-09-26 13:55:04 +0000 [trace]: #0 fluent/log.rb:319:: registered output plugin 'null'
2025-09-26 13:55:04 +0000 [trace]: #0 fluent/log.rb:319:: registered buffer plugin 'memory'
2025-09-26 13:55:04 +0000 [info]: fluent/log.rb:362:: adding match in @ERROR pattern="**" type="null"
2025-09-26 13:55:04 +0000 [info]: fluent/log.rb:362:: adding match pattern="**" type="label_router"
2025-09-26 13:55:04 +0000 [trace]: #0 fluent/log.rb:319:: registered output plugin 'label_router'
2025-09-26 13:55:04 +0000 [info]: fluent/log.rb:362:: adding match pattern="**" type="null"
2025-09-26 13:55:04 +0000 [info]: fluent/log.rb:362:: adding source type="forward"
2025-09-26 13:55:04 +0000 [trace]: #0 fluent/log.rb:319:: registered input plugin 'forward'
2025-09-26 13:55:04 +0000 [info]: #0 fluent/log.rb:362:: starting fluentd worker pid=28 ppid=7 worker=0
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] buffer started instance=904 stage_size=0 queue_size=0
2025-09-26 13:55:04 +0000 [info]: #0 [main_forward] listening port port=24240 bind="x.x.x.x"
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] flush_thread actually running
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] flush_thread actually running
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] flush_thread actually running
2025-09-26 13:55:04 +0000 [info]: #0 fluent/log.rb:362:: fluentd worker is now running worker=0
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] flush_thread actually running
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] flush_thread actually running
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] flush_thread actually running
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] flush_thread actually running
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueue_thread actually running
2025-09-26 13:55:04 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:04 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:04 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] flush_thread actually running
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=38754
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=38754
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=38766
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=38766
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=56462
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=56462
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=47074
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=47074
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=47076
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=47076
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=47088
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=47088
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=47102
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=47102
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=47114
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=47114
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=47130
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=47130
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=47134
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=47134
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=47138
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=47138
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=47148
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=47148
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=47154
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=47154
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=53162
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=53162
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=53166
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=53166
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=53178
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=53178
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=53190
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=53190
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=53194
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=53194
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=53204
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=53204
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=53218
2025-09-26 13:55:04 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=53218
2025-09-26 13:55:05 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:05 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:05 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=56476
2025-09-26 13:55:05 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=56476
2025-09-26 13:55:05 +0000 [trace]: #0 [main_forward] connected fluent socket addr="x.x.x.x" port=56484
2025-09-26 13:55:05 +0000 [trace]: #0 [main_forward] accepted fluent socket addr="x.x.x.x" port=56484
2025-09-26 13:55:05 +0000 [info]: #0 fluent/log.rb:362:: disable filter chain optimization because [Fluent::Plugin::ParserFilter] uses `#filter_stream` method.
2025-09-26 13:55:05 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] writing events into buffer instance=904 metadata_size=1
2025-09-26 13:55:05 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] Created new chunk chunk_id="63fb4a20ce94e3ed4d025e8d143b2118" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=1758894600, tag="kubernetes.var.log.containers.some-container-6657c94685-7zdhd_xxxx_some-container-bf0183abc950f4dd3e1a87f7a604a1f8162c292c09d2f8c069e530b898d6e885.log", variables=nil, seq=0>
2025-09-26 13:55:05 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] chunk /buffers/flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output.b63fb4a20ce94e3ed4d025e8d143b2118.buffer size_added: 2280 new_size: 2280
2025-09-26 13:55:06 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:06 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:06 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] writing events into buffer instance=904 metadata_size=1
2025-09-26 13:55:06 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] chunk /buffers/flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output.b63fb4a20ce94e3ed4d025e8d143b2118.buffer size_added: 2280 new_size: 4560
2025-09-26 13:55:07 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:07 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:08 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:08 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:08 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] writing events into buffer instance=904 metadata_size=1
2025-09-26 13:55:08 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] chunk /buffers/flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output.b63fb4a20ce94e3ed4d025e8d143b2118.buffer size_added: 2280 new_size: 6840
2025-09-26 13:55:08 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] writing events into buffer instance=904 metadata_size=1
2025-09-26 13:55:08 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] chunk /buffers/flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output.b63fb4a20ce94e3ed4d025e8d143b2118.buffer size_added: 2277 new_size: 9117
2025-09-26 13:55:08 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] writing events into buffer instance=904 metadata_size=1
2025-09-26 13:55:08 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] chunk /buffers/flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output.b63fb4a20ce94e3ed4d025e8d143b2118.buffer size_added: 2280 new_size: 11397
2025-09-26 13:55:09 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:09 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:09 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] writing events into buffer instance=904 metadata_size=1
2025-09-26 13:55:09 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] chunk /buffers/flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output.b63fb4a20ce94e3ed4d025e8d143b2118.buffer size_added: 2278 new_size: 13675
2025-09-26 13:55:10 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:10 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing chunk instance=904 metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=1758894600, tag="kubernetes.var.log.containers.some-container-6657c94685-7zdhd_xxxx_some-container-bf0183abc950f4dd3e1a87f7a604a1f8162c292c09d2f8c069e530b898d6e885.log", variables=nil, seq=0>
2025-09-26 13:55:10 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:11 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] dequeueing a chunk instance=904
2025-09-26 13:55:11 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] chunk dequeued instance=904 metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=1758894600, tag="kubernetes.var.log.containers.some-container-6657c94685-7zdhd_xxxx_some-container-bf0183abc950f4dd3e1a87f7a604a1f8162c292c09d2f8c069e530b898d6e885.log", variables=nil, seq=0>
2025-09-26 13:55:11 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] trying flush for a chunk chunk="63fb4a20ce94e3ed4d025e8d143b2118"
2025-09-26 13:55:11 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] adding write count instance=1248
2025-09-26 13:55:11 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] executing sync write chunk="63fb4a20ce94e3ed4d025e8d143b2118"
2025-09-26 13:55:11 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:11 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] enqueueing all chunks in buffer instance=904
2025-09-26 13:55:11 +0000 [debug]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] taking back chunk for errors. chunk="63fb4a20ce94e3ed4d025e8d143b2118"
2025-09-26 13:55:11 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] taking back a chunk instance=904 chunk_id="63fb4a20ce94e3ed4d025e8d143b2118"
2025-09-26 13:55:11 +0000 [trace]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] chunk taken back instance=904 chunk_id="63fb4a20ce94e3ed4d025e8d143b2118" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=1758894600, tag="kubernetes.var.log.containers.some-container-6657c94685-7zdhd_xxxx_some-container-bf0183abc950f4dd3e1a87f7a604a1f8162c292c09d2f8c069e530b898d6e885.log", variables=nil, seq=0>
2025-09-26 13:55:11 +0000 [warn]: #0 [flow:xxxx:logs-microservices.flow:output:xxxx:logs-microservices.output] failed to flush the buffer. retry_times=0 next_retry_time=2025-09-26 13:55:12 +0000 chunk="63fb4a20ce94e3ed4d025e8d143b2118" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster (ror-structured-logs-microservices-dev): [400] {\"error\":{\"root_cause\":[{\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Content-Type, Accept]\"}],\"type\":\"media_type_header_exception\",\"reason\":\"Invalid media-type value on headers [Content-Type, Accept]\",\"caused_by\":{\"type\":\"status_exception\",\"reason\":\"Content-Type version must be either version 8 or 7, but found 9. Content-Type=application/vnd.elasticsearch+x-ndjson; compatible-with=9\"}},\"status\":400}"
  2025-09-26 13:55:11 +0000 [warn]: #0 /usr/local/bundle/gems/fluent-plugin-elasticsearch-6.0.0/lib/fluent/plugin/out_elasticsearch_data_stream.rb:293:in 'Fluent::Plugin::ElasticsearchOutputDataStream#write'
  2025-09-26 13:55:11 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/output.rb:1225:in 'Fluent::Plugin::Output#try_flush'
  2025-09-26 13:55:11 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/output.rb:1540:in 'Fluent::Plugin::Output#flush_thread_run'
  2025-09-26 13:55:11 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin/output.rb:510:in 'block (2 levels) in Fluent::Plugin::Output#start'
  2025-09-26 13:55:11 +0000 [warn]: #0 /usr/local/bundle/gems/fluentd-1.18.0/lib/fluent/plugin_helper/thread.rb:78:in 'block in Fluent::PluginHelper::Thread#thread_create'

Using Fluentd and ES plugin versions

• OS version
• Bare Metal or within Docker or Kubernetes or others? Kubernetes + ArgoCD
• Fluentd v0.12 or v0.14/v1.0

$ fluentd --version
fluentd 1.18.0

• ES plugin 3.x.y/2.x.y or 1.x.y

$ fluent-gem list

*** LOCAL GEMS ***

abbrev (0.1.2)
activemodel (7.1.1)
activesupport (7.1.1)
addressable (2.8.7)
aes_key_wrap (1.1.0)
aliyun-sdk (0.7.0)
attr_required (1.0.1)
aws-eventstream (1.3.2)
aws-partitions (1.1105.0)
aws-sdk-cloudwatchlogs (1.116.0)
aws-sdk-core (3.224.0)
aws-sdk-firehose (1.65.0)
aws-sdk-kinesis (1.55.0)
aws-sdk-kms (1.95.0)
aws-sdk-s3 (1.169.0)
aws-sdk-sqs (1.64.0)
aws-sigv4 (1.11.0)
base64 (0.2.0)
benchmark (default: 0.4.0)
bigdecimal (3.1.8, 3.1.6)
bindata (2.5.0)
bundler (2.6.9, default: 2.6.7)
cgi (default: 0.4.2)
concurrent-ruby (1.3.5, 1.2.3)
connection_pool (2.4.1)
cool.io (1.9.0, 1.8.1)
csv (3.3.4, 3.3.2)
date (default: 3.4.1)
debug (1.10.0)
declarative (0.0.20)
delegate (default: 0.4.0)
did_you_mean (default: 2.0.0)
dig_rb (1.0.1)
digest (default: 3.2.0)
digest-crc (0.6.5)
domain_name (0.5.20190701)
drb (2.2.1)
elastic-transport (8.4.0)
elasticsearch (9.0.2)
elasticsearch-api (9.0.2)
english (default: 0.8.0)
erb (default: 4.0.4)
error_highlight (default: 0.7.0)
etc (default: 1.4.5)
excon (0.104.0)
faraday (2.7.12)
faraday-excon (2.1.0)
faraday-follow_redirects (0.3.0)
faraday-net_http (3.0.2)
faraday_middleware-aws-sigv4 (1.0.1)
fcntl (default: 1.2.0)
ffi (1.17.1)
ffi-compiler (1.0.1)
fiddle (default: 1.1.6)
fileutils (default: 1.7.3)
find (default: 0.2.0)
fluent-config-regexp-type (1.0.0)
fluent-mixin-config-placeholders (0.4.0)
fluent-plugin-aliyun-oss (0.0.1)
fluent-plugin-cloudwatch-logs (0.15.0)
fluent-plugin-concat (2.5.0)
fluent-plugin-datadog (0.14.4)
fluent-plugin-dedot_filter (1.0.0)
fluent-plugin-detect-exceptions (0.0.16)
fluent-plugin-elasticsearch (6.0.0)
fluent-plugin-gcs (0.4.4)
fluent-plugin-gelf-best (1.4.1)
fluent-plugin-geoip (1.3.2)
fluent-plugin-grafana-loki (1.2.20)
fluent-plugin-grok-parser (2.6.2)
fluent-plugin-kafka (0.19.4)
fluent-plugin-kinesis (3.5.0)
fluent-plugin-kube-events-timestamp (0.1.3)
fluent-plugin-label-router (0.5.0)
fluent-plugin-lm-logs (1.2.5)
fluent-plugin-logzio (0.2.2)
fluent-plugin-mattermost (0.2.2)
fluent-plugin-multi-format-parser (1.1.0)
fluent-plugin-mysqlslowquery (0.0.9)
fluent-plugin-newrelic (1.2.3)
fluent-plugin-opensearch (1.1.5)
fluent-plugin-parser-logfmt (0.0.0)
fluent-plugin-prometheus (2.2.1)
fluent-plugin-record-modifier (2.2.0)
fluent-plugin-redis (0.3.5)
fluent-plugin-remote-syslog (1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.8.3)
fluent-plugin-splunk-hec (1.3.3)
fluent-plugin-sqs (3.0.0)
fluent-plugin-syslog_rfc5424 (0.9.0.rc.8)
fluent-plugin-tag-normaliser (0.1.3)
fluent-plugin-throttle (0.0.5)
fluent-plugin-ua-parser (1.2.0)
fluent-plugin-vmware-log-intelligence (2.0.8)
fluent-plugin-vmware-loginsight (1.4.2)
fluent-plugin-webhdfs (1.6.0)
fluentd (1.18.0, 1.17.1)
forwardable (default: 1.3.3)
gelf_redux (4.1.0)
geoip-c (0.9.1)
geoip2_c (0.3.4)
getoptlong (0.2.1)
google-apis-core (0.15.1)
google-apis-iamcredentials_v1 (0.22.0)
google-apis-storage_v1 (0.47.0)
google-cloud-core (1.7.1)
google-cloud-env (2.2.1)
google-cloud-errors (1.4.0)
google-cloud-storage (1.52.0)
google-protobuf (3.25.5)
googleauth (1.11.2)
http (5.1.1)
http-accept (1.7.0)
http-cookie (1.0.5)
http-form_data (2.3.0)
http_parser.rb (0.8.0)
httpclient (2.8.3)
i18n (1.14.4)
io-console (default: 0.8.0)
io-nonblock (default: 0.3.1)
io-wait (default: 0.3.1)
ipaddr (default: 1.2.7)
irb (default: 1.14.3)
jmespath (1.6.2)
json (2.10.2, default: 2.9.1)
json-jwt (1.15.3.1)
jwt (2.9.3)
llhttp-ffi (0.4.0)
logfmt (0.0.10)
logger (1.7.0, 1.6.6, default: 1.6.4)
lru_redux (1.1.0)
ltsv (0.1.2)
mail (2.8.1)
matrix (0.4.2)
mime-types (3.5.1)
mime-types-data (3.2023.1003)
mini_mime (1.1.5)
mini_portile2 (2.8.8)
minitest (5.25.4, 5.22.2)
msgpack (1.8.0, 1.7.5)
multi_json (1.15.0)
mutex_m (0.3.0, 0.2.0)
myslog (0.1.1)
net-ftp (0.3.8)
net-http (default: 0.6.0)
net-http-persistent (4.0.2)
net-imap (0.5.8, 0.4.20)
net-pop (0.1.2)
net-protocol (default: 0.2.2)
net-smtp (0.5.1, 0.4.0)
netrc (0.11.0)
nkf (0.2.0)
nokogiri (1.18.8)
observer (0.1.2)
oj (3.16.10)
open-uri (default: 0.5.0)
open3 (default: 0.2.1)
openid_connect (1.1.8)
opensearch-ruby (3.0.1)
openssl (default: 3.3.0)
optparse (default: 0.6.0)
os (1.1.4)
ostruct (default: 0.6.1)
pathname (default: 0.4.0)
power_assert (2.0.5)
pp (default: 0.6.2)
prettyprint (default: 0.2.0)
prime (0.1.3)
prism (default: 1.2.0)
prometheus-client (4.2.4, 4.2.1)
pstore (default: 0.1.4)
psych (default: 5.2.2)
public_suffix (5.0.3)
racc (1.8.1)
rack (3.0.16)
rack-oauth2 (1.21.3)
rake (13.2.1)
rbs (3.8.0)
rdkafka (0.21.0)
rdoc (default: 6.10.0)
readline (default: 0.0.4)
redis (3.3.5)
reline (default: 0.6.0)
repl_type_completor (0.1.9)
representable (3.2.0)
resolv (default: 0.6.0)
resolv-replace (0.1.1)
rest-client (2.1.0)
retriable (3.1.2)
rexml (3.4.0)
rinda (0.2.0)
rss (0.3.1)
ruby-kafka (1.5.0)
ruby2_keywords (default: 0.0.5)
securerandom (default: 0.4.1)
serverengine (2.4.0, 2.3.2)
set (default: 1.1.1)
shellwords (default: 0.2.2)
sigdump (0.2.5)
signet (0.19.0)
singleton (default: 0.3.0)
specific_install (0.3.8)
stringio (default: 3.1.2)
strptime (0.2.5)
strscan (default: 3.1.2)
swd (2.0.2)
syntax_suggest (default: 2.0.2)
syslog (0.2.0)
syslog_protocol (0.9.2)
tempfile (default: 0.3.1)
test-unit (3.6.7)
time (default: 0.4.1)
timeout (default: 0.4.3)
tmpdir (default: 0.3.1)
trailblazer-option (0.1.2)
tsort (default: 0.2.0)
typeprof (0.30.1)
tzinfo (2.0.6)
tzinfo-data (1.2025.2, 1.2023.4)
uber (0.1.0)
un (default: 0.3.0)
unf (0.1.4)
unf_ext (0.0.8.2)
uri (default: 1.0.3)
user_agent_parser (2.16.0)
uuidtools (2.2.0)
validate_email (0.1.6)
validate_url (1.0.15)
weakref (default: 0.1.3)
webfinger (2.1.2)
webhdfs (0.11.0)
webrick (1.9.1)
yajl-ruby (1.4.3)
yaml (default: 0.4.0)
zlib (default: 3.2.1)

• ES version : 8.15.2
• ES template(s) (optional)

[loic-brtd]

I think the problem might be related to the elasticsearch gem, which wrongly overrides the version of Elasticsearch which is passed by headers. There's an issue for this : elastic/elasticsearch-ruby#2758.

I managed to work around this problem by retrograding to version 5.1.1 of the Logging Operator, which uses version 1.17.1 of fluentd, version 5.4.3 of the fluent-plugin-elasticsearch gem and version 8.11.0 of the elasticsearch gem :

$ fluent-gem list

*** LOCAL GEMS ***

abbrev (0.1.2)
activemodel (7.1.1)
activesupport (7.1.1)
addressable (2.8.7)
aes_key_wrap (1.1.0)
aliyun-sdk (0.7.0)
attr_required (1.0.1)
aws-eventstream (1.3.0)
aws-partitions (1.1001.0)
aws-sdk-cloudwatchlogs (1.71.0)
aws-sdk-core (3.212.0)
aws-sdk-firehose (1.65.0)
aws-sdk-kinesis (1.55.0)
aws-sdk-kms (1.95.0)
aws-sdk-s3 (1.169.0)
aws-sdk-sqs (1.64.0)
aws-sigv4 (1.10.1)
base64 (0.2.0)
benchmark (default: 0.4.0)
bigdecimal (3.1.8, 3.1.6)
bindata (2.5.0)
bundler (2.6.3, default: 2.6.2)
cgi (default: 0.4.1)
concurrent-ruby (1.3.5, 1.2.3)
connection_pool (2.4.1)
cool.io (1.9.0, 1.8.1)
csv (3.3.2, 3.3.1, 3.3.0)
date (default: 3.4.1, 3.3.3)
debug (1.10.0)
declarative (0.0.20)
delegate (default: 0.4.0)
did_you_mean (default: 2.0.0)
dig_rb (1.0.1)
digest (default: 3.2.0)
digest-crc (0.6.5)
domain_name (0.5.20190701)
drb (2.2.1)
elastic-transport (8.3.0)
elasticsearch (8.11.0)
elasticsearch-api (8.11.0)
english (default: 0.8.0)
erb (default: 4.0.4)
error_highlight (default: 0.7.0)
etc (default: 1.4.5)
excon (0.104.0)
faraday (2.7.12)
faraday-excon (2.1.0)
faraday-follow_redirects (0.3.0)
faraday-net_http (3.0.2)
faraday_middleware-aws-sigv4 (1.0.1)
fcntl (default: 1.2.0)
ffi (1.17.1)
ffi-compiler (1.0.1)
fiddle (default: 1.1.6)
fileutils (default: 1.7.3)
find (default: 0.2.0)
fluent-config-regexp-type (1.0.0)
fluent-mixin-config-placeholders (0.4.0)
fluent-plugin-aliyun-oss (0.0.1)
fluent-plugin-cloudwatch-logs (0.14.3)
fluent-plugin-concat (2.5.0)
fluent-plugin-datadog (0.14.4)
fluent-plugin-dedot_filter (1.0.0)
fluent-plugin-detect-exceptions (0.0.16)
fluent-plugin-elasticsearch (5.4.3)
fluent-plugin-gcs (0.4.4)
fluent-plugin-gelf-best (1.3.4)
fluent-plugin-geoip (1.3.2)
fluent-plugin-grafana-loki (1.2.20)
fluent-plugin-grok-parser (2.6.2)
fluent-plugin-kafka (0.19.3)
fluent-plugin-kinesis (3.5.0)
fluent-plugin-kube-events-timestamp (0.1.3)
fluent-plugin-label-router (0.4.0)
fluent-plugin-logzio (0.2.2)
fluent-plugin-mattermost (0.2.2)
fluent-plugin-multi-format-parser (1.1.0)
fluent-plugin-mysqlslowquery (0.0.9)
fluent-plugin-newrelic (1.2.3)
fluent-plugin-opensearch (1.1.5)
fluent-plugin-parser-logfmt (0.0.0)
fluent-plugin-prometheus (2.2.0)
fluent-plugin-record-modifier (2.2.0)
fluent-plugin-redis (0.3.5)
fluent-plugin-remote-syslog (1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.8.2)
fluent-plugin-splunk-hec (1.3.3)
fluent-plugin-sqs (3.0.0)
fluent-plugin-syslog_rfc5424 (0.9.0.rc.8)
fluent-plugin-tag-normaliser (0.1.3)
fluent-plugin-throttle (0.0.5)
fluent-plugin-ua-parser (1.2.0)
fluent-plugin-vmware-log-intelligence (2.0.8)
fluent-plugin-vmware-loginsight (1.4.2)
fluent-plugin-webhdfs (1.6.0)
fluentd (1.17.1)
forwardable (default: 1.3.3)
gelf (3.1.0)
geoip-c (0.9.1)
geoip2_c (0.3.4)
getoptlong (0.2.1)
google-apis-core (0.15.1)
google-apis-iamcredentials_v1 (0.22.0)
google-apis-storage_v1 (0.47.0)
google-cloud-core (1.7.1)
google-cloud-env (2.2.1)
google-cloud-errors (1.4.0)
google-cloud-storage (1.52.0)
google-protobuf (3.25.5)
googleauth (1.11.2)
http (5.1.1)
http-accept (1.7.0)
http-cookie (1.0.5)
http-form_data (2.3.0)
http_parser.rb (0.8.0)
httpclient (2.8.3)
i18n (1.14.4)
io-console (default: 0.8.0)
io-nonblock (default: 0.3.1)
io-wait (default: 0.3.1)
ipaddr (default: 1.2.7)
irb (default: 1.14.3)
jmespath (1.6.2)
json (default: 2.9.1, 2.6.3)
json-jwt (1.15.3.1)
jwt (2.9.3)
llhttp-ffi (0.4.0)
logfmt (0.0.10)
logger (default: 1.6.4, 1.6.3, 1.6.2)
lru_redux (1.1.0)
ltsv (0.1.2)
mail (2.8.1)
matrix (0.4.2)
mime-types (3.5.1)
mime-types-data (3.2023.1003)
mini_mime (1.1.5)
mini_portile2 (2.8.8)
minitest (5.25.4, 5.22.2)
msgpack (1.7.5)
multi_json (1.15.0)
mutex_m (0.3.0, 0.2.0)
myslog (0.1.1)
net-ftp (0.3.8)
net-http (default: 0.6.0)
net-http-persistent (4.0.2)
net-imap (0.5.4, 0.4.2)
net-pop (0.1.2)
net-protocol (default: 0.2.2, 0.2.1)
net-smtp (0.5.0, 0.4.0)
netrc (0.11.0)
nkf (0.2.0)
nokogiri (1.16.6)
observer (0.1.2)
open-uri (default: 0.5.0)
open3 (default: 0.2.1)
openid_connect (1.1.8)
opensearch-ruby (3.0.1)
openssl (default: 3.3.0)
optparse (default: 0.6.0)
os (1.1.4)
ostruct (default: 0.6.1)
pathname (default: 0.4.0)
power_assert (2.0.5)
pp (default: 0.6.2)
prettyprint (default: 0.2.0)
prime (0.1.3)
prism (default: 1.2.0)
prometheus-client (4.2.4, 4.2.1)
pstore (default: 0.1.4)
psych (default: 5.2.2)
public_suffix (5.0.3)
racc (1.8.1, 1.8.0)
rack (3.0.9.1)
rack-oauth2 (1.21.3)
rake (13.2.1)
rbs (3.8.0)
rdkafka (0.20.0)
rdoc (default: 6.10.0)
readline (default: 0.0.4)
redis (3.3.5)
reline (default: 0.6.0)
repl_type_completor (0.1.9)
representable (3.2.0)
resolv (default: 0.6.0)
resolv-replace (0.1.1)
rest-client (2.1.0)
retriable (3.1.2)
rexml (3.4.0)
rinda (0.2.0)
rss (0.3.1)
ruby-kafka (1.5.0)
ruby2_keywords (default: 0.0.5)
securerandom (default: 0.4.1)
serverengine (2.4.0, 2.3.2)
set (default: 1.1.1)
shellwords (default: 0.2.2)
sigdump (0.2.5)
signet (0.19.0)
singleton (default: 0.3.0)
specific_install (0.3.8)
stringio (default: 3.1.2)
strptime (0.2.5)
strscan (default: 3.1.2)
swd (2.0.2)
syntax_suggest (default: 2.0.2)
syslog (0.2.0)
syslog_protocol (0.9.2)
tempfile (default: 0.3.1)
test-unit (3.6.7)
time (default: 0.4.1)
timeout (default: 0.4.3, 0.4.0)
tmpdir (default: 0.3.1)
trailblazer-option (0.1.2)
tsort (default: 0.2.0)
typeprof (0.30.1)
tzinfo (2.0.6)
tzinfo-data (1.2025.1, 1.2024.2, 1.2023.4)
uber (0.1.0)
un (default: 0.3.0)
unf (0.1.4)
unf_ext (0.0.8.2)
uri (default: 1.0.2)
user_agent_parser (2.16.0)
uuidtools (2.2.0)
validate_email (0.1.6)
validate_url (1.0.15)
weakref (default: 0.1.3)
webfinger (2.1.2)
webhdfs (0.11.0)
webrick (1.9.1)
yajl-ruby (1.4.3)
yaml (default: 0.4.0)
zlib (default: 3.2.1)

[jeremych1000]

I think the problem might be related to the elasticsearch gem, which wrongly overrides the version of Elasticsearch which is passed by headers. There's an issue for this : elastic/elasticsearch-ruby#2758.

I managed to work around this problem by retrograding to version 5.1.1 of the Logging Operator, which uses version 1.17.1 of fluentd, version 5.4.3 of the fluent-plugin-elasticsearch gem and version 8.11.0 of the elasticsearch gem :

$ fluent-gem list

Heya - I think I've bumped into the same problem as you. Did you manage to find a working setup? 5.1.1 is quite old now, latest is 6.2.1.

[loic-brtd]

Heya - I think I've bumped into the same problem as you. Did you manage to find a working setup? 5.1.1 is quite old now, latest is 6.2.1.

Hi, unfortunately I haven't found any other solution as of now. But I haven't tested the latest versions.