elastic search 9.2.3 not compatible with fluentd

[gbhushan86]

We have delpoyed elastic cache version 8.17 and 8.19 in our env's where fluentd aggregator is running as stateful set with out any issue.
But when Elastic search is deployed with docker.elastic.co/elasticsearch/elasticsearch:9.2.3 with fluentd:v1.19.1-debian-1.0 and fluentd:v1.18-debian-1

im getting error in the logs stating ruby path is not found due to which plugins are failing to install.

efk-upgrade-fluentd-aggregator-0.log

as we are planning for upgrade from 8-9 fluentd is blocking the upgrade.

Problem

Unable to upgrade elastic search to 9.2.3 since fluentd is not getting upgraded its not getting compatible
...

Steps to replicate

Deploy ELastic search 9.2.3 with fluentd:v1.19.1-debian-1.0 or fluentd:v1.18-debian-1

OR

Provide example config and message

Expected Behavior or What you need to ask

efk-upgrade-fluentd-aggregator-0.log

...

Using Fluentd and ES plugin versions

FLuent plugin elasticsearch 6.0.0

• OS version
  *Kubernetes AKS
• Fluentd
  fluentd 1.18.0
• ES plugin 3.2.0
  • paste boot log of fluentd or td-agent
  • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock
• ES version 9.2.3

efk-upgrade-fluentd-aggregator-0.log

• ES template(s) (optional)
  Attaching sample yaml file we used to delpoy in aks
  platform: "azure" #valid values are openshift, azure
  externalSecret:
  enabled: true
  ingress:
  enabled: true
  elasticHost: efk-upgrade-elastic.idit.sapiens.com
  kibanaHost: efk-upgrade-kibana.idit.sapiens.com
  #Parameters for elasticsearch
  eck-elasticsearch:
  enabled: true
  fullnameOverride: "efk-upgrade"
  version: 9.2.3
  image: harbor.idit.sapiens.com/infrastructure-releases/elasticsearch:9.2.3
  http:
  tls:
  selfSignedCertificate:
  disabled: false
  secureSettings:
  • secretName: snapshot-secret
    nodeSets:
  • name: idit-cluster
    config:
    node.roles: ["master","data", "ingest"]
    node.store.allow_mmap: false
    path.repo:
    - /usr/share/elasticsearch/backup
    count: 1
    podTemplate:
    spec:
    imagePullSecrets: []
    securityContext:
    fsGroup: 1000
    runAsUser: 1000
    runAsGroup: 0
    volumes:
    - name: es-shared
    persistentVolumeClaim:
    claimName: es-shared-pvc
    affinity:
    nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
    - key: "nodetype"
    operator: In
    values:
    - infra
    podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
    podAffinityTerm:
    labelSelector:
    matchExpressions:
    - key: "app.kubernetes.io/name"
    operator: In
    values:
    - elasticsearch
    topologyKey: "kubernetes.io/hostname"
    initContainers:
    - name: elastic-internal-init-filesystem
    securityContext:
    runAsUser: 0
    runAsGroup: 0
    - name: install-plugins
    command:
    - sh
    - -c
    - |
    bin/elasticsearch-plugin remove repository-azure
    bin/elasticsearch-plugin install --batch repository-azure
    containers:
    - name: elasticsearch
    resources:
    limits:
    cpu: 2
    memory: "12Gi"
    env:
    - name: ES_JAVA_OPTS
    value: "-Xms8g -Xmx8g"
    volumeMounts:
    - name: es-shared
    mountPath: /usr/share/elasticsearch/backup
    volumeClaimTemplates:
    • metadata:
      name: elasticsearch-data
      spec:
      accessModes:
      • ReadWriteOnce
        resources:
        requests:
        storage: 1000Gi
        storageClassName: managed-csi

#Parameters for Kibana
eck-kibana:
enabled: true
fullnameOverride: efk-upgrade
image: harbor.idit.sapiens.com/infrastructure-releases/kibana:9.2.3
spec:
version: 9.2.3
elasticsearchRef:
name: "efk-upgrade"

Parameters for fluentd chart

fluentd:
enabled: true
image:
repository: "harbor.idit.sapiens.com/infrastructure-releases/fluentd"
volumes:
- name: es-credentials
secret:
secretName: efk-upgrade-es-elastic-user
backup:
enabled: true # set true to enable azure snapshot
storageaccountName:
storageaccountKey: