DNS lookup does not work properly #1050
Description
(check apply)
- read the contribution guideline
- (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.
Problem
In order to increase security, we recently wanted to switch to Consul Transparent Proxy. However, due to that localhost will no longer be properly resolved. Instead we need to use the service name on Kubernetes.
However, due to that change, the fluentd output plugin for elasticsearch does not work anymore. It always complains, that it cannot resolve the hostname. It seems that this is a known issue, as I already found multiple issues here.
Steps to replicate
Deploy Consul with Transparent Proxy
Deploy the Fluentd Helm Chart on K8s
Try to establish a connection between fluentd and elasticsearch, which is protected by transparent proxy.
We are using the following output config:
<label @out_container_log>
<match *.**>
@type copy
<store>
@type elasticsearch
default_elasticsearch_version 8
host elasticsearch-master
port 9200
user elastic
password PASS
type_name fluentd
logstash_format true
logstash_prefix fluentd
include_timestamp true
reload_connections false
reload_on_failure true
reconnect_on_error true
flush_mode default
flush_interval 10s
</store>
</match>
</label>
<match apc_app_log.**>
@type elasticsearch
default_elasticsearch_version 8
host elasticsearch-master
port 9200
user elastic
password PASS
suppress_type_name true
logstash_format true
target_index_key index
include_timestamp true
id_key _id
routing_key routing
remove_keys routing_key, index, _id
reload_connections false
reload_on_failure true
reconnect_on_error true
<buffer>
@type file
path /var/log/fluentd/buffer
flush_mode immediate
flush_thread_count 20
chunk_limit_size 256m
total_limit_size 20G
queue_limit_length 200
overflow_action throw_exception
retry_type periodic
</buffer>
</match>
Expected Behavior or What you need to ask
Other services can resolve the elasticsearch-master service, which results to the following output. However, unfortunately, something seems to be wrong with the way that DNS works in the plugin, if it cannot resolve to localhost.
Below is the nslookup command executed from another pod container
nslookup elasticsearch-master
;; Got recursion not available from 127.0.0.1, trying next server
;; Got recursion not available from 10.0.0.10
Server: 10.0.0.10
Address: 10.0.0.10#53
Name: elasticsearch-master.default.svc.cluster.local
Address: 172.28.160.107
Name: elasticsearch-master.default.svc.cluster.local
Address: 172.28.162.22
Using Fluentd and ES plugin versions
- Bare Metal or within Docker or Kubernetes or others? Kubernetes
- Fluentd v0.12 or v0.14/v1.0 - fluentd 1.17.0
- ES plugin 3.x.y/2.x.y or 1.x.y
- paste boot log of fluentd or td-agent
2024-08-07 12:02:45 +0000 [info]: #0 'flush_interval' is configured at out side of <buffer>. 'flush_mode' is set to 'interval' to keep existing behaviour
The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the server is running an unsupported product.
2024-08-07 12:02:46 +0000 [error]: #0 unexpected error error_class=Elastic::Transport::Transport::Error error="no address for elasticsearch-master (Resolv::ResolvError)"
2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elastic-transport-8.3.2/lib/elastic/transport/transport/base.rb:324:in `rescue in perform_request'
2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elastic-transport-8.3.2/lib/elastic/transport/transport/base.rb:285:in `perform_request'
2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elastic-transport-8.3.2/lib/elastic/transport/transport/http/faraday.rb:36:in `perform_request'
2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elastic-transport-8.3.2/lib/elastic/transport/client.rb:197:in `perform_request'
2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-8.13.0/lib/elasticsearch.rb:86:in `verify_elasticsearch'
2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-8.13.0/lib/elasticsearch.rb:69:in `method_missing'
2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/elasticsearch-api-8.13.0/lib/elasticsearch/api/actions/info.rb:43:in `info'
2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/out_elasticsearch.rb:498:in `detect_es_major_version'
2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/out_elasticsearch.rb:489:in `block in handle_last_seen_es_major_version'
2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/elasticsearch_index_template.rb:56:in `retry_operate'
2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/out_elasticsearch.rb:486:in `handle_last_seen_es_major_version'
2024-08-07 12:02:46 +0000 [error]: #0 /opt/bitnami/fluentd/gems/fluent-plugin-elasticsearch-5.4.3/lib/fluent/plugin/out_elasticsearch.rb:338:in `configure'
- paste result of
fluent-gem list,td-agent-gem listor your Gemfile.lock
abbrev (default: 0.1.0)
activesupport (7.1.3.2)
addressable (2.8.6)
aws-eventstream (1.3.0)
aws-partitions (1.927.0)
aws-sdk-core (3.195.0)
aws-sdk-kms (1.80.0)
aws-sdk-s3 (1.149.1)
aws-sdk-sqs (1.73.0)
aws-sigv4 (1.8.0)
base64 (0.2.0, default: 0.1.1)
benchmark (default: 0.2.0)
bigdecimal (3.1.8, default: 3.1.1)
bundler (2.5.10)
cgi (default: 0.3.6)
concurrent-ruby (1.2.3)
connection_pool (2.4.1)
cool.io (1.8.0)
csv (3.3.0, default: 3.2.5)
date (default: 3.2.2)
debug (1.6.3)
delegate (default: 0.2.0)
did_you_mean (default: 1.6.1)
digest (default: 3.1.0)
digest-crc (0.6.5)
domain_name (0.6.20240107)
drb (2.2.1, default: 2.1.0)
elastic-transport (8.3.2)
elasticsearch (8.14.0, 8.13.0)
elasticsearch-api (8.14.0, 8.13.0)
elasticsearch-xpack (7.17.10)
english (default: 0.7.1)
erb (default: 2.2.3)
error_highlight (default: 0.3.0)
etc (default: 1.3.0)
excon (0.110.0)
faraday (2.9.0)
faraday-excon (2.1.0)
faraday-net_http (3.1.0)
fcntl (default: 1.0.1)
ffi (1.16.3)
ffi-compiler (1.3.2)
fiddle (default: 1.1.0)
fileutils (default: 1.6.0)
find (default: 0.1.1)
fluent-config-regexp-type (1.0.0)
fluent-plugin-concat (2.5.0)
fluent-plugin-detect-exceptions (0.0.15)
fluent-plugin-elasticsearch (5.4.3)
fluent-plugin-grafana-loki (1.2.20)
fluent-plugin-kafka (0.19.2)
fluent-plugin-kubernetes_metadata_filter (3.4.0)
fluent-plugin-multi-format-parser (1.1.0)
fluent-plugin-parser-cri (0.1.1)
fluent-plugin-prometheus (2.1.0)
fluent-plugin-record-modifier (2.2.0)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.7.2)
fluent-plugin-systemd (1.0.5)
fluentd (1.17.0, 1.16.5)
forwardable (default: 1.3.2)
getoptlong (default: 0.1.1)
http (5.2.0)
http-accept (1.7.0)
http-cookie (1.0.5)
http-form_data (2.3.0)
http_parser.rb (0.8.0)
i18n (1.14.5)
io-console (default: 0.5.11)
io-nonblock (default: 0.1.0)
io-wait (default: 0.2.1)
ipaddr (default: 1.2.4)
irb (default: 1.4.1)
jmespath (1.6.2)
json (default: 2.6.1, 2.1.0)
jsonpath (1.1.5)
kubeclient (4.11.0)
llhttp-ffi (0.5.0)
logger (default: 1.5.0)
lru_redux (1.1.0)
ltsv (0.1.2)
matrix (0.4.2)
mime-types (3.5.2)
mime-types-data (3.2024.0507)
minitest (5.22.3, 5.15.0)
msgpack (1.7.2)
multi_json (1.15.0)
mutex_m (0.2.0, default: 0.1.1)
net-ftp (0.1.4)
net-http (0.4.1, default: 0.3.0.1)
net-imap (0.2.4)
net-pop (0.1.1)
net-protocol (default: 0.1.2)
net-smtp (0.3.1.1)
netrc (0.11.0)
nkf (default: 0.1.1)
observer (default: 0.1.1)
oj (3.3.10)
open-uri (default: 0.2.0)
open3 (default: 0.1.1)
openssl (default: 3.0.1)
optparse (default: 0.2.0)
ostruct (default: 0.5.2)
pathname (default: 0.2.0)
power_assert (2.0.1)
pp (default: 0.3.0)
prettyprint (default: 0.1.1)
prime (0.1.2)
prometheus-client (4.2.2)
pstore (default: 0.1.1)
psych (default: 4.0.4)
public_suffix (5.0.5)
racc (default: 1.6.0)
rake (13.2.1, 13.0.6)
rbs (2.7.0)
rdoc (default: 6.4.1.1)
readline (default: 0.0.3)
readline-ext (default: 0.1.4)
recursive-open-struct (1.1.3)
reline (default: 0.3.1)
resolv (default: 0.2.1)
resolv-replace (default: 0.1.0)
rest-client (2.1.0)
rexml (3.2.5)
rinda (default: 0.1.1)
rss (0.2.9)
ruby-kafka (1.5.0)
ruby2_keywords (default: 0.0.5)
rubygems-update (3.5.10)
securerandom (default: 0.2.0)
serverengine (2.3.2)
set (default: 1.0.2)
shellwords (default: 0.1.0)
sigdump (0.2.5)
singleton (default: 0.1.1)
stringio (default: 3.0.1.2)
strptime (0.2.5)
strscan (default: 3.0.1)
syslog (default: 0.1.0)
systemd-journal (1.4.2)
tempfile (default: 0.1.2)
test-unit (3.5.3)
time (default: 0.2.2)
timeout (default: 0.2.0)
tmpdir (default: 0.1.2)
tsort (default: 0.1.0)
typeprof (0.21.3)
tzinfo (2.0.6)
tzinfo-data (1.2024.1)
un (default: 0.2.0)
uri (0.13.0, default: 0.12.2)
weakref (default: 0.1.1)
webrick (1.8.1)
yajl-ruby (1.4.3)
yaml (default: 0.2.0)
zlib (default: 2.1.1)
- ES version (optional) - 8.14.1
- ES template(s) (optional)