EFS mount fails on Bottlerocket: efs-proxy cannot bind to localhost

[mike-ainsel]

What happened?

EFS mounts fail on EKS 1.34 with Bottlerocket nodes. The efs-proxy component starts but immediately panics when trying to bind to localhost, causing mount failures with DeadlineExceeded errors.

When manually running efs-proxy inside the CSI node container, it crashes with:

thread 'main' (127) panicked at src/controller.rs:89:13:
Failed to bind 127.0.0.1:20381
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The mount.log shows:

2026-01-14 16:12:01 UTC - INFO - Starting efs-proxy: "/sbin/efs-proxy /var/run/efs/stunnel-config... --tls"
2026-01-14 16:12:01 UTC - INFO - Started efs-proxy, pid: 26
2026-01-14 16:12:01 UTC - WARNING - Error connecting to 127.0.0.1:20241, [Errno 111] Connection refused
2026-01-14 16:12:16 UTC - ERROR - Mounting ... failed due to timeout after 15 sec

What you expected to happen?

EFS should mount successfully. The efs-proxy should be able to bind to localhost and proxy NFS traffic over TLS.

How to reproduce it (as minimally and precisely as possible)?

1. Deploy EKS 1.34 cluster with Bottlerocket nodes (tested with Bottlerocket OS 1.52.0)
2. Install EFS CSI driver v2.2.0 as EKS add-on
3. Create EFS with mount targets in node subnets
4. Create StorageClass, PVC, and Pod:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: fs-XXXXXXXXX
  directoryPerms: "755"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: efs-test
spec:
  containers:
  - name: app
    image: amazonlinux:2023
    command: ["/bin/sh", "-c", "sleep 3600"]
    volumeMounts:
    - name: efs
      mountPath: /data
  volumes:
  - name: efs
    persistentVolumeClaim:
      claimName: efs-pvc

5. Pod stays in ContainerCreating with FailedMount events

Anything else we need to know?

• Plain NFS mount works (without TLS/efs-proxy):

  mount -t nfs4 -o nfsvers=4.1 fs-xxx.efs.region.amazonaws.com:/ /mnt/efs

• TLS connection to EFS works (tested with openssl s_client)

• The CSI node container is properly configured:

  • privileged: true
  • hostNetwork: true
  • Loopback interface exists and has traffic

• The stunnel config generated includes socket = a:SO_BINDTODEVICE=lo which may have issues in containerized environments

• Tested with both encrypted (KMS) and unencrypted EFS - same failure

• Also reproduced on Amazon Linux 2023 EC2 instance with efs-utils v2.4.1

• Warning in logs: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"

Environment

• Kubernetes version (use kubectl version): v1.34 (EKS platform eks.9)
• Driver version: v2.2.0-eksbuild.1 (EKS Add-on)
• Node OS: Bottlerocket OS 1.52.0 (aws-k8s-1.34)
• Kernel: 6.12.58
• Container Runtime: containerd://2.1.5+bottlerocket
• Region: eu-central-1

Please also attach debug logs to help us better diagnose

EFS CSI Node Pod logs:

I0114 16:09:14.552835       1 config_dir.go:88] Creating symlink from '/etc/amazon/efs' to '/var/amazon/efs'
I0114 16:09:14.568524       1 driver.go:131] Registering Node Server
I0114 16:09:14.568583       1 driver.go:133] Registering Controller Server
I0114 16:09:14.568623       1 driver.go:136] Starting efs-utils watchdog
I0114 16:09:14.569367       1 driver.go:151] Listening for connections on address: &net.UnixAddr{Name:"/csi/csi.sock", Net:"unix"}
I0114 16:12:00.975226       1 mount_linux.go:285] Detected OS without systemd

Mount log from inside container (/var/log/amazon/efs/mount.log):

2026-01-14 16:12:01 UTC - INFO - version=2.4.1 options={'rw': None, 'accesspoint': 'fsap-xxx', 'tls': None}
2026-01-14 16:12:01 UTC - INFO - binding 20241
2026-01-14 16:12:01 UTC - WARNING - Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"
2026-01-14 16:12:01 UTC - INFO - Starting efs-proxy: "/sbin/efs-proxy /var/run/efs/stunnel-config.fs-xxx... --tls"
2026-01-14 16:12:01 UTC - INFO - Started efs-proxy, pid: 26
2026-01-14 16:12:01 UTC - WARNING - Error connecting to 127.0.0.1:20241, [Errno 111] Connection refused
2026-01-14 16:12:01 UTC - INFO - Executing: "/sbin/mount.nfs4 127.0.0.1:/ /var/lib/kubelet/pods/.../mount -o rw,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,port=20241" with 15 sec time limit.
2026-01-14 16:12:16 UTC - ERROR - Mounting fs-xxx.efs.eu-central-1.amazonaws.com to ... failed due to timeout after 15 sec, mount attempt 1/3

Manual efs-proxy execution (from inside CSI node container):

$ /sbin/efs-proxy "/var/run/efs/stunnel-config.fs-xxx..." --tls
thread 'main' (127) panicked at src/controller.rs:89:13:
Failed to bind 127.0.0.1:20381
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Stunnel config content:

fips = no
foreground = yes
socket = l:SO_REUSEADDR=yes
socket = a:SO_BINDTODEVICE=lo
pid = /var/run/efs/.../stunnel.pid
[efs]
client = yes
accept = 127.0.0.1:20381
connect = fs-xxx.efs.eu-central-1.amazonaws.com:2049
sslVersion = TLSv1.2
renegotiation = no
TIMEOUTbusy = 20
TIMEOUTclose = 0
TIMEOUTidle = 70
delay = yes
verify = 2
CAfile = /etc/amazon/efs/efs-utils.crt
cert = /var/run/efs/.../certificate.pem
key = /etc/amazon/efs/privateKey.pem
checkHost = fs-xxx.efs.eu-central-1.amazonaws.com

[mike-ainsel]

✅ RESOLUTION

Root Cause

The VPC had DNS64 enabled on subnets (enable_dns64 = true). This caused the VPC DNS resolver to synthesize NAT64 IPv6 addresses (e.g., 64:ff9b::afc:5bf) for EFS mount targets, even though the EFS only had IPv4 addresses.

When efs-proxy resolved the EFS DNS name, it received:

64:ff9b::afc:5bf  (NAT64 synthesized - listed FIRST)
10.252.5.191      (real IPv4 - listed second)

Since IPv6 was listed first, efs-proxy tried to connect via IPv6, but:

• Pods only have IPv4 addresses (no routable IPv6)
• NAT64 gateway was not configured for internal VPC traffic
• Result: "Network unreachable" → mount timeout

Why EKS 1.33 Worked

The test cluster on EKS 1.33 was created in a different AWS account with a VPC that did NOT have DNS64 enabled. This is why it appeared to be an EKS version issue, but it was actually a VPC configuration difference.

The Fix

Option 1: Disable DNS64 on subnets (Recommended)

# In Terraform VPC module
public_subnet_enable_dns64  = false
private_subnet_enable_dns64 = false

Or via AWS CLI:

aws ec2 modify-subnet-attribute --subnet-id subnet-xxx --no-enable-dns64

Option 2: Use IPv4-only EFS mount targets

# In Terraform EFS module
mount_targets = {
  for idx, subnet_id in local.private_subnet_ids :
  local.azs[idx] => {
    subnet_id       = subnet_id
    ip_address_type = "IPV4_ONLY"
  }
}

Option 3: Use mounttargetip mount option (workaround)

mountOptions:
  - tls
  - iam
  - mounttargetip=10.252.5.191  # Force IPv4 (AZ-specific)

Verification

After applying the fix:

# DNS should return only IPv4
kubectl exec -it test-pod -- getent ahosts fs-xxx.efs.eu-central-1.amazonaws.com
# Expected: 10.252.x.x (IPv4 only, no 64:ff9b:: prefix)

Key Takeaway

If you have a dual-stack VPC with DNS64 enabled, but your pods only have IPv4 addresses, EFS TLS mounts will fail because efs-proxy tries the synthesized (unreachable) IPv6 address first.

Check your subnet settings:

aws ec2 describe-subnets --subnet-ids subnet-xxx --query "Subnets[*].[SubnetId,EnableDns64]"