exporter does not work and leaks memory

[nishant-dash]

Description

I am running

openstack-exporter                active      1  openstack-exporter  latest/stable   59  no       

on charmed jammy caracal

and I see that the exporter does not work, no metrics are available

root@juju-d65dda-26-lxd-3:~# curl http://127.0.0.1:9180/metrics 
root@juju-d65dda-26-lxd-3:~# 

Additionally the service is leaking a lot of memory and triggering the OOM killer.
It's currently at 14G, (it was restarted as part of the charm refresh), and previously at 32G.

Over a minute it went from

# uptime && ps aux | grep 195647
root      195647 38.6  0.4 18507972 14478180 ?   Ssl  12:13   3:03 /snap/charmed-openstack-exporter/44/bin/openstack-exporter --cache --cache-ttl=300s --os-client-config=/var/snap/charmed-openstack-exporter/common/clouds.yaml --web.listen-address=:9180 openstack

to

 12:25:45 up 45 days, 17:08,  2 users,  load average: 3.33, 6.14, 10.40
root      195647 38.1  0.4 18780884 14748516 ?   Ssl  12:13   3:23 /snap/charmed-openstack-exporter/44/bin/openstack-exporter --cache --cache-ttl=300s --os-client-config=/var/snap/charmed-openstack-exporter/common/clouds.yaml --web.listen-address=:9180 openstack
root      196712  0.0  0.0   6484     0 pts/2    S+   12:25   0:00 grep --color=auto 195647

thats about 0.25 G per minute, which is really high

Charm revision

59

Juju version

3.6.0

Cloud

OpenStack, MAAS

Expected behaviour

metrics appear

Reproduce / Test

juju deploy it in jammy caracal and relate it to the required entities, we have it with

keystone:identity-admin                                     openstack-exporter:credentials                                keystone-admin           regular      
openstack-exporter:cos-agent                                grafana-agent-container:cos-agent                             cos_agent                subordinate  
openstack-exporter:juju-info                                grafana-agent-container:juju-info                             juju-info                subordinate  
openstack-exporter:juju-info                                landscape-client:container                                    juju-info                subordinate  
openstack-exporter:juju-info                                logrotated:juju-info                                          juju-info                subordinate  
openstack-exporter:juju-info                                nrpe:general-info                                             juju-info                subordinate  
openstack-exporter:juju-info                                ubuntu-advantage:juju-info                                    juju-info                subordinate  

Notes & References

There is no way to set debug in the charm config so I tried to run it manually with

OS_COMPUTE_API_VERSION=2.87 /snap/charmed-openstack-exporter/44/bin/openstack-exporter --cache --cache-ttl=300s --os-client-config=/var/snap/charmed-openstack-exporter/common/clouds.yaml --web.listen-address=:9180 openstack --log.level=debug

but I dont see anything useful in debug logs, but there are a bunch of errors
However, none of the error logs seems to justify why the exporter itself should not be running

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/SOLENG-1782.

This message was autogenerated

[nishant-dash]

output from

OS_COMPUTE_API_VERSION=2.87 /snap/charmed-openstack-exporter/44/bin/openstack-exporter --cache --cache-ttl=300s --os-client-config=/var/snap/charmed-openstack-exporter/common/clouds.yaml --web.listen-address=:9180 openstack --log.level=debug

o7k-exporter-debug.log

[nishant-dash]

looking at the oom kill message it was

Jan 10 09:22:43 REDACTED kernel: [3938928.323388] Out of memory: Killed process 863430 (openstack-expor) total-vm:112153904kB, anon-rss:102979056kB, file-rss:0kB, shmem-rss:0kB, UID:1000000 pgtables:202440kB oom_score_adj:0
Jan 10 09:22:51 REDACTED kernel: [3938935.918063] oom_reaper: reaped process 863430 (openstack-expor), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jan 10 10:32:21 REDACTED kernel: [3943106.373619] CPU 3/KVM invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
Jan 10 10:32:21 REDACTED kernel: [3943106.373656]  oom_kill_process+0x118/0x280
Jan 10 10:32:21 REDACTED kernel: [3943106.373664]  ? oom_evaluate_task+0x143/0x1e0
Jan 10 10:32:21 REDACTED kernel: [3943106.373671]  __alloc_pages_may_oom+0x112/0x1e0
Jan 10 10:32:21 REDACTED kernel: [3943106.373959] [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Jan 10 10:32:21 REDACTED kernel: [3943106.374879] oom-kill:constraint=CONSTRAINT_CPUSET,nodemask=(null),cpuset=vcpu3,mems_allowed=0,global_oom,task_memcg=/lxc.payload.juju-d65dda-26-lxd-3/system.slice/snap.charmed-openstack-exporter.service.service,task=openstack-expor,pid=1012000,uid=1000000

which is about 92 G of memory

[jneo8]

Hi @nishant-dash,

To help us with the root cause analysis of this issue, could you try running the exporter with specific service flags?

For example, using --disable-service.network would help us determine if the networking component is the source of the problem through a process of elimination. You can find the full list of available flags in the command-line options documentation.

[nishant-dash]

Hey @jneo8

I ran the exporter with gnocchi disabled and it works now, no mem leaks and metrics for other components are generated

snap set charmed-openstack-exporter disable-service.gnocchi=true

[nishant-dash]

I believe this might be because gnocchi is stopped in the cloud where this exporter is running

Which means to reproduce, you should be able to deploy an o7k cloud, the exporter and stop gnocchi (or perhaps any relevant api service).

If this breaks on stopping any api service, then something is fundamentally wrong with the exporter

[H-M-Quang-Ngo]

I'm having a jammy caracal with openstack-exporter in rev 59 latest/stable, so I quickly check to see I can reproduce this issue.
What I do is simply pause all gnocchi units via juju action and restart the exporter. However seems like the exporter still correctly reports gnocchi down instead of dying:

ubuntu@pc6a-infra:~$ juju status | grep gnocchi
gnocchi                                4.6.0            blocked      3  gnocchi                      2024.1/stable  245  no       Services should be paused but these ports which should be closed, but are open: 8041
gnocchi-mysql-router                   8.0.44           active       3  mysql-router                 8.0/stable     257  no       Unit is ready
hacluster-gnocchi                      2.1.2            active       3  hacluster                    2.4/stable     131  no       Unit is ready and clustered
gnocchi/0                                   blocked   idle   3/lxd/7   10.17.2.159     8041/tcp        Services should be paused but these ports which should be closed, but are open: 8041
  gnocchi-mysql-router/1                    active    idle             10.17.2.159                     Unit is ready
  hacluster-gnocchi/1                       active    idle             10.17.2.159                     Unit is ready and clustered
gnocchi/1                                   blocked   idle   4/lxd/7   10.17.2.177     8041/tcp        Services should be paused but these ports which should be closed, but are open: 8041
  gnocchi-mysql-router/2                    active    idle             10.17.2.177                     Unit is ready
  hacluster-gnocchi/2                       active    idle             10.17.2.177                     Unit is ready and clustered
gnocchi/2*                                  blocked   idle   5/lxd/7   10.17.2.102     8041/tcp        Services should be paused but these ports which should be closed, but are open: 8041
  gnocchi-mysql-router/0*                   active    idle             10.17.2.102                     Unit is ready
  hacluster-gnocchi/0*                      active    idle             10.17.2.102                     Unit is ready and clustered

ubuntu@pc6a-infra:~$ juju exec -u openstack-exporter/leader -- "ps aux | grep openstack-exporter | grep -v grep"
root        8682  0.0  0.0 377284  6144 ?        Ssl  Jan12   0:04 snapfuse /var/lib/snapd/snaps/charmed-openstack-exporter_44.snap /snap/charmed-openstack-exporter/44 -o ro,nodev,allow_other,suid
root      982975  0.0  0.0 3828340 10752 ?       Ssl  07:09   0:02 /snap/charmed-openstack-exporter/44/bin/openstack-exporter --cache --cache-ttl=300s --os-client-config=/var/snap/charmed-openstack-exporter/common/clouds.yaml --web.listen-address=:9180 openstack

ubuntu@pc6a-infra:~$ juju exec -u openstack-exporter/leader -- "curl -s http://127.0.0.1:9180/metrics | grep gnocchi"
# HELP openstack_gnocchi_up up
# TYPE openstack_gnocchi_up gauge
openstack_gnocchi_up 0

I'm in PC6a, not sure it is a specific issue in PS7 or we need more proper ways to reproduce the issue.

[jneo8]

What I do is simply pause all gnocchi units via juju action and restart the exporter. However seems like the exporter still correctly reports gnocchi down instead of dying:

You shouldn't restart the exporter, right?
Can you observer the behavior of exporter once you stop the gnocchi service?

[H-M-Quang-Ngo]

@jneo8 Without restarting, I can still see the metrics available when checking the endpoint.
However, the exporter is serving from cache by default so if I didn't restart then when checking the endpoint it show the cache and we will still see the gnocchi up.

[chanchiwai-ray]

Deploying OpenStack Jammy Caracal with stsstack-bundle also cannot reproduce the mem leak

[nishant-dash]

hey @H-M-Quang-Ngo can you clarify some details about the reproducer? Are there vms in your test cloud? what kind of gnocchi metrics did the exporter pick up from gnocchi before you stopped it?

[H-M-Quang-Ngo]

@nishant-dash This is what I got before stopping gnocchi:

ubuntu@pc6a-infra:~$ juju exec -u openstack-exporter/leader -- "curl -s http://127.0.0.1:9180/metrics | grep gnocchi"
# HELP openstack_gnocchi_status_metric_having_measures_to_process status_metric_having_measures_to_process
# TYPE openstack_gnocchi_status_metric_having_measures_to_process gauge
openstack_gnocchi_status_metric_having_measures_to_process 0
# HELP openstack_gnocchi_status_measures_to_process status_measures_to_process
# TYPE openstack_gnocchi_status_measures_to_process gauge
openstack_gnocchi_status_measures_to_process 0
# HELP openstack_gnocchi_status_metricd_processors status_metricd_processors
# TYPE openstack_gnocchi_status_metricd_processors gauge
openstack_gnocchi_status_metricd_processors 12
# HELP openstack_gnocchi_total_metrics total_metrics
# TYPE openstack_gnocchi_total_metrics gauge
openstack_gnocchi_total_metrics 402
# HELP openstack_gnocchi_up up
# TYPE openstack_gnocchi_up gauge
openstack_gnocchi_up 1

I used to have few lightweight testing VMs, and at the time of checking I only have 1. So most of the time the cluster is pretty idle and not really under serious workload.

I guess it’s reasonable to put some stress onto the cluster which could help us see the issue happen. But I’m not sure are there specific conditions - like we just have to spin ups dozen VMs, or we need to make Gnocchi really busy somehow.

It’s nice to have insights from you, which will be helpful for @chanchiwai-ray to reproduce the issue.