uWSGI crashes during startup causing nginx 502 and CrashLoopBackOff in ITRB pod

[hodgesf]

The ITRB pod is entering a CrashLoopBackOff state due to uWSGI exiting during startup. Nginx subsequently returns HTTP 502 because it cannot connect to uWSGI via unix:///tmp/uwsgi.sock (connection refused). The healthcheck and init logs show the failure, but they do not include the underlying reason for the uWSGI crash.

Based on the logs, uWSGI starts, reads its configuration, and then exits almost immediately. Because the uWSGI process never successfully binds the socket, nginx fails downstream.

[uWSGI] getting INI configuration from /app/uwsgi.ini
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini
...
WARN exited: uwsgi (exit status 22; not expected)

CI ARAX was pointed to the self-hosted Plover instance by @bazarkua
Pod was manually restarted by Kanna and CrashLoop behavior was reproduced with the latest image (December 24th, 2025). When reverting back to an older image (July 30th, 2025) the pod came up successfully.

Potential fix suggested by Eric was to push a commit without any code changes, and see if redeployment is successful. Will attempt this and report results back to this issue.

[hodgesf]

Rebuild triggered by 8411cfb was not successful. Working on getting the build logs from ITRB (Kanna).

[hodgesf]

Logs provided by Kanna:

Checking for script in /app/prestart.sh
Running script /app/prestart.sh
Running inside /app/prestart.sh, you could add migrations to this file, e.g.:

#! /usr/bin/env sh

# Let the DB start
sleep 10;
# Run migrations
alembic upgrade head

/usr/lib/python3/dist-packages/supervisor/options.py:474: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
  self.warnings.warn(
2026-01-08 17:34:19,483 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2026-01-08 17:34:19,483 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
2026-01-08 17:34:19,486 INFO RPC interface 'supervisor' initialized
2026-01-08 17:34:19,486 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2026-01-08 17:34:19,486 INFO supervisord started with pid 1
2026-01-08 17:34:20,489 INFO spawned: 'quit_on_failure' with pid 8
2026-01-08 17:34:20,490 INFO spawned: 'nginx' with pid 9
2026-01-08 17:34:20,491 INFO spawned: 'uwsgi' with pid 10
2026-01-08 17:34:20,493 INFO success: nginx entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2026-01-08 17:34:20,493 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
[uWSGI] getting INI configuration from /app/uwsgi.ini
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini
2026-01-08 17:34:21,501 INFO success: quit_on_failure entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2026-01-08 17:34:21,501 WARN exited: uwsgi (exit status 22; not expected)
2026-01-08 17:34:22,503 WARN received SIGTERM indicating exit request
2026-01-08 17:34:22,503 INFO waiting for quit_on_failure, nginx to die
2026/01/08 17:34:22 [error] 11#11: *1 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 10.11.0.61, server: , request: "GET /healthcheck HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "10.11.108.91"
10.11.0.61 - - [08/Jan/2026:17:34:22 +0000] "GET /healthcheck HTTP/1.1" 502 157 "-" "ELB-HealthChecker/2.0" "-"
2026-01-08 17:34:22,631 INFO stopped: nginx (exit status 0)
2026-01-08 17:34:22,632 WARN stopped: quit_on_failure (terminated by SIGTERM)

From these logs, uwsgi is starting and then exiting almost immediately with exit status 22, before binding its socket. This points to an early uwsgi startup failure, likely during application import or configuration parsing. I have requested Kanna to attempt to rerun the container with increased verbosity or to bypass nginx and supervisor and run uwsgi directly

[hodgesf]

After reviewing the code with Dr. Ramsey and Adil, we confirmed there are no code changes between the last known stable build and the current version. To validate this, I built kg2.10.2c on an r5a.8xlarge instance and was able to successfully query it using the current codebase. This strongly suggests the failure is due to either an ITRB configuration issue or an out-of-memory condition. We have asked Kanna to increase the instance memory to see whether the issue persists.