Player Service says content service is not healthy

[jona-r]

Hello @pallakartheekreddy
Please guide me on this issue

We are facing the following issue
CONTENT_SERVICE_HEALTH_FAILED on /content/composite/v1/search though Content & Knowledge-MW health are OK (LMS 4.10)

More Details

Calls to https://{host}/content/composite/v1/search fail with SERVICE_UNAVAILABLE and CONTENT_SERVICE_HEALTH_FAILED.
However, direct health checks from knowledge-mw and content-service pods report healthy=true.
If we restart the Player pod, composite search briefly returns 200 OK, then reverts to failure.

Postman response -

{
  "id": "api.error",
  "ver": "1.0",
  "ts": "2025-09-09 06:48:34:951+0000",
  "params": {
    "resmsgid": "01b8e970-8d49-11f0-93dd-2b7be56737c2",
    "msgid": null,
    "status": "failed",
    "err": "SERVICE_UNAVAILABLE",
    "errmsg": "Service is unavailable"
  },
  "responseCode": "SERVICE_UNAVAILABLE",
  "result": {
    "check": [
      { "name": "contentservice.api", "err": "CONTENT_SERVICE_HEALTH_FAILED", "errmsg": "content service is not healthy" },
      { "name": "cassandra.db", "healthy": true, "err": "", "errmsg": "" },
      { "name": "cassandra.db", "healthy": true, "err": "", "errmsg": "" },
      { "name": "learnerservice.api", "healthy": true, "err": "", "errmsg": "" },
      { "name": "cassandra.db", "healthy": true, "err": "", "errmsg": "" }
    ]
  }
}

Health from pods (same window)

knowledge-mw:

curl http://localhost:5000/health
{"id":"api.health","ver":"1.0","ts":"2025-09-09T06:54:35.712Z","params":{"resmsgid":"d8c0ac00-8d49-11f0-ac0b-a56d5bd48b35","msgid":"d8b9f540-8d49-11f0-b27d-796604e662b0","status":"successful","err":null,"errmsg":null},"responseCode":"OK","result":{"name":"ContentService","version":"1.0","healthy":true,"check":[{"name":"learnerservice.api","healthy":true,"err":""},{"name":"cassandra.db","healthy":true,"err":""},{"name":"ekstep.api","healthy":true,"err":""}]}}

content-service:

curl http://localhost:9000/health
{"id":"api.content.health","ver":"3.0","ts":"2025-09-09T06:57:56ZZ","params":{"resmsgid":"caef91f9-c8b8-4552-b42d-f4b0b3c0603f","msgid":null,"err":null,"status":"successful","errmsg":null},"responseCode":"OK","result":{"checks":[{"name":"redis cache","healthy":true},{"name":"graph db","healthy":true},{"name":"cassandra db","healthy":true}],"healthy":true}}

Behavioral clue: Restarting the Player pod temporarily restores composite search to 200 OK; after a short period, failures resume.

Impact

• /content/composite/v1/search is unreliable for learners and creators via the gateway.
• Periodic service unavailability affects search and course discovery.

Environment

• Sunbird LMS: 4.10.0
• Affected services: content-service, knowledge-mw, player, gateway (Kong/Nginx)
• Data stores healthy per health checks: Cassandra, Graph/Neo4j, Redis cache, Elastic-search
• Deployment: Kubernetes (pods for content-service, knowledge-mw, player)

Steps to Reproduce

• Call POST https://{host}/content/composite/v1/search repeatedly with a standard payload.
• Observe SERVICE_UNAVAILABLE and CONTENT_SERVICE_HEALTH_FAILED in response.
• From within pods, curl health endpoints for knowledge-mw and content-service → both healthy.
• Restart Player pod → composite search returns 200 OK for a short time, then fails again.

Expected

• Composite search should succeed if downstream health is green.
• Health gating should reflect actual upstream status, not a stale/incorrect failure.

Actual

• Gateway/composite path reports content service unhealthy while direct health shows healthy.
• Temporary recovery after Player restart suggests cache/limiter/circuit state is influencing upstream health.

cc: @vinod303, @komalm

[aimansharief]

Hey @jona-r , If knowledge-mw and content service health checks are okay. There might be some misconfiguration in the player config. Can you verify that once?

[jona-r]

Hi @aimansharief as requested I am attaching the logs
content.txt
knowledge-mw.txt
player.txt

[jona-r]

Hi @aimansharief Any update on this

[jona-r]

Hi @aimansharief if required we could get on a call and check this issue out please let me know you availability.

[pallakartheekreddy]

@jona-r lets connect today 4:00PM

[jona-r]

@pallakartheekreddy
I have set up the meeting for 4:00pm
https://meet.google.com/dex-ufta-hmg