Content creation issue for QuestionSet type #821

simhead · 2025-09-10T03:28:42Z

simhead
Sep 10, 2025

As below, I am trying to create 'QuestionSet' Content and this is the error I am getting:

Version: v7.7.0 tag

Error Details:

POST /action/questionset/v2/create HTTP/1.1
Host: sunbird-ed-terra.oke.ocloud.today
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:143.0) Gecko/20100101 Firefox/143.0
Accept: application/json
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
X-Source: web
ts: 2025-09-10T13:03:32+10:00
X-msgid: d1f059a3-6fa4-4fb8-89e7-87fae843b899
X-Request-ID: d1f059a3-6fa4-4fb8-89e7-87fae843b899
X-App-Version: 7.0.0
X-Session-ID: 45wXPR7nMG5LbA2TAWp0F0k1qedI6B0l
X-Device-ID: 145785e16d3707eb384536852930931a
X-App-Id: staging.sunbird.portal
X-User-ID: 8eaec873-c728-440e-80ed-03c13d6d1330
Content-Type: application/json
Content-Length: 302
Origin: https://sunbird-ed-terra.oke.ocloud.today
Connection: keep-alive
Referer: https://sunbird-ed-terra.oke.ocloud.today/workspace/content/create/questionset
Cookie: connect.sid=s%3A45wXPR7nMG5LbA2TAWp0F0k1qedI6B0l.S5keCIalxaCB%2BFqk6huQXzCUWm35UH1EG0WAG5AgdYY
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
Priority: u=0
Request:
{"request":{"questionset":{"name":"Untitled QuestionSet","mimeType":"application/vnd.sunbird.questionset","primaryCategory":"Practice Question Set","createdBy":"8eaec873-c728-440e-80ed-03c13d6d1330","createdFor":["0143966106886799360"],"framework":"NCF","code":"5ee76bff-769e-45ad-9e4a-20c015b994b4"}}}
Response:
{"id":"api.questionset.create","ver":"5.0","ts":"2025-09-10T03:03:33ZZ","params":{"resmsgid":"7827dbf8-522a-4afe-a6d2-7fa992a906b8","msgid":null,"err":"ERR_SYSTEM_EXCEPTION","status":"failed","errmsg":"Something went wrong in server while processing the request"},"responseCode":"SERVER_ERROR","result":{}}

What could be missing?

JeraldJF · 2025-09-10T05:08:49Z

JeraldJF
Sep 10, 2025

@simhead Could you please share the logs from the assessment service when the API is triggered. This would provide more context on the issue.

0 replies

simhead · 2025-09-10T07:35:15Z

simhead
Sep 10, 2025
Author

Ok, first I saw the same error as in https://github.com/orgs/Sunbird-Ed/discussions/812
So updated schema.base_path= "/home/sunbird/assessment-service-1.0-SNAPSHOT/schemas" to fix the above error.

Now new problem occured as below:

and the message from network monitor:
Request:
{"request":{"resourceId":"do_21439735451621785615","resourceType":"Content","resourceInfo":"{"contentType":"QuestionSet","identifier":"do_21439735451621785615","mimeType":"application/vnd.sunbird.questionset"}","creatorInfo":"{"name":"Content Creator","id":"8eaec873-c728-440e-80ed-03c13d6d1330"}","createdBy":"8eaec873-c728-440e-80ed-03c13d6d1330","isRootOrgAdmin":false}}

POST /content/lock/v1/create HTTP/1.1
Host: sunbird-ed-terra.oke.ocloud.today
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:143.0) Gecko/20100101 Firefox/143.0
Accept: application/json
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
X-Source: web
ts: 2025-09-10T17:32:45+10:00
X-msgid: ca07f1b6-d043-4715-b4fc-3f211d2aef32
X-Request-ID: ca07f1b6-d043-4715-b4fc-3f211d2aef32
X-App-Version: 7.0.0
X-Session-ID: TjNMk2vxEPAIv5o64PdectQF2FdUMBYy
X-Device-ID: 508ada0d65f1e1bdcaea6e61241fe29d
X-Org-code: 0143966106886799360
X-Channel-Id: 0143966106886799360
X-App-Id: staging.sunbird.portal
X-User-ID: 8eaec873-c728-440e-80ed-03c13d6d1330
Content-Type: application/json
Content-Length: 393
Origin: https://sunbird-ed-terra.oke.ocloud.today
Connection: keep-alive
Referer: https://sunbird-ed-terra.oke.ocloud.today/workspace/edit/QuestionSet/do_21439735451621785615/allcontent/Draft
Cookie: connect.sid=s%3ATjNMk2vxEPAIv5o64PdectQF2FdUMBYy.pBq8El5zRvhT7ZprdtlQ5jQFtKubScjUxQHyhlQmIaI
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin

Response:
{"id":"api.lock.create","ver":"1.0","ts":"2025-09-10T07:32:45.805Z","params":{"resmsgid":"582ae1d0-8e18-11f0-8c65-41956bef60ad","msgid":null,"status":"failed","err":"ERR_LOCK_CREATION_FAILED","errmsg":"Something went wrong in server while processing the request"},"responseCode":"CLIENT_ERROR","result":{}}

HTTP/1.1 412 Precondition Failed
Date: Wed, 10 Sep 2025 07:32:45 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 306
Connection: keep-alive
X-DNS-Prefetch-Control: off
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Download-Options: noopen
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
x-ratelimit-limit-hour: 5000
x-ratelimit-remaining-hour: 4994
x-powered-by: Express
vary: X-HTTP-Method-Override
access-control-allow-origin: *
access-control-allow-methods: GET,PUT,POST,PATCH,DELETE,OPTIONS
access-control-allow-headers: *
etag: W/"132-V/eJibvyvVTUYTUm9iSK3Y6iM6Q"
x-envoy-upstream-service-time: 23
x-kong-upstream-latency: 27
x-kong-proxy-latency: 8
via: kong/0.14.1

0 replies

simhead · 2025-09-10T07:36:23Z

simhead
Sep 10, 2025
Author

and here is assessment log:

2025-09-10 07:35:49,786 CSPMetaUtil ::: updateRelativePath util.Map[String, AnyRef] ::: data before url replace :: {name=Untitled QuestionSet, mimeType=application/vnd.sunbird.questionset, primaryCategory=Practice Question Set, createdBy=8eaec873-c728-440e-80ed-03c13d6d1330, createdFor=[0143966106886799360], framework=NCF, code=8032d20e-3896-445b-af17-a72283883ab6, consumerId=81ebf182-7574-4fec-8fd4-39ea1e12052b, channel=0143966106886799360, qumlVersion=1.1, schemaVersion=1.1, visibility=Default, license=CC BY 4.0, contentEncoding=gzip, contentDisposition=inline, compatibilityLevel=6, status=Draft, language=[English], allowAnonymousAccess=Yes, setType=materialised, showTimer=false, showHints=false, showFeedback=false, showSolutions=false, navigationMode=non-linear, allowSkip=Yes, requiresSubmit=No, shuffle=true, containsUserData=No, generateDIALCodes=No, allowBranching=No, trackable={enabled=No, autoBatch=No}, scoreCutoffType=AssessmentLevel}
2025-09-10 07:35:49,786 CSPMetaUtil ::: updateRelativePath util.Map[String, AnyRef] ::: data after url replace :: {code=8032d20e-3896-445b-af17-a72283883ab6, allowSkip=Yes, containsUserData=No, qumlVersion=1.1, channel=0143966106886799360, language=[English], mimeType=application/vnd.sunbird.questionset, showHints=false, scoreCutoffType=AssessmentLevel, primaryCategory=Practice Question Set, contentDisposition=inline, contentEncoding=gzip, generateDIALCodes=No, showSolutions=false, allowAnonymousAccess=Yes, trackable={enabled=No, autoBatch=No}, createdFor=[0143966106886799360], schemaVersion=1.1, visibility=Default, requiresSubmit=No, showTimer=false, consumerId=81ebf182-7574-4fec-8fd4-39ea1e12052b, setType=materialised, showFeedback=false, license=CC BY 4.0, framework=NCF, createdBy=8eaec873-c728-440e-80ed-03c13d6d1330, compatibilityLevel=6, name=Untitled QuestionSet, navigationMode=non-linear, allowBranching=No, shuffle=true, status=Draft}
2025-09-10 07:35:49,804 CSPMetaUtil ::: updateAbsolutePath util.Map[String, AnyRef] ::: data before url replace :: {code=8032d20e-3896-445b-af17-a72283883ab6, allowSkip=Yes, containsUserData=No, qumlVersion=1.1, channel=0143966106886799360, language=[English], mimeType=application/vnd.sunbird.questionset, showHints=false, scoreCutoffType=AssessmentLevel, primaryCategory=Practice Question Set, contentDisposition=inline, contentEncoding=gzip, generateDIALCodes=No, showSolutions=false, allowAnonymousAccess=Yes, trackable={"enabled":"No","autoBatch":"No"}, createdFor=[0143966106886799360], schemaVersion=1.1, visibility=Default, requiresSubmit=No, showTimer=false, consumerId=81ebf182-7574-4fec-8fd4-39ea1e12052b, setType=materialised, showFeedback=false, versionKey=1757489749786, license=CC BY 4.0, framework=NCF, createdBy=8eaec873-c728-440e-80ed-03c13d6d1330, compatibilityLevel=6, name=Untitled QuestionSet, navigationMode=non-linear, allowBranching=No, shuffle=true, status=Draft}
2025-09-10 07:35:49,804 CSPMetaUtil ::: updateAbsolutePath util.Map[String, AnyRef] ::: updateAbsolutePath returnData :: {code=8032d20e-3896-445b-af17-a72283883ab6, allowSkip=Yes, containsUserData=No, qumlVersion=1.1, channel=0143966106886799360, language=[English], mimeType=application/vnd.sunbird.questionset, showHints=false, scoreCutoffType=AssessmentLevel, primaryCategory=Practice Question Set, contentDisposition=inline, contentEncoding=gzip, generateDIALCodes=No, showSolutions=false, allowAnonymousAccess=Yes, trackable={"enabled":"No","autoBatch":"No"}, createdFor=[0143966106886799360], schemaVersion=1.1, visibility=Default, requiresSubmit=No, showTimer=false, consumerId=81ebf182-7574-4fec-8fd4-39ea1e12052b, setType=materialised, showFeedback=false, versionKey=1757489749786, license=CC BY 4.0, framework=NCF, createdBy=8eaec873-c728-440e-80ed-03c13d6d1330, compatibilityLevel=6, name=Untitled QuestionSet, navigationMode=non-linear, allowBranching=No, shuffle=true, status=Draft}
2025-09-10 07:35:49,807 {"eid":"LOG","ets":1757489749806,"ver":"3.0","mid":"LP.1757489749806.348c1031-ceb2-4962-ae08-92c00dbc1d72","actor":{"id":"org.sunbird.learning.platform","type":"System"},"context":{"channel":"in.ekstep","pdata":{"id":"staging.sunbird.portal","pid":"","ver":"1.0"},"env":"assessment","sid":"TjNMk2vxEPAIv5o64PdectQF2FdUMBYy","did":"508ada0d65f1e1bdcaea6e61241fe29d"},"edata":{"level":"INFO","type":"api_access","message":"","params":[{"duration":27},{"protocol":"http"},{"size":292},{"method":"POST"},{"uip":"10.244.6.150"},{"url":"/questionset/v5/create"},{"status":200}]},"syncts":1757489749806}
2025-09-10 07:35:50,113 CSPMetaUtil ::: updateAbsolutePath util.Map[String, AnyRef] ::: data before url replace :: {code=8032d20e-3896-445b-af17-a72283883ab6, allowSkip=Yes, containsUserData=No, qumlVersion=1.1, channel=0143966106886799360, language=[Ljava.lang.String;@51374b9c, mimeType=application/vnd.sunbird.questionset, showHints=false, createdOn=2025-09-10T07:35:49.786+0000, scoreCutoffType=AssessmentLevel, primaryCategory=Practice Question Set, contentDisposition=inline, contentEncoding=gzip, lastUpdatedOn=2025-09-10T07:35:49.786+0000, generateDIALCodes=No, showSolutions=false, trackable={"enabled":"No","autoBatch":"No"}, allowAnonymousAccess=Yes, createdFor=[Ljava.lang.String;@284c2bde, lastStatusChangedOn=2025-09-10T07:35:49.786+0000, schemaVersion=1.1, requiresSubmit=No, visibility=Default, showTimer=false, consumerId=81ebf182-7574-4fec-8fd4-39ea1e12052b, setType=materialised, showFeedback=false, versionKey=1757489749786, license=CC BY 4.0, framework=NCF, createdBy=8eaec873-c728-440e-80ed-03c13d6d1330, compatibilityLevel=6, allowBranching=No, navigationMode=non-linear, name=Untitled QuestionSet, shuffle=true, status=Draft}
2025-09-10 07:35:50,114 CSPMetaUtil ::: updateAbsolutePath util.Map[String, AnyRef] ::: updateAbsolutePath returnData :: {code=8032d20e-3896-445b-af17-a72283883ab6, allowSkip=Yes, containsUserData=No, qumlVersion=1.1, channel=0143966106886799360, language=[Ljava.lang.String;@51374b9c, mimeType=application/vnd.sunbird.questionset, showHints=false, createdOn=2025-09-10T07:35:49.786+0000, scoreCutoffType=AssessmentLevel, primaryCategory=Practice Question Set, contentDisposition=inline, contentEncoding=gzip, lastUpdatedOn=2025-09-10T07:35:49.786+0000, generateDIALCodes=No, showSolutions=false, trackable={"enabled":"No","autoBatch":"No"}, allowAnonymousAccess=Yes, createdFor=[Ljava.lang.String;@284c2bde, lastStatusChangedOn=2025-09-10T07:35:49.786+0000, schemaVersion=1.1, requiresSubmit=No, visibility=Default, showTimer=false, consumerId=81ebf182-7574-4fec-8fd4-39ea1e12052b, setType=materialised, showFeedback=false, versionKey=1757489749786, license=CC BY 4.0, framework=NCF, createdBy=8eaec873-c728-440e-80ed-03c13d6d1330, compatibilityLevel=6, allowBranching=No, navigationMode=non-linear, name=Untitled QuestionSet, shuffle=true, status=Draft}
2025-09-10 07:35:50,119 {"eid":"LOG","ets":1757489750119,"ver":"3.0","mid":"LP.1757489750119.d5cf3397-88ce-4569-9c64-f4c8cb720279","actor":{"id":"org.sunbird.learning.platform","type":"System"},"context":{"channel":"in.ekstep","pdata":{"id":"staging.sunbird.portal","pid":"","ver":"1.0"},"env":"assessment","sid":"TjNMk2vxEPAIv5o64PdectQF2FdUMBYy","did":"508ada0d65f1e1bdcaea6e61241fe29d"},"edata":{"level":"INFO","type":"api_access","message":"","params":[{"duration":20},{"protocol":"http"},{"size":502},{"method":"GET"},{"uip":"10.244.6.150"},{"url":"/questionset/v5/read/do_21439735603023052816?fields=createdBy,status,mimeType,contentType,resourceType,collaborators,contentDisposition,primaryCategory,framework,targetFWIds,qumlVersion&mode=edit"},{"status":200}]},"syncts":1757489750119}

0 replies

simhead · 2025-09-10T07:41:08Z

simhead
Sep 10, 2025
Author

and interesting log from content service:

javax.json.JsonException: JSON schema does not exist at the path "/home/sunbird/content-service-1.0-SNAPSHOT/schemas/questionset/1.1/schema.json".
	at org.leadpony.justify.internal.schema.io.JsonSchemaReaderFactoryImpl.newJsonException(JsonSchemaReaderFactoryImpl.java:166)
	at org.leadpony.justify.internal.schema.io.JsonSchemaReaderFactoryImpl.createSchemaReader(JsonSchemaReaderFactoryImpl.java:115)
	at org.sunbird.schema.impl.BaseSchemaValidator.readSchema(BaseSchemaValidator.java:78)
	at org.sunbird.schema.impl.JsonSchemaValidator.loadSchema(JsonSchemaValidator.java:40)
	at org.sunbird.schema.impl.JsonSchemaValidator.<init>(JsonSchemaValidator.java:24)
	at org.sunbird.schema.SchemaValidatorFactory.initSchema(SchemaValidatorFactory.java:49)
	at org.sunbird.schema.SchemaValidatorFactory.getInstance(SchemaValidatorFactory.java:21)
	at org.sunbird.graph.schema.IDefinition.<init>(IDefinition.scala:11)
	at org.sunbird.graph.schema.validator.BaseDefinitionNode.<init>(BaseDefinitionNode.scala:17)
	at org.sunbird.graph.schema.DefinitionDTO.<init>(DefinitionDTO.scala:17)
	at org.sunbird.graph.schema.DefinitionFactory$.$anonfun$getDefinition$1(DefinitionFactory.scala:14)
	at scala.collection.immutable.Map$Map4.getOrElse(Map.scala:324)
	at org.sunbird.graph.schema.DefinitionFactory$.getDefinition(DefinitionFactory.scala:14)
	at org.sunbird.graph.schema.DefinitionNode$.getExternalProps(DefinitionNode.scala:35)
	at org.sunbird.graph.nodes.DataNode$.$anonfun$read$1(DataNode.scala:66)
	at scala.util.Success.$anonfun$map$1(Try.scala:255)

and when I go to content service to check for "/home/sunbird/content-service-1.0-SNAPSHOT/schemas/questionset/1.1/schema.json" and this is missing

1 reply

JeraldJF Sep 10, 2025

@simhead To resolve the issue, can you update the schema.base_path to point here https://github.com/project-sunbird/sunbird-ed-installer/blob/331aa19221e27ac8edf8c453c4c7d504052a9fa7/helmcharts/knowledgebb/charts/content/configs/application.conf#L352
which will point to the blob.

simhead · 2025-09-11T00:21:47Z

simhead
Sep 11, 2025
Author

@JeraldJF that link is using:
schema.base_path="{{ include "common.tplvalues.render" (dict "value" .Values.global.schema.base_path "context" $) }}"
but in your template/global-values.yaml does not have this variable "global.schema.base_path" defined.

Where is it from?

1 reply

aimansharief Sep 11, 2025

@simhead It will pick the value from here - https://github.com/project-sunbird/sunbird-ed-installer/blob/331aa19221e27ac8edf8c453c4c7d504052a9fa7/helmcharts/knowledgebb/values.yaml#L173

simhead · 2025-09-11T06:56:39Z

simhead
Sep 11, 2025
Author

@JeraldJF, ok I changed and it is setup as below:
schema.base_path="https://idnlppwjcf2n.compat.objectstorage.us-ashburn-1.oraclecloud.com/ed-ocioketest-public-b67a2dc1f5/schemas/local"

but I am getting this error:

javax.json.stream.JsonParsingException: Unexpected char 0 at (line no=89, column no=19, offset=1957)
	at org.glassfish.json.JsonTokenizer.unexpectedChar(JsonTokenizer.java:577)
	at org.glassfish.json.JsonTokenizer.readString(JsonTokenizer.java:165)
	at org.glassfish.json.JsonTokenizer.nextToken(JsonTokenizer.java:355)
	at org.glassfish.json.JsonParserImpl$ArrayContext.getNextEvent(JsonParserImpl.java:535)
	at org.glassfish.json.JsonParserImpl.next(JsonParserImpl.java:352)
	at org.leadpony.justify.internal.base.json.JsonParserDecorator.fetchNextEvent(JsonParserDecorator.java:110)
	at org.leadpony.justify.internal.base.json.AbstractStreamJsonParser.next(AbstractStreamJsonParser.java:52)
	at org.leadpony.justify.internal.schema.io.JsonSchemaReaderImpl.parseArray(JsonSchemaReaderImpl.java:205)
	at org.leadpony.justify.internal.schema.io.JsonSchemaReaderImpl.parseValue(JsonSchemaReaderImpl.java:185)
	at org.leadpony.justify.internal.schema.io.JsonSchemaReaderImpl.parseObject(JsonSchemaReaderImpl.java:227)
	at org.leadpony.justify.internal.schema.io.JsonSchemaReaderImpl.parseValue(JsonSchemaReaderImpl.java:187)
	at org.leadpony.justify.internal.schema.io.JsonSchemaReaderImpl.parseObject(JsonSchemaReaderImpl.java:227)
	at org.leadpony.justify.internal.schema.io.JsonSchemaReaderImpl.parseValue(JsonSchemaReaderImpl.java:187)
	at org.leadpony.justify.internal.schema.io.JsonSchemaReaderImpl.parseObject(JsonSchemaReaderImpl.java:227)
	at org.leadpony.justify.internal.schema.io.JsonSchemaReaderImpl.parseValue(JsonSchemaReaderImpl.java:187)
	at org.leadpony.justify.internal.schema.io.JsonSchemaReaderImpl.readRootSchema(JsonSchemaReaderImpl.java:174)
	at org.leadpony.justify.internal.schema.io.JsonSchemaReaderImpl.readSchema(JsonSchemaReaderImpl.java:97)
	at org.leadpony.justify.internal.schema.io.AbstractJsonSchemaReader.read(AbstractJsonSchemaReader.java:73)
	at org.leadpony.justify.internal.schema.io.AbstractProbeSchemaReader.readSchema(AbstractProbeSchemaReader.java:53)
	at org.leadpony.justify.internal.schema.io.AbstractJsonSchemaReader.read(AbstractJsonSchemaReader.java:73)
	at org.sunbird.schema.impl.BaseSchemaValidator.readSchema(BaseSchemaValidator.java:73)
	at org.sunbird.schema.impl.JsonSchemaValidator.loadSchema(JsonSchemaValidator.java:37)
	at org.sunbird.schema.impl.JsonSchemaValidator.<init>(JsonSchemaValidator.java:24)
	at org.sunbird.schema.SchemaValidatorFactory.initSchema(SchemaValidatorFactory.java:49)
	at org.sunbird.schema.SchemaValidatorFactory.getInstance(SchemaValidatorFactory.java:21)
	at org.sunbird.graph.schema.IDefinition.<init>(IDefinition.scala:11)
	at org.sunbird.graph.schema.validator.BaseDefinitionNode.<init>(BaseDefinitionNode.scala:17)
	at org.sunbird.graph.schema.DefinitionDTO.<init>(DefinitionDTO.scala:17)
	at org.sunbird.graph.schema.DefinitionFactory$.$anonfun$getDefinition$1(DefinitionFactory.scala:14)
	at scala.collection.immutable.Map$EmptyMap$.getOrElse(Map.scala:107)
	at org.sunbird.graph.schema.DefinitionFactory$.getDefinition(DefinitionFactory.scala:14)
	at org.sunbird.graph.schema.DefinitionNode$.getNode(DefinitionNode.scala:74)
	at org.sunbird.graph.nodes.DataNode$.read(DataNode.scala:56)
	at org.sunbird.content.actors.ContentActor.read(ContentActor.scala:95)
	at org.sunbird.content.actors.ContentActor.onReceive(ContentActor.scala:43)
	at org.sunbird.actor.core.BaseActor.internalOnReceive(BaseActor.java:29)
	at org.sunbird.actor.core.BaseActor.lambda$createReceive$0(BaseActor.java:43)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
	at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
	at akka.actor.Actor.aroundReceive(Actor.scala:539)
	at akka.actor.Actor.aroundReceive$(Actor.scala:537)
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:226)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:610)
	at akka.actor.ActorCell.invoke(ActorCell.scala:579)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
	at akka.dispatch.Mailbox.run(Mailbox.scala:229)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Exception occurred - class :javax.json.stream.JsonParsingException with message :Unexpected char 0 at (line no=89, column no=19, offset=1957)
2025-09-11 06:48:00,117 {"eid":"LOG","ets":1757573280116,"ver":"3.0","mid":"LP.1757573280116.fbfa3b91-d972-4ecb-ac88-2d800288bf55","actor":{"id":"org.sunbird.learning.platform","type":"System"},"context":{"channel":"in.ekstep","pdata":{"id":"staging.sunbird.portal","pid":"","ver":"1.0"},"env":"content","sid":"TjNMk2vxEPAIv5o64PdectQF2FdUMBYy","did":"508ada0d65f1e1bdcaea6e61241fe29d"},"edata":{"level":"INFO","type":"api_access","message":"","params":[{"duration":911},{"protocol":"http"},{"size":300},{"method":"GET"},{"uip":"10.0.2.235"},{"url":"/content/v3/read/do_214398040296333312119?mode=edit"},{"status":500}]},"syncts":1757573280116}

0 replies

aimansharief · 2025-09-12T04:56:00Z

aimansharief
Sep 12, 2025

@simhead can we get into a call and look into this?

0 replies

simhead · 2025-09-12T05:50:58Z

simhead
Sep 12, 2025
Author

@aimansharief , sure how about either Wed or Thur?

1 reply

aimansharief Sep 12, 2025

@simhead That Works for me.

simhead · 2025-09-12T06:00:26Z

simhead
Sep 12, 2025
Author

great! @JeraldJF or @aimansharief can you please share meeting invite? I can open zoom meeting but I need email address

1 reply

aimansharief Sep 12, 2025

@simhead Here are the email ids - aimansharief@sanketika.in , jeraldj@sanketika.in . Please setup a call.

simhead · 2025-09-12T06:34:01Z

simhead
Sep 12, 2025
Author

zoom invite sent.

0 replies

simhead · 2025-09-17T06:42:02Z

simhead
Sep 17, 2025
Author

@aimansharief , please share new image tag to try out

1 reply

JeraldJF Sep 23, 2025

@simhead For the content service, please use the image ghcr.io/sunbird-knowlg/knowledge-platform/content-service:release-6.1.0_rc8_d630a49. Once updated, please modify the config map to point the schema path to the blob path, as referenced in our previous messages. Can you try this out and let me know.

simhead · 2025-09-23T05:37:35Z

simhead
Sep 23, 2025
Author

@JeraldJF , looks like earlier issue has been resolved but encountered another issue.
Say I created a questionset and send for Review:

then from Content Reviewer account, approve it:

then nothing happens. If I refresh browser, I still see QuestionSet is still in "Review" state as below:

I checked content, content job, content task pods but no errors found.

Please advise

1 reply

JeraldJF Sep 23, 2025

@simhead Ok, can you just check the logs of async-questionset-publish-taskmanager for any error, on clicking of publish button.

simhead · 2025-09-23T06:17:53Z

simhead
Sep 23, 2025
Author

@JeraldJF
logs are here:

2025-09-23 06:11:38,143 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (fa2d301a05cbd91a2631e5fcac27cbdf) switched from RUNNING to FAILED on 10.244.5.34:6122-53b8fc @ 10-244-5-34.async-questionset-publish-taskmanager.sunbird.svc.cluster.local (dataPort=44213).
java.lang.Exception: Error While Generating FULL ECAR Bundle For : do_214406495294537728127
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:168) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl$(QuestionPublisher.scala:141) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.updateArtifactUrl(QuestionPublishUtil.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.$anonfun$publishQuestions$2(QuestionPublishUtil.scala:35) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.collection.immutable.List.map(List.scala:286) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.publishQuestions(QuestionPublishUtil.scala:23) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:96) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:27) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:82) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.ProcessOperator$ContextImpl.output(ProcessOperator.java:103) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:46) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: org.sunbird.cloud.storage.exception.StorageServiceException: Unknown storage type found
	at org.sunbird.cloud.storage.factory.StorageServiceFactory$.getStorageService(StorageServiceFactory.scala:29) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.util.CloudStorageUtil.getService(CloudStorageUtil.scala:22) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.util.CloudStorageUtil.deleteFile(CloudStorageUtil.scala:57) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.uploadArtifactToCloud(QuestionPublisher.scala:195) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:161) ~[async-questionset-publish-1.0.0.jar:?]
	... 31 more
2025-09-23 06:11:38,154 INFO  org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task ffd47792f88da570f313a05a9be7b6b0_0.
2025-09-23 06:11:38,155 INFO  org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 2 tasks should be restarted to recover the failed task ffd47792f88da570f313a05a9be7b6b0_0.
2025-09-23 06:11:38,156 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job async-questionset-publish (4d766e38d73f19e2f89511771fada79c) switched from state RUNNING to RESTARTING.
2025-09-23 06:11:38,157 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (8f51193335207911f24ef6946034ac24) switched from RUNNING to CANCELING.
2025-09-23 06:11:38,171 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (8f51193335207911f24ef6946034ac24) switched from CANCELING to CANCELED.
2025-09-23 06:11:38,173 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Clearing resource requirements of job 4d766e38d73f19e2f89511771fada79c
2025-09-23 06:12:08,164 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job async-questionset-publish (4d766e38d73f19e2f89511771fada79c) switched from state RESTARTING to RUNNING.
2025-09-23 06:12:08,166 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - No checkpoint found during restore.
2025-09-23 06:12:08,166 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (74fc99982c310bee1c350c7bd91c5515) switched from CREATED to SCHEDULED.
2025-09-23 06:12:08,166 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (5e9f87a99a1432d37896d33b03227124) switched from CREATED to SCHEDULED.
2025-09-23 06:12:08,167 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job 4d766e38d73f19e2f89511771fada79c: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
2025-09-23 06:12:08,168 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (74fc99982c310bee1c350c7bd91c5515) switched from SCHEDULED to DEPLOYING.
2025-09-23 06:12:08,168 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: questionset-publish-consumer (1/1) (attempt #1) with attempt id 74fc99982c310bee1c350c7bd91c5515 to 10.244.5.34:6122-53b8fc @ 10-244-5-34.async-questionset-publish-taskmanager.sunbird.svc.cluster.local (dataPort=44213) with allocation id 855f5346e60e9e9ea6b118eb547e9d1f
2025-09-23 06:12:08,168 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (5e9f87a99a1432d37896d33b03227124) switched from SCHEDULED to DEPLOYING.
2025-09-23 06:12:08,168 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (attempt #1) with attempt id 5e9f87a99a1432d37896d33b03227124 to 10.244.5.34:6122-53b8fc @ 10-244-5-34.async-questionset-publish-taskmanager.sunbird.svc.cluster.local (dataPort=44213) with allocation id 855f5346e60e9e9ea6b118eb547e9d1f
2025-09-23 06:12:08,176 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (74fc99982c310bee1c350c7bd91c5515) switched from DEPLOYING to INITIALIZING.
2025-09-23 06:12:08,181 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (5e9f87a99a1432d37896d33b03227124) switched from DEPLOYING to INITIALIZING.
2025-09-23 06:12:08,192 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (74fc99982c310bee1c350c7bd91c5515) switched from INITIALIZING to RUNNING.
2025-09-23 06:12:08,521 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (5e9f87a99a1432d37896d33b03227124) switched from INITIALIZING to RUNNING.
2025-09-23 06:12:08,728 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (5e9f87a99a1432d37896d33b03227124) switched from RUNNING to FAILED on 10.244.5.34:6122-53b8fc @ 10-244-5-34.async-questionset-publish-taskmanager.sunbird.svc.cluster.local (dataPort=44213).
java.lang.Exception: Error While Generating FULL ECAR Bundle For : do_214406495294537728127
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:168) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl$(QuestionPublisher.scala:141) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.updateArtifactUrl(QuestionPublishUtil.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.$anonfun$publishQuestions$2(QuestionPublishUtil.scala:35) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.collection.immutable.List.map(List.scala:286) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.publishQuestions(QuestionPublishUtil.scala:23) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:96) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:27) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:82) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.ProcessOperator$ContextImpl.output(ProcessOperator.java:103) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:46) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: org.sunbird.cloud.storage.exception.StorageServiceException: Unknown storage type found
	at org.sunbird.cloud.storage.factory.StorageServiceFactory$.getStorageService(StorageServiceFactory.scala:29) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.util.CloudStorageUtil.getService(CloudStorageUtil.scala:22) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.util.CloudStorageUtil.deleteFile(CloudStorageUtil.scala:57) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.uploadArtifactToCloud(QuestionPublisher.scala:195) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:161) ~[async-questionset-publish-1.0.0.jar:?]
	... 31 more
2025-09-23 06:12:08,730 INFO  org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task ffd47792f88da570f313a05a9be7b6b0_0.
2025-09-23 06:12:08,730 INFO  org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 2 tasks should be restarted to recover the failed task ffd47792f88da570f313a05a9be7b6b0_0.
2025-09-23 06:12:08,730 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job async-questionset-publish (4d766e38d73f19e2f89511771fada79c) switched from state RUNNING to RESTARTING.
2025-09-23 06:12:08,730 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (74fc99982c310bee1c350c7bd91c5515) switched from RUNNING to CANCELING.
2025-09-23 06:12:08,747 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (74fc99982c310bee1c350c7bd91c5515) switched from CANCELING to CANCELED.
2025-09-23 06:12:08,748 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Clearing resource requirements of job 4d766e38d73f19e2f89511771fada79c
2025-09-23 06:12:38,732 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job async-questionset-publish (4d766e38d73f19e2f89511771fada79c) switched from state RESTARTING to RUNNING.
2025-09-23 06:12:38,734 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - No checkpoint found during restore.
2025-09-23 06:12:38,734 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (e6a91bcdae4ee80f7f093e2130d2136c) switched from CREATED to SCHEDULED.
2025-09-23 06:12:38,734 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (cd6a9b4ce24e971e28534ba17769f1d0) switched from CREATED to SCHEDULED.
2025-09-23 06:12:38,735 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job 4d766e38d73f19e2f89511771fada79c: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
2025-09-23 06:12:38,735 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (e6a91bcdae4ee80f7f093e2130d2136c) switched from SCHEDULED to DEPLOYING.
2025-09-23 06:12:38,735 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: questionset-publish-consumer (1/1) (attempt #2) with attempt id e6a91bcdae4ee80f7f093e2130d2136c to 10.244.5.34:6122-53b8fc @ 10-244-5-34.async-questionset-publish-taskmanager.sunbird.svc.cluster.local (dataPort=44213) with allocation id 855f5346e60e9e9ea6b118eb547e9d1f
2025-09-23 06:12:38,735 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (cd6a9b4ce24e971e28534ba17769f1d0) switched from SCHEDULED to DEPLOYING.
2025-09-23 06:12:38,736 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (attempt #2) with attempt id cd6a9b4ce24e971e28534ba17769f1d0 to 10.244.5.34:6122-53b8fc @ 10-244-5-34.async-questionset-publish-taskmanager.sunbird.svc.cluster.local (dataPort=44213) with allocation id 855f5346e60e9e9ea6b118eb547e9d1f
2025-09-23 06:12:38,744 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (e6a91bcdae4ee80f7f093e2130d2136c) switched from DEPLOYING to INITIALIZING.
2025-09-23 06:12:38,751 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (cd6a9b4ce24e971e28534ba17769f1d0) switched from DEPLOYING to INITIALIZING.
2025-09-23 06:12:38,756 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (e6a91bcdae4ee80f7f093e2130d2136c) switched from INITIALIZING to RUNNING.
2025-09-23 06:12:39,022 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (cd6a9b4ce24e971e28534ba17769f1d0) switched from INITIALIZING to RUNNING.
2025-09-23 06:12:39,154 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (cd6a9b4ce24e971e28534ba17769f1d0) switched from RUNNING to FAILED on 10.244.5.34:6122-53b8fc @ 10-244-5-34.async-questionset-publish-taskmanager.sunbird.svc.cluster.local (dataPort=44213).
java.lang.Exception: Error While Generating FULL ECAR Bundle For : do_214406495294537728127
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:168) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl$(QuestionPublisher.scala:141) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.updateArtifactUrl(QuestionPublishUtil.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.$anonfun$publishQuestions$2(QuestionPublishUtil.scala:35) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.collection.immutable.List.map(List.scala:286) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.publishQuestions(QuestionPublishUtil.scala:23) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:96) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:27) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:82) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.ProcessOperator$ContextImpl.output(ProcessOperator.java:103) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:46) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: org.sunbird.cloud.storage.exception.StorageServiceException: Unknown storage type found
	at org.sunbird.cloud.storage.factory.StorageServiceFactory$.getStorageService(StorageServiceFactory.scala:29) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.util.CloudStorageUtil.getService(CloudStorageUtil.scala:22) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.util.CloudStorageUtil.deleteFile(CloudStorageUtil.scala:57) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.uploadArtifactToCloud(QuestionPublisher.scala:195) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:161) ~[async-questionset-publish-1.0.0.jar:?]
	... 31 more
2025-09-23 06:12:39,157 INFO  org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task ffd47792f88da570f313a05a9be7b6b0_0.
2025-09-23 06:12:39,157 INFO  org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 2 tasks should be restarted to recover the failed task ffd47792f88da570f313a05a9be7b6b0_0.
2025-09-23 06:12:39,157 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job async-questionset-publish (4d766e38d73f19e2f89511771fada79c) switched from state RUNNING to RESTARTING.
2025-09-23 06:12:39,157 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (e6a91bcdae4ee80f7f093e2130d2136c) switched from RUNNING to CANCELING.
2025-09-23 06:12:39,172 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (e6a91bcdae4ee80f7f093e2130d2136c) switched from CANCELING to CANCELED.
2025-09-23 06:12:39,173 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Clearing resource requirements of job 4d766e38d73f19e2f89511771fada79c
2025-09-23 06:13:09,159 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job async-questionset-publish (4d766e38d73f19e2f89511771fada79c) switched from state RESTARTING to RUNNING.
2025-09-23 06:13:09,160 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - No checkpoint found during restore.
2025-09-23 06:13:09,160 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (d4421eb0390b83e9ae541bbf62a6062e) switched from CREATED to SCHEDULED.
2025-09-23 06:13:09,160 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (978c8bd7e34da85a6e53b1b0de86278e) switched from CREATED to SCHEDULED.
2025-09-23 06:13:09,161 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job 4d766e38d73f19e2f89511771fada79c: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=1}]
2025-09-23 06:13:09,161 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (d4421eb0390b83e9ae541bbf62a6062e) switched from SCHEDULED to DEPLOYING.
2025-09-23 06:13:09,161 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying Source: questionset-publish-consumer (1/1) (attempt #3) with attempt id d4421eb0390b83e9ae541bbf62a6062e to 10.244.5.34:6122-53b8fc @ 10-244-5-34.async-questionset-publish-taskmanager.sunbird.svc.cluster.local (dataPort=44213) with allocation id 855f5346e60e9e9ea6b118eb547e9d1f
2025-09-23 06:13:09,161 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (978c8bd7e34da85a6e53b1b0de86278e) switched from SCHEDULED to DEPLOYING.
2025-09-23 06:13:09,161 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Deploying publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (attempt #3) with attempt id 978c8bd7e34da85a6e53b1b0de86278e to 10.244.5.34:6122-53b8fc @ 10-244-5-34.async-questionset-publish-taskmanager.sunbird.svc.cluster.local (dataPort=44213) with allocation id 855f5346e60e9e9ea6b118eb547e9d1f
2025-09-23 06:13:09,168 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (d4421eb0390b83e9ae541bbf62a6062e) switched from DEPLOYING to INITIALIZING.
2025-09-23 06:13:09,184 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (978c8bd7e34da85a6e53b1b0de86278e) switched from DEPLOYING to INITIALIZING.
2025-09-23 06:13:09,194 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (d4421eb0390b83e9ae541bbf62a6062e) switched from INITIALIZING to RUNNING.
2025-09-23 06:13:09,459 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (978c8bd7e34da85a6e53b1b0de86278e) switched from INITIALIZING to RUNNING.
2025-09-23 06:13:09,574 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - publish-event-router -> (question-publish-process, questionset-publish-process) (1/1) (978c8bd7e34da85a6e53b1b0de86278e) switched from RUNNING to FAILED on 10.244.5.34:6122-53b8fc @ 10-244-5-34.async-questionset-publish-taskmanager.sunbird.svc.cluster.local (dataPort=44213).
java.lang.Exception: Error While Generating FULL ECAR Bundle For : do_214406495294537728127
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:168) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl$(QuestionPublisher.scala:141) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.updateArtifactUrl(QuestionPublishUtil.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.$anonfun$publishQuestions$2(QuestionPublishUtil.scala:35) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.collection.immutable.List.map(List.scala:286) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.publishQuestions(QuestionPublishUtil.scala:23) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:96) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:27) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:82) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.ProcessOperator$ContextImpl.output(ProcessOperator.java:103) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:46) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: org.sunbird.cloud.storage.exception.StorageServiceException: Unknown storage type found
	at org.sunbird.cloud.storage.factory.StorageServiceFactory$.getStorageService(StorageServiceFactory.scala:29) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.util.CloudStorageUtil.getService(CloudStorageUtil.scala:22) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.util.CloudStorageUtil.deleteFile(CloudStorageUtil.scala:57) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.uploadArtifactToCloud(QuestionPublisher.scala:195) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:161) ~[async-questionset-publish-1.0.0.jar:?]
	... 31 more
2025-09-23 06:13:09,577 INFO  org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - Calculating tasks to restart to recover the failed task ffd47792f88da570f313a05a9be7b6b0_0.
2025-09-23 06:13:09,577 INFO  org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy [] - 2 tasks should be restarted to recover the failed task ffd47792f88da570f313a05a9be7b6b0_0.
2025-09-23 06:13:09,578 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job async-questionset-publish (4d766e38d73f19e2f89511771fada79c) switched from state RUNNING to FAILING.
org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=30000)
	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:216) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:206) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:197) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:682) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:435) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at jdk.internal.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) [async-questionset-publish-1.0.0.jar:?]
	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) [async-questionset-publish-1.0.0.jar:?]
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) [async-questionset-publish-1.0.0.jar:?]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [async-questionset-publish-1.0.0.jar:?]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [async-questionset-publish-1.0.0.jar:?]
	at akka.actor.Actor.aroundReceive(Actor.scala:517) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.actor.Actor.aroundReceive$(Actor.scala:515) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.actor.ActorCell.invoke(ActorCell.scala:561) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.12-1.13.5.jar:1.13.5]
Caused by: java.lang.Exception: Error While Generating FULL ECAR Bundle For : do_214406495294537728127
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:168) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl$(QuestionPublisher.scala:141) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.updateArtifactUrl(QuestionPublishUtil.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.$anonfun$publishQuestions$2(QuestionPublishUtil.scala:35) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.collection.immutable.List.map(List.scala:286) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.publishQuestions(QuestionPublishUtil.scala:23) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:96) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:27) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:82) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.ProcessOperator$ContextImpl.output(ProcessOperator.java:103) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:46) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: org.sunbird.cloud.storage.exception.StorageServiceException: Unknown storage type found
	at org.sunbird.cloud.storage.factory.StorageServiceFactory$.getStorageService(StorageServiceFactory.scala:29) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.util.CloudStorageUtil.getService(CloudStorageUtil.scala:22) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.util.CloudStorageUtil.deleteFile(CloudStorageUtil.scala:57) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.uploadArtifactToCloud(QuestionPublisher.scala:195) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:161) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl$(QuestionPublisher.scala:141) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.updateArtifactUrl(QuestionPublishUtil.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.$anonfun$publishQuestions$2(QuestionPublishUtil.scala:35) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.collection.immutable.List.map(List.scala:286) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.publishQuestions(QuestionPublishUtil.scala:23) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:96) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:27) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:82) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.ProcessOperator$ContextImpl.output(ProcessOperator.java:103) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:46) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
2025-09-23 06:13:09,585 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (d4421eb0390b83e9ae541bbf62a6062e) switched from RUNNING to CANCELING.
2025-09-23 06:13:09,661 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Source: questionset-publish-consumer (1/1) (d4421eb0390b83e9ae541bbf62a6062e) switched from CANCELING to CANCELED.
2025-09-23 06:13:09,662 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Clearing resource requirements of job 4d766e38d73f19e2f89511771fada79c
2025-09-23 06:13:09,662 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Job async-questionset-publish (4d766e38d73f19e2f89511771fada79c) switched from state FAILING to FAILED.
org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=30000)
	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:216) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:206) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:197) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:682) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:435) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at jdk.internal.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) [async-questionset-publish-1.0.0.jar:?]
	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) [async-questionset-publish-1.0.0.jar:?]
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) [async-questionset-publish-1.0.0.jar:?]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [async-questionset-publish-1.0.0.jar:?]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) [async-questionset-publish-1.0.0.jar:?]
	at akka.actor.Actor.aroundReceive(Actor.scala:517) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.actor.Actor.aroundReceive$(Actor.scala:515) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.actor.ActorCell.invoke(ActorCell.scala:561) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.12-1.13.5.jar:1.13.5]
Caused by: java.lang.Exception: Error While Generating FULL ECAR Bundle For : do_214406495294537728127
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:168) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl$(QuestionPublisher.scala:141) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.updateArtifactUrl(QuestionPublishUtil.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.$anonfun$publishQuestions$2(QuestionPublishUtil.scala:35) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.collection.immutable.List.map(List.scala:286) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.publishQuestions(QuestionPublishUtil.scala:23) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:96) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:27) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:82) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.ProcessOperator$ContextImpl.output(ProcessOperator.java:103) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:46) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: org.sunbird.cloud.storage.exception.StorageServiceException: Unknown storage type found
	at org.sunbird.cloud.storage.factory.StorageServiceFactory$.getStorageService(StorageServiceFactory.scala:29) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.util.CloudStorageUtil.getService(CloudStorageUtil.scala:22) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.util.CloudStorageUtil.deleteFile(CloudStorageUtil.scala:57) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.uploadArtifactToCloud(QuestionPublisher.scala:195) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:161) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl$(QuestionPublisher.scala:141) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.updateArtifactUrl(QuestionPublishUtil.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.$anonfun$publishQuestions$2(QuestionPublishUtil.scala:35) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.collection.immutable.List.map(List.scala:286) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.publishQuestions(QuestionPublishUtil.scala:23) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:96) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:27) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:82) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.ProcessOperator$ContextImpl.output(ProcessOperator.java:103) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:46) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
2025-09-23 06:13:09,664 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Stopping checkpoint coordinator for job 4d766e38d73f19e2f89511771fada79c.
2025-09-23 06:13:09,665 INFO  org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore [] - Shutting down
2025-09-23 06:13:09,667 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Job 4d766e38d73f19e2f89511771fada79c reached terminal state FAILED.
org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=30000)
	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
	at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:216)
	at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:206)
	at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:197)
	at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:682)
	at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79)
	at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:435)
	at jdk.internal.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212)
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
	at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.Exception: Error While Generating FULL ECAR Bundle For : do_214406495294537728127
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:168)
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl$(QuestionPublisher.scala:141)
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.updateArtifactUrl(QuestionPublishUtil.scala:14)
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.$anonfun$publishQuestions$2(QuestionPublishUtil.scala:35)
	at scala.collection.immutable.List.map(List.scala:286)
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.publishQuestions(QuestionPublishUtil.scala:23)
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:96)
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:27)
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48)
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66)
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71)
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
	at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:82)
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
	at org.apache.flink.streaming.api.operators.ProcessOperator$ContextImpl.output(ProcessOperator.java:103)
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:46)
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:14)
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48)
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66)
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205)
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.sunbird.cloud.storage.exception.StorageServiceException: Unknown storage type found
	at org.sunbird.cloud.storage.factory.StorageServiceFactory$.getStorageService(StorageServiceFactory.scala:29)
	at org.sunbird.job.util.CloudStorageUtil.getService(CloudStorageUtil.scala:22)
	at org.sunbird.job.util.CloudStorageUtil.deleteFile(CloudStorageUtil.scala:57)
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.uploadArtifactToCloud(QuestionPublisher.scala:195)
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:161)
	... 31 more
2025-09-23 06:13:09,669 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Stopping the JobMaster for job async-questionset-publish(4d766e38d73f19e2f89511771fada79c).
2025-09-23 06:13:09,672 INFO  org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool [] - Releasing slot [855f5346e60e9e9ea6b118eb547e9d1f].
2025-09-23 06:13:09,677 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Close ResourceManager connection 160c20f02fae1287b64b8f3e81596462: Stopping JobMaster for job async-questionset-publish(4d766e38d73f19e2f89511771fada79c)..
2025-09-23 06:13:09,678 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Disconnect job manager 00000000000000000000000000000000@akka.tcp://flink@async-questionset-publish-jobmanager:6123/user/rpc/jobmanager_2 for job 4d766e38d73f19e2f89511771fada79c from the resource manager.
2025-09-23 06:13:10,491 INFO  org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap [] - Application FAILED:
java.util.concurrent.CompletionException: org.apache.flink.client.deployment.application.UnsuccessfulExecutionException: Application Status: FAILED
	at org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$unwrapJobResultException$4(ApplicationDispatcherBootstrap.java:304) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]
	at org.apache.flink.client.deployment.application.JobStatusPollingUtils.lambda$null$2(JobStatusPollingUtils.java:101) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]
	at org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]
	at org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1081) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.OnComplete.internal(Future.scala:264) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.OnComplete.internal(Future.scala:261) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:191) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.japi$CallbackBridge.apply(Future.scala:188) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288) ~[async-questionset-publish-1.0.0.jar:?]
	at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:573) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at scala.concurrent.Future.$anonfun$andThen$1(Future.scala:536) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) ~[async-questionset-publish-1.0.0.jar:?]
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) [async-questionset-publish-1.0.0.jar:?]
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85) [async-questionset-publish-1.0.0.jar:?]
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flink-dist_2.12-1.13.5.jar:1.13.5]
Caused by: org.apache.flink.client.deployment.application.UnsuccessfulExecutionException: Application Status: FAILED
	at org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:71) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	... 43 more
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
	at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:60) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	... 43 more
Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=3, backoffTimeMS=30000)
	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:216) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:206) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:197) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:682) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:79) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:435) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at jdk.internal.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) ~[async-questionset-publish-1.0.0.jar:?]
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) ~[async-questionset-publish-1.0.0.jar:?]
	at akka.actor.Actor.aroundReceive(Actor.scala:517) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.actor.Actor.aroundReceive$(Actor.scala:515) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	... 4 more
Caused by: java.lang.Exception: Error While Generating FULL ECAR Bundle For : do_214406495294537728127
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:168) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl$(QuestionPublisher.scala:141) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.updateArtifactUrl(QuestionPublishUtil.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.$anonfun$publishQuestions$2(QuestionPublishUtil.scala:35) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.collection.immutable.List.map(List.scala:286) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.publishQuestions(QuestionPublishUtil.scala:23) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:96) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:27) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:82) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.ProcessOperator$ContextImpl.output(ProcessOperator.java:103) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:46) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: org.sunbird.cloud.storage.exception.StorageServiceException: Unknown storage type found
	at org.sunbird.cloud.storage.factory.StorageServiceFactory$.getStorageService(StorageServiceFactory.scala:29) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.util.CloudStorageUtil.getService(CloudStorageUtil.scala:22) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.util.CloudStorageUtil.deleteFile(CloudStorageUtil.scala:57) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.uploadArtifactToCloud(QuestionPublisher.scala:195) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl(QuestionPublisher.scala:161) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.helpers.QuestionPublisher.updateArtifactUrl$(QuestionPublisher.scala:141) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.updateArtifactUrl(QuestionPublishUtil.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.$anonfun$publishQuestions$2(QuestionPublishUtil.scala:35) ~[async-questionset-publish-1.0.0.jar:?]
	at scala.collection.immutable.List.map(List.scala:286) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.publish.util.QuestionPublishUtil$.publishQuestions(QuestionPublishUtil.scala:23) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:96) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.QuestionSetPublishFunction.processElement(QuestionSetPublishFunction.scala:27) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:82) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.api.operators.ProcessOperator$ContextImpl.output(ProcessOperator.java:103) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:46) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.questionset.function.PublishEventRouter.processElement(PublishEventRouter.scala:14) ~[async-questionset-publish-1.0.0.jar:?]
	at org.sunbird.job.BaseProcessFunction.processElement(BaseProcessFunction.scala:48) ~[async-questionset-publish-1.0.0.jar:?]
	at org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:205) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:66) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:423) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:204) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
2025-09-23 06:13:10,494 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting StandaloneApplicationClusterEntryPoint down with application status FAILED. Diagnostics null.
2025-09-23 06:13:10,494 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shutting down rest endpoint.
2025-09-23 06:13:10,504 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Removing cache directory /tmp/flink-web-4973eac0-2147-4789-87f2-68c705f4ea5f/flink-web-ui
2025-09-23 06:13:10,504 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - http://async-questionset-publish-jobmanager:8081 lost leadership
2025-09-23 06:13:10,504 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shut down complete.
2025-09-23 06:13:10,506 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Shut down cluster because application is in FAILED, diagnostics null.
2025-09-23 06:13:10,506 INFO  org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent [] - Closing components.
2025-09-23 06:13:10,507 INFO  org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunner [] - DefaultDispatcherRunner was revoked the leadership with leader id 00000000-0000-0000-0000-000000000000. Stopping the DispatcherLeaderProcess.
2025-09-23 06:13:10,507 INFO  org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Stopping SessionDispatcherLeaderProcess.
2025-09-23 06:13:10,507 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Stopping dispatcher akka.tcp://flink@async-questionset-publish-jobmanager:6123/user/rpc/dispatcher_1.
2025-09-23 06:13:10,507 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Stopping all currently running jobs of dispatcher akka.tcp://flink@async-questionset-publish-jobmanager:6123/user/rpc/dispatcher_1.
2025-09-23 06:13:10,508 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Closing the slot manager.
2025-09-23 06:13:10,508 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Suspending the slot manager.
2025-09-23 06:13:10,508 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Stopped dispatcher akka.tcp://flink@async-questionset-publish-jobmanager:6123/user/rpc/dispatcher_1.
2025-09-23 06:13:10,510 INFO  org.apache.flink.runtime.blob.BlobServer                     [] - Stopped BLOB server at 0.0.0.0:6124
2025-09-23 06:13:10,512 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Stopping Akka RPC service.
2025-09-23 06:13:10,515 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Stopping Akka RPC service.
2025-09-23 06:13:10,566 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Stopped Akka RPC service.
2025-09-23 06:13:10,645 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Stopped Akka RPC service.
2025-09-23 06:13:10,645 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Terminating cluster entrypoint process StandaloneApplicationClusterEntryPoint with exit code 1443.

0 replies

simhead · 2025-09-23T06:19:45Z

simhead
Sep 23, 2025
Author

just in case if you want cm for async is here:

data:
  base-config: "kafka {\n    broker-servers = \"kafka:9092\"\n    zookeeper = \"zookeeper:2181\"\n
    \   producer {\n      max-request-size = 1.572864e+06\n    }\n    input {\n    user_pii_topic
    = \"test.delete.user.job.request\"\n    ownership_transfer_topic = \"test.user.ownership.transfer\"\n
    \ }\n  groupId = \"test-user-pii-data-updater-group\"\n}\n\njob {\n  env = \"test\"\n
    \ enable.distributed.checkpointing = true\n  statebackend {\n    base.url = \"s3://ed-ocioketest-private-b67a2dc1f5/checkpoint\"
    \n  }\n}\n\n  task {\n    parallelism = 1\n    consumer.parallelism = 1\n    checkpointing.compressed
    = true\n    checkpointing.interval = 60000\n    checkpointing.pause.between.seconds
    = 5000\n    restart-strategy.attempts = 3\n    restart-strategy.delay = 30000
    # in milli-seconds\n  }\n  redis {\n    host = redis-master\n    port = 6379\n
    \ }\n  lms-cassandra {\n    host = \"cassandra\"\n    port = 9042\n  }\n  neo4j
    {\n    routePath = \"bolt://neo4j:7687\"\n    graph = \"domain\"\n  }\n  es {\n
    \     basePath = \"elasticsearch:9200\"\n  }\n  schema {\n    basePath = \"https://idnlppwjcf2n.compat.objectstorage.us-ashburn-1.oraclecloud.com/ed-ocioketest-public-b67a2dc1f5/schemas/local\"\n
    \   supportedVersion = {\n      itemset = \"2.0\"\n    }\n  }"
  config: "include file(\"/data/flink/conf/base-config.conf\")\nkafka {\n  input.topic
    =test.assessment.publish.request\n  post_publish.topic =test.assessment.postpublish.request\n
    \ groupId =test-questionset-publish-group\n}\ntask {\n  consumer.parallelism =
    1\n  parallelism = 1\n  router.parallelism = 1\n}\nquestion {\n  keyspace =test_question_store\n
    \ table = \"question_data\"\n}\nquestionset {\n  keyspace =test_hierarchy_store\n
    \ table = \"questionset_hierarchy\"\n}\nprint_service.base_url = \"http://print:5000\"\n#Cloud
    Storage Config\ncloud_storage_type: \"s3\"\ncloud_storage_key: \"d5886c2c69507044654c9d826931e2b1482d9c60\"
    \ \ncloud_storage_secret: \"rxIH9O+zG0CMnzUrQx16qvu/hRTen5CQgKqsj8TLuYs=\" \ncloud_storage_endpoint:
    \"\"\ncloud_storage_container: \"ed-ocioketest-public-b67a2dc1f5\"\n\nmaster.category.validation.enabled
    =\"Yes\"\ncloudstorage {\n  metadata.replace_absolute_path=false\n  metadata.list=
    [\"appIcon\",\"posterImage\",\"artifactUrl\",\"downloadUrl\",\"variants\",\"previewUrl\",\"pdfUrl\"]\n
    \ relative_path_prefix=\"CONTENT_STORAGE_BASE_PATH\"\n  read_base_path=\"https://idnlppwjcf2n.compat.objectstorage.us-ashburn-1.oraclecloud.com\"\n
    \ write_base_path=[\"https://idnlppwjcf2n.compat.objectstorage.us-ashburn-1.oraclecloud.com\"]\n}\nschema.basePath=\"https://idnlppwjcf2n.compat.objectstorage.us-ashburn-1.oraclecloud.com/ed-ocioketest-public-b67a2dc1f5/schemas/local\""

1 reply

JeraldJF Sep 23, 2025

@simhead For Unknown storage type found error, I see that cloud_storage_type is pointing to "s3" here in the configmap. Can you please check and update the Cloud storage configs with valid ones. Once updated can you try publishing question set again.

simhead · 2025-09-23T07:12:19Z

simhead
Sep 23, 2025
Author

@JeraldJF , great. I changed to "oci" and questionset is live.

But if I click on delete, I am getting this error:

and async or content pods don't have any errors while doing this, exept I get this error in browser network:

DELETE /content/content/v1/retire HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-AU,en;q=0.9,ko-KR;q=0.8,ko;q=0.7,en-GB;q=0.6,en-US;q=0.5,ar;q=0.4
Connection: keep-alive
Content-Length: 55
Content-Type: application/json
Cookie: connect.sid=s%3ArXNxYnHBlguuOZzQTrqXEmoGIArsd7Aa.NCbkZpDdC9nwrUFKKxFg%2FPUWlI%2FGyVlVxN6E0coU6F8
Host: sunbird-ed-terra.oke.ocloud.today
Origin: https://sunbird-ed-terra.oke.ocloud.today
Referer: https://sunbird-ed-terra.oke.ocloud.today/workspace/content/allcontent/1
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36
X-App-Id: staging.sunbird.portal
X-App-Version: 7.0.0
X-Channel-Id: 0143966106886799360
X-Device-ID: 29898784ef083211d28651e97c6576c2
X-Org-code: 0143966106886799360
X-Request-ID: 619c5ebe-d8af-4346-9c2c-6cff2ec9cf6f
X-Session-ID: rXNxYnHBlguuOZzQTrqXEmoGIArsd7Aa
X-Source: web
X-User-ID: 8eaec873-c728-440e-80ed-03c13d6d1330
X-msgid: 619c5ebe-d8af-4346-9c2c-6cff2ec9cf6f
sec-ch-ua: "Chromium";v="140", "Not=A?Brand";v="24", "Google Chrome";v="140"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
ts: 2025-09-23T17:07:15+10:00

Respond:

{
    "id": "api.content.retire",
    "ver": "1.0",
    "ts": "2025-09-23T07:07:15.172Z",
    "params": {
        "resmsgid": "ef359240-984b-11f0-8cd9-fbd0507c7839",
        "msgid": null,
        "status": "failed",
        "err": "ERR_TOKEN_INVALID",
        "errmsg": "Access denied"
    },
    "responseCode": "UNAUTHORIZED_ACCESS",
    "result": {}
}

It is happening just for questionset, other contents can be deleted.

What could be the problem?

2 replies

JeraldJF Sep 23, 2025

@simhead Ok, will be looking into that issue and get back to you with the solution

JeraldJF Sep 23, 2025

@simhead As i see, this is an existing bug which needs to be fixed.
If this is a blocker, please run the content retire API from Postman.

simhead · 2025-09-24T00:35:40Z

simhead
Sep 24, 2025
Author

@JeraldJF is it this one?
'https://staging.sunbirded.org/api/content/v2/retire/ea laborum'
I see there are multiple versions, so v1, v2, v3 etc. which one should I use?

1 reply

JeraldJF Sep 24, 2025

@simhead
You can use
curl --location --globoff --request DELETE '{{hostUrl}}/api/content/v1/retire' \ --header 'Content-Type: application/json' \ --header 'X-Channel-Id: {{channelId}}' \ --header 'Authorization: Bearer {{apiKey}}' \ --header 'x-authenticated-user-token: {{userAccessToken}}' \ --data '{ "request": { "contentIds": [ "do_214321561610330112119" ] } }'

To obtain the accessToken:

1. Import the Postman Collection and Configure Environment

Import the Postman collection from the provided URL: https://github.com/project-sunbird/sunbird-ed-installer/blob/v7.7.0/postman-collection/collectionrelease700.json.
The env.json file, generated during the cluster setup, contains crucial environment variables. From this file, retrieve the values for channelId, apiKey, client_id, and client_secret.

2. Generate a User Token

In the imported collection, go to the "2 - Knowlg >> Cart as Asset" folder.
Within this folder, execute the first two requests. These are pre-configured to generate the necessary user token for authentication. Please provide content creator credentials in the request to generate creator access token

3. Retire the Question Set

In the request body, you will need to provide the identifier of the question set inside the contentIds in place of do_214321561610330112119.
Execute the content retire api.

simhead · 2025-09-26T02:18:37Z

simhead
Sep 26, 2025
Author

@JeraldJF , when I run this curl

curl --location 'https://sunbird-ed-terra.oke.ocloud.today/api/content/v1/read/do_214406492198150144123' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json'

it works but if I run delete curl:

curl --location --request DELETE 'https://sunbird-ed-terra.oke.ocloud.today/api/content/v1/retire' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer <apikey>' \
--header 'x-authenticated-user-token: <user_token>' \
--header 'X-Channel-Id: 0143964915123322880' \
--data '{
    "request": {
        "contentIds": [
            "do_214406492198150144123"
        ]
    }
}'

I am getting:

{
    "id": "api.content.retire",
    "ver": "1.0",
    "ts": "2025-09-26T02:12:32.433Z",
    "params": {
        "resmsgid": "42b6fa10-9a7e-11f0-855e-e97e71a8383b",
        "msgid": null,
        "status": "failed",
        "err": "ERR_TOKEN_INVALID",
        "errmsg": "Access denied"
    },
    "responseCode": "UNAUTHORIZED_ACCESS",
    "result": {}
}

upon checking knowledge-mw service, I see these errors:

[2025-09-26T02:02:16.103] [ERROR] api - {"id":"d35a5460-9a7c-11f0-9aed-79d0688b1284","msg":"validateToken token failed, Invalid token","err":{"err":"INVALID_KID","errCode":"ERR_TOKEN_INVALID","errMsg":"Access denied","responseCode":"UNAUTHORIZED_ACCESS"}}
Offline token - jwt verification is success. kid: <kid-key>
Offline token - jwt verificate error. kid: <kid-key> TokenExpiredError: jwt expired
    at /home/sunbird/mw/content/libs/sb_api_interceptor/node_modules/jsonwebtoken/verify.js:152:21
    at getSecret (/home/sunbird/mw/content/libs/sb_api_interceptor/node_modules/jsonwebtoken/verify.js:90:14)
    at Object.module.exports [as verify] (/home/sunbird/mw/content/libs/sb_api_interceptor/node_modules/jsonwebtoken/verify.js:94:10)
    at ApiInterceptor.validateToken (/home/sunbird/mw/content/libs/sb_api_interceptor/index.js:62:10)
    at validateToken (/home/sunbird/mw/content/middlewares/request.middleware.js:119:18)
    at Layer.handle [as handle_request] (/home/sunbird/mw/content/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/sunbird/mw/content/node_modules/express/lib/router/route.js:137:13)
    at createAndValidateRequestBody (/home/sunbird/mw/content/middlewares/request.middleware.js:85:3)
    at Layer.handle [as handle_request] (/home/sunbird/mw/content/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/sunbird/mw/content/node_modules/express/lib/router/route.js:137:13)
    at compression (/home/sunbird/mw/content/node_modules/compression/index.js:220:5)
    at /home/sunbird/mw/content/middlewares/request.middleware.js:159:7
    at Layer.handle [as handle_request] (/home/sunbird/mw/content/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/sunbird/mw/content/node_modules/express/lib/router/route.js:137:13)
    at /home/sunbird/mw/content/service/healthCheckService.js:180:9
    at Layer.handle [as handle_request] (/home/sunbird/mw/content/node_modules/express/lib/router/layer.js:95:5) {
  expiredAt: 2025-09-10T20:40:25.000Z
}

Looks like keycloak kid key got expired, where can I update or fix this?

1 reply

JeraldJF Sep 26, 2025

@simhead Can you verify whether the api key, channel id and the content creator user token generated are the valid ones and whether they exist in the headers.

simhead · 2025-09-26T06:05:09Z

simhead
Sep 26, 2025
Author

@JeraldJF yes they are valid and I am able to repeately use token service to create and refresh etc.

I think issue is due to "Offline token - jwt verificate error. kid: TokenExpiredError: jwt expired" as I noted earlier.
As you can see from the error: expiredAt: 2025-09-10T20:40:25.000Z, indeed the cert got expired hence I renewed the cert in global-values.yaml to update the cert. But looks like 'keycloak-kids-keys' configmap's 'KEYCLOAK_PUBLIC_KEY_KID' still using old cert.

So the question is how to update or check this cert/key?

3 replies

JeraldJF Sep 26, 2025

@simhead Can we connect to check on the issue.

simhead Sep 26, 2025
Author

@JeraldJF sure how about tue or wed next week? please share your availabilities

JeraldJF Sep 26, 2025

@simhead Tuesday would be fine, please share the meeting details, email id : jeraldj@sanketika.in

simhead · 2025-09-30T07:01:39Z

simhead
Sep 30, 2025
Author

thanks @JeraldJF for the temp fix.
Just for recording, the fix for now is to invoke this specific api:

https://sunbird-ed-terra.oke.ocloud.today/api/questionset/v2/retire/{{questionset_id}}
{ "request": { "questionset": {} } }

0 replies

Content creation issue for QuestionSet type #821

Uh oh!

Uh oh!

Replies: 19 comments · 14 replies

Uh oh!

Uh oh!

simhead Sep 10, 2025 Author

Uh oh!

simhead Sep 10, 2025 Author

Uh oh!

simhead Sep 10, 2025 Author

Uh oh!

Uh oh!

Uh oh!

simhead Sep 11, 2025 Author

Uh oh!

Uh oh!

simhead Sep 11, 2025 Author

Uh oh!

Uh oh!

simhead Sep 12, 2025 Author

Uh oh!

Uh oh!

simhead Sep 12, 2025 Author

Uh oh!

Uh oh!

simhead Sep 12, 2025 Author

Uh oh!

Uh oh!

simhead Sep 17, 2025 Author

Uh oh!

Uh oh!

simhead Sep 23, 2025 Author

Uh oh!

Uh oh!

simhead Sep 23, 2025 Author

Uh oh!

simhead Sep 23, 2025 Author

Uh oh!

Uh oh!

Uh oh!

simhead Sep 23, 2025 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simhead Sep 24, 2025 Author

Uh oh!

Uh oh!

1. Import the Postman Collection and Configure Environment

2. Generate a User Token

3. Retire the Question Set

Uh oh!

Uh oh!

Replies: 19 comments 14 replies

simhead
Sep 10, 2025
Author

simhead
Sep 10, 2025
Author

simhead
Sep 10, 2025
Author

simhead
Sep 11, 2025
Author

simhead
Sep 11, 2025
Author

simhead
Sep 12, 2025
Author

simhead
Sep 12, 2025
Author

simhead
Sep 12, 2025
Author

simhead
Sep 17, 2025
Author

simhead
Sep 23, 2025
Author

simhead
Sep 23, 2025
Author

simhead
Sep 23, 2025
Author

simhead
Sep 23, 2025
Author

simhead
Sep 24, 2025
Author