AS3 package state may be stuck at INSTALLING state

[zhaoqin-github]

I attempted to install AS3 rpm via TrustedExtensions api today. I totally attempted 10 times, 3 of them failed. A new BIG-IP VE was provisioned every time, before invoking TrustedExtensions API. However, after invoking API to install AS3 rpm, the rpm package state was stuck at INSTALLING state, am it never moved to AVAILABLE state. There was no error output in ASG container log.

I do not know how to debug AS3 installation in BIG-IP side. If anyone know a way to debug it, please let me know.

Some example output of the failure:

EI-ML-00005209:tmp qzhao$ curl -X GET https://localhost:8443/mgmt/shared/TrustedDevices | jq
{
  "devices": [
    {
      "targetUUID": "c3c81c58-f956-4159-b105-4918162a7426",
      "targetHost": "10.250.15.131",
      "targetPort": 443,
      "state": "ACTIVE",
      "targetHostname": "cd8d67d8-096b-455c-96cc-3227d7cc9e13.f5bigip.local",
      "targetVersion": "13.1.1",
      "targetRESTVersion": "13.1.1-0.0.4",
      "available": true,
      "lastValidated": "2019-05-29T02:07:11.197Z"
    }
  ]
}

BEI-ML-00005209:tmp qzhao$ curl -X GET --insecure https://localhost:8443/mgmt/shared/TrustedExtensions/c3c81c58-f956-4159-b105-4918162a7426 | jq
[
  {
    "rpmFile": "f5-appsvcs-3.10.0-5.noarch.rpm",
    "downloadUrl": "https://github.com/F5Networks/f5-appsvcs-extension/releases/download/v3.10.0/f5-appsvcs-3.10.0-5.noarch.rpm",
    "state": "INSTALLING",
    "name": "",
    "version": "",
    "release": "",
    "arch": "",
    "packageName": "",
    "tags": []
  }
]

BEI-ML-00005209:tmp qzhao$ curl -X GET --insecure https://admin:admin@10.250.15.131:443/mgmt/shared/appsvcs/info |jq
{
  "code": 404,
  "message": "Public URI path not registered. Please see /var/log/restjavad.0.log and /var/log/restnoded/restnoded.log for details.",
  "referer": "10.250.63.27",
  "restOperationId": 4108098,
  "errorStack": [
    "com.f5.rest.common.RestWorkerUriNotFoundException: Public URI path not registered. Please see /var/log/restjavad.0.log and /var/log/restnoded/restnoded.log for details.",
    "at com.f5.rest.workers.ForwarderPassThroughWorker.cloneAndForwardRequest(ForwarderPassThroughWorker.java:572)",
    "at com.f5.rest.workers.ForwarderPassThroughWorker.access$000(ForwarderPassThroughWorker.java:44)",
    "at com.f5.rest.workers.ForwarderPassThroughWorker$1.completed(ForwarderPassThroughWorker.java:314)",
    "at com.f5.rest.workers.ForwarderPassThroughWorker$1.completed(ForwarderPassThroughWorker.java:311)",
    "at com.f5.rest.workers.EvaluatePermissions.completeEvaluatePermission(EvaluatePermissions.java:150)",
    "at com.f5.rest.workers.EvaluatePermissions.evaluatePermission(EvaluatePermissions.java:78)",
    "at com.f5.rest.workers.ForwarderPassThroughWorker.onForward(ForwarderPassThroughWorker.java:342)",
    "at com.f5.rest.workers.ForwarderPassThroughWorker.onGet(ForwarderPassThroughWorker.java:753)",
    "at com.f5.rest.common.RestWorker.callDerivedRestMethod(RestWorker.java:1154)",
    "at com.f5.rest.common.RestWorker.callRestMethodHandler(RestWorker.java:1119)",
    "at com.f5.rest.common.RestServer.processQueuedRequests(RestServer.java:1385)",
    "at com.f5.rest.common.RestServer.access$000(RestServer.java:48)",
    "at com.f5.rest.common.RestServer$1.run(RestServer.java:329)",
    "at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)",
    "at java.util.concurrent.FutureTask.run(FutureTask.java:262)",
    "at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)",
    "at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)",
    "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)",
    "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)",
    "at java.lang.Thread.run(Thread.java:745)\n"
  ],
  "kind": ":resterrorresponse"
}