kubectl get pods :lcm ContainerCreating,prometheus trainer and trainingdata STATUS CrashLoopBackOff

[Earl-chen]

After I installed FfDl according to the prompts, check the status and get the following prompt:

and use helm list,get follow :

Then, for these incorrect pods, I use kubectl describe pods <pods Name> to view the information. The results are as follows:

Is there any friend who can give me some advice? What is the problem with me? I would like to express my heartfelt thanks.

[Tomcli]

Hi @Earl-chen, it looks like the volume configmap is not being created. Can you run the following script to generate the necessary configmap? Thanks.

pushd bin
./create_static_volumes.sh
./create_static_volumes_config.sh
popd

[Earl-chen]

Thank you very much for your help. @Tomcli 
I tried the method you said, and also tried to use the command` helm delete $(helm list | grep ffdl | awk '{print $1}' | head -n 1) `After deleting FfDL, use the command of` helm install .` to install FFDL. The final result is still a failure. The specific description is as follows:

When I just installed FfDL, in the first 10 seconds or so, all the STATUS was found by kubectl get pods.

However, after a while, ffdl-trainer-6777dd5756-rjlfl, prometheus-67fb854b59-mxdrn and ffdl-trainingdata-696b99ff5c-2hsc5 will often become CrashLoopBackOff or ERROR.

After about 10 minutes, ffdl-lcm-8d555c7bf-r6fj8, ffdl-trainer-6777dd5756-rjlfl, ffdl-trainingdata-696b99ff5c-2hsc5 and prometheus-67fb854b59-mxdrn are not becoming Running. At the same time, by command: helm status $(helm list | grep ffdl | awk '{print $1}' | head -n 1) | grep STATUS:
The result is also changed from the previous DEPLOYED to FAILED.

See the details by kubectl describe pods as follows:

I can't understand why this is happening?

[Tomcli]

Thank you for taking the time to redeploy FfDL. It looks like many of the pods failed liveness probe. Which means those microservices might not able to communicate with each other via the KubeDNS server on your cluster. Can you display some logs from your KubeDNS pod in the kube-system namespace?

[Earl-chen]

@Tomcli Thank you for your prompt reply. Unfortunately, I missed KubeDNS. KubeDNS is not installed. I install it now, and then give you feedback.

[Eric-Zhang1990]

@Tomcli I also have the problem above

I run command $kubectl describe pods ffdl-lcm-8d555c7bf-6pg7z --namespace kube-system, the result is:

192.168.110.158 is the node of k8s, I run the command
$pushd bin
$./create_static_volumes.sh
$./create_static_volumes_config.sh
$popd
on 192.168.110.25 (master of k8s) and 192.168.110.158 (node).
How can I solve problem above? Thank you.

[Tomcli]

@Eric-Zhang1990 ./create_static_volumes.sh and ./create_static_volumes_config.sh should able to create the static-volumes and v2 configmap for you. Do you still encounter problems with configmap "static-volumes" not found?

[Eric-Zhang1990]

@Tomcli I can get following info, it shows static-volumes and v2 are there:

However, I restart FfDL, and still encounter problems with "SetUp failed for volume "static-volumes-config-volume-v2" : configmap "static-volumes-v2" not found".

And I rerun the command ./create_static_volumes.sh and ./create_static_volumes_config.sh, I got these:

How can I solve it? Thanks.
Another question is: which kind of type should I set for SHARED_VOLUME_STORAGE_CLASS? Thanks.

[Tomcli]

@Eric-Zhang1990 It looks like you deployed the static-volumes at the default namespace while FfDL is at kube-system namespace. You could deploy FfDL using helm with the namespace flag (e.g. helm install . --set namespace=default) to deploy FfDL on your default namespace.

[Tomcli]

The SHARED_VOLUME_STORAGE_CLASS should be your default storageclass in your Kubernetes cluster. You can check the storageclass with kubectl get storageclass

[Eric-Zhang1990]

@Tomcli Does following state cause the problem above?

The status of static-volume-1 is always in pending, is something wrong?

[Eric-Zhang1990]

The SHARED_VOLUME_STORAGE_CLASS should be your default storageclass in your Kubernetes cluster. You can check the storageclass with kubectl get storageclass

I run command kubectl get storageclass but get nothing.

[Eric-Zhang1990]

It looks like you deployed the static-volumes at the default namespace while FfDL is at kube-system namespace. You could deploy FfDL using helm with the namespace flag (e.g. helm install . --set namespace=default) to deploy FfDL on your default namespace.

Now I change the static-volumes namespace into kube-system namespace, and I also deploy FfDL in kube-system namespace, then pod 'ffdl-lcm' now run ok, but status of pod 'ffdl-trainer' and 'ffdl-trainingdata' are not stable,

Which reason can cause this problem? Thank you.

[Eric-Zhang1990]

@Tomcli After running one more hour, the statues of 'ffdl-trainingdata*' is still changing, sometimes is in status 'running', sometimes is 'CrashLoopBackOff'.

I run command "kubectl describe pods ffdl-trainingdata-74f7cdf66c-lkk2p", get following info:

And I run command "kubectl logs ffdl-trainingdata-74f7cdf66c-lkk2p", logs info are:

time="2019-01-23T07:06:18Z" level=debug msg="Log level set to 'debug'"
time="2019-01-23T07:06:18Z" level=debug msg="Milli CPU is: 60"
time="2019-01-23T07:06:18Z" level=info msg="GetTrainingDataMemInMB() returns 300"
time="2019-01-23T07:06:18Z" level=debug msg="Training Data Mem in MB is: 300"
time="2019-01-23T07:06:18Z" level=debug msg="No config file 'config-dev.yml' found. Using environment variables only."
{"caller_info":"metrics/main.go:36 main -","level":"debug","module":"training-data-service","msg":"function entry","time":"2019-01-23T07:06:18Z"}
{"caller_info":"metrics/main.go:42 main -","level":"debug","module":"training-data-service","msg":"Port is: 8443","time":"2019-01-23T07:06:18Z"}
{"caller_info":"metrics/main.go:44 main -","level":"debug","module":"training-data-service","msg":"Creating dlaas-training-metrics-service","time":"2019-01-23T07:06:18Z"}
{"caller_info":"service/service_impl.go:147 NewService -","level":"debug","module":"training-data-service","msg":"es address #0: http://elasticsearch:9200","time":"2019-01-23T07:06:18Z"}
{"caller_info":"service/service_impl.go:885 createIndexWithLogsIfDoesNotExist -","level":"debug","module":"training-data-service","msg":"function entry","time":"2019-01-23T07:06:18Z"}
{"caller_info":"service/service_impl.go:887 createIndexWithLogsIfDoesNotExist -","level":"info","module":"training-data-service","msg":"calling IndexExists for dlaas_learner_data","time":"2019-01-23T07:06:18Z"}
{"caller_info":"service/service_impl.go:888 createIndexWithLogsIfDoesNotExist -","error":"Head http://elasticsearch:9200/dlaas_learner_data: dial tcp: lookup elasticsearch on 10.254.0.2:53: read udp 172.17.0.6:53791-\u003e10.254.0.2:53: i/o timeout","level":"error","module":"training-data-service","msg":"IndexExists for dlaas_learner_data failed","time":"2019-01-23T07:06:58Z"}
{"caller_info":"elastic.v5/indices_create.go:31 createIndexWithLogsIfDoesNotExist -","level":"debug","module":"training-data-service","msg":"calling CreateIndex","time":"2019-01-23T07:06:58Z"}
{"caller_info":"service/service_impl.go:907 createIndexWithLogsIfDoesNotExist -","error":"no available connection: no Elasticsearch node available","level":"debug","module":"training-data-service","msg":"CreateIndex failed","time":"2019-01-23T07:06:58Z"}
panic: no available connection: no Elasticsearch node available

goroutine 1 [running]:
github.com/IBM/FfDL/metrics/service.NewService(0xc420479f68, 0xe23640)
/Users/tommyli/go/src/github.com/IBM/FfDL/metrics/service/service_impl.go:167 +0x980
main.main()
/Users/tommyli/go/src/github.com/IBM/FfDL/metrics/main.go:44 +0x16c

Is the problem of "no available connection: no Elasticsearch node available"?
Thanks.

[Tomcli]

Thank you for taking time to debug this. Elasticsearch should be part of the storage-0 container. It could be the Elasticsearch service didn't properly enabled. Can you run kubectl get svc to check is the elasticsearch is deployed? Also, you might want to run kubectl logs storage-0 to check is there any error related to elastic search.

Thanks.

[Eric-Zhang1990]

@Tomcli I check elasticsearch is deployed, and logs of storage-0 shows "Failed to find a usable hardware address from the network interfaces; using random bytes: 64:4b:61:9d:da:79:4a:d3", which reason can cause this problem?
Thanks.