Operator installation script fails to apply 3 yaml files due to missing namespace

[amitsadaphule]

SUMMARY

I built the operator code and tried to install on k8s cluster by following these steps. But the application of the last 3 yaml files fails due to missing namespace. Log pasted below:

# curl -sL https://raw.githubusercontent.com/IBM/cloud-operators/master/hack/configure-operator.sh | bash -s -- -v 1.0.7 install
secret/ibmcloud-operator-secret unchanged
configmap/ibmcloud-operator-defaults unchanged
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 29049    0 29049    0     0   146k      0 --:--:-- --:--:-- --:--:--  146k
Downloaded:
apiextensions.k8s.io_v1beta1_customresourcedefinition_bindings.ibmcloud.ibm.com.yaml
apiextensions.k8s.io_v1beta1_customresourcedefinition_services.ibmcloud.ibm.com.yaml
apps_v1_deployment_ibmcloud-operator-controller-manager.yaml
monitoring.coreos.com_v1_servicemonitor_ibmcloud-operator-controller-manager-metrics-monitor.yaml
rbac.authorization.k8s.io_v1beta1_clusterrole_ibmcloud-operator-metrics-reader.yaml
rbac.authorization.k8s.io_v1_clusterrolebinding_ibmcloud-operator-manager-rolebinding.yaml
rbac.authorization.k8s.io_v1_clusterrolebinding_ibmcloud-operator-proxy-rolebinding.yaml
rbac.authorization.k8s.io_v1_clusterrole_ibmcloud-operator-manager-role.yaml
rbac.authorization.k8s.io_v1_clusterrole_ibmcloud-operator-proxy-role.yaml
rbac.authorization.k8s.io_v1_rolebinding_ibmcloud-operator-leader-election-rolebinding.yaml
rbac.authorization.k8s.io_v1_role_ibmcloud-operator-leader-election-role.yaml
v1_namespace_ibmcloud-operator-system.yaml
v1_service_ibmcloud-operator-controller-manager-metrics-service.yaml
error: unable to recognize "/tmp/tmp.jPq8p2azYA/monitoring.coreos.com_v1_servicemonitor_ibmcloud-operator-controller-manager-metrics-monitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
Failed to install monitoring, skipping...
Install the Prometheus Operator and re-run this script to include monitoring.
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/bindings.ibmcloud.ibm.com created
customresourcedefinition.apiextensions.k8s.io/services.ibmcloud.ibm.com created
clusterrole.rbac.authorization.k8s.io/ibmcloud-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/ibmcloud-operator-proxy-role created
clusterrolebinding.rbac.authorization.k8s.io/ibmcloud-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/ibmcloud-operator-proxy-rolebinding created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/ibmcloud-operator-metrics-reader created
namespace/ibmcloud-operator-system created
service/ibmcloud-operator-controller-manager-metrics-service created
Error from server (NotFound): error when creating "/tmp/tmp.jPq8p2azYA/apps_v1_deployment_ibmcloud-operator-controller-manager.yaml": namespaces "ibmcloud-operator-system" not found
Error from server (NotFound): error when creating "/tmp/tmp.jPq8p2azYA/rbac.authorization.k8s.io_v1_role_ibmcloud-operator-leader-election-role.yaml": namespaces "ibmcloud-operator-system" not found
Error from server (NotFound): error when creating "/tmp/tmp.jPq8p2azYA/rbac.authorization.k8s.io_v1_rolebinding_ibmcloud-operator-leader-election-rolebinding.yaml": namespaces "ibmcloud-operator-system" not found
+ rm -rf /tmp/tmp.jPq8p2azYA

However, it was seen that the namespace gets created as expected. The issue seems to be that the remaining dependent yaml files are applied too soon. There needs to be some check for namespace before applying the rest of the dependent yamls:

apps_v1_deployment_ibmcloud-operator-controller-manager.yaml
rbac.authorization.k8s.io_v1_role_ibmcloud-operator-leader-election-role.yaml
rbac.authorization.k8s.io_v1_rolebinding_ibmcloud-operator-leader-election-rolebinding.yaml

Manually applying the yamls later works fine and the operator pod comes up properly.

VERSION

v1.0.7

OS, environment

Ubuntu 16.04 x86_64
Go 1.15.7

Steps to reproduce the problem

apt-get install -y make git wget gcc
wget https://golang.org/dl/go1.15.7.linux-amd64.tar.gz
tar -xzf go1.15.7.linux-amd64.tar.gz
rm -rf go1.15.7.linux-amd64.tar.gz
export GOPATH=`pwd`/gopath
export PATH=`pwd`/go/bin:$GOPATH/bin:$PATH
mkdir -p $GOPATH/src/github.com/IBM
cd $GOPATH/src/github.com/IBM
git clone https://github.com/IBM/cloud-operators.git
cd cloud-operators
git checkout v1.0.7
make controller-gen
make kustomize
make -e RELEASE_VERSION=1.0.7 release-prep
make -e RELEASE_VERSION=1.0.7 docker-build

curl -sL https://raw.githubusercontent.com/IBM/cloud-operators/master/hack/configure-operator.sh | bash -s -- -v 1.0.7 install
OR
./hack/configure-operator.sh -v 1.0.7 install

Please help me resolve this issue. Thanks!

[JohnStarich]

@amitsadaphule Thanks for opening. The order yaml files are applied isn't great sometimes. To work around this issue for now, you can retry the installer script multiple times. If a resource is not created on the first pass, a second pass should fix it.

[amitsadaphule]

Thanks @JohnStarich for the response! I'm using that as a workaround currently. :)