From 363af5bd20fad22edd4be322ac834c43c67455db Mon Sep 17 00:00:00 2001 From: Ravi Ranjan Date: Tue, 19 May 2020 12:13:18 +0530 Subject: [PATCH] Updated manager.py for reading secrets to check for the existence of a secret instead of listing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Issue: When someone runs Kubeflow Faring from Kubeflow Pipeline, he/she will get below error: IOError: [Errno 2] No such file or directory: ‘/etc/secrets/user-gcp-sa.json’ Reason: When Kubeflow Fairing tries to launch the pod for model training like LightGBM distributed parallel training, it tries to check the presence of the secret by using list instead of get and when list fails due to lack of permission, it just does not attach any secrets to the pod. This causes the whole Kubeflow pipeline to fail. Description of your changes: The secret_exists function is incorrectly using list_namespaced_secret (https://github.com/kubeflow/fairing/blob/e0d44e870b467bbd773f836a8b1b648b79019dd1/kubeflow/fairing/kubernetes/manager.py#L220) to check for the existence of a secret instead of read_namespaced_secret. The default pipeline-runner SA used by Kubeflow Pipelines does not have the list permission for [secrets](https://github.com/kubeflow/manifests/blob/9e6ef8681003c195560fb7b3b75211830c19b7a0/pipeline/pipelines-runner/base/cluster-role.yaml#L9), so fairing fails to check that the secret exists and fails to attach it. kubeflow/pipelines#3742 This helped me run Kubeflow fairing from the Kubeflow pipeline successfully. --- kubeflow/fairing/kubernetes/manager.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kubeflow/fairing/kubernetes/manager.py b/kubeflow/fairing/kubernetes/manager.py index 71bb5e7c..a849a4b3 100644 --- a/kubeflow/fairing/kubernetes/manager.py +++ b/kubeflow/fairing/kubernetes/manager.py @@ -217,7 +217,7 @@ def secret_exists(self, name, namespace): :returns: bool: True if the secret exists, otherwise return False. """ - secrets = client.CoreV1Api().list_namespaced_secret(namespace) + secrets = client.CoreV1Api().read_namespaced_secret(namespace) secret_names = [secret.metadata.name for secret in secrets.items] return name in secret_names