From 318f1f893115c810d519978e8716e4619d9867c3 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Fri, 1 Jul 2022 10:02:18 +0530 Subject: [PATCH 01/46] updated with AKS provision workflow in github action --- .../setup-aks-provisioning-pipeline.asciidoc | 9 ++ .../setup-aks-provisioning-pipeline.asciidoc | 107 ++++++++++++++++++ .../setup-aks-provisioning-pipeline.asciidoc | 9 ++ .../azure/aks/terraform.tfvars | 4 +- .../pipelines/github/pipeline_generator.sh | 3 +- .../github/templates/aks/aks-pipeline.cfg | 61 ++++++++++ .../aks/aks-provisioning.yml.template | 106 +++++++++++++++++ .../github/templates/aks/obtain-dns.sh | 21 ++++ .../kubernetes/install-nginx-ingress.sh | 4 + .../common/kubernetes/install-rancher.sh | 7 ++ 10 files changed, 328 insertions(+), 3 deletions(-) create mode 100644 documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc create mode 100644 documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc create mode 100644 documentation/src/github/setup-aks-provisioning-pipeline.asciidoc create mode 100644 scripts/pipelines/github/templates/aks/aks-pipeline.cfg create mode 100644 scripts/pipelines/github/templates/aks/aks-provisioning.yml.template create mode 100644 scripts/pipelines/github/templates/aks/obtain-dns.sh create mode 100644 scripts/pipelines/github/templates/common/kubernetes/install-nginx-ingress.sh create mode 100644 scripts/pipelines/github/templates/common/kubernetes/install-rancher.sh diff --git a/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc new file mode 100644 index 000000000..c9224fd18 --- /dev/null +++ b/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -0,0 +1,9 @@ +:provider: Azure Devops +:pipeline_type: Pipeline +:trigger_sentence: * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. * Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. +:pipeline_type2: pipeline +:path_provider: azure-devops +:usage_type: +:cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w +:aks_variables: Azure DevOps > Pipelines > Library > `aks-variables` +include::../common_templates/setup-build-pipeline.asciidoc[] \ No newline at end of file diff --git a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc new file mode 100644 index 000000000..320b8e0e2 --- /dev/null +++ b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc @@ -0,0 +1,107 @@ += Setting up a Azure AKS provisioning {pipeline_type} on {provider} + +In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. + +The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. + +Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. + +The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. + +== Prerequisites + +{trigger_sentence} + +* An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: + +``` +az group create -n -l +``` + +* An Azure storage account within the previous resource group. You can use an existing one or create a new one with the following command: + +``` +az storage account create -n -g -l +``` + +* An Azure storage container in Azure within the previous storage account. You can use an existing one or create a new one with the following command: + +``` +az storage container create -n --account-name +``` + +* This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`). + +== Creating the {pipeline_type} using provided script + +Before executing the script you will need to customize some input variables about the environment. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/azure/aks`, which allows you to create or update values for the required variables, passing them as flags. As a full example: + +``` +./set-terraform-variables.sh --cluster_name --location --resource_group_name --instance_type --worker_node_count --dns_prefix +``` + +=== Usage +``` +pipeline_generator.sh \ + -c \ + -n \ + -d \ +{usage_type} + --resource-group \ + --storage-account \ + --storage-container \ + [--rancher] \ + [-b ] \ + [-w] +``` + +NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`. + +=== Flags +``` +-c, --config-file [Required] Configuration file containing pipeline definition. +-n, --pipeline-name [Required] Name that will be set to the pipeline. +-d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). +{usage_type} + --resource-group [Required] Name of the resource group for the cluster. + --storage--account [Required] Name of the storage account for the cluster. + --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. + --rancher Install Rancher to manage the cluster. +-b, --target-branch Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided. +-w Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag. +``` + +=== Example + +``` +{cmd_add} +``` + +NOTE: Rancher is installed on the cluster after provisioning when using the above command. + +=== Appendix: Interacting with the cluster + +NOTE: Make sure you have https://kubernetes.io/docs/tasks/tools/#kubectl[kubectl] installed. + +In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning pipeline on the location it is expected by default (`~/.kube/config`) or either: + +``` +# via environment variable (you can add this on your profile) +export KUBECONFIG= +kubectl + +# via command-line flag +kubectl --kubeconfig= +``` + +To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables}. + +Rancher, if installed, will be available on `https:///dashboard`. You will be asked for an initial password, which can be retrieved with: + +``` +kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}' +``` + +=== Appendix: Destroying the cluster + +To destroy the provisioned resources, set `operation` pipeline variable value to `destroy` and run the pipeline. \ No newline at end of file diff --git a/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc new file mode 100644 index 000000000..d4af31d8c --- /dev/null +++ b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc @@ -0,0 +1,9 @@ +:provider: Github +:pipeline_type: workflow +:trigger_sentence: * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`,`AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. +:pipeline_type2: Github action +:path_provider: github +:usage_type: +:cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w +:aks_variables: .github > vars > `aks-variables.env` +include::../common_templates/setup-build-pipeline.asciidoc[] \ No newline at end of file diff --git a/scripts/environment-provisioning/azure/aks/terraform.tfvars b/scripts/environment-provisioning/azure/aks/terraform.tfvars index a18c5b3a8..088b1cbd9 100644 --- a/scripts/environment-provisioning/azure/aks/terraform.tfvars +++ b/scripts/environment-provisioning/azure/aks/terraform.tfvars @@ -1,8 +1,8 @@ -cluster_name = "hangar-k8s" +#cluster_name = "hangar-k8s" location = "westeurope" -resource_group_name = "hangar-aks-rg" +resource_group_name = "az-devops-rg" instance_type = "Standard_DS2_v2" diff --git a/scripts/pipelines/github/pipeline_generator.sh b/scripts/pipelines/github/pipeline_generator.sh index 274f0b753..b6ea8619f 100644 --- a/scripts/pipelines/github/pipeline_generator.sh +++ b/scripts/pipelines/github/pipeline_generator.sh @@ -1,6 +1,6 @@ #!/bin/bash set -e -FLAGS=$(getopt -a --options c:n:d:a:b:l:i:u:p:hw --long "config-file:,pipeline-name:,local-directory:,artifact-path:,target-branch:,language:,build-pipeline-name:,sonar-url:,sonar-token:,image-name:,registry-user:,registry-password:,resource-group:,storage-account:,storage-container:,cluster-name:,s3-bucket:,s3-key-path:,quality-pipeline-name:,dockerfile:,test-pipeline-name:,aws-access-key:,aws-secret-access-key:,aws-region:,help" -- "$@") +FLAGS=$(getopt -a --options c:n:d:a:b:l:i:u:p:hw --long "config-file:,pipeline-name:,local-directory:,artifact-path:,target-branch:,language:,build-pipeline-name:,sonar-url:,sonar-token:,image-name:,registry-user:,registry-password:,resource-group:,storage-account:,storage-container:,cluster-name:,s3-bucket:,s3-key-path:,quality-pipeline-name:,dockerfile:,test-pipeline-name:,aws-access-key:,aws-secret-access-key:,aws-region:,help,rancher" -- "$@") eval set -- "$FLAGS" while true; do @@ -20,6 +20,7 @@ while true; do --resource-group) resourceGroupName=$2; shift 2;; --storage-account) storageAccountName=$2; shift 2;; --storage-container) storageContainerName=$2; shift 2;; + --rancher) installRancher="true"; shift 1;; --cluster-name) clusterName=$2; shift 2;; --s3-bucket) s3Bucket=$2; shift 2;; --s3-key-path) s3KeyPath=$2; shift 2;; diff --git a/scripts/pipelines/github/templates/aks/aks-pipeline.cfg b/scripts/pipelines/github/templates/aks/aks-pipeline.cfg new file mode 100644 index 000000000..b7dc51aa2 --- /dev/null +++ b/scripts/pipelines/github/templates/aks/aks-pipeline.cfg @@ -0,0 +1,61 @@ +# Mandatory flags. +mandatoryFalgs="$pipelineName,$localDirectory,$clusterName,$resourceGroupName,$storageAccountName,$storageContainerName," +# Path to the templates. +templatesPath="scripts/pipelines/github/templates/aks" +#Path to common kubernetes templates. +commonKubernetesPath="scripts/pipelines/github/templates/common/kubernetes" +# YAML file name. +yamlFile="aks-provisioning.yml" +# Source branch. +sourceBranch="feature/aks-provisioning" +# Path to terraform templates. +terraformTemplatesPath="scripts/environment-provisioning/azure/aks" +# Path to terraform scripts. +terraformPath=".terraform/aks" +# Default cluster operation. +operation="create" +# Install Rancher on AKS cluster. +if test -z $installRancher +then + installRancher=false +fi + +# Function that copies the necessary scripts into the directory. +function copyScript { + # Create .terraform/aks folder if it does not exist. + mkdir -p "${localDirectory}/${terraformPath}" + + # Copy the terraform files. + cd "${hangarPath}/${terraformTemplatesPath}" + cp * "${localDirectory}/${terraformPath}" + + # Copy the script for the DNS name into the directory. + cp "${hangarPath}/${templatesPath}/obtain-dns.sh" "${localDirectory}/${scriptFilePath}/obtain-dns.sh" + + # Copy the common files for kubernetes + cp "${hangarPath}/${commonKubernetesPath}"/*.sh "${localDirectory}/${scriptFilePath}" +} + +function commitFiles { + # Add the terraform files. + git add .terraform -f + + # Changing all files to be executable. + find .terraform -type f -name '*.sh' -exec git update-index --chmod=+x {} \; + + # Git commit and push it into the repository. + git commit -m "Adding the terraform files" + git push -u origin ${sourceBranch} +} + +# Function that adds the variables to be used in the pipeline. +function addPipelineVariables { + + export clusterName + export resourceGroupName + export storageAccountName + export storageContainerName + export installRancher + export operation + specificEnvSubstList='${clusterName} ${resourceGroupName} ${storageAccountName} ${storageContainerName} ${installRancher} ${operation}' +} diff --git a/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template b/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template new file mode 100644 index 000000000..ac92d1759 --- /dev/null +++ b/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template @@ -0,0 +1,106 @@ +name: $pipelineName + +on: + workflow_dispatch: + inputs: + cluster_name: + description: 'Name for the cluster.' + required: true + type: string + default: $clusterName + install_rancher: + description: 'Installs Rancher on AKS when set to true.' + required: true + type: string + default: $installRancher + cluster_operation: + description: 'Operation to be performed on AKS Cluster. create or destroy' + required: false + type: string + default: $operation + +env: + terraformWorkingDirectory: "${{ github.workspace }}/.terraform/aks" + +jobs: + terraform-actions: + name: Provisioning AKS + runs-on: ubuntu-latest + steps: + - name: login cli + run: | + az login -u "${{ secrets.AZURE_USERNAME }}" -p "${{ secrets.AZURE_PASSWORD }}" + + - name: Checkout Repository + uses: actions/checkout@master + + - name: Terraform - Init + id: init + run: terraform init -backend-config=resource_group_name=$resourceGroupName -backend-config=storage_account_name=$storageAccountName -backend-config=container_name=$storageContainerName -backend-config=key=terraform.tfstate + working-directory: ${{ env.terraformWorkingDirectory }} + continue-on-error: false + + - name: Terraform - apply + if: ${{ github.event.inputs.cluster_operation == 'create' }} + id: apply + working-directory: ${{ env.terraformWorkingDirectory }} + run: terraform apply -var 'cluster_name=${{ github.event.inputs.cluster_name }}' --auto-approve + continue-on-error: false + + - name: Terraform Destroy + if: ${{ github.event.inputs.cluster_operation == 'destroy' }} + id: destroy + working-directory: ${{ env.terraformWorkingDirectory }} + run: terraform apply -destroy -var 'cluster_name=${{ github.event.inputs.cluster_name }}' --auto-approve + continue-on-error: false + + - name: Install nginx-ingress controller + if: ${{ github.event.inputs.cluster_operation == 'create' }} + id: installnginx + env: + KUBECONFIG: "${{ env.terraformWorkingDirectory }}/kubeconfig" + run: .github/workflows/scripts/install-nginx-ingress.sh + + - name: Obtain DNS name + if: ${{ github.event.inputs.cluster_operation == 'create' }} + id: obtaindns + env: + KUBECONFIG: "${{ env.terraformWorkingDirectory }}/kubeconfig" + run: | + .github/workflows/scripts/obtain-dns.sh '${{ github.event.inputs.cluster_name }}' + dnsname="${{ github.event.inputs.cluster_name }}.westeurope.cloudapp.azure.com" + echo "::set-output name=dnsname::$dnsname" + + - name: Publish DNS name + if: ${{ github.event.inputs.cluster_operation == 'create' }} + id: publishdns + run: | + cd .github + mkdir vars + cd vars + touch "aks_variables.env" + echo "dns_name=${{ steps.obtaindns.outputs.dnsname }}" >> aks_variables.env + cat aks_variables.env + + - name: Publish kubeconfig + uses: actions/upload-artifact@v3.1.0 + if: ${{ github.event.inputs.cluster_operation == 'create' }} + with: + name: kubeconfig + path: ${{ env.terraformWorkingDirectory }}/kubeconfig + if-no-files-found: error + + - name: Push AKS variables + uses: EndBug/add-and-commit@v7 + if: ${{ github.event.inputs.cluster_operation == 'create' }} + with: + default_author: github_actions + cwd: '.github/vars/' + + - name: Install Rancher + if: ${{ github.event.inputs.install_rancher == 'true' && github.event.inputs.cluster_operation == 'create'}} + id: installrancher + env: + KUBECONFIG: "${{ env.terraformWorkingDirectory }}/kubeconfig" + run: .github/workflows/scripts/install-rancher.sh ${{ steps.obtaindns.outputs.dnsname }} + \ No newline at end of file diff --git a/scripts/pipelines/github/templates/aks/obtain-dns.sh b/scripts/pipelines/github/templates/aks/obtain-dns.sh new file mode 100644 index 000000000..c6f431573 --- /dev/null +++ b/scripts/pipelines/github/templates/aks/obtain-dns.sh @@ -0,0 +1,21 @@ +#!/bin/bash +ip="$(kubectl get svc nginx-ingress-nginx-ingress-controller --namespace nginx-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" + +while test -z "$ip" +do + sleep 5s + ip="$(kubectl get svc nginx-ingress-nginx-ingress-controller --namespace nginx-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" +done + +# Obtain the AKS cluster name +dnsname=$1 + +ipname=$(az network public-ip list --query "[?ipAddress!=null]|[?contains(ipAddress, '$ip')].[name]" --output tsv) + +iprg=$(az network public-ip list --query "[?ipAddress!=null]|[?contains(ipAddress, '$ip')].[resourceGroup]" --output tsv) + +az network public-ip update --resource-group "$iprg" --name "$ipname" --dns-name "$dnsname" + +dns="$(az network public-ip show --resource-group "$iprg" --name "$ipname" --query "[dnsSettings.fqdn]" --output tsv)" + +echo "$dns" diff --git a/scripts/pipelines/github/templates/common/kubernetes/install-nginx-ingress.sh b/scripts/pipelines/github/templates/common/kubernetes/install-nginx-ingress.sh new file mode 100644 index 000000000..0914cf122 --- /dev/null +++ b/scripts/pipelines/github/templates/common/kubernetes/install-nginx-ingress.sh @@ -0,0 +1,4 @@ +#!/bin/bash +helm repo add bitnami https://charts.bitnami.com/bitnami +helm repo update +helm install nginx-ingress bitnami/nginx-ingress-controller --set ingressClassResource.default=true --set containerSecurityContext.allowPrivilegeEscalation=false --namespace nginx-ingress --create-namespace \ No newline at end of file diff --git a/scripts/pipelines/github/templates/common/kubernetes/install-rancher.sh b/scripts/pipelines/github/templates/common/kubernetes/install-rancher.sh new file mode 100644 index 000000000..66c61ac78 --- /dev/null +++ b/scripts/pipelines/github/templates/common/kubernetes/install-rancher.sh @@ -0,0 +1,7 @@ +#!/bin/bash +helm repo add rancher-latest https://releases.rancher.com/server-charts/latest +helm repo add jetstack https://charts.jetstack.io +helm repo add +kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.1/cert-manager.crds.yaml +helm install cert-manager "jetstack/cert-manager" --namespace cert-manager --create-namespace --version v1.5.1 +helm install rancher "rancher-latest/rancher" --namespace cattle-system --create-namespace --set hostname="$1" \ No newline at end of file From 069c94ae3c9b164c87da360ff8f16d305cf4f107 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Fri, 1 Jul 2022 10:08:02 +0530 Subject: [PATCH 02/46] updated with AKS provision workflow in github action --- .../src/azure-devops/setup-aks-provisioning-pipeline.asciidoc | 1 - .../common_templates/setup-aks-provisioning-pipeline.asciidoc | 2 -- .../src/github/setup-aks-provisioning-pipeline.asciidoc | 1 - 3 files changed, 4 deletions(-) diff --git a/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc index c9224fd18..dc3092d34 100644 --- a/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -3,7 +3,6 @@ :trigger_sentence: * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. * Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. :pipeline_type2: pipeline :path_provider: azure-devops -:usage_type: :cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w :aks_variables: Azure DevOps > Pipelines > Library > `aks-variables` include::../common_templates/setup-build-pipeline.asciidoc[] \ No newline at end of file diff --git a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc index 320b8e0e2..e5553d9c3 100644 --- a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc @@ -46,7 +46,6 @@ pipeline_generator.sh \ -c \ -n \ -d \ -{usage_type} --resource-group \ --storage-account \ --storage-container \ @@ -62,7 +61,6 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). -{usage_type} --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. diff --git a/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc index d4af31d8c..617b9114a 100644 --- a/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc @@ -3,7 +3,6 @@ :trigger_sentence: * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`,`AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. :pipeline_type2: Github action :path_provider: github -:usage_type: :cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w :aks_variables: .github > vars > `aks-variables.env` include::../common_templates/setup-build-pipeline.asciidoc[] \ No newline at end of file From 7d284030efc26cfeca8f5d97c81f997eb1b28f3d Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Fri, 1 Jul 2022 04:43:03 +0000 Subject: [PATCH 03/46] automatic commit dor docs --- .../setup-aks-provisioning-pipeline.asciidoc | 97 ++----------------- .../setup-aks-provisioning-pipeline.asciidoc | 8 ++ 2 files changed, 16 insertions(+), 89 deletions(-) create mode 100644 documentation/github/setup-aks-provisioning-pipeline.asciidoc diff --git a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc index 07d0dd68d..0cb8ca031 100644 --- a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -1,89 +1,8 @@ -= Setting up a Azure AKS provisioning pipeline on Azure DevOps - -In this section we will create a pipeline which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix to retrieve the DNS name of the Ingress controller independently. - -The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the pipeline and the terraform files for creating the cluster will be pushed to it. - -Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. - -The script located at `/scripts/pipelines/azure-devops/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. - -== Prerequisites - -* Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. -* Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. -* An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: - -``` -az group create -n -l -``` - -* An Azure storage account within the previous resource group. You can use an existing one or create a new one with the following command: - -``` -az storage account create -n -g -l -``` - -* An Azure storage container in Azure within the previous storage account. You can use an existing one or create a new one with the following command: - -``` -az storage container create -n --account-name -``` - -* This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`). - -== Creating the pipeline using provided script - -Before executing the script you will need to customize some input variables about the environment. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/azure/aks`, which allows you to create or update values for the required variables, passing them as flags. As a full example: - -``` -./set-terraform-variables.sh --cluster_name --location --resource_group_name --instance_type --worker_node_count --dns_prefix -``` - -=== Usage -``` -pipeline_generator.sh \ - -c \ - -n \ - -d \ - --resource-group \ - --storage-account \ - --storage-container \ - [-b ] \ - [-w] -``` - -NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/azure-devops/templates/aks/aks-pipeline.cfg`. - -=== Flags -``` --c, --config-file [Required] Configuration file containing pipeline definition. --n, --pipeline-name [Required] Name that will be set to the pipeline. --d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). - --resource-group [Required] Name of the resource group for the cluster. - --storage--account [Required] Name of the storage account for the cluster. - --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. --b, --target-branch Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided. --w Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag. -``` - -=== Example - -``` -./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state -b develop -w -``` - -=== Appendix: Interacting with the cluster - -NOTE: Make sure you have https://kubernetes.io/docs/tasks/tools/#kubectl[kubectl] installed. - -In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning pipeline on the location it is expected by default (`~/.kube/config`) or either: - -``` -# via environment variable (you can add this on your profile) -export KUBECONFIG= -kubectl - -# via command-line flag -kubectl --kubeconfig= -``` +:provider: Azure Devops +:pipeline_type: Pipeline +:trigger_sentence: * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. * Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. +:pipeline_type2: pipeline +:path_provider: azure-devops +:cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w +:aks_variables: Azure DevOps > Pipelines > Library > `aks-variables` +Unresolved directive in setup-aks-provisioning-pipeline.asciidoc - include::../common_templates/setup-build-pipeline.asciidoc[] diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc new file mode 100644 index 000000000..3c126a575 --- /dev/null +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -0,0 +1,8 @@ +:provider: Github +:pipeline_type: workflow +:trigger_sentence: * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`,`AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. +:pipeline_type2: Github action +:path_provider: github +:cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w +:aks_variables: .github > vars > `aks-variables.env` +Unresolved directive in setup-aks-provisioning-pipeline.asciidoc - include::../common_templates/setup-build-pipeline.asciidoc[] From 5ea4d242a7fb665f78b956613c24aa79425e85d6 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Fri, 1 Jul 2022 10:19:18 +0530 Subject: [PATCH 04/46] Update setup-aks-provisioning-pipeline.asciidoc --- .../src/azure-devops/setup-aks-provisioning-pipeline.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc index dc3092d34..0ddf9a1a8 100644 --- a/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -5,4 +5,4 @@ :path_provider: azure-devops :cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w :aks_variables: Azure DevOps > Pipelines > Library > `aks-variables` -include::../common_templates/setup-build-pipeline.asciidoc[] \ No newline at end of file +include::../common_templates/setup-aks-provisioning-pipeline.asciidoc[] From f70b4238c8e2001c2750f254fba5fce51594ec0d Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Fri, 1 Jul 2022 10:19:36 +0530 Subject: [PATCH 05/46] Update setup-aks-provisioning-pipeline.asciidoc --- .../src/github/setup-aks-provisioning-pipeline.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc index 617b9114a..00de7c7c1 100644 --- a/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc @@ -5,4 +5,4 @@ :path_provider: github :cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w :aks_variables: .github > vars > `aks-variables.env` -include::../common_templates/setup-build-pipeline.asciidoc[] \ No newline at end of file +include::../common_templates/setup-aks-provisioning-pipeline.asciidoc[] From ec025649d2d200c4fb29cf72c6200495769801b7 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Fri, 1 Jul 2022 04:49:54 +0000 Subject: [PATCH 06/46] automatic commit dor docs --- .../setup-aks-provisioning-pipeline.asciidoc | 106 +++++++++++++++++- .../setup-aks-provisioning-pipeline.asciidoc | 106 +++++++++++++++++- 2 files changed, 210 insertions(+), 2 deletions(-) diff --git a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc index 0cb8ca031..6fbf960aa 100644 --- a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -5,4 +5,108 @@ :path_provider: azure-devops :cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w :aks_variables: Azure DevOps > Pipelines > Library > `aks-variables` -Unresolved directive in setup-aks-provisioning-pipeline.asciidoc - include::../common_templates/setup-build-pipeline.asciidoc[] += Setting up a Azure AKS provisioning {pipeline_type} on {provider} + +In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. + +The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. + +Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. + +The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. + +== Prerequisites + +{trigger_sentence} + +* An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: + +``` +az group create -n -l +``` + +* An Azure storage account within the previous resource group. You can use an existing one or create a new one with the following command: + +``` +az storage account create -n -g -l +``` + +* An Azure storage container in Azure within the previous storage account. You can use an existing one or create a new one with the following command: + +``` +az storage container create -n --account-name +``` + +* This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`). + +== Creating the {pipeline_type} using provided script + +Before executing the script you will need to customize some input variables about the environment. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/azure/aks`, which allows you to create or update values for the required variables, passing them as flags. As a full example: + +``` +./set-terraform-variables.sh --cluster_name --location --resource_group_name --instance_type --worker_node_count --dns_prefix +``` + +=== Usage +``` +pipeline_generator.sh \ + -c \ + -n \ + -d \ + --resource-group \ + --storage-account \ + --storage-container \ + [--rancher] \ + [-b ] \ + [-w] +``` + +NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`. + +=== Flags +``` +-c, --config-file [Required] Configuration file containing pipeline definition. +-n, --pipeline-name [Required] Name that will be set to the pipeline. +-d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). + --resource-group [Required] Name of the resource group for the cluster. + --storage--account [Required] Name of the storage account for the cluster. + --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. + --rancher Install Rancher to manage the cluster. +-b, --target-branch Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided. +-w Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag. +``` + +=== Example + +``` +{cmd_add} +``` + +NOTE: Rancher is installed on the cluster after provisioning when using the above command. + +=== Appendix: Interacting with the cluster + +NOTE: Make sure you have https://kubernetes.io/docs/tasks/tools/#kubectl[kubectl] installed. + +In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning pipeline on the location it is expected by default (`~/.kube/config`) or either: + +``` +# via environment variable (you can add this on your profile) +export KUBECONFIG= +kubectl + +# via command-line flag +kubectl --kubeconfig= +``` + +To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables}. + +Rancher, if installed, will be available on `https:///dashboard`. You will be asked for an initial password, which can be retrieved with: + +``` +kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}' +``` + +=== Appendix: Destroying the cluster + +To destroy the provisioned resources, set `operation` pipeline variable value to `destroy` and run the pipeline. diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc index 3c126a575..3b74d2b44 100644 --- a/documentation/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -5,4 +5,108 @@ :path_provider: github :cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w :aks_variables: .github > vars > `aks-variables.env` -Unresolved directive in setup-aks-provisioning-pipeline.asciidoc - include::../common_templates/setup-build-pipeline.asciidoc[] += Setting up a Azure AKS provisioning {pipeline_type} on {provider} + +In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. + +The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. + +Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. + +The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. + +== Prerequisites + +{trigger_sentence} + +* An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: + +``` +az group create -n -l +``` + +* An Azure storage account within the previous resource group. You can use an existing one or create a new one with the following command: + +``` +az storage account create -n -g -l +``` + +* An Azure storage container in Azure within the previous storage account. You can use an existing one or create a new one with the following command: + +``` +az storage container create -n --account-name +``` + +* This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`). + +== Creating the {pipeline_type} using provided script + +Before executing the script you will need to customize some input variables about the environment. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/azure/aks`, which allows you to create or update values for the required variables, passing them as flags. As a full example: + +``` +./set-terraform-variables.sh --cluster_name --location --resource_group_name --instance_type --worker_node_count --dns_prefix +``` + +=== Usage +``` +pipeline_generator.sh \ + -c \ + -n \ + -d \ + --resource-group \ + --storage-account \ + --storage-container \ + [--rancher] \ + [-b ] \ + [-w] +``` + +NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`. + +=== Flags +``` +-c, --config-file [Required] Configuration file containing pipeline definition. +-n, --pipeline-name [Required] Name that will be set to the pipeline. +-d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). + --resource-group [Required] Name of the resource group for the cluster. + --storage--account [Required] Name of the storage account for the cluster. + --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. + --rancher Install Rancher to manage the cluster. +-b, --target-branch Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided. +-w Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag. +``` + +=== Example + +``` +{cmd_add} +``` + +NOTE: Rancher is installed on the cluster after provisioning when using the above command. + +=== Appendix: Interacting with the cluster + +NOTE: Make sure you have https://kubernetes.io/docs/tasks/tools/#kubectl[kubectl] installed. + +In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning pipeline on the location it is expected by default (`~/.kube/config`) or either: + +``` +# via environment variable (you can add this on your profile) +export KUBECONFIG= +kubectl + +# via command-line flag +kubectl --kubeconfig= +``` + +To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables}. + +Rancher, if installed, will be available on `https:///dashboard`. You will be asked for an initial password, which can be retrieved with: + +``` +kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}' +``` + +=== Appendix: Destroying the cluster + +To destroy the provisioned resources, set `operation` pipeline variable value to `destroy` and run the pipeline. From 81cb809f5e9abf5532323d4f97bffaf64763a9f7 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Fri, 1 Jul 2022 10:28:41 +0530 Subject: [PATCH 07/46] Update setup-aks-provisioning-pipeline.asciidoc --- documentation/github/setup-aks-provisioning-pipeline.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc index 3b74d2b44..44984e673 100644 --- a/documentation/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -1,6 +1,6 @@ :provider: Github :pipeline_type: workflow -:trigger_sentence: * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`,`AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. +:trigger_sentence: Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it AZURE_USERNAME, AZURE_PASSWORD. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. :pipeline_type2: Github action :path_provider: github :cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w @@ -17,7 +17,7 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites -{trigger_sentence} +* {trigger_sentence} * An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: From 90080879ba222a1681bb6c42f5a6aaaf05ac86a6 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Fri, 1 Jul 2022 15:00:36 +0530 Subject: [PATCH 08/46] Update setup-aks-provisioning-pipeline.asciidoc --- .../common_templates/setup-aks-provisioning-pipeline.asciidoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc index e5553d9c3..cfba92dab 100644 --- a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc @@ -46,6 +46,7 @@ pipeline_generator.sh \ -c \ -n \ -d \ + --resource-group \ --resource-group \ --storage-account \ --storage-container \ @@ -61,6 +62,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). + --resource-group [Required] Name for the cluster. --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. @@ -102,4 +104,4 @@ kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ === Appendix: Destroying the cluster -To destroy the provisioned resources, set `operation` pipeline variable value to `destroy` and run the pipeline. \ No newline at end of file +To destroy the provisioned resources, set `operation` pipeline variable value to `destroy` and run the pipeline. From be22db05de37324781060c16a9a6eb653433fb6c Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Fri, 1 Jul 2022 09:30:55 +0000 Subject: [PATCH 09/46] Automatic generation of documentation --- .../azure-devops/setup-aks-provisioning-pipeline.asciidoc | 2 ++ .../github/setup-aks-provisioning-pipeline.asciidoc | 6 ++++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc index 6fbf960aa..b13c186d6 100644 --- a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -53,6 +53,7 @@ pipeline_generator.sh \ -c \ -n \ -d \ + --resource-group \ --resource-group \ --storage-account \ --storage-container \ @@ -68,6 +69,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). + --resource-group [Required] Name for the cluster. --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc index 44984e673..51923a34f 100644 --- a/documentation/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -1,6 +1,6 @@ :provider: Github :pipeline_type: workflow -:trigger_sentence: Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it AZURE_USERNAME, AZURE_PASSWORD. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. +:trigger_sentence: * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`,`AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. :pipeline_type2: Github action :path_provider: github :cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w @@ -17,7 +17,7 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites -* {trigger_sentence} +{trigger_sentence} * An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: @@ -53,6 +53,7 @@ pipeline_generator.sh \ -c \ -n \ -d \ + --resource-group \ --resource-group \ --storage-account \ --storage-container \ @@ -68,6 +69,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). + --resource-group [Required] Name for the cluster. --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. From d81e5ab01fd689e2e7f9c61946899ba6421cb352 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Fri, 1 Jul 2022 15:04:11 +0530 Subject: [PATCH 10/46] Update setup-aks-provisioning-pipeline.asciidoc --- .../common_templates/setup-aks-provisioning-pipeline.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc index cfba92dab..1afd1ae8d 100644 --- a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc @@ -74,7 +74,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ === Example ``` -{cmd_add} +./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w ``` NOTE: Rancher is installed on the cluster after provisioning when using the above command. From 22f07044b6eafef17ebe26b138c8203ef8dc06bc Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Fri, 1 Jul 2022 09:34:36 +0000 Subject: [PATCH 11/46] Automatic generation of documentation --- .../azure-devops/setup-aks-provisioning-pipeline.asciidoc | 2 +- documentation/github/setup-aks-provisioning-pipeline.asciidoc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc index b13c186d6..40f2dd9f3 100644 --- a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -81,7 +81,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ === Example ``` -{cmd_add} +./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w ``` NOTE: Rancher is installed on the cluster after provisioning when using the above command. diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc index 51923a34f..399dc468a 100644 --- a/documentation/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -81,7 +81,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ === Example ``` -{cmd_add} +./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w ``` NOTE: Rancher is installed on the cluster after provisioning when using the above command. From 5c30817fb9ef68f9826963a25f2855fc3d65b330 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Fri, 1 Jul 2022 15:20:56 +0530 Subject: [PATCH 12/46] Create setup-aks-provisioning-pipeline.asciidoc --- .../setup-aks-provisioning-pipeline.asciidoc | 27 +++++++------------ 1 file changed, 10 insertions(+), 17 deletions(-) diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc index 399dc468a..9292778a5 100644 --- a/documentation/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -1,23 +1,16 @@ -:provider: Github -:pipeline_type: workflow -:trigger_sentence: * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`,`AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. -:pipeline_type2: Github action -:path_provider: github -:cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w -:aks_variables: .github > vars > `aks-variables.env` -= Setting up a Azure AKS provisioning {pipeline_type} on {provider} += Setting up a Azure AKS provisioning workflow on Github -In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. +In this section we will create a workflow which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. -The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. +The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the workflow and the terraform files for creating the cluster will be pushed to it. Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. -The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. +The script located at `/scripts/pipelines/github/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. == Prerequisites -{trigger_sentence} +* Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. * An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: @@ -39,7 +32,7 @@ az storage container create -n --account-name \ -n \ -d \ - --resource-group \ + --cluster-name \ --resource-group \ --storage-account \ --storage-container \ @@ -62,14 +55,14 @@ pipeline_generator.sh \ [-w] ``` -NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`. +NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/github/templates/aks/aks-pipeline.cfg`. === Flags ``` -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). - --resource-group [Required] Name for the cluster. + --cluster-name [Required] Name for the cluster. --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. @@ -101,7 +94,7 @@ kubectl kubectl --kubeconfig= ``` -To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables}. +To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into .github > vars > `aks-variables.env`. Rancher, if installed, will be available on `https:///dashboard`. You will be asked for an initial password, which can be retrieved with: From 74f6b5898f8f2407755a32af394dd285d365197c Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Fri, 1 Jul 2022 15:21:20 +0530 Subject: [PATCH 13/46] Delete documentation/src directory --- .../setup-aks-provisioning-pipeline.asciidoc | 8 -- .../setup-aks-provisioning-pipeline.asciidoc | 107 ------------------ .../setup-aks-provisioning-pipeline.asciidoc | 8 -- 3 files changed, 123 deletions(-) delete mode 100644 documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc delete mode 100644 documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc delete mode 100644 documentation/src/github/setup-aks-provisioning-pipeline.asciidoc diff --git a/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc deleted file mode 100644 index 0ddf9a1a8..000000000 --- a/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ /dev/null @@ -1,8 +0,0 @@ -:provider: Azure Devops -:pipeline_type: Pipeline -:trigger_sentence: * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. * Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. -:pipeline_type2: pipeline -:path_provider: azure-devops -:cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w -:aks_variables: Azure DevOps > Pipelines > Library > `aks-variables` -include::../common_templates/setup-aks-provisioning-pipeline.asciidoc[] diff --git a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc deleted file mode 100644 index 1afd1ae8d..000000000 --- a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc +++ /dev/null @@ -1,107 +0,0 @@ -= Setting up a Azure AKS provisioning {pipeline_type} on {provider} - -In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. - -The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. - -Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. - -The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. - -== Prerequisites - -{trigger_sentence} - -* An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: - -``` -az group create -n -l -``` - -* An Azure storage account within the previous resource group. You can use an existing one or create a new one with the following command: - -``` -az storage account create -n -g -l -``` - -* An Azure storage container in Azure within the previous storage account. You can use an existing one or create a new one with the following command: - -``` -az storage container create -n --account-name -``` - -* This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`). - -== Creating the {pipeline_type} using provided script - -Before executing the script you will need to customize some input variables about the environment. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/azure/aks`, which allows you to create or update values for the required variables, passing them as flags. As a full example: - -``` -./set-terraform-variables.sh --cluster_name --location --resource_group_name --instance_type --worker_node_count --dns_prefix -``` - -=== Usage -``` -pipeline_generator.sh \ - -c \ - -n \ - -d \ - --resource-group \ - --resource-group \ - --storage-account \ - --storage-container \ - [--rancher] \ - [-b ] \ - [-w] -``` - -NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`. - -=== Flags -``` --c, --config-file [Required] Configuration file containing pipeline definition. --n, --pipeline-name [Required] Name that will be set to the pipeline. --d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). - --resource-group [Required] Name for the cluster. - --resource-group [Required] Name of the resource group for the cluster. - --storage--account [Required] Name of the storage account for the cluster. - --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. - --rancher Install Rancher to manage the cluster. --b, --target-branch Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided. --w Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag. -``` - -=== Example - -``` -./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w -``` - -NOTE: Rancher is installed on the cluster after provisioning when using the above command. - -=== Appendix: Interacting with the cluster - -NOTE: Make sure you have https://kubernetes.io/docs/tasks/tools/#kubectl[kubectl] installed. - -In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning pipeline on the location it is expected by default (`~/.kube/config`) or either: - -``` -# via environment variable (you can add this on your profile) -export KUBECONFIG= -kubectl - -# via command-line flag -kubectl --kubeconfig= -``` - -To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables}. - -Rancher, if installed, will be available on `https:///dashboard`. You will be asked for an initial password, which can be retrieved with: - -``` -kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}' -``` - -=== Appendix: Destroying the cluster - -To destroy the provisioned resources, set `operation` pipeline variable value to `destroy` and run the pipeline. diff --git a/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc deleted file mode 100644 index 00de7c7c1..000000000 --- a/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc +++ /dev/null @@ -1,8 +0,0 @@ -:provider: Github -:pipeline_type: workflow -:trigger_sentence: * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`,`AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. -:pipeline_type2: Github action -:path_provider: github -:cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w -:aks_variables: .github > vars > `aks-variables.env` -include::../common_templates/setup-aks-provisioning-pipeline.asciidoc[] From 8fc8de788c8b9a9ad603c61bc855522f91bf17ed Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 10:28:29 +0530 Subject: [PATCH 14/46] updating aks provision document --- .../setup-aks-provisioning-pipeline.asciidoc | 11 ++ .../setup-aks-provisioning-pipeline.asciidoc | 110 ++++++++++++++++++ .../setup-aks-provisioning-pipeline.asciidoc | 10 ++ 3 files changed, 131 insertions(+) create mode 100644 documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc create mode 100644 documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc create mode 100644 documentation/src/github/setup-aks-provisioning-pipeline.asciidoc diff --git a/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc new file mode 100644 index 000000000..1cf59b3e2 --- /dev/null +++ b/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -0,0 +1,11 @@ +:provider: Azure Devops +:pipeline_type: Pipeline +:trigger_sentence_1: Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. +:trigger_sentence_2: Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. +:pipeline_type2: pipeline +:path_provider: azure-devops +:Usage: +:flags: +:cmd_add: +:aks_variables: Azure DevOps > Pipelines > Library > `aks-variables` +include::../common_templates/setup-aks-provisioning-pipeline.asciidoc[] \ No newline at end of file diff --git a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc new file mode 100644 index 000000000..ece922003 --- /dev/null +++ b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc @@ -0,0 +1,110 @@ += Setting up a Azure AKS provisioning {pipeline_type} on {provider} + +In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. + +The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the workflow and the terraform files for creating the cluster will be pushed to it. + +Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. + +The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. + +== Prerequisites + +* {trigger_sentence_1} +{trigger_sentence_2} + +* Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. + +* An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: + +``` +az group create -n -l +``` + +* An Azure storage account within the previous resource group. You can use an existing one or create a new one with the following command: + +``` +az storage account create -n -g -l +``` + +* An Azure storage container in Azure within the previous storage account. You can use an existing one or create a new one with the following command: + +``` +az storage container create -n --account-name +``` + +* This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`). + +== Creating the {pipeline_type} using provided script + +Before executing the script you will need to customize some input variables about the environment. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/azure/aks`, which allows you to create or update values for the required variables, passing them as flags. As a full example: + +``` +./set-terraform-variables.sh --cluster_name --location --resource_group_name --instance_type --worker_node_count --dns_prefix +``` + +=== Usage +``` +pipeline_generator.sh \ + -c \ + -n \ + -d \ + {Usage} \ + --resource-group \ + --storage-account \ + --storage-container \ + [--rancher] \ + [-b ] \ + [-w] +``` + +NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`. + +=== Flags +``` +-c, --config-file [Required] Configuration file containing pipeline definition. +-n, --pipeline-name [Required] Name that will be set to the pipeline. +-d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). + {flags} + --resource-group [Required] Name of the resource group for the cluster. + --storage--account [Required] Name of the storage account for the cluster. + --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. + --rancher Install Rancher to manage the cluster. +-b, --target-branch Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided. +-w Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag. +``` + +=== Example + +``` +./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project {cmd_add} --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w +``` + +NOTE: Rancher is installed on the cluster after provisioning when using the above command. + +=== Appendix: Interacting with the cluster + +NOTE: Make sure you have https://kubernetes.io/docs/tasks/tools/#kubectl[kubectl] installed. + +In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning pipeline on the location it is expected by default (`~/.kube/config`) or either: + +``` +# via environment variable (you can add this on your profile) +export KUBECONFIG= +kubectl + +# via command-line flag +kubectl --kubeconfig= +``` + +To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables}. + +Rancher, if installed, will be available on `https:///dashboard`. You will be asked for an initial password, which can be retrieved with: + +``` +kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}' +``` + +=== Appendix: Destroying the cluster + +To destroy the provisioned resources, set `operation` pipeline variable value to `destroy` and run the pipeline. diff --git a/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc new file mode 100644 index 000000000..ca3a14c3a --- /dev/null +++ b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc @@ -0,0 +1,10 @@ +:provider: Github +:pipeline_type: workflow +:trigger_sentence_1: Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. +:pipeline_type2: Github action +:path_provider: github +:Usage: --cluster-name +:flags: --cluster-name [Required] Name for the cluster. +:cmd_add: --cluster-name devon-hangar +:aks_variables: .github > vars > `aks-variables.env` +include::../common_templates/setup-aks-provisioning-pipeline.asciidoc[] \ No newline at end of file From 62f4addbb0a3ba5e99742391b8a0abc44436d31d Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 04:59:11 +0000 Subject: [PATCH 15/46] Automatic generation of documentation --- .../setup-aks-provisioning-pipeline.asciidoc | 20 ++++++++---- .../setup-aks-provisioning-pipeline.asciidoc | 32 +++++++++++++------ 2 files changed, 35 insertions(+), 17 deletions(-) diff --git a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc index 40f2dd9f3..a96725221 100644 --- a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -1,15 +1,18 @@ :provider: Azure Devops :pipeline_type: Pipeline -:trigger_sentence: * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. * Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. +:trigger_sentence_1: Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. +:trigger_sentence_2: Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. :pipeline_type2: pipeline :path_provider: azure-devops -:cmd_add: ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w +:Usage: +:flags: +:cmd_add: :aks_variables: Azure DevOps > Pipelines > Library > `aks-variables` = Setting up a Azure AKS provisioning {pipeline_type} on {provider} In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. -The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. +The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the workflow and the terraform files for creating the cluster will be pushed to it. Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. @@ -17,7 +20,10 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites -{trigger_sentence} +* {trigger_sentence_1} +{trigger_sentence_2} + +* Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. * An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: @@ -53,7 +59,7 @@ pipeline_generator.sh \ -c \ -n \ -d \ - --resource-group \ + {Usage} \ --resource-group \ --storage-account \ --storage-container \ @@ -69,7 +75,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). - --resource-group [Required] Name for the cluster. + {flags} --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. @@ -81,7 +87,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ === Example ``` -./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w +./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project {cmd_add} --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w ``` NOTE: Rancher is installed on the cluster after provisioning when using the above command. diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc index 9292778a5..5f45b9e23 100644 --- a/documentation/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -1,15 +1,27 @@ -= Setting up a Azure AKS provisioning workflow on Github - -In this section we will create a workflow which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. +:provider: Github +:pipeline_type: workflow +:trigger_sentence_1: Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. +:pipeline_type2: Github action +:path_provider: github +:Usage: --cluster-name +:flags: --cluster-name [Required] Name for the cluster. +:cmd_add: --cluster-name devon-hangar +:aks_variables: .github > vars > `aks-variables.env` += Setting up a Azure AKS provisioning {pipeline_type} on {provider} + +In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the workflow and the terraform files for creating the cluster will be pushed to it. Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. -The script located at `/scripts/pipelines/github/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. +The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. == Prerequisites +* {trigger_sentence_1} +{trigger_sentence_2} + * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. * An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: @@ -32,7 +44,7 @@ az storage container create -n --account-name \ -n \ -d \ - --cluster-name \ + {Usage} \ --resource-group \ --storage-account \ --storage-container \ @@ -55,14 +67,14 @@ pipeline_generator.sh \ [-w] ``` -NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/github/templates/aks/aks-pipeline.cfg`. +NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`. === Flags ``` -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). - --cluster-name [Required] Name for the cluster. + {flags} --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. @@ -74,7 +86,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ === Example ``` -./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w +./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project {cmd_add} --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w ``` NOTE: Rancher is installed on the cluster after provisioning when using the above command. @@ -94,7 +106,7 @@ kubectl kubectl --kubeconfig= ``` -To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into .github > vars > `aks-variables.env`. +To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables}. Rancher, if installed, will be available on `https:///dashboard`. You will be asked for an initial password, which can be retrieved with: From 1a486b6cd84727d6a93324bdccf258c8b5047317 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 15:16:17 +0530 Subject: [PATCH 16/46] Update setup-aks-provisioning-pipeline.asciidoc --- .../setup-aks-provisioning-pipeline.asciidoc | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc index ece922003..0c58fed39 100644 --- a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc @@ -10,10 +10,10 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites -* {trigger_sentence_1} -{trigger_sentence_2} - -* Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. +ifdef::trigger_sentence_github[ * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps.] +ifdef::trigger_sentence_github[ *Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly.] +ifdef::trigger_sentence_github[ * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly.] +ifdef::cluster_name[ --cluster-name [Required] Name for the cluster.] * An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: @@ -49,7 +49,7 @@ pipeline_generator.sh \ -c \ -n \ -d \ - {Usage} \ +ifdef::cluster_name[ --cluster-name \] --resource-group \ --storage-account \ --storage-container \ @@ -65,7 +65,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). - {flags} +ifdef::cluster_name[ --cluster-name [Required] Name for the cluster.] --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. @@ -77,7 +77,8 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ === Example ``` -./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project {cmd_add} --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w +./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w \ +ifdef::cluster_name[ --cluster-name devon-hangar] ``` NOTE: Rancher is installed on the cluster after provisioning when using the above command. @@ -97,7 +98,7 @@ kubectl kubectl --kubeconfig= ``` -To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables}. +To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables_path}. Rancher, if installed, will be available on `https:///dashboard`. You will be asked for an initial password, which can be retrieved with: From 2df51771836b002947cec6919614863332c5eb15 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 09:46:37 +0000 Subject: [PATCH 17/46] Automatic generation of documentation --- .../setup-aks-provisioning-pipeline.asciidoc | 10 ++-------- .../github/setup-aks-provisioning-pipeline.asciidoc | 10 ++-------- 2 files changed, 4 insertions(+), 16 deletions(-) diff --git a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc index a96725221..03f368c80 100644 --- a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -20,10 +20,6 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites -* {trigger_sentence_1} -{trigger_sentence_2} - -* Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. * An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: @@ -59,7 +55,6 @@ pipeline_generator.sh \ -c \ -n \ -d \ - {Usage} \ --resource-group \ --storage-account \ --storage-container \ @@ -75,7 +70,6 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). - {flags} --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. @@ -87,7 +81,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ === Example ``` -./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project {cmd_add} --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w +./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w \ ``` NOTE: Rancher is installed on the cluster after provisioning when using the above command. @@ -107,7 +101,7 @@ kubectl kubectl --kubeconfig= ``` -To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables}. +To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables_path}. Rancher, if installed, will be available on `https:///dashboard`. You will be asked for an initial password, which can be retrieved with: diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc index 5f45b9e23..fb9e027d5 100644 --- a/documentation/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -19,10 +19,6 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites -* {trigger_sentence_1} -{trigger_sentence_2} - -* Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. * An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: @@ -58,7 +54,6 @@ pipeline_generator.sh \ -c \ -n \ -d \ - {Usage} \ --resource-group \ --storage-account \ --storage-container \ @@ -74,7 +69,6 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). - {flags} --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. @@ -86,7 +80,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ === Example ``` -./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project {cmd_add} --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w +./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w \ ``` NOTE: Rancher is installed on the cluster after provisioning when using the above command. @@ -106,7 +100,7 @@ kubectl kubectl --kubeconfig= ``` -To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables}. +To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables_path}. Rancher, if installed, will be available on `https:///dashboard`. You will be asked for an initial password, which can be retrieved with: From 4002c10bbae48df47bad055c9565971823ebf682 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 15:16:47 +0530 Subject: [PATCH 18/46] Update setup-aks-provisioning-pipeline.asciidoc --- .../setup-aks-provisioning-pipeline.asciidoc | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc index 1cf59b3e2..6afd8fad9 100644 --- a/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -1,11 +1,7 @@ :provider: Azure Devops :pipeline_type: Pipeline -:trigger_sentence_1: Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. -:trigger_sentence_2: Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. +:trigger_sentence_azure: :pipeline_type2: pipeline -:path_provider: azure-devops -:Usage: -:flags: -:cmd_add: -:aks_variables: Azure DevOps > Pipelines > Library > `aks-variables` -include::../common_templates/setup-aks-provisioning-pipeline.asciidoc[] \ No newline at end of file +:path_provider: azure-devops +:aks_variables_path: Azure DevOps > Pipelines > Library > `aks-variables` +include::../common_templates/setup-aks-provisioning-pipeline.asciidoc[] From 9927dd82db535e0fabcc6d0c067ac0901ebadca5 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 15:17:07 +0530 Subject: [PATCH 19/46] Update setup-aks-provisioning-pipeline.asciidoc --- .../github/setup-aks-provisioning-pipeline.asciidoc | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc index ca3a14c3a..9563082f1 100644 --- a/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc @@ -1,10 +1,8 @@ :provider: Github :pipeline_type: workflow -:trigger_sentence_1: Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. +:trigger_sentence_github: :pipeline_type2: Github action :path_provider: github -:Usage: --cluster-name -:flags: --cluster-name [Required] Name for the cluster. -:cmd_add: --cluster-name devon-hangar -:aks_variables: .github > vars > `aks-variables.env` -include::../common_templates/setup-aks-provisioning-pipeline.asciidoc[] \ No newline at end of file +:cluster_name: +:aks_variables_path: .github > vars > `aks-variables.env` +include::../common_templates/setup-aks-provisioning-pipeline.asciidoc[] From 6008fd094713f3b7b122f3c899369be07d85c9ad Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 09:47:27 +0000 Subject: [PATCH 20/46] Automatic generation of documentation --- .../setup-aks-provisioning-pipeline.asciidoc | 8 ++------ .../setup-aks-provisioning-pipeline.asciidoc | 15 ++++++++++----- 2 files changed, 12 insertions(+), 11 deletions(-) diff --git a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc index 03f368c80..1c8703297 100644 --- a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -1,13 +1,9 @@ :provider: Azure Devops :pipeline_type: Pipeline -:trigger_sentence_1: Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. -:trigger_sentence_2: Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. +:trigger_sentence_azure: :pipeline_type2: pipeline :path_provider: azure-devops -:Usage: -:flags: -:cmd_add: -:aks_variables: Azure DevOps > Pipelines > Library > `aks-variables` +:aks_variables_path: Azure DevOps > Pipelines > Library > `aks-variables` = Setting up a Azure AKS provisioning {pipeline_type} on {provider} In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc index fb9e027d5..e0c296fe7 100644 --- a/documentation/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -1,12 +1,10 @@ :provider: Github :pipeline_type: workflow -:trigger_sentence_1: Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. +:trigger_sentence_github: :pipeline_type2: Github action :path_provider: github -:Usage: --cluster-name -:flags: --cluster-name [Required] Name for the cluster. -:cmd_add: --cluster-name devon-hangar -:aks_variables: .github > vars > `aks-variables.env` +:cluster_name: +:aks_variables_path: .github > vars > `aks-variables.env` = Setting up a Azure AKS provisioning {pipeline_type} on {provider} In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. @@ -19,6 +17,10 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites + * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. + *Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. + * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. + --cluster-name [Required] Name for the cluster. * An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: @@ -54,6 +56,7 @@ pipeline_generator.sh \ -c \ -n \ -d \ + --cluster-name \ --resource-group \ --storage-account \ --storage-container \ @@ -69,6 +72,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). + --cluster-name [Required] Name for the cluster. --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. @@ -81,6 +85,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ ``` ./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w \ + --cluster-name devon-hangar ``` NOTE: Rancher is installed on the cluster after provisioning when using the above command. From b87f5e7dc3e8495b35e212bba5b1354a050b1c43 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 15:21:39 +0530 Subject: [PATCH 21/46] Update setup-aks-provisioning-pipeline.asciidoc --- .../setup-aks-provisioning-pipeline.asciidoc | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc index 0c58fed39..34b7b4d9e 100644 --- a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc @@ -10,10 +10,11 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites -ifdef::trigger_sentence_github[ * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps.] -ifdef::trigger_sentence_github[ *Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly.] +ifdef::trigger_sentence_azure[ * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps.] + +ifdef::trigger_sentence_azure[ *Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly.] + ifdef::trigger_sentence_github[ * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly.] -ifdef::cluster_name[ --cluster-name [Required] Name for the cluster.] * An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: From 43d0a9c4dd2381dd86f2bfa0cb7a74ab2be0035a Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 09:52:03 +0000 Subject: [PATCH 22/46] Automatic generation of documentation --- .../azure-devops/setup-aks-provisioning-pipeline.asciidoc | 4 ++++ .../github/setup-aks-provisioning-pipeline.asciidoc | 5 ++--- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc index 1c8703297..0d80e6b4e 100644 --- a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -16,6 +16,10 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites + * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. + + *Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. + * An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc index e0c296fe7..50d470634 100644 --- a/documentation/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -17,10 +17,9 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites - * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. - *Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. + + * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. - --cluster-name [Required] Name for the cluster. * An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: From dd0141842e62e70f45f74ff3e89c7b6400873c86 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 15:36:13 +0530 Subject: [PATCH 23/46] Update setup-aks-provisioning-pipeline.asciidoc --- .../common_templates/setup-aks-provisioning-pipeline.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc index 34b7b4d9e..6060dc93f 100644 --- a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc @@ -11,7 +11,6 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites ifdef::trigger_sentence_azure[ * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps.] - ifdef::trigger_sentence_azure[ *Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly.] ifdef::trigger_sentence_github[ * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly.] From 895e452d9e65abfd78e051d9a866b828b6413838 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 10:06:31 +0000 Subject: [PATCH 24/46] Automatic generation of documentation --- .../azure-devops/setup-aks-provisioning-pipeline.asciidoc | 1 - documentation/github/setup-aks-provisioning-pipeline.asciidoc | 1 - 2 files changed, 2 deletions(-) diff --git a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc index 0d80e6b4e..6609e5f7c 100644 --- a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -17,7 +17,6 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. - *Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc index 50d470634..3b97e4359 100644 --- a/documentation/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -18,7 +18,6 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites - * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly. * An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: From 4ac04d00d817628e15aa5201566add5c1d08296a Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 15:37:44 +0530 Subject: [PATCH 25/46] Update setup-aks-provisioning-pipeline.asciidoc --- .../common_templates/setup-aks-provisioning-pipeline.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc index 6060dc93f..4665a7195 100644 --- a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc @@ -11,7 +11,7 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites ifdef::trigger_sentence_azure[ * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps.] -ifdef::trigger_sentence_azure[ *Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly.] +ifdef::trigger_sentence_azure[ * Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly.] ifdef::trigger_sentence_github[ * Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly.] From 1250980727f360b4c1ca27709452991e93d3e16e Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 10:08:05 +0000 Subject: [PATCH 26/46] Automatic generation of documentation --- .../azure-devops/setup-aks-provisioning-pipeline.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc index 6609e5f7c..6124ea8ee 100644 --- a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -17,7 +17,7 @@ The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` == Prerequisites * Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps. - *Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. + * Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly. * An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command: From 24d98eda569008eca8a9d36fea63f572955b9623 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 17:32:00 +0530 Subject: [PATCH 27/46] Update setup-aks-provisioning-pipeline.asciidoc --- .../common_templates/setup-aks-provisioning-pipeline.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc index 4665a7195..81b1c3927 100644 --- a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc @@ -65,7 +65,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). -ifdef::cluster_name[ --cluster-name [Required] Name for the cluster.] +ifdef::cluster_name[ --cluster-name [Required] Name for the cluster.] --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. From c5554a135ae4a05043801c9ff40bc644d356a5e5 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 5 Jul 2022 12:02:27 +0000 Subject: [PATCH 28/46] Automatic generation of documentation --- documentation/github/setup-aks-provisioning-pipeline.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc index 3b97e4359..dc1ee693d 100644 --- a/documentation/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -70,7 +70,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). - --cluster-name [Required] Name for the cluster. + --cluster-name [Required] Name for the cluster. --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. From e27a7533c0d1034f60bc51484b841869c5778307 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Thu, 7 Jul 2022 13:02:08 +0530 Subject: [PATCH 29/46] updated cluster name in azure devops pipeline --- .../setup-aks-provisioning-pipeline.asciidoc | 19 +++--- .../setup-aks-provisioning-pipeline.asciidoc | 1 - .../azure-devops/pipeline_generator.sh | 66 +++++++++---------- .../templates/aks/aks-pipeline.cfg | 15 ++--- .../aks/aks-provisioning.yml.template | 2 + .../azure-devops/templates/aks/obtain-dns.sh | 6 +- .../github/templates/aks/aks-pipeline.cfg | 1 - 7 files changed, 53 insertions(+), 57 deletions(-) diff --git a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc index 81b1c3927..c3d14f53d 100644 --- a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc @@ -1,8 +1,8 @@ = Setting up a Azure AKS provisioning {pipeline_type} on {provider} -In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. +In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This {pipeline_type} will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. -The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the workflow and the terraform files for creating the cluster will be pushed to it. +The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. @@ -40,7 +40,7 @@ az storage container create -n --account-name --location --resource_group_name --instance_type --worker_node_count --dns_prefix +./set-terraform-variables.sh --location --resource_group_name --instance_type --worker_node_count --dns_prefix ``` === Usage @@ -49,7 +49,7 @@ pipeline_generator.sh \ -c \ -n \ -d \ -ifdef::cluster_name[ --cluster-name \] + --cluster-name \ --resource-group \ --storage-account \ --storage-container \ @@ -58,14 +58,14 @@ ifdef::cluster_name[ --cluster-name \] [-w] ``` -NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`. +NOTE: The config file for the AKS provisioning {pipeline_type} is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`. === Flags ``` -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). -ifdef::cluster_name[ --cluster-name [Required] Name for the cluster.] + --cluster-name [Required] Name for the cluster. --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. @@ -77,8 +77,7 @@ ifdef::cluster_name[ --cluster-name [Required] Name for the cluster.] === Example ``` -./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w \ -ifdef::cluster_name[ --cluster-name devon-hangar] +./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w ``` NOTE: Rancher is installed on the cluster after provisioning when using the above command. @@ -87,7 +86,7 @@ NOTE: Rancher is installed on the cluster after provisioning when using the abov NOTE: Make sure you have https://kubernetes.io/docs/tasks/tools/#kubectl[kubectl] installed. -In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning pipeline on the location it is expected by default (`~/.kube/config`) or either: +In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning {pipeline_type} on the location it is expected by default (`~/.kube/config`) or either: ``` # via environment variable (you can add this on your profile) @@ -108,4 +107,4 @@ kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ === Appendix: Destroying the cluster -To destroy the provisioned resources, set `operation` pipeline variable value to `destroy` and run the pipeline. +To destroy the provisioned resources, set `operation` {pipeline_type} variable value to `destroy` and run the {pipeline_type}. diff --git a/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc index 9563082f1..05abd79ba 100644 --- a/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc @@ -3,6 +3,5 @@ :trigger_sentence_github: :pipeline_type2: Github action :path_provider: github -:cluster_name: :aks_variables_path: .github > vars > `aks-variables.env` include::../common_templates/setup-aks-provisioning-pipeline.asciidoc[] diff --git a/scripts/pipelines/azure-devops/pipeline_generator.sh b/scripts/pipelines/azure-devops/pipeline_generator.sh index 6fb791164..e0376704a 100644 --- a/scripts/pipelines/azure-devops/pipeline_generator.sh +++ b/scripts/pipelines/azure-devops/pipeline_generator.sh @@ -4,40 +4,40 @@ FLAGS=$(getopt -a --options c:n:d:a:b:l:t:i:u:p:hw --long "config-file:,pipeline eval set -- "$FLAGS" while true; do case "$1" in - -c | --config-file) configFile=$2; shift 2;; - -n | --pipeline-name) pipelineName=$2; shift 2;; - -d | --local-directory) localDirectory=$2; shift 2;; - -a | --artifact-path) artifactPath=$2; shift 2;; - -b | --target-branch) targetBranch=$2; shift 2;; - -l | --language) language=$2; shift 2;; - -t | --target-directory) targetDirectory=$2; shift 2;; - --build-pipeline-name) export buildPipelineName=$2; shift 2;; - --sonar-url) sonarUrl=$2; shift 2;; - --sonar-token) sonarToken=$2; shift 2;; - -i | --image-name) imageName=$2; shift 2;; - -u | --registry-user) dockerUser=$2; shift 2;; - -p | --registry-password) dockerPassword=$2; shift 2;; - --resource-group) resourceGroupName=$2; shift 2;; - --storage-account) storageAccountName=$2; shift 2;; - --storage-container) storageContainerName=$2; shift 2;; - --rancher) installRancher="true"; shift 1;; - --cluster-name) clusterName=$2; shift 2;; - --s3-bucket) s3Bucket=$2; shift 2;; - --s3-key-path) s3KeyPath=$2; shift 2;; - --quality-pipeline-name) export qualityPipelineName=$2; shift 2;; - --test-pipeline-name) export testPipelineName=$2; shift 2;; - --dockerfile) dockerFile=$2; shift 2;; - --aws-access-key) awsAccessKey="$2"; shift 2;; - --aws-secret-access-key) awsSecretAccessKey="$2"; shift 2;; - --aws-region) awsRegion="$2"; shift 2;; - --package-pipeline-name) export packagePipelineName=$2; shift 2;; + -c | --config-file) configFile=$2; shift 2;; + -n | --pipeline-name) pipelineName=$2; shift 2;; + -d | --local-directory) localDirectory=$2; shift 2;; + -a | --artifact-path) artifactPath=$2; shift 2;; + -b | --target-branch) targetBranch=$2; shift 2;; + -l | --language) language=$2; shift 2;; + -t | --target-directory) targetDirectory=$2; shift 2;; + --build-pipeline-name) export buildPipelineName=$2; shift 2;; + --sonar-url) sonarUrl=$2; shift 2;; + --sonar-token) sonarToken=$2; shift 2;; + -i | --image-name) imageName=$2; shift 2;; + -u | --registry-user) dockerUser=$2; shift 2;; + -p | --registry-password) dockerPassword=$2; shift 2;; + --resource-group) resourceGroupName=$2; shift 2;; + --storage-account) storageAccountName=$2; shift 2;; + --storage-container) storageContainerName=$2; shift 2;; + --rancher) installRancher="true"; shift 1;; + --cluster-name) clusterName=$2; shift 2;; + --s3-bucket) s3Bucket=$2; shift 2;; + --s3-key-path) s3KeyPath=$2; shift 2;; + --quality-pipeline-name) export qualityPipelineName=$2; shift 2;; + --test-pipeline-name) export testPipelineName=$2; shift 2;; + --dockerfile) dockerFile=$2; shift 2;; + --aws-access-key) awsAccessKey="$2"; shift 2;; + --aws-secret-access-key) awsSecretAccessKey="$2"; shift 2;; + --aws-region) awsRegion="$2"; shift 2;; + --package-pipeline-name) export packagePipelineName=$2; shift 2;; --env-provision-pipeline-name) envProvisionPipelineName="$2"; shift 2;; - --k8s-provider) k8sProvider=$2; shift 2;; - --k8s-namespace) k8sNamespace="$2"; shift 2;; - --k8s-deploy-files-path) k8sDeployFiles=$2; shift 2;; - --k8s-image-pull-secret-name) k8sImagePullSecret=$2; shift 2;; - -h | --help) help="true"; shift 1;; - -w) webBrowser="true"; shift 1;; + --k8s-provider) k8sProvider=$2; shift 2;; + --k8s-namespace) k8sNamespace="$2"; shift 2;; + --k8s-deploy-files-path) k8sDeployFiles=$2; shift 2;; + --k8s-image-pull-secret-name) k8sImagePullSecret=$2; shift 2;; + -h | --help) help="true"; shift 1;; + -w) webBrowser="true"; shift 1;; --) shift; break;; esac done diff --git a/scripts/pipelines/azure-devops/templates/aks/aks-pipeline.cfg b/scripts/pipelines/azure-devops/templates/aks/aks-pipeline.cfg index 52053f491..83846ce98 100644 --- a/scripts/pipelines/azure-devops/templates/aks/aks-pipeline.cfg +++ b/scripts/pipelines/azure-devops/templates/aks/aks-pipeline.cfg @@ -1,5 +1,5 @@ # Mandatory flags. -mandatoryFalgs="$pipelineName,$localDirectory,$resourceGroupName,$storageAccountName,$storageContainerName," +mandatoryFalgs="$pipelineName,$localDirectory,$clusterName,$resourceGroupName,$storageAccountName,$storageContainerName," # Path to the templates. templatesPath="scripts/pipelines/azure-devops/templates/aks" #Path to common kubernetes templates. @@ -8,24 +8,20 @@ commonKubernetesPath="scripts/pipelines/azure-devops/templates/common/kubernetes yamlFile="aks-provisioning.yml" # Source branch. sourceBranch="feature/aks-provisioning" -# Path to the pipelines. -pipelinePath=".pipelines" # Path to terraform templates. terraformTemplatesPath="scripts/environment-provisioning/azure/aks" # Path to terraform scripts. terraformPath=".terraform/aks" # Service connection to AKS. serviceConnection="aks-connection" -# Path to the scripts. -scriptFilePath=".pipelines/scripts" # Default cluster operation. operation="create" # Install Rancher on AKS cluster. -if test -z ${installRancher} +if test -z $installRancher +then installRancher=false fi - # Function that copies the necessary scripts into the directory. function copyScript { # Create .terraform/aks folder if it does not exist. @@ -56,6 +52,9 @@ function commitFiles { # Function that adds the variables to be used in the pipeline. function addPipelineVariables { + # Add the AKS cluster name variable. + az pipelines variable create --name "clusterName" --pipeline-name $pipelineName --value ${clusterName} + # Add the resource group name variable. az pipelines variable create --name "resourceGroupName" --pipeline-name $pipelineName --value ${resourceGroupName} @@ -72,7 +71,7 @@ function addPipelineVariables { az pipelines variable create --name "installRancher" --pipeline-name $pipelineName --value ${installRancher} # Add the variable to choose create or destroy AKS cluster - az pipelines variable create --name "operation" --pipeline-name ${pipelineName} --value ${operation} + az pipelines variable create --name "operation" --pipeline-name $pipelineName --value ${operation} # Create the PAT. organization=$(git config --get remote.origin.url) diff --git a/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template b/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template index a9a8b91b6..2fe052f33 100644 --- a/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template +++ b/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template @@ -36,6 +36,7 @@ steps: condition: eq(variables.operation, 'create') inputs: command: 'apply' + commandOptions: '-var cluster_name=$(clusterName)' workingDirectory: '$(terraformWorkingDirectory)' environmentServiceNameAzureRm: '$(serviceConnection)' @@ -88,5 +89,6 @@ steps: inputs: provider: 'azurerm' command: 'destroy' + commandOptions: '-destroy -var cluster_name=$(clusterName)' workingDirectory: '$(terraformWorkingDirectory)' environmentServiceNameAzureRm: '$(serviceConnection)' diff --git a/scripts/pipelines/azure-devops/templates/aks/obtain-dns.sh b/scripts/pipelines/azure-devops/templates/aks/obtain-dns.sh index 60a315a88..439e3d42e 100644 --- a/scripts/pipelines/azure-devops/templates/aks/obtain-dns.sh +++ b/scripts/pipelines/azure-devops/templates/aks/obtain-dns.sh @@ -7,10 +7,8 @@ do ip="$(kubectl get svc nginx-ingress-nginx-ingress-controller --namespace nginx-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" done -url=$2 -length=${#url} -# Obtain the organization name by taking away the first 22 characters (https://dev.azure.com/) and the last /. -dnsname=${url:22:length-22-1} +# Obtain the AKS cluster name +dnsname=$2 ipname=$(az network public-ip list --query "[?ipAddress!=null]|[?contains(ipAddress, '$ip')].[name]" --output tsv) diff --git a/scripts/pipelines/github/templates/aks/aks-pipeline.cfg b/scripts/pipelines/github/templates/aks/aks-pipeline.cfg index b7dc51aa2..e466c2dd5 100644 --- a/scripts/pipelines/github/templates/aks/aks-pipeline.cfg +++ b/scripts/pipelines/github/templates/aks/aks-pipeline.cfg @@ -50,7 +50,6 @@ function commitFiles { # Function that adds the variables to be used in the pipeline. function addPipelineVariables { - export clusterName export resourceGroupName export storageAccountName From 5a20455e4028ce400916c3dff47075a8f419a3d1 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Thu, 7 Jul 2022 07:33:27 +0000 Subject: [PATCH 30/46] Automatic generation of documentation --- .../setup-aks-provisioning-pipeline.asciidoc | 16 +++++++++------- .../setup-aks-provisioning-pipeline.asciidoc | 16 +++++++--------- 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc index 6124ea8ee..a32a0eb41 100644 --- a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -6,9 +6,9 @@ :aks_variables_path: Azure DevOps > Pipelines > Library > `aks-variables` = Setting up a Azure AKS provisioning {pipeline_type} on {provider} -In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. +In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This {pipeline_type} will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. -The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the workflow and the terraform files for creating the cluster will be pushed to it. +The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. @@ -45,7 +45,7 @@ az storage container create -n --account-name --location --resource_group_name --instance_type --worker_node_count --dns_prefix +./set-terraform-variables.sh --location --resource_group_name --instance_type --worker_node_count --dns_prefix ``` === Usage @@ -54,6 +54,7 @@ pipeline_generator.sh \ -c \ -n \ -d \ + --cluster-name \ --resource-group \ --storage-account \ --storage-container \ @@ -62,13 +63,14 @@ pipeline_generator.sh \ [-w] ``` -NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`. +NOTE: The config file for the AKS provisioning {pipeline_type} is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`. === Flags ``` -c, --config-file [Required] Configuration file containing pipeline definition. -n, --pipeline-name [Required] Name that will be set to the pipeline. -d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\'). + --cluster-name [Required] Name for the cluster. --resource-group [Required] Name of the resource group for the cluster. --storage--account [Required] Name of the storage account for the cluster. --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored. @@ -80,7 +82,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ === Example ``` -./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w \ +./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w ``` NOTE: Rancher is installed on the cluster after provisioning when using the above command. @@ -89,7 +91,7 @@ NOTE: Rancher is installed on the cluster after provisioning when using the abov NOTE: Make sure you have https://kubernetes.io/docs/tasks/tools/#kubectl[kubectl] installed. -In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning pipeline on the location it is expected by default (`~/.kube/config`) or either: +In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning {pipeline_type} on the location it is expected by default (`~/.kube/config`) or either: ``` # via environment variable (you can add this on your profile) @@ -110,4 +112,4 @@ kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ === Appendix: Destroying the cluster -To destroy the provisioned resources, set `operation` pipeline variable value to `destroy` and run the pipeline. +To destroy the provisioned resources, set `operation` {pipeline_type} variable value to `destroy` and run the {pipeline_type}. diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc index dc1ee693d..0de5e073f 100644 --- a/documentation/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -3,13 +3,12 @@ :trigger_sentence_github: :pipeline_type2: Github action :path_provider: github -:cluster_name: :aks_variables_path: .github > vars > `aks-variables.env` = Setting up a Azure AKS provisioning {pipeline_type} on {provider} -In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. +In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This {pipeline_type} will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. -The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the workflow and the terraform files for creating the cluster will be pushed to it. +The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. @@ -45,7 +44,7 @@ az storage container create -n --account-name --location --resource_group_name --instance_type --worker_node_count --dns_prefix +./set-terraform-variables.sh --location --resource_group_name --instance_type --worker_node_count --dns_prefix ``` === Usage @@ -63,7 +62,7 @@ pipeline_generator.sh \ [-w] ``` -NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`. +NOTE: The config file for the AKS provisioning {pipeline_type} is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`. === Flags ``` @@ -82,8 +81,7 @@ NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/ === Example ``` -./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w \ - --cluster-name devon-hangar +./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w ``` NOTE: Rancher is installed on the cluster after provisioning when using the above command. @@ -92,7 +90,7 @@ NOTE: Rancher is installed on the cluster after provisioning when using the abov NOTE: Make sure you have https://kubernetes.io/docs/tasks/tools/#kubectl[kubectl] installed. -In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning pipeline on the location it is expected by default (`~/.kube/config`) or either: +In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning {pipeline_type} on the location it is expected by default (`~/.kube/config`) or either: ``` # via environment variable (you can add this on your profile) @@ -113,4 +111,4 @@ kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ === Appendix: Destroying the cluster -To destroy the provisioned resources, set `operation` pipeline variable value to `destroy` and run the pipeline. +To destroy the provisioned resources, set `operation` {pipeline_type} variable value to `destroy` and run the {pipeline_type}. From f2b19b1f147ece947dbf213b94f628c142ab04cb Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Thu, 7 Jul 2022 13:18:49 +0530 Subject: [PATCH 31/46] Update aks-provisioning.yml.template --- .../azure-devops/templates/aks/aks-provisioning.yml.template | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template b/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template index 2fe052f33..09ee74457 100644 --- a/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template +++ b/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template @@ -88,7 +88,7 @@ steps: condition: eq(variables.operation, 'destroy') inputs: provider: 'azurerm' - command: 'destroy' + command: 'apply' commandOptions: '-destroy -var cluster_name=$(clusterName)' workingDirectory: '$(terraformWorkingDirectory)' environmentServiceNameAzureRm: '$(serviceConnection)' From 26b457f2704d58123308d89d985ec2048926e734 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Thu, 7 Jul 2022 13:19:41 +0530 Subject: [PATCH 32/46] Update setup-aks-provisioning-pipeline.asciidoc --- .../common_templates/setup-aks-provisioning-pipeline.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc index c3d14f53d..208c34980 100644 --- a/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/common_templates/setup-aks-provisioning-pipeline.asciidoc @@ -2,11 +2,11 @@ In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This {pipeline_type} will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. -The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. +The creation of the {pipeline_type} will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. -The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. +The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning {pipeline_type} based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. == Prerequisites From e62c914b06af74d0b2b1aad5ab1bd208e394f9c3 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Thu, 7 Jul 2022 07:49:57 +0000 Subject: [PATCH 33/46] Automatic generation of documentation --- .../azure-devops/setup-aks-provisioning-pipeline.asciidoc | 4 ++-- documentation/github/setup-aks-provisioning-pipeline.asciidoc | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc index a32a0eb41..613820317 100644 --- a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -8,11 +8,11 @@ In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This {pipeline_type} will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. -The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. +The creation of the {pipeline_type} will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. -The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. +The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning {pipeline_type} based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. == Prerequisites diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc index 0de5e073f..690a77c14 100644 --- a/documentation/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -8,11 +8,11 @@ In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This {pipeline_type} will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. -The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. +The creation of the {pipeline_type} will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it. Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. -The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. +The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning {pipeline_type} based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. == Prerequisites From 21868f9b5ab9a19527daece816e84de1ae136ded Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Thu, 7 Jul 2022 13:24:14 +0530 Subject: [PATCH 34/46] Update terraform.tfvars --- scripts/environment-provisioning/azure/aks/terraform.tfvars | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/environment-provisioning/azure/aks/terraform.tfvars b/scripts/environment-provisioning/azure/aks/terraform.tfvars index 088b1cbd9..95a7e33ba 100644 --- a/scripts/environment-provisioning/azure/aks/terraform.tfvars +++ b/scripts/environment-provisioning/azure/aks/terraform.tfvars @@ -2,7 +2,7 @@ location = "westeurope" -resource_group_name = "az-devops-rg" +resource_group_name = "hangar-aks-rg" instance_type = "Standard_DS2_v2" From 821c228049607631cc827a1fbbbadd70c3d7cb60 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Thu, 7 Jul 2022 13:25:40 +0530 Subject: [PATCH 35/46] Update setup-aks-provisioning-pipeline.asciidoc --- .../src/azure-devops/setup-aks-provisioning-pipeline.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc index 6afd8fad9..41cf611d2 100644 --- a/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -1,5 +1,5 @@ :provider: Azure Devops -:pipeline_type: Pipeline +:pipeline_type: pipeline :trigger_sentence_azure: :pipeline_type2: pipeline :path_provider: azure-devops From 42b78f3f56291a32600637167754df2ef0119f5a Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Thu, 7 Jul 2022 07:55:56 +0000 Subject: [PATCH 36/46] Automatic generation of documentation --- .../azure-devops/setup-aks-provisioning-pipeline.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc index 613820317..9aa13b597 100644 --- a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -1,5 +1,5 @@ :provider: Azure Devops -:pipeline_type: Pipeline +:pipeline_type: pipeline :trigger_sentence_azure: :pipeline_type2: pipeline :path_provider: azure-devops From ae6d73a5f8c902080039e4c08b41a87038a0f803 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Sat, 9 Jul 2022 15:12:30 +0530 Subject: [PATCH 37/46] Update aks-provisioning.yml.template --- .../azure-devops/templates/aks/aks-provisioning.yml.template | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template b/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template index 09ee74457..987e7a3a1 100644 --- a/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template +++ b/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template @@ -3,8 +3,7 @@ trigger: variables: terraformWorkingDirectory: $(System.DefaultWorkingDirectory)/.terraform/aks - organization: $(System.TeamFoundationCollectionUri) - + pool: vmImage: ubuntu-latest @@ -58,7 +57,7 @@ steps: scriptType: 'bash' scriptLocation: 'scriptPath' scriptPath: '$(Build.Repository.LocalPath)/.pipelines/scripts/obtain-dns.sh' - arguments: $(terraformWorkingDirectory) $(organization) + arguments: $(terraformWorkingDirectory) $(clusterName) - publish: $(terraformWorkingDirectory)/kubeconfig displayName: 'Publish kubeconfig' From aa8a072eec5535f4cf62d846269eae2d1763e008 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 12 Jul 2022 15:56:52 +0530 Subject: [PATCH 38/46] Update aks-provisioning.yml.template --- .../github/templates/aks/aks-provisioning.yml.template | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template b/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template index ac92d1759..ea326344a 100644 --- a/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template +++ b/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template @@ -10,7 +10,7 @@ on: default: $clusterName install_rancher: description: 'Installs Rancher on AKS when set to true.' - required: true + required: false type: string default: $installRancher cluster_operation: @@ -103,4 +103,4 @@ jobs: env: KUBECONFIG: "${{ env.terraformWorkingDirectory }}/kubeconfig" run: .github/workflows/scripts/install-rancher.sh ${{ steps.obtaindns.outputs.dnsname }} - \ No newline at end of file + From c05d1a2208b7e6a1aee311fbab4e61e1e82d476e Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Mon, 18 Jul 2022 09:17:35 +0530 Subject: [PATCH 39/46] updated with common variable group --- .../templates/aks/aks-pipeline.cfg | 18 ++++++++++-------- .../aks/aks-provisioning.yml.template | 11 +++++------ .../azure-devops/templates/aks/obtain-dns.sh | 8 +++++--- .../aks/aks-provisioning.yml.template | 10 +++++----- 4 files changed, 25 insertions(+), 22 deletions(-) diff --git a/scripts/pipelines/azure-devops/templates/aks/aks-pipeline.cfg b/scripts/pipelines/azure-devops/templates/aks/aks-pipeline.cfg index 54502fbe0..427bb5357 100644 --- a/scripts/pipelines/azure-devops/templates/aks/aks-pipeline.cfg +++ b/scripts/pipelines/azure-devops/templates/aks/aks-pipeline.cfg @@ -1,5 +1,5 @@ # Mandatory flags. -mandatoryFalgs="$pipelineName,$localDirectory,$clusterName,$resourceGroupName,$storageAccountName,$storageContainerName," +mandatoryFalgs="$pipelineName,$localDirectory,$resourceGroupName,$storageAccountName,$storageContainerName," # Path to the templates. templatesPath="scripts/pipelines/azure-devops/templates/aks" # Path to common kubernetes templates. @@ -8,20 +8,25 @@ commonKubernetesPath="scripts/pipelines/azure-devops/templates/common/kubernetes yamlFile="aks-provisioning.yml" # Source branch. sourceBranch="feature/aks-provisioning" +# Path to the pipelines. +pipelinePath=".pipelines" # Path to terraform templates. terraformTemplatesPath="scripts/environment-provisioning/azure/aks" # Path to terraform scripts. terraformPath=".terraform/aks" # Service connection to AKS. serviceConnection="aks-connection" +# Path to the scripts. +scriptFilePath=".pipelines/scripts" # Default cluster operation. operation="create" # Install Rancher on AKS cluster. -if test -z $installRancher +if test -z ${installRancher} then installRancher=false fi + # Function that copies the necessary scripts into the directory. function copyScript { # Create .terraform/aks folder if it does not exist. @@ -52,9 +57,6 @@ function commitFiles { # Function that adds the variables to be used in the pipeline. function addPipelineVariables { - # Add the AKS cluster name variable. - az pipelines variable create --name "clusterName" --pipeline-name $pipelineName --value ${clusterName} - # Add the resource group name variable. az pipelines variable create --name "resourceGroupName" --pipeline-name $pipelineName --value ${resourceGroupName} @@ -70,9 +72,9 @@ function addPipelineVariables { # Add the install rancher variable. az pipelines variable create --name "installRancher" --pipeline-name $pipelineName --value ${installRancher} - # Add the variable to choose create or destroy AKS cluster - az pipelines variable create --name "operation" --pipeline-name $pipelineName --value ${operation} - + # Add the variable to choose create or destroy AKS cluster + az pipelines variable create --name "operation" --pipeline-name ${pipelineName} --value ${operation} + # Create the PAT. organization=$(git config --get remote.origin.url) organization=$(echo $organization | cut -d/ -f4) diff --git a/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template b/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template index 987e7a3a1..fc3bf26b4 100644 --- a/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template +++ b/scripts/pipelines/azure-devops/templates/aks/aks-provisioning.yml.template @@ -3,7 +3,8 @@ trigger: variables: terraformWorkingDirectory: $(System.DefaultWorkingDirectory)/.terraform/aks - + organization: $(System.TeamFoundationCollectionUri) + pool: vmImage: ubuntu-latest @@ -35,7 +36,6 @@ steps: condition: eq(variables.operation, 'create') inputs: command: 'apply' - commandOptions: '-var cluster_name=$(clusterName)' workingDirectory: '$(terraformWorkingDirectory)' environmentServiceNameAzureRm: '$(serviceConnection)' @@ -57,7 +57,7 @@ steps: scriptType: 'bash' scriptLocation: 'scriptPath' scriptPath: '$(Build.Repository.LocalPath)/.pipelines/scripts/obtain-dns.sh' - arguments: $(terraformWorkingDirectory) $(clusterName) + arguments: $(terraformWorkingDirectory) $(organization) - publish: $(terraformWorkingDirectory)/kubeconfig displayName: 'Publish kubeconfig' @@ -87,7 +87,6 @@ steps: condition: eq(variables.operation, 'destroy') inputs: provider: 'azurerm' - command: 'apply' - commandOptions: '-destroy -var cluster_name=$(clusterName)' + command: 'destroy' workingDirectory: '$(terraformWorkingDirectory)' - environmentServiceNameAzureRm: '$(serviceConnection)' + environmentServiceNameAzureRm: '$(serviceConnection)' \ No newline at end of file diff --git a/scripts/pipelines/azure-devops/templates/aks/obtain-dns.sh b/scripts/pipelines/azure-devops/templates/aks/obtain-dns.sh index 439e3d42e..fd8ef20a9 100644 --- a/scripts/pipelines/azure-devops/templates/aks/obtain-dns.sh +++ b/scripts/pipelines/azure-devops/templates/aks/obtain-dns.sh @@ -7,8 +7,10 @@ do ip="$(kubectl get svc nginx-ingress-nginx-ingress-controller --namespace nginx-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}')" done -# Obtain the AKS cluster name -dnsname=$2 +url=$2 +length=${#url} +# Obtain the organization name by taking away the first 22 characters (https://dev.azure.com/) and the last /. +dnsname=${url:22:length-22-1} ipname=$(az network public-ip list --query "[?ipAddress!=null]|[?contains(ipAddress, '$ip')].[name]" --output tsv) @@ -18,4 +20,4 @@ az network public-ip update --resource-group "$iprg" --name "$ipname" --dns-name dns="$(az network public-ip show --resource-group "$iprg" --name "$ipname" --query "[dnsSettings.fqdn]" --output tsv)" -echo "##vso[task.setvariable variable=dns;]$dns" +echo "##vso[task.setvariable variable=dns;]$dns" \ No newline at end of file diff --git a/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template b/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template index ea326344a..c3b5a41aa 100644 --- a/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template +++ b/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template @@ -76,12 +76,12 @@ jobs: id: publishdns run: | cd .github - mkdir vars + mkdir -p vars cd vars - touch "aks_variables.env" - echo "dns_name=${{ steps.obtaindns.outputs.dnsname }}" >> aks_variables.env - cat aks_variables.env - + touch "common-k8s-variables.env" + echo "dns_name=${{ steps.obtaindns.outputs.dnsname }}" >> common-k8s-variables.env + cat common-k8s-variables.env + - name: Publish kubeconfig uses: actions/upload-artifact@v3.1.0 if: ${{ github.event.inputs.cluster_operation == 'create' }} From 30f9517b805910d8e61ebabacbbe504f840cd968 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Mon, 18 Jul 2022 09:24:40 +0530 Subject: [PATCH 40/46] Update aks-provisioning.yml.template --- .../github/templates/aks/aks-provisioning.yml.template | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template b/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template index c3b5a41aa..5a3fd236c 100644 --- a/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template +++ b/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template @@ -79,7 +79,7 @@ jobs: mkdir -p vars cd vars touch "common-k8s-variables.env" - echo "dns_name=${{ steps.obtaindns.outputs.dnsname }}" >> common-k8s-variables.env + echo "aks_dns_name=${{ steps.obtaindns.outputs.dnsname }}" >> common-k8s-variables.env cat common-k8s-variables.env - name: Publish kubeconfig From 89cd9b5dfeac1f6486b5e9aac13f8f8aac22c715 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Wed, 20 Jul 2022 19:50:53 +0530 Subject: [PATCH 41/46] fixed end of the lines --- .../setup-repository-script.asciidoc | 258 +++++++++--------- .../setup-package-pipeline.asciidoc | 20 +- .../setup-quality-pipeline.asciidoc | 20 +- .../setup-package-pipeline.asciidoc | 208 +++++++------- .../github/setup-package-pipeline.asciidoc | 18 +- .../github/setup-quality-pipeline.asciidoc | 22 +- .../src/github/setup-test-pipeline.asciidoc | 20 +- 7 files changed, 283 insertions(+), 283 deletions(-) diff --git a/documentation/azure-devops/setup-repository-script.asciidoc b/documentation/azure-devops/setup-repository-script.asciidoc index 04530f42e..d77d4a682 100644 --- a/documentation/azure-devops/setup-repository-script.asciidoc +++ b/documentation/azure-devops/setup-repository-script.asciidoc @@ -1,129 +1,129 @@ -= Setting up a repository on Azure DevOps - -By the end of this guide, a repository on Azure DevOps will be created in an automated way using a script. - -== Prerequisites -* Have an Azure DevOps project already setup and Azure CLI installed and configured. If it is not the case, please go back to corresponding guide. - -* Install https://git-scm.com/book/en/v2/Getting-Started-Installing-Git[Git]. - -== Creating repository using provided script - -The script located at `/scripts/repositories/azure-devops/create-repo.sh` allows you to either: - -. Create an empty repository with just a README file and clone it to your computer into the directory you set. Useful when starting a project from scratch. - -. Import an already existing directory or Git repository into your project giving a path or an URL. Useful for taking to Azure DevOps the development of an existing project. - -=== Usage - -``` -create-repo.sh \ - -a \ - -d \ - -o \ - -p \ - [-n ] \ - [-g ] \ - [-b ] \ - [-r] \ - [-s ] \ - [-f] \ - [--subpath ] \ -``` - -=== Flags - -``` --a, --action [Required] Use case to fulfil: create, import. --d, --directory [Required] Path to the directory where your repository will be cloned or initialized. --o, --org [Required] Name of the Azure DevOps organization. --p, --project [Required] Name of the Azure DevOps project. --n, --name Name for the Azure DevOps repository. By default, the source repository or directory name (either new or existing, depending on use case) is used. --g, --source-git-url Source URL of the Git repository to import. --b, --source-branch Source branch to be used as a basis to initialize the repository on import, as master branch. --r, --remove-other-branches Removes branches other than the (possibly new) default one. --s, --setup-branch-strategy Creates branches and policies required for the desired workflow. Requires -b on import. Accepted values: gitflow. --f, --force Skips any user confirmation. - --subpath When combined with -g and -r, imports only the specified subpath of the source Git repository. -``` - -=== Typical use cases - -TIP: This is non-exhaustive list. Make your own combination of flags if all of the following use cases does not fit your needs. - -==== Creating an empty repository - - ./create-repo.sh -a create -o -p -n -d - -In case repository name is not specified, destination directory name will be used. - -==== Creating an empty repository with Gitflow branching strategy - - ./create-repo.sh -a create -o -p -n -d -s gitflow - -==== Importing a remote Git repository - - ./create-repo.sh -a import -g -o -p -n -d - -In case repository name is not specified, source repository name (in URL) will be used. - -==== Importing a specific remote Git repository branch as source for a new repository with Gitflow branching strategy - - ./create-repo.sh -a import -g -b -s gitflow -r -o -p -n -d - -This will create `master` (and `develop` since a branching strategy is specified) from the ``, removing any other branch (including ``). - -==== Importing local directory or Git repository - - ./create-repo.sh -a import -d -o -p -n - -In case repository name is not specified, source directory name will be used. - -==== Importing a specific local Git repository branch as source for a new repository with Gitflow branching strategy - - ./create-repo.sh -a 'import' -d -b -s gitflow -r -o -p -n - -This will create `master` (and `develop` since a branching strategy is specified) from the ``, removing any other branch (including ``). - -WARNING: This operation is destructive regarding branches on the local repository. - -NOTE: Same command could also be used with a local directory, but then using `-b` and `-r` would be redundant. - - -== Branching strategies - -To ensure the quality of development, it is crucial to keep a clean Git workflow. The following branching strategies are supported (using `-s` flag): - -=== Gitflow - -This is not an explanation of Gitflow (there are plenty of them on the web), but the actions performed by the script to help you start using this worflow. - -==== Branches - -* `master` is the initial (seed) branch. -* `develop` branch is created from `master` branch. - -Any other branch part the strategy (feature, release, and hotfix branches) will be created by developers during the lifecycle of the project. - -=== Policies - -In Azure DevOps it is possible to protect important branches against bad practices using branch policies. - -The following branch policies are applied to `master` and `develop` branches: - -* Require a minimum number of reviewers: ON -** Minimum number of reviewers: 1 -** Allow requestors to approve their own changes: ON -** Prohibit the most recent pusher from approving their own changes: OFF -** Allow completion even if some reviewers vote to wait or reject: OFF -** When new changes are pushed: Reset all approval votes (does not reset votes to reject or wait) -* Check for linked work items: OFF -* Check for comment resolution: REQUIRED -* Limit merge types: OFF - -The above policies are defined in a configuration file located at `/scripts/repositories/common/config/strategy.cfg`. Feel free to adapt it to your needs. - -NOTE: This is the bare minimum standard for any project. We do not prohibit the most recent pusher from approving their own changes, although being more than recommendable, because if we do, it will block the auto-approval of Pull Requests generated during pipelines creation on the following guides. - -You can find more information about branch policies in the https://docs.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=azure-devops[official documentation]. += Setting up a repository on Azure DevOps + +By the end of this guide, a repository on Azure DevOps will be created in an automated way using a script. + +== Prerequisites +* Have an Azure DevOps project already setup and Azure CLI installed and configured. If it is not the case, please go back to corresponding guide. + +* Install https://git-scm.com/book/en/v2/Getting-Started-Installing-Git[Git]. + +== Creating repository using provided script + +The script located at `/scripts/repositories/azure-devops/create-repo.sh` allows you to either: + +. Create an empty repository with just a README file and clone it to your computer into the directory you set. Useful when starting a project from scratch. + +. Import an already existing directory or Git repository into your project giving a path or an URL. Useful for taking to Azure DevOps the development of an existing project. + +=== Usage + +``` +create-repo.sh \ + -a \ + -d \ + -o \ + -p \ + [-n ] \ + [-g ] \ + [-b ] \ + [-r] \ + [-s ] \ + [-f] \ + [--subpath ] \ +``` + +=== Flags + +``` +-a, --action [Required] Use case to fulfil: create, import. +-d, --directory [Required] Path to the directory where your repository will be cloned or initialized. +-o, --org [Required] Name of the Azure DevOps organization. +-p, --project [Required] Name of the Azure DevOps project. +-n, --name Name for the Azure DevOps repository. By default, the source repository or directory name (either new or existing, depending on use case) is used. +-g, --source-git-url Source URL of the Git repository to import. +-b, --source-branch Source branch to be used as a basis to initialize the repository on import, as master branch. +-r, --remove-other-branches Removes branches other than the (possibly new) default one. +-s, --setup-branch-strategy Creates branches and policies required for the desired workflow. Requires -b on import. Accepted values: gitflow. +-f, --force Skips any user confirmation. + --subpath When combined with -g and -r, imports only the specified subpath of the source Git repository. +``` + +=== Typical use cases + +TIP: This is non-exhaustive list. Make your own combination of flags if all of the following use cases does not fit your needs. + +==== Creating an empty repository + + ./create-repo.sh -a create -o -p -n -d + +In case repository name is not specified, destination directory name will be used. + +==== Creating an empty repository with Gitflow branching strategy + + ./create-repo.sh -a create -o -p -n -d -s gitflow + +==== Importing a remote Git repository + + ./create-repo.sh -a import -g -o -p -n -d + +In case repository name is not specified, source repository name (in URL) will be used. + +==== Importing a specific remote Git repository branch as source for a new repository with Gitflow branching strategy + + ./create-repo.sh -a import -g -b -s gitflow -r -o -p -n -d + +This will create `master` (and `develop` since a branching strategy is specified) from the ``, removing any other branch (including ``). + +==== Importing local directory or Git repository + + ./create-repo.sh -a import -d -o -p -n + +In case repository name is not specified, source directory name will be used. + +==== Importing a specific local Git repository branch as source for a new repository with Gitflow branching strategy + + ./create-repo.sh -a 'import' -d -b -s gitflow -r -o -p -n + +This will create `master` (and `develop` since a branching strategy is specified) from the ``, removing any other branch (including ``). + +WARNING: This operation is destructive regarding branches on the local repository. + +NOTE: Same command could also be used with a local directory, but then using `-b` and `-r` would be redundant. + + +== Branching strategies + +To ensure the quality of development, it is crucial to keep a clean Git workflow. The following branching strategies are supported (using `-s` flag): + +=== Gitflow + +This is not an explanation of Gitflow (there are plenty of them on the web), but the actions performed by the script to help you start using this worflow. + +==== Branches + +* `master` is the initial (seed) branch. +* `develop` branch is created from `master` branch. + +Any other branch part the strategy (feature, release, and hotfix branches) will be created by developers during the lifecycle of the project. + +=== Policies + +In Azure DevOps it is possible to protect important branches against bad practices using branch policies. + +The following branch policies are applied to `master` and `develop` branches: + +* Require a minimum number of reviewers: ON +** Minimum number of reviewers: 1 +** Allow requestors to approve their own changes: ON +** Prohibit the most recent pusher from approving their own changes: OFF +** Allow completion even if some reviewers vote to wait or reject: OFF +** When new changes are pushed: Reset all approval votes (does not reset votes to reject or wait) +* Check for linked work items: OFF +* Check for comment resolution: REQUIRED +* Limit merge types: OFF + +The above policies are defined in a configuration file located at `/scripts/repositories/common/config/strategy.cfg`. Feel free to adapt it to your needs. + +NOTE: This is the bare minimum standard for any project. We do not prohibit the most recent pusher from approving their own changes, although being more than recommendable, because if we do, it will block the auto-approval of Pull Requests generated during pipelines creation on the following guides. + +You can find more information about branch policies in the https://docs.microsoft.com/en-us/azure/devops/repos/git/branch-policies?view=azure-devops[official documentation]. diff --git a/documentation/src/azure-devops/setup-package-pipeline.asciidoc b/documentation/src/azure-devops/setup-package-pipeline.asciidoc index c1ca8de6d..afe64c7e3 100644 --- a/documentation/src/azure-devops/setup-package-pipeline.asciidoc +++ b/documentation/src/azure-devops/setup-package-pipeline.asciidoc @@ -1,11 +1,11 @@ -:provider: Azure DevOps -:pipeline_type: pipeline -:trigger_sentence: This pipeline will be configured in order to be triggered every time quality pipeline is executed successfully on a commit for release/* and develop branches, requiring manual launch for other branches but still enforcing that quality pipeline has passed -:pipeline_type2: pipeline -:path_provider: azure-devops -:build-pipeline: -:quality-pipeline: -:extra_args_quarkus: --build-pipeline-name quarkus-project-build --quality-pipeline-name quarkus-project-quality -:extra_args_node: --build-pipeline-name node-project-build --quality-pipeline-name node-project-quality -:extra_args_angular: --build-pipeline-name angular-project-build --quality-pipeline-name angular-project-quality +:provider: Azure DevOps +:pipeline_type: pipeline +:trigger_sentence: This pipeline will be configured in order to be triggered every time quality pipeline is executed successfully on a commit for release/* and develop branches, requiring manual launch for other branches but still enforcing that quality pipeline has passed +:pipeline_type2: pipeline +:path_provider: azure-devops +:build-pipeline: +:quality-pipeline: +:extra_args_quarkus: --build-pipeline-name quarkus-project-build --quality-pipeline-name quarkus-project-quality +:extra_args_node: --build-pipeline-name node-project-build --quality-pipeline-name node-project-quality +:extra_args_angular: --build-pipeline-name angular-project-build --quality-pipeline-name angular-project-quality include::../common_templates/setup-package-pipeline.asciidoc[] \ No newline at end of file diff --git a/documentation/src/azure-devops/setup-quality-pipeline.asciidoc b/documentation/src/azure-devops/setup-quality-pipeline.asciidoc index 17239e63d..1e6d9b66e 100644 --- a/documentation/src/azure-devops/setup-quality-pipeline.asciidoc +++ b/documentation/src/azure-devops/setup-quality-pipeline.asciidoc @@ -1,11 +1,11 @@ -:provider: Azure DevOps -:pipeline_type: pipeline -:trigger_sentence: This pipeline will be configured in order to be triggered every time the test pipeline is executed successfully on a commit -:pipeline_type2: pipeline -:path_provider: azure-devops -:build-pipeline: -:test-pipeline: -:extra_args_quarkus: --build-pipeline-name quarkus-project-build --test-pipeline-name quarkus-project-test -:extra_args_node: --build-pipeline-name node-project-build --test-pipeline-name node-project-test -:extra_args_angular: --build-pipeline-name angular-project-build --test-pipeline-name angular-project-test +:provider: Azure DevOps +:pipeline_type: pipeline +:trigger_sentence: This pipeline will be configured in order to be triggered every time the test pipeline is executed successfully on a commit +:pipeline_type2: pipeline +:path_provider: azure-devops +:build-pipeline: +:test-pipeline: +:extra_args_quarkus: --build-pipeline-name quarkus-project-build --test-pipeline-name quarkus-project-test +:extra_args_node: --build-pipeline-name node-project-build --test-pipeline-name node-project-test +:extra_args_angular: --build-pipeline-name angular-project-build --test-pipeline-name angular-project-test include::../common_templates/setup-quality-pipeline.asciidoc[] \ No newline at end of file diff --git a/documentation/src/common_templates/setup-package-pipeline.asciidoc b/documentation/src/common_templates/setup-package-pipeline.asciidoc index 840de0a1a..8dcf98eb2 100644 --- a/documentation/src/common_templates/setup-package-pipeline.asciidoc +++ b/documentation/src/common_templates/setup-package-pipeline.asciidoc @@ -1,105 +1,105 @@ -= Setting up a Package {pipeline_type} on {provider} - -In this section we will create a package {pipeline_type} to build and push a container image of the project application into the specified container registry. {trigger_sentence}. It consumes the artifact produced by the build {pipeline_type}. - -The creation of the {pipeline_type2} will follow the project workflow, so a new branch named `feature/package-pipeline` will be created and the YAML file for the {pipeline_type} will be pushed to it. - -Then, a Pull Request (PR) will be created in order to merge the new branch into the appropriate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. - -The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create a package {pipeline_type} based on a YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. - -== Prerequisites - -This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled latest changes with `git pull`). - -== Creating the {pipeline_type} using provided script - -=== Usage -[subs=attributes+] -``` -pipeline_generator.sh \ - -c \ - -n <{pipeline_type} name> \ - -l \ - --dockerfile \ - -d \ -ifdef::build-pipeline[ --build-pipeline-name \] -ifdef::quality-pipeline[ --quality-pipeline-name \] -ifdef::ci-pipeline[ --ci-pipeline-name \] - -i \ - [-u ] \ - [-p ] \ - [--aws-access-key ] \ - [--aws-secret-access-key ] \ - [--aws-region ] \ - [-b ] \ - [-w] - -``` - -NOTE: The config file for the package {pipeline_type} is located at `/scripts/pipelines/{path_provider}/templates/package/package-pipeline.cfg`. - - -=== Flags -[subs=attributes+] -``` --c, --config-file [Required] Configuration file containing {pipeline_type} definition. --n, --pipeline-name [Required] Name that will be set to the {pipeline_type}. --l, --language [Required, if dockerfile not set] Language or framework of the project. - --dockerfile [Required, if language not set] Path from the root of the project to its Dockerfile. Takes precedence over the language/framework default one. --d, --local-directory [Required] Local directory of your project. -ifdef::build-pipeline[ --build-pipeline-name [Required] Build {pipeline_type} name.] -ifdef::quality-pipeline[ --quality-pipeline-name [Required] Quality {pipeline_type} name.] -ifdef::ci-pipeline[ --ci-pipeline-name [Required] CI {pipeline_type} name.] --i, --image-name [Required] Name (excluding tag) for the generated container image. --u, --registry-user [Required, unless AWS] Container registry login user. --p, --registry-password [Required, unless AWS] Container registry login password. - --aws-access-key [Required, if AWS] AWS account access key ID. Takes precedence over registry credentials." - --aws-secret-access-key [Required, if AWS] AWS account secret access key." - --aws-region [Required, if AWS] AWS region for ECR." --b, --target-branch Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided. --w Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag. -``` - -=== Examples - -==== Quarkus project - -===== Generic container registry -[subs=attributes+] -``` -./pipeline_generator.sh -c ./templates/package/package-pipeline.cfg -n quarkus-project-package -l quarkus -d C:/Users/$USERNAME/Desktop/quarkus-project -i username/quarkus-project -u username -p password {extra_args_quarkus} -b develop -w -``` - -===== AWS ECR -[subs=attributes+] -``` -./pipeline_generator.sh -c ./templates/package/package-pipeline.cfg -n quarkus-project-package -l quarkus -d C:/Users/$USERNAME/Desktop/quarkus-project -i username/quarkus-project --aws-access-key AKIAIOSFODNN7EXAMPLE --aws-secret-access-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY --aws-region eu-west-1 {extra_args_quarkus} -b develop -w -``` - -==== Node.js project - -===== Generic container registry -[subs=attributes+] -``` -./pipeline_generator.sh -c ./templates/package/package-pipeline.cfg -n node-project-package -l node -d C:/Users/$USERNAME/Desktop/node-project -i username/node-project -u username -p password {extra_args_node} -b develop -w -``` - -===== AWS ECR -[subs=attributes+] -``` -./pipeline_generator.sh -c ./templates/package/package-pipeline.cfg -n node-project-package -l node -d C:/Users/$USERNAME/Desktop/node-project -i username/node-project --aws-access-key AKIAIOSFODNN7EXAMPLE --aws-secret-access-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY --aws-region eu-west-1 {extra_args_node} -b develop -w -``` -==== Angular project - -===== Generic container registry -[subs=attributes+] -``` -./pipeline_generator.sh -c ./templates/package/package-pipeline.cfg -n angular-project-package -l angular -d C:/Users/$USERNAME/Desktop/angular-project --build-pipeline-name angular-project-build --quality-pipeline-name angular-project-quality -i username/angular-project -u username -p password -b develop -w -``` - -===== AWS ECR -[subs=attributes+] -``` -./pipeline_generator.sh -c ./templates/package/package-pipeline.cfg -n angular-project-package -l angular -d C:/Users/$USERNAME/Desktop/angular-project --build-pipeline-name angular-project-build --quality-pipeline-name angular-project-quality -i username/angular-project --aws-access-key AKIAIOSFODNN7EXAMPLE --aws-secret-access-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY --aws-region eu-west-1 -b develop -w += Setting up a Package {pipeline_type} on {provider} + +In this section we will create a package {pipeline_type} to build and push a container image of the project application into the specified container registry. {trigger_sentence}. It consumes the artifact produced by the build {pipeline_type}. + +The creation of the {pipeline_type2} will follow the project workflow, so a new branch named `feature/package-pipeline` will be created and the YAML file for the {pipeline_type} will be pushed to it. + +Then, a Pull Request (PR) will be created in order to merge the new branch into the appropriate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag. + +The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create a package {pipeline_type} based on a YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch. + +== Prerequisites + +This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled latest changes with `git pull`). + +== Creating the {pipeline_type} using provided script + +=== Usage +[subs=attributes+] +``` +pipeline_generator.sh \ + -c \ + -n <{pipeline_type} name> \ + -l \ + --dockerfile \ + -d \ +ifdef::build-pipeline[ --build-pipeline-name \] +ifdef::quality-pipeline[ --quality-pipeline-name \] +ifdef::ci-pipeline[ --ci-pipeline-name \] + -i \ + [-u ] \ + [-p ] \ + [--aws-access-key ] \ + [--aws-secret-access-key ] \ + [--aws-region ] \ + [-b ] \ + [-w] + +``` + +NOTE: The config file for the package {pipeline_type} is located at `/scripts/pipelines/{path_provider}/templates/package/package-pipeline.cfg`. + + +=== Flags +[subs=attributes+] +``` +-c, --config-file [Required] Configuration file containing {pipeline_type} definition. +-n, --pipeline-name [Required] Name that will be set to the {pipeline_type}. +-l, --language [Required, if dockerfile not set] Language or framework of the project. + --dockerfile [Required, if language not set] Path from the root of the project to its Dockerfile. Takes precedence over the language/framework default one. +-d, --local-directory [Required] Local directory of your project. +ifdef::build-pipeline[ --build-pipeline-name [Required] Build {pipeline_type} name.] +ifdef::quality-pipeline[ --quality-pipeline-name [Required] Quality {pipeline_type} name.] +ifdef::ci-pipeline[ --ci-pipeline-name [Required] CI {pipeline_type} name.] +-i, --image-name [Required] Name (excluding tag) for the generated container image. +-u, --registry-user [Required, unless AWS] Container registry login user. +-p, --registry-password [Required, unless AWS] Container registry login password. + --aws-access-key [Required, if AWS] AWS account access key ID. Takes precedence over registry credentials." + --aws-secret-access-key [Required, if AWS] AWS account secret access key." + --aws-region [Required, if AWS] AWS region for ECR." +-b, --target-branch Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided. +-w Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag. +``` + +=== Examples + +==== Quarkus project + +===== Generic container registry +[subs=attributes+] +``` +./pipeline_generator.sh -c ./templates/package/package-pipeline.cfg -n quarkus-project-package -l quarkus -d C:/Users/$USERNAME/Desktop/quarkus-project -i username/quarkus-project -u username -p password {extra_args_quarkus} -b develop -w +``` + +===== AWS ECR +[subs=attributes+] +``` +./pipeline_generator.sh -c ./templates/package/package-pipeline.cfg -n quarkus-project-package -l quarkus -d C:/Users/$USERNAME/Desktop/quarkus-project -i username/quarkus-project --aws-access-key AKIAIOSFODNN7EXAMPLE --aws-secret-access-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY --aws-region eu-west-1 {extra_args_quarkus} -b develop -w +``` + +==== Node.js project + +===== Generic container registry +[subs=attributes+] +``` +./pipeline_generator.sh -c ./templates/package/package-pipeline.cfg -n node-project-package -l node -d C:/Users/$USERNAME/Desktop/node-project -i username/node-project -u username -p password {extra_args_node} -b develop -w +``` + +===== AWS ECR +[subs=attributes+] +``` +./pipeline_generator.sh -c ./templates/package/package-pipeline.cfg -n node-project-package -l node -d C:/Users/$USERNAME/Desktop/node-project -i username/node-project --aws-access-key AKIAIOSFODNN7EXAMPLE --aws-secret-access-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY --aws-region eu-west-1 {extra_args_node} -b develop -w +``` +==== Angular project + +===== Generic container registry +[subs=attributes+] +``` +./pipeline_generator.sh -c ./templates/package/package-pipeline.cfg -n angular-project-package -l angular -d C:/Users/$USERNAME/Desktop/angular-project --build-pipeline-name angular-project-build --quality-pipeline-name angular-project-quality -i username/angular-project -u username -p password -b develop -w +``` + +===== AWS ECR +[subs=attributes+] +``` +./pipeline_generator.sh -c ./templates/package/package-pipeline.cfg -n angular-project-package -l angular -d C:/Users/$USERNAME/Desktop/angular-project --build-pipeline-name angular-project-build --quality-pipeline-name angular-project-quality -i username/angular-project --aws-access-key AKIAIOSFODNN7EXAMPLE --aws-secret-access-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY --aws-region eu-west-1 -b develop -w ``` \ No newline at end of file diff --git a/documentation/src/github/setup-package-pipeline.asciidoc b/documentation/src/github/setup-package-pipeline.asciidoc index b0a1152b8..94bf6e632 100644 --- a/documentation/src/github/setup-package-pipeline.asciidoc +++ b/documentation/src/github/setup-package-pipeline.asciidoc @@ -1,10 +1,10 @@ -:provider: GitHub -:pipeline_type: workflow -:trigger_sentence: This workflow will be configured in order to be triggered every time CI workflow is executed successfully on a commit for release/* and develop branches, requiring manual launch for other branches but still enforcing that CI workflow has passed -:pipeline_type2: GitHub action -:path_provider: github -:ci-pipeline: -:extra_args_quarkus: --ci-pipeline-name quarkus-project-ci -:extra_args_node: --ci-pipeline-name node-project-ci -:extra_args_angular: --ci-pipeline-name angular-project-ci +:provider: GitHub +:pipeline_type: workflow +:trigger_sentence: This workflow will be configured in order to be triggered every time CI workflow is executed successfully on a commit for release/* and develop branches, requiring manual launch for other branches but still enforcing that CI workflow has passed +:pipeline_type2: GitHub action +:path_provider: github +:ci-pipeline: +:extra_args_quarkus: --ci-pipeline-name quarkus-project-ci +:extra_args_node: --ci-pipeline-name node-project-ci +:extra_args_angular: --ci-pipeline-name angular-project-ci include::../common_templates/setup-package-pipeline.asciidoc[] \ No newline at end of file diff --git a/documentation/src/github/setup-quality-pipeline.asciidoc b/documentation/src/github/setup-quality-pipeline.asciidoc index 6e79574f8..1b3c75e92 100644 --- a/documentation/src/github/setup-quality-pipeline.asciidoc +++ b/documentation/src/github/setup-quality-pipeline.asciidoc @@ -1,11 +1,11 @@ -:provider: GitHub -:pipeline_type: workflow -:trigger_sentence: This workflow will be configured to be executed as a job inside a CI workflow after the test (or build, if no test) job -:pipeline_type2: GitHub action -:path_provider: github -:extra_args_quarkus: -:extra_args_node: -:extra_args_angular: -:extra_sentence_ci: Please note that this workflow, although manually triggerable, is designed to be executed as part of a CI workflow, which you can create following this xref:setup-ci-pipeline.asciidoc[guide]. -include::../common_templates/setup-quality-pipeline.asciidoc[] - +:provider: GitHub +:pipeline_type: workflow +:trigger_sentence: This workflow will be configured to be executed as a job inside a CI workflow after the test (or build, if no test) job +:pipeline_type2: GitHub action +:path_provider: github +:extra_args_quarkus: +:extra_args_node: +:extra_args_angular: +:extra_sentence_ci: Please note that this workflow, although manually triggerable, is designed to be executed as part of a CI workflow, which you can create following this xref:setup-ci-pipeline.asciidoc[guide]. +include::../common_templates/setup-quality-pipeline.asciidoc[] + diff --git a/documentation/src/github/setup-test-pipeline.asciidoc b/documentation/src/github/setup-test-pipeline.asciidoc index ba547babe..66da72033 100644 --- a/documentation/src/github/setup-test-pipeline.asciidoc +++ b/documentation/src/github/setup-test-pipeline.asciidoc @@ -1,10 +1,10 @@ -:provider: GitHub -:pipeline_type: workflow -:trigger_sentence: This workflow will be configured to be executed as a job inside a CI workflow after the build job -:pipeline_type2: GitHub action -:path_provider: github -:extra_args_quarkus: -:extra_args_node: -:extra_args_angular: -:extra_sentence_ci: Please note that this workflow, although manually triggerable, is designed to be executed as part of a CI workflow, which you can create following this xref:setup-ci-pipeline.asciidoc[guide]. -include::../common_templates/setup-test-pipeline.asciidoc[] +:provider: GitHub +:pipeline_type: workflow +:trigger_sentence: This workflow will be configured to be executed as a job inside a CI workflow after the build job +:pipeline_type2: GitHub action +:path_provider: github +:extra_args_quarkus: +:extra_args_node: +:extra_args_angular: +:extra_sentence_ci: Please note that this workflow, although manually triggerable, is designed to be executed as part of a CI workflow, which you can create following this xref:setup-ci-pipeline.asciidoc[guide]. +include::../common_templates/setup-test-pipeline.asciidoc[] From 58929fd5d7ccb5f998a0ec2caa5441c966f78815 Mon Sep 17 00:00:00 2001 From: TIMOTHE PATY Date: Thu, 4 Aug 2022 14:07:23 +0200 Subject: [PATCH 42/46] cluster name in help for aks --- scripts/pipelines/common/pipeline_generator.lib | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/pipelines/common/pipeline_generator.lib b/scripts/pipelines/common/pipeline_generator.lib index 1c6f092dc..d1d556746 100644 --- a/scripts/pipelines/common/pipeline_generator.lib +++ b/scripts/pipelines/common/pipeline_generator.lib @@ -49,6 +49,7 @@ function help { echo " -l, --language [Required] Language or framework of the project." echo "" echo "Azure AKS provisioning $pipeline_type flags:" + echo " --cluster-name [Required] Name for the cluster." echo " --resource-group [Required] Name of the resource group for the cluster." echo " --storage-account [Required] Name of the storage account for the cluster." echo " --storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored." From 8f5029ab137775ff3276a26bc7d309ee501c5526 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Thu, 8 Sep 2022 09:27:11 +0530 Subject: [PATCH 43/46] Update aks-provisioning.yml.template --- .../github/templates/aks/aks-provisioning.yml.template | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template b/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template index 5a3fd236c..43b97e230 100644 --- a/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template +++ b/scripts/pipelines/github/templates/aks/aks-provisioning.yml.template @@ -78,9 +78,8 @@ jobs: cd .github mkdir -p vars cd vars - touch "common-k8s-variables.env" - echo "aks_dns_name=${{ steps.obtaindns.outputs.dnsname }}" >> common-k8s-variables.env - cat common-k8s-variables.env + touch "deployment-env-variables.env" + echo "aks_dns_name=${{ steps.obtaindns.outputs.dnsname }}" >> deployment-env-variables.env - name: Publish kubeconfig uses: actions/upload-artifact@v3.1.0 From d97e86a431c6c5e624903f32b4a919016cc16714 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Thu, 8 Sep 2022 09:28:18 +0530 Subject: [PATCH 44/46] Update setup-aks-provisioning-pipeline.asciidoc --- .../src/github/setup-aks-provisioning-pipeline.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc index 05abd79ba..0b50c4a5d 100644 --- a/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/src/github/setup-aks-provisioning-pipeline.asciidoc @@ -3,5 +3,5 @@ :trigger_sentence_github: :pipeline_type2: Github action :path_provider: github -:aks_variables_path: .github > vars > `aks-variables.env` +:aks_variables_path: .github > vars > `deployment-env-variables.env` include::../common_templates/setup-aks-provisioning-pipeline.asciidoc[] From 384309653198643b6cc6e405737f3867ce2e6856 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Thu, 8 Sep 2022 03:58:45 +0000 Subject: [PATCH 45/46] Automatic generation of documentation --- .../azure-devops/setup-aks-provisioning-pipeline.asciidoc | 2 +- documentation/github/setup-aks-provisioning-pipeline.asciidoc | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc index 2728e9ae6..9aa13b597 100644 --- a/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc @@ -112,4 +112,4 @@ kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ === Appendix: Destroying the cluster -To destroy the provisioned resources, set `operation` {pipeline_type} variable value to `destroy` and run the {pipeline_type}. \ No newline at end of file +To destroy the provisioned resources, set `operation` {pipeline_type} variable value to `destroy` and run the {pipeline_type}. diff --git a/documentation/github/setup-aks-provisioning-pipeline.asciidoc b/documentation/github/setup-aks-provisioning-pipeline.asciidoc index 690a77c14..439e483de 100644 --- a/documentation/github/setup-aks-provisioning-pipeline.asciidoc +++ b/documentation/github/setup-aks-provisioning-pipeline.asciidoc @@ -3,7 +3,7 @@ :trigger_sentence_github: :pipeline_type2: Github action :path_provider: github -:aks_variables_path: .github > vars > `aks-variables.env` +:aks_variables_path: .github > vars > `deployment-env-variables.env` = Setting up a Azure AKS provisioning {pipeline_type} on {provider} In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This {pipeline_type} will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details. From 1f8e2e00ec8763e1a7a649eabddbcff02c30c9c4 Mon Sep 17 00:00:00 2001 From: prathibhapadma Date: Tue, 18 Oct 2022 04:03:34 +0000 Subject: [PATCH 46/46] Automatic generation of documentation --- documentation/azure-devops/setup-repository-script.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/documentation/azure-devops/setup-repository-script.asciidoc b/documentation/azure-devops/setup-repository-script.asciidoc index 55fc8504f..a7b2c2c10 100644 --- a/documentation/azure-devops/setup-repository-script.asciidoc +++ b/documentation/azure-devops/setup-repository-script.asciidoc @@ -34,7 +34,6 @@ create-repo.sh \ ``` === Flags [subs="attributes"] - ``` -a, --action [Required] Use case to fulfil: create, import. -d, --directory [Required] Path to the directory where your repository will be cloned or initialized.