Gitlab pipeline to build and publish a helm chart to artifactory

Lets see how we can use gitlab pipeline to build a helm chart and publish it to artifactory using its API.

Before that it is good to have some basic understanding on gitlab repository and its pipeline before following this guide.

Identify Required Stages and Jobs

Gitlab pipeline executes the stages in defined order, a stage may contain one or more jobs. Job in a stage will not start until the previous stage is completed.

For the purpose of build and publish the helm chart, we can think about following jobs build, render, lint, package and publish that can be grouped into following 3 stages build, test and release, which has to be executed in the defined order.

Lets define the stages in the pipeline file .gitlab-ci.yml

stages:
  - build
  - test
  - release

Docker image for job

Gitlab jobs are executed in a docker image by a runner. In our case we want to execute helm commands, therefore it is better to run the jobs in a docker container where the helm command is supported.

Lets use alpine/helm:3.13.3 docker image for this purpose. Since this is going to be used by almost all the jobs, it can be defined in global level in the pipeline like this.

default:
  image:
    name: alpine/helm:3.13.3
    entrypoint: [""]

Note that the default entrypoint of the image is overridden to value empty to get the access to the shell when executing the scripts in the job

Build helm chart

First thing we should do is the helm dependency build to make sure that helm chart dependency are valid and reachable.

Lets define the build job for it.

build:
  stage: build
  script:
    - helm dep build

Note down the stage of the job which is assigned to the stage build. Upon a trigger of the pipeline, given script, helm dep build will be executed by the gitlab pipeline.

Test helm chart

Helm chart can be tested by running helm template and helm lint. These commands will make sure that our rendered chart content is valid as per the mock values given to it.

These two verification can be performed in the test stage that will be executed once build stage is completed.

For the both helm template and lint commands, we can use a same value file, which would contain the mock value that required to render the helm chart. Therefore, this sample file can be declared as global variable.

variables:
  LINT_VALUE_FILE: values-lint.yaml

Both jobs can be defined as this

render:
  stage: test
  script:
    - helm template . -f $LINT_VALUE_FILE > rendered.yaml

lint:
  stage: test
  script:
    - helm lint -f $LINT_VALUE_FILE

make sure you have a file named values-lint.yaml created in your chart's root folder for the commands to get success.

Package chart

Helm chart should be packaged first before it pushed into artifactory. Fortunately helm has a command package for this purpose.

One thing to note is, whenever we run the pipeline from master branch, we shall package with a snapshot version, and when run from TAG then we shall package with concreate chart version to support continues integration of latest changes.

For the snapshot we can toss up a version suffix like this -dev.$CI_JOB_ID+$CI_COMMIT_SHORT_SHA where the gitlab inbuilt variable $CI_JOB_ID is the current job id and the $CI_COMMIT_SHORT_SHA is the latest short commit SHA of the branch/TAG.

Job for the package can be declared like this.

package:
  stage: release
  before_script:
    - CURRENT_CHART_VERSION=$(helm show chart . | grep ^version | cut -d ':' -f 2 | cut -d ' ' -f 2)
  script:
    - PACKAGE_VERSION=$CURRENT_CHART_VERSION$VERSION_SUFFIX
    - echo "Version to create the package is $PACKAGE_VERSION"
    - helm package . --version $PACKAGE_VERSION -d build/
  rules:
    - if: $CI_COMMIT_BRANCH == "master"
      variables:
        VERSION_SUFFIX: -dev.$CI_JOB_ID+$CI_COMMIT_SHORT_SHA
    - if: $CI_COMMIT_TAG
      variables:
        VERSION_SUFFIX: ""
  artifacts:
    paths:
      - build/*.tgz
    expire_in: 1 day

Few things to notice in this job declarations are

• Chart version extracted from the root Chart.yaml version with the help of helm show chart result, where the version value is extracted using the cut tool.
• Version suffix is appended when the pipeline run in master branch and suffix omitted when it is run from a TAG.
• Result of the package command attached as a job artifact, so the next job can use this artifact to publish it to the artifactory.

Publish chart

Lets assume there is a artifactory deployed in the following address artifacts.devops.xyz.com and it contains two repositories hlm-tmp and hlm-prd for the purpose of snapshot artifacts and production artifacts.

Lets define this as global variables

variables:
  HELM_TMP_REPOSITORY: hlm-tmp
  HELM_PRD_REPOSITORY: hlm-prd
  ARTIFACTORY_HOST: artifacts.devops.xyz.com

It is a good idea to keep snapshot artifact(build from master) in one location and production artifact(build from TAG) in another location to maintain them in a better way.

Artifactory has REST API which can be used to push artifacts into repositories, which we can also use from a gitlab pipeline, with the help of tool like curl.

With that, a job to push a artifact to artifactory would look like this.

publish:
  stage: release
  image: 
    name: alpine/curl:8.8.0
    entrypoint: [""]
  needs:
    - package
  before_script:
    - HELM_PACKAGE_NAME=$(ls build/)
  script:
    - echo "Helm package to push $HELM_PACKAGE_NAME"
    - curl -u $ARTIFACTORY_USERNAME:$ARTIFACTORY_PASSWORD -X PUT "https://$ARTIFACTORY_HOST/$REPOSITORY/$HELM_PACKAGE_NAME" -T build/$HELM_PACKAGE_NAME
  rules:
    - if: $CI_COMMIT_BRANCH == "master"
      variables:
        REPOSITORY: $HELM_TMP_REPOSITORY
    - if: $CI_COMMIT_TAG
      variables:
        REPOSITORY: $HELM_PRD_REPOSITORY

Few things to notice in this job description are

• Default image is overridden to use alpine/curl for the purpose of using the tool curl.
• Job declared as depends on the job package through keyword needs to access the job artifact (helm package generated) from the job package.
• Repository to push is decided based on where the pipeline is being executed. If it is in master, then temporary repository used. If it is in TAG then production repository is used.
• Variables ARTIFACTORY_USERNAME and ARTIFACTORY_PASSWORD are expected to have username and password configured to access the artifactory API

That's pretty much all what we needed to build, test, package and publish a helm chart to artifactory using a gitlab pipeline.

To sum it up, complete pipeline configuration would look like this after each variable and job declarations.

default:
  image:
    name: alpine/helm:3.13.3
    entrypoint: [""]

variables:
  HELM_TMP_REPOSITORY: hlm-tmp
  HELM_PRD_REPOSITORY: hlm-prd
  ARTIFACTORY_HOST: artifacts.devops.xyz.com
  LINT_VALUE_FILE: values-lint.yaml

stages:
  - build
  - test
  - release

build:
  stage: build
  script:
    - helm dep build

render:
  stage: test
  script:
    - helm template . -f $LINT_VALUE_FILE > rendered.yaml

lint:
  stage: test
  script:
    - helm lint -f $LINT_VALUE_FILE

package:
  stage: release
  before_script:
    - CURRENT_CHART_VERSION=$(helm show chart . | grep ^version | cut -d ':' -f 2 | cut -d ' ' -f 2)
  script:
    - PACKAGE_VERSION=$CURRENT_CHART_VERSION$VERSION_SUFFIX
    - echo "Version to create the package is $PACKAGE_VERSION"
    - helm package . --version $PACKAGE_VERSION -d build/
  rules:
    - if: $CI_COMMIT_BRANCH == "master"
      variables:
        VERSION_SUFFIX: -dev.$CI_JOB_ID+$CI_COMMIT_SHORT_SHA
    - if: $CI_COMMIT_TAG
      variables:
        VERSION_SUFFIX: ""
  artifacts:
    paths:
      - build/*.tgz
    expire_in: 1 day

publish:
  stage: release
  image: 
    name: alpine/curl:8.8.0
    entrypoint: [""]
  needs:
    - package
  before_script:
    - HELM_PACKAGE_NAME=$(ls build/)
  script:
    - echo "Helm package to push $HELM_PACKAGE_NAME"
    - curl -u $ARTIFACTORY_USERNAME:$ARTIFACTORY_PASSWORD -X PUT "https://$ARTIFACTORY_HOST/$REPOSITORY/$HELM_PACKAGE_NAME" -T build/$HELM_PACKAGE_NAME
  rules:
    - if: $CI_COMMIT_BRANCH == "master"
      variables:
        REPOSITORY: $HELM_TMP_REPOSITORY
    - if: $CI_COMMIT_TAG
      variables:
        REPOSITORY: $HELM_PRD_REPOSITORY