How to read and process yaml files in POSIX using yq and jq

Prerequisites

yq and jq tools installed

Lets assume we have following yaml file called snapshots.yaml where the fields reference and version should be read for each item in the snapshots key in POSIX script

documentation:
  snapshots:
    - reference: master
      version: latest
    - reference: '2024.1'
      version: '1.2.1'
    - reference: '2023.12'
      version: '1.0.1'

In order to read them in POSIX script lets use the tool yq and jq, it provide support to process yaml and json contents respectively.

Use following command to read documentation.snapshots yaml content as json

yq eval -o=j -I=0 '.documentation.snapshots[]' snapshots.yaml

this will produce the following result, where each line will be a json object

{"reference":"master","version":"latest"}
{"reference":"2024.1","version":"1.2.1"}
{"reference":"2023.12","version":"1.0.1"}

these result can be iterated in POSIX script and each json line can be processed using the jq tool to read the fields reference and version like this.

echo '{"reference":"master","version":"latest"}' | jq -r .reference
echo '{"reference":"master","version":"latest"}' | jq -r .version

With all that the complete POSIX script would look like this.

#!/usr/bin/env sh
set -eu

for snapshot in $(yq eval -o=j -I=0 '.documentation.snapshots[]' snapshots.yaml); do
  reference=$(echo "$snapshot" | jq -r .reference)
  version=$(echo "$snapshot" | jq -r .version)

  echo "Reference: $reference, Version: $version"
done

Result

Reference: master, Version: latest
Reference: 2024.1, Version: 1.2.1
Reference: 2023.12, Version: 1.0.1

Create Azure AKS cluster with static egress IP using Terraform

Lets see how we can use a static public IP as a egress IP for the AKS cluster

Prerequisite

• Azure Account
• Configured Azure CLI for the Azure Subscription
• Terraform AKS cluster project 

Create IP resource

We first need to create a Public IP Address for us to use it in our AKS cluster.

Login to Azure portal  > Create a Resource Group

Choose your subscription and provide a name for the resource group E.g. Name StaticIpExample

Under the newly created StaticIpExample resource group, create a Public IP address resource.

Provide a name. E.g. Name: MyStaticIp, and make sure subscription and resource group are selected to the correct value. Once verified create the resource.

This will create a public IP for you.

Load the Public IP into Terraform

Open aks-cluster.tf file from the Terraform project and add following new block into the file.

data "azurerm_public_ip" "egress" {
  name                = "MyStaticIp"
  resource_group_name = "StaticIpExample"
}

once added, run terraform plan and verify the changes. This change will load the azurerm_public_ip resource named MyStaticIp from StaticIpExample resource group.

Use the static IP in AKS cluster

Now the above loaded static IP should be used in the AKS cluster declaration to use it as egress IP.

Open aks-cluster.tf file and add following block into the resource "azurerm_kubernetes_cluster" "default" {  block.

network_profile {
  network_plugin = "kubenet"
  load_balancer_profile {
    outbound_ip_address_ids = [data.azurerm_public_ip.egress.id]
  }
}

The block sets the kubenet as the default network plugin and sets the already defined static public IP as the outbound IP address for the load balancer.

Provide Access to AKS Cluster

If the Public IP address resource and AKS cluster are in two different resource groups then the AKS cluster needs to be provided with the access to use the IP address in the different resource group.

Create User Assigned Managed Identity resource in the resource group where the Public IP address resource present, in our case StaticIpExample, and provide a name E.g. Name: IpIdentity.

Open Public IP address resource MyStaticIp, navigate to Access Control (IAM) > Add role assignment > choose Contributor role under privileged administrator roles (or choose a role best match for you) > Next > Choose Managed Identity > Select newly created Managed Identity > Next > Review + assign

This will provide access to Managed Identity IpIdentity to handle Public IP address resource MyStaticIp.

Load the created managed identity into Terraform.

Open the aks-cluster.tf file and add following block

data "azurerm_user_assigned_identity" "ip_identity" {
  name                = "IpIdentity"
  resource_group_name = "StaticIpExample"
}

above declared managed identity data should be used in AKS cluster to be able to access the Public IP address resource present in the different resource group.

Open aks-cluster.tf file and add following block into the resource "azurerm_kubernetes_cluster" "default" {  block.

identity {
  type = "UserAssigned"
  identity_ids = [
    data.azurerm_user_assigned_identity.ip_identity.id
  ]
}

that's all and terraform plan then terraform apply should now create an AKS cluster which uses our defined static Ip address as its egress IP. 

This will make sure the cluster egress IP doesn't change when cluster deleted and re-created again.