A look into Kubernetes Operator Lifecycle Manager (OLM)

Operators are vital part in Kubernetes ecosystem. It helps us install and manage software/service without the need for any manual intervention throughout the usage. 

One such use case could be installing a fully managed PostgreSQL cluster in a Kubenetes native way.

Such Operators could be installed in many different ways and installing it through the Operator Lifecycle Manager (OLM) is one such way.

OLM helps us to install an operator by requesting it through set of Kubernetes CRD that are known to it.

Lets go through OLM process and understand Kubernetes CRDs involved with that.

Prerequisites

OLM is installed in your cluster.

Create OperatorGroup

Typically OLM has a cluster wide access and it could provide any access which is requested by an operator, therefore, OperatorGroup is used by OLM to limit the scope of the operator to a single namespace or multiple namespaces.

Scuh OperatorGroup should be requested to target a set of namespaces using the OperatorGroup CRD provided by OLM. 

Any operators created in those targeted namespaces will become a part of that OperatorGroup and this enabled cluster admins to scope the operator permission to limited namespaces.

Following OperatorGroup is one such example where it targets a single namespace called my-namespace, it uses targetNamespace field to assign an operators, created in a namespace, within its group.

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: my-group
  namespace: my-namespace
spec:
  targetNamespaces:
  - my-namespace

An OperatorGroup may target one or more namespaces through its targetNamespaces field.

Create Subscription

Subscription is the CRD contains enough information to let OLM know what kind of Operator we need in our namespace.

Subscription simply contains the information about the CatalogSource name, its namespace, operator name, and the operator version.

Following is one such sample Subscription requesting a CloudNativePG operator.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cloudnative-pg
  namespace: my-namespace
spec:
  channel: v1.22.1
  name: cloudnative-pg
  source: community-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic

what is meant by above Subscription is, look for an operator named cloudnative-pg with release line v1.22.1 from the CatalogSource named community-operators which present in the namespace openshift-marketplace.

CatalogSource is more like a store which contains set of operators it knows on how to install them or more specifically its ClusterServiceVersion.

What is ClusterServiceVersion (CSV)

ClusterServiceVersion resource contains operator installation specification such as InstallModes, required permission, deployment template, CustomResourceDefinitions and etc.

One thing to note is that a operator will be assigned to a specific OperatorGroup only when OperatorGroup targetNamespaces type matches one of ClusterServiceVersion InstallModes.

E.g. Operator which supports OwnNamespace install mode can get assinged to an OperatorGroup that targets single namespace and its where the operator is installed.

InstallPlan

With the help of Subscription OLM will create an InstallPlan resource which contains the selected ClusterServiceVersion for the installation and its state of approval for the installation procedure.

If the Subscription's installPlanApproval is automatic then OLM will go ahead and install the operator components into the namespace using the selected ClusterServiceVersion resource. Otherwise an approval should be given explicitly to install the operator.

To sum all up, following is a simple illustrate on how resources are connected when an operator is deployed using OLM.

Java 22 Preview: Would Gatherer extension point revolutionize the Stream API in Java ?

Java 22 released few weeks ago and with that comes the gather() method that accepts Gatherer interface in Stream API as a preview feature.

When Stream API was released with Java 8 there was plenty of joy across java developers, but its limited function hid its true potential. Developers often create a list out of the Stream and perform any additional operation imperatively, which often leads to complicated, error-prone code.

To overcome this limitation, with Java 22, Gatherer interface and gather() method introduced to perform complex transformation as 1:1, 1:n, n:1 and n:m.

Now lets go thorough some of the inbuilt Gatherers and its use cases.

Install JDK 22 and enable preview mode by parsing the flag --enable-preview when executing the program.

Gatherers.windowFixed(int)

Window fixed implementation provides a way to group the stream elements in the order into group of fixed number of elements. 

Key points are 

1. No window will be provided if the list is empty

2. Last window may have less number of element than the fixed size given.

Suppose if we have following list of number 1, 2, 3, 4, 5, 6, 7, 8 and if want to group them into group of 3 elements and find the max number among them, then it can be done in this way

var numbers = List.of(1, 2, 3, 4, 5, 6, 7, 8);
var maxNumbers = numbers.stream()
    .gather(Gatherers.windowFixed(3))
    .peek(System.out::println)
    .map(values -> values.stream().max(Comparator.naturalOrder()).orElseThrow())
    .toList();
System.out.println(maxNumbers);

and the result would be

[1, 2, 3]
[4, 5, 6]
[7, 8]
Result [3, 6, 8]

Gatherers.windowSliding(int)

This is similar to window fixed except that it makes the group in sliding one by one, , until the last element is included. 

In case if there are numbers 1, 2, 3 then windowSliding(2) would provide groups as [1,2] and [2, 3]

Key points are 

1. No window provided for empty list

2. If the list contains less elements than sliding size then single window with all the elements provided

3. If the list contains same amount of element or more, than the sliding size then one or more windows where all of its size same as sliding size are provided.

Suppose if we have following list of number 1, 2, 3, 4, 5 and if want to group them into slide of 3 elements and find the max number among them, then it can be done in this way.

var numbers = List.of(1, 2, 3, 4, 5);
var maxNumbers = numbers.stream()
	.gather(Gatherers.windowSliding(3))
	.peek(System.out::println)
	.map(values -> values.stream().max(Comparator.naturalOrder()).orElseThrow())
	.toList();
System.out.println("Result " + maxNumbers);

and the result would be

[1, 2, 3]
[2, 3, 4]
[3, 4, 5]
Result [3, 4, 5]

Gatherers.fold(Supplier, BiFunction)

This is similar to reduce, where a single element is generated from list of element.

Suppose if we have following list of numbers 1, 2, 3, 4, 5 then fold can be used to add all this numbers.

var numbers = List.of(1, 2, 3, 4, 5);
var maxNumbers = numbers.stream()
	.gather(Gatherers.fold(() -> 0, Integer::sum))
	.toList();
System.out.println("Result " + maxNumbers); 

and the result would be

Result [15]

Gatherers.scan(Supplier, BiFunction)

scan is similar to fold except that scan will push the each iteration and fold will push only in the end of the stream.

Suppose if we have following list of numbers 1, 2, 3, 4, 5 and if we want to accumulate the value of each elements with its previous values and get the result as 1, 3, 6, 10, 15.

var numbers = List.of(1, 2, 3, 4, 5);
var maxNumbers = numbers.stream()
	.gather(Gatherers.scan(() -> 0, Integer::sum))
	.toList();
System.out.println("Result " + maxNumbers);

and the result is

Result [1, 3, 6, 10, 15]

More interesting thing about Gatherer is that it brings the possibility of creating our own Gatherer and integrate into Stream.

In the Gatherers.windowFixed example there was a part where max number is found in each window. For that map(values -> values.stream().max(Comparator.naturalOrder()).orElseThrow()) was used and lets try to achieve same thing using our own Gatherer implementation.

public static Gatherer<List<Integer>, ?, Integer> maxNumber() {
	class MaxNumber {
		boolean integrate(List<Integer> elements, Gatherer.Downstream<? super Integer> downstream) {
			var maxValue = elements.stream().max(Comparator.naturalOrder()).orElseThrow();
			return downstream.push(maxValue);
		}
	}

	return Gatherer.ofSequential(
		MaxNumber::new,
		Gatherer.Integrator.<MaxNumber, List<Integer>, Integer>ofGreedy(MaxNumber::integrate)
	);
}

var numbers = List.of(1, 2, 3, 4, 5, 6, 7, 8);
var maxNumbers = numbers.stream()
	.gather(Gatherers.windowFixed(3))
	.gather(maxNumber()) // Our maxNumber gatherer is used
	.toList();
System.out.println("Result " + maxNumbers);

and the result is 

Result [3, 6, 8]

To sum up, Gatherer interface introduced with Stream API looks promising as it allows great level of extendibility to the Stream processing and hopefully would allow us to do many more stream magics in future.

Use PostgreSQL function to trigger a task on a table upon row insert or update

PostgreSQL provides support to create function which can then be executed on a table upon a specific event.

Suppose if we want to keep only a specific number of latest rows in the table upon each insertion or update, we can use the trigger function to do that, without the need to run any background clean up task.

Lets see how we can do that

Create a table which has id, value and createdDataTime as its columns

create table test (
  id INT PRIMARY KEY, 
  value VARCHAR(50), 
  createdDateTime TIMESTAMP DEFAULT now()
);

then lets create a PostgreSQL procedure function, which upon execution should keep only the last 5 rows created and drop all the others.

CREATE FUNCTION drop_old_records() RETURNS trigger AS 
    '
    BEGIN
        DELETE FROM test WHERE id IN (SELECT id FROM test ORDER BY createdDateTime DESC OFFSET 5);
        RETURN NEW;
    END;
    '
LANGUAGE plpgsql;

Now the above created procedure function can be configured to trigger it whenever an insert or update is performed on the table test.

CREATE TRIGGER trigger_on_insert_or_update
AFTER INSERT OR UPDATE
ON test
EXECUTE PROCEDURE drop_old_records();

With that when insert or update performed on the table test then the drop_old_records function will be executed and only the last 5 rows will be kept.

Sample insertions

INSERT INTO test (id, value) VALUES (1, 'one');
INSERT INTO test (id, value) VALUES (2, 'two');
INSERT INTO test (id, value) VALUES (3, 'three');
INSERT INTO test (id, value) VALUES (4, 'four');
INSERT INTO test (id, value) VALUES (5, 'five');
INSERT INTO test (id, value) VALUES (6, 'six');

Result

id	value	createddatetime
2	two	2024-02-24T08:35:23.367Z
3	three	2024-02-24T08:35:23.368Z
4	four	2024-02-24T08:35:23.368Z
5	five	2024-02-24T08:35:23.368Z
6	six	2024-02-24T08:35:23.369Z

How to read and process yaml files in POSIX using yq and jq

Prerequisites

yq and jq tools installed

Lets assume we have following yaml file called snapshots.yaml where the fields reference and version should be read for each item in the snapshots key in POSIX script

documentation:
  snapshots:
    - reference: master
      version: latest
    - reference: '2024.1'
      version: '1.2.1'
    - reference: '2023.12'
      version: '1.0.1'

In order to read them in POSIX script lets use the tool yq and jq, it provide support to process yaml and json contents respectively.

Use following command to read documentation.snapshots yaml content as json

yq eval -o=j -I=0 '.documentation.snapshots[]' snapshots.yaml

this will produce the following result, where each line will be a json object

{"reference":"master","version":"latest"}
{"reference":"2024.1","version":"1.2.1"}
{"reference":"2023.12","version":"1.0.1"}

these result can be iterated in POSIX script and each json line can be processed using the jq tool to read the fields reference and version like this.

echo '{"reference":"master","version":"latest"}' | jq -r .reference
echo '{"reference":"master","version":"latest"}' | jq -r .version

With all that the complete POSIX script would look like this.

#!/usr/bin/env sh
set -eu

for snapshot in $(yq eval -o=j -I=0 '.documentation.snapshots[]' snapshots.yaml); do
  reference=$(echo "$snapshot" | jq -r .reference)
  version=$(echo "$snapshot" | jq -r .version)

  echo "Reference: $reference, Version: $version"
done

Result

Reference: master, Version: latest
Reference: 2024.1, Version: 1.2.1
Reference: 2023.12, Version: 1.0.1