Crafting rolling releases for a Quarkus CLI application

I've been working on a tool called JReleaser for sometime now. JReleaser shortens the distance between your binaries and potential consumers by packaging and publishing said binaries using formats and tools that consumers enjoy, such as Homebrew, Scoop, Docker, etc. All kind of Java (and since v0.5.0 also non-Java) applications are supported, this being said CLI applications are ideal. There are many ways to build CLI applications with Java, it so happens that Quarkus is one of them. I've been looking for an excuse to setup a Quarkus project to try out JReleaser with it, little did I know that Gunnar Morling (@gunnarmorling) would provide the initial spark:

The kcctl project is a Quarkus application that leverages PicoCLI, offering a command line experience similar to kubectl but for Kafka Connect. At the time of the announcement you had to build your own binaries if you wanted to give kcctl a try. That's OK to get started but Dan Allen said it best:

Automated releases? Yes! Takes ages to setup? Not quite, with JReleaser it's a snap! What follows is a description of the configuration added to kcctl to create rolling releases (early-access) on every push to the main branch; credit goes to Christian Stein (@sormuras) for showing me how it's done with "plain" GitHub Actions. His ideas and advice have been incorporated into JReleaser.

There are 3 aspects that need to be taken care of:

  • Updating the Maven configuration to produce binary distributions.
  • Providing the JReleaser configuration that will create a release.
  • Configure a GitHub Actions workflow that can package binaries for Linux, OSX, and Windows.

Here goes.

Native Image Distributions

JReleaser expects distributions to follow a specific file structure, where the executable goes inside a bin directory, additional files may be found at the root or at sibling directories; in short it should look like this

.
├── LICENSE
├── README
└── bin
    └── executable

Unfortunately the initial Maven setup created for a Quarkus project does not provide such structure out of the box, also the name of the executable matches a convention set as ${project.artifactId}-${project.version}-runner but we'd need it to be kcctl. Given that the distribution is platform specific it would be a good idea if the platform were to be part of the distribution name thus we need a way to capture that value. We can perform these two tasks by applying the os-maven-plugin and assembly-maven-plugin plugins. The first will let us capture the platform and expose it as a project property, the second will let us assemble a Zip file with the desired file structure.

It's recommended to configure the os-maven-plugin as a extension, that way it will be able to detect the platform and expose properties much earlier in the lifecycle. It's a simple as adding the following to the <build> section of the pom.xml

  <build>
    <!-- detect OS classifier, needed for distribution Zip file -->
    <extensions>
      <extension>
        <groupId>kr.motd.maven</groupId>
        <artifactId>os-maven-plugin</artifactId>
        <version>1.7.0</version>
      </extension>
    </extensions>
  </build>  

Now, assembling a binary distribution is ... a bit troublesome because ... Windows paths and filename conventions. You see, while both Linux and OSX are happy with executables files not having an extension at all, Windows prefers using .exe, thus the Quarkus build creates ${project.artifactId}-${project.version}-runner for Linux and OSX and ${project.artifactId}-${project.version}-runner.exe for Windows. Unfortunately the assembly descriptors do not accept conditionals (at least not that I could find) hence an alternative is required. Luckily Maven profiles can be used to solve this problem.

Here's how it's going to be played out:

  1. Add the assembly-maven-plugin with default configuration, with no descriptor and disabled by default. Why? So that its assembly:single goal may be bound to the package lifecycle phase using a profile that must be enabled explicitly. Why? So that the assembly is only created exactly when we want it, separate from the default lifecycle.
  2. Add a profile that activates the assembly plugin, let's name it "dist". This profile also configures the assembly descriptor used by both Linux and OSX.
  3. Add another profile that activates on a condition (os family = 'windows') setting the assembly descriptor that's specific to Windows.

The first step is covered by adding the following to the <plugins> section

      <plugins>
        <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-assembly-plugin</artifactId>
        <version>3.2.0</version>
        <configuration>
          <attach>false</attach>
          <appendAssemblyId>false</appendAssemblyId>
          <finalName>kcctl-${project.version}-${os.detected.classifier}</finalName>
          <workDirectory>${project.build.directory}/assembly/work</workDirectory>
          <skipAssembly>true</skipAssembly>
        </configuration>
        <executions>
          <execution>
            <id>make-distribution</id>
            <phase>package</phase>
            <goals>
              <goal>single</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>  

Steps #2 and #3 require defining a pair of profiles, like so

  <profiles>
    <profile>
      <id>dist</id>
      <activation>
        <property>
          <name>dist</name>
        </property>
      </activation>
      <build>
        <plugins>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-assembly-plugin</artifactId>
            <configuration combine.self="append">
              <skipAssembly>false</skipAssembly>
              <descriptors>
                <descriptor>src/main/assembly/assembly.xml</descriptor>
              </descriptors>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>
    <profile>
      <id>dist-windows</id>
      <activation>
        <os>
          <family>windows</family>
        </os>
      </activation>
      <build>
        <plugins>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-assembly-plugin</artifactId>
            <configuration combine.self="append">
              <skipAssembly>false</skipAssembly>
              <descriptors>
                <descriptor>src/main/assembly/assembly-windows.xml</descriptor>
              </descriptors>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </profile>
  </profiles>  

The first profile, "dist", enables the assembly plugin and sets the configuration for Linux and OSX. The second profile, "dist-windows", is activated by default when the running platform is Windows, and it configures the Windows specific descriptor. By the way, these descriptors look like this:

assembly.xml

<?xml version="1.0"?>
<assembly
        xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
    <id>dist</id>
    <formats>
        <format>zip</format>
        <format>dir</format>
    </formats>
    <files>
        <file>
            <source>LICENSE.txt</source>
            <outputDirectory>./</outputDirectory>
        </file>
        <file>
            <source>kcctl_completion</source>
            <outputDirectory>./</outputDirectory>
        </file>
        <file>
            <source>${project.build.directory}/${project.artifactId}-${project.version}-runner</source>
            <outputDirectory>./bin</outputDirectory>
            <destName>kcctl</destName>
        </file>
    </files>
</assembly>

assembly-windows.xml

<?xml version="1.0"?>
<assembly
        xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">
    <id>dist</id>
    <formats>
        <format>zip</format>
        <format>dir</format>
    </formats>
    <files>
        <file>
            <source>LICENSE.txt</source>
            <outputDirectory>./</outputDirectory>
        </file>
        <file>
            <source>kcctl_completion</source>
            <outputDirectory>./</outputDirectory>
        </file>
        <file>
            <source>${project.build.directory}/${project.artifactId}-${project.version}-runner.exe</source>
            <outputDirectory>./bin</outputDirectory>
            <destName>kcctl.exe</destName>
        </file>
    </files>
</assembly>

Notice that the only difference between them is the use of the .exe file extension for the executable files. If someone knows a shorter, more concise version to get the job done please do let me know!

With these settings in place we can now generate a distribution with the following command invocations:

$ mvn -Pnative package
$ mvn -Ddist package -DskipTests

That should take care of the Maven configuration for now, moving on to the next task: configuring JReleaser.

Configuring JReleaser

There are several ways to configure and run JReleaser on a build. Given that the current build is Maven based you'd think that the natural choice would be to use the jreleaser-maven-plugin option but no, I chose to stick with the CLI option because that allows using the early-access builds that deliver new features on every commit. That's right, we're using a rolling release of a tool to configure rolling releases for another tool. Inception! Of course, at some point the kcctl release may switch to the stable JReleaser release stream.

Given that the Maven build already produces binary distributions as JReleaser expects them it's just a simple matter of defining the basic configuration, as follows

jreleaser.yml

project:
  name: kcctl
  description: kcctl -- A CLI for Apache Kafka Connect
  longDescription: kcctl -- A CLI for Apache Kafka Connect
  website: https://github.com/gunnarmorling/kcctl
  authors:
    - Gunnar Morling
  license: Apache-2.0
  snapshot:
    label: "{{ Env.KCCTL_VERSION }}-early-access"
  extraProperties:
    inceptionYear: 2021

distributions:
  kcctl:
    type: NATIVE_IMAGE
    artifacts:
      - path: "artifacts/{{distributionName}}-{{projectVersion}}-linux-x86_64.zip"
        transform: "artifacts/{{distributionName}}-{{projectEffectiveVersion}}-linux-x86_64.zip"
        platform: linux-x86_64
      - path: "artifacts/{{distributionName}}-{{projectVersion}}-windows-x86_64.zip"
        transform: "artifacts/{{distributionName}}-{{projectEffectiveVersion}}-windows-x86_64.zip"
        platform: windows-x86_64
      - path: "artifacts/{{distributionName}}-{{projectVersion}}-osx-x86_64.zip"
        transform: "artifacts/{{distributionName}}-{{projectEffectiveVersion}}-osx-x86_64.zip"
        platform: osx-x86_64

We can appreciate project metadata in the project: section. Take special note of the snapshot.label setting, this is a new feature coming up in v0.6.0 and the reason for choosing early-access builds. Gunnar requested that the tag name for rolling releases match 1.0.0-early-access instead of the default early-access. the configuration makes use of Mustache templates to resolve and environment variable named KCCTL_VERSION to compose the final value for the snapshot.label property. We'll see shortly how and when that environment variable is set.

Next, on the distributions: section we can see a list of artifacts that match the conventions laid out in the Maven build. Each artifact is comprised of 3 properties:

  • the path of the artifact as created by the Maven build.
  • the platform that's associated with the artifact; you can see the value repeated in the artifact's filename.
  • a transformed path that looks similar to the original path except for the version template. This is required to change the path from containing 1.0.0-SNAPSHOT to 1.0.0-early-access.

This is it for now, nothing more needs to be done for JReleaser, as the tool is smart enough to harvest metadata from the existing Git repository and thus is able to post a release to GitHub.

Now for the final part, assembling the binaries and pushing the release using a GitHub Actions workflow.

Releasing with GitHub Actions

This is the part that took me the longest to figure out, because once again Windows threw a wrench into the plan. Let me explain. Compiling native executables using GraalVM Native Image requires a platform specific compiler, obviously. Both Linux and OSX runners are setup with the right settings from the start, however Windows runners require one extra step, not to mention the use of a file extension that the other runners do not require. This extra step requires setting up paths in such way that cl.exe becomes available to the GraalVM Native Image toolchain.

A quick googling and I found just the action I need to configure, ilammy/msvc-dev-cmd, along with DeLaGuardo/setup-graalvm for setting up a Graal distribution to run the build.

Once these two aspects are covered it's just a matter of configuring a build matrix that invokes the previously mentioned Maven commands to build and package each distribution. We'll collect all distributions into a single bucket, then consume all of them from a "release" job which invokes JReleaser via its jreleaser/release-action, the easiest way to get the job done if you ask me.

.github/workflows/early-access.yml

name: EarlyAccess

on:
  push:
    branches: [ main ]

jobs:
  # Build native executable per runner
  build:
    name: 'Build with Graal on ${{ matrix.os }}'
    if: github.repository == 'gunnarmorling/kcctl' && startsWith(github.event.head_commit.message, 'Releasing version') != true
    strategy:
      fail-fast: true
      matrix:
        os: [ ubuntu-latest, macOS-latest, windows-latest ]
        gu-binary: [ gu, gu.cmd ]
        exclude:
          - os: ubuntu-latest
            gu-binary: gu.cmd
          - os: macos-latest
            gu-binary: gu.cmd
          - os: windows-latest
            gu-binary: gu
    runs-on: ${{ matrix.os }}

    steps:
      - name: 'Check out repository'
        uses: actions/checkout@v2

      - name: 'Add Developer Command Prompt for Microsoft Visual C++ '
        if: ${{ runner.os == 'Windows' }}
        uses: ilammy/msvc-dev-cmd@v1

      - name: 'Set up Graal'
        uses: DeLaGuardo/setup-graalvm@4.0
        with:
          graalvm: '21.1.0'
          java: 'java11'

      - name: 'Install native-image component'
        run: |
          ${{ matrix.gu-binary }} install native-image

      - name: 'Cache Maven packages'
        uses: actions/cache@v1
        with:
          path: ~/.m2
          key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }}
          restore-keys: ${{ runner.os }}-m2

      - name: 'Build Native Image'
        run: mvn -B --file pom.xml -Pnative package

      - name: 'Create distribution'
        run: mvn -B --file pom.xml -Pdist package -DskipTests

      - name: 'Upload build artifact'
        uses: actions/upload-artifact@v2
        with:
          name: artifacts
          path: target/*.zip

  # Collect all executables and release
  release:
    needs: [ build ]
    runs-on: ubuntu-latest

    steps:
      - name: 'Check out repository'
        uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: 'Download all build artifacts'
        uses: actions/download-artifact@v2

      - name: 'Set up Java'
        uses: actions/setup-java@v2
        with:
          java-version: 11
          distribution: 'zulu'

      - name: 'Version'
        id: version
        run: |
          POM_VERSION=grep -oE -m 1 "<version>(.*)</version>" pom.xml | awk 'match($0, />.*?</) { print substr($0, RSTART+1, RLENGTH-2); }'
          KCCTL_VERSION=echo $POM_VERSION | awk 'match($0, /^(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)/) { print substr($0, RSTART, RLENGTH); }'
          echo "POM_VERSION = $POM_VERSION"
          echo "KCCTL_VERSION = $KCCTL_VERSION"
          echo "::set-output name=POM_VERSION::$POM_VERSION"
          echo "::set-output name=KCCTL_VERSION::$KCCTL_VERSION"

      - name: 'Release with JReleaser'
        uses: jreleaser/release-action@v1
        with:
          version: early-access
        env:
          JRELEASER_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          JRELEASER_PROJECT_VERSION: ${{ steps.version.outputs.POM_VERSION }}
          KCCTL_VERSION: ${{ steps.version.outputs.KCCTL_VERSION }}

The last piece of the puzzle is reading the version number from the pom.xml, parsing said version and generating two environment variables: the KCCTL_VERSION required by the jreleaser.yml configuration file and POM_VERSION which is used untouched as the project's version. Note that the release action requires an access token, in this case we can use the default GITHUB_TOKEN that's supplied to every workflow.

With all this setup in place, the only thing left is push a commit to the main branch to kickstart the process; triggering it again with every subsequent push to the main branch. This configuration was put together as a Pull Request that was merged by Gunnar, a few minutes later a release became available, or as Gunnar himself put it:


Conclusion

Building CLI applications with Quarkus is a straight forward task, just ask Gunnar or any other Quarkus user. Publishing binaries to a Git release is also a straight forward task if you happen to use JReleaser.

Keep on coding!

Image by David Mark from Pixabay

Liked it? Take a second to support aalmiray on Patreon!
Become a patron at Patreon!

1 comment

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

ˆ Back To Top