GithubActions: Build and test on HuaweiCloud ARM64

If you have read my previous articles you might have noticed that ARM64 CPU architecture conquerors the server side!

Any software product that takes itself serious should use Continuous Integration (CI) system to regularly build and to run its tests!

If you manage the CI system yourself and you have access to ARM64 hardware then almost any CI system could do the job. For example I know about these projects:

• several Apache projects (e.g. Hadoop, Cassandra, Kafka) use Jenkins with ARM64 agents
• Memcached uses Buildbot
• JBoss Wildfly uses Teamcity

But many open source projects prefer to use cloud based CI systems and not bother with maintaining peripheral things like CI. Few years ago the most famous cloud based CI was TravisCI! TravisCI is very ARM64 friendly! By just adding arch: arm64[-graviton2] to your .travis.yml is enough to run your jobs on ARM64 hardware! TravisCI is still used by many but recently they changed their billing plans and now every project has up to 10000 free credits. After that one needs to change to a paid plan.

What are the alternatives ?

Github Actions is the new favorite cloud based CI for almost every OSS project out there. But it does not support ARM64! There were plans to add such but recently this ticket has been edited and the mentioning of “ARM64” has been removed from it! Also there are many issues at https://github.com/actions/virtual-environments/issues?q=arm closed with “Azure does not support ARM64 (Linux and Mac) and we (Github) don’t have plans to support it”. Not good!

But this should not stop us! With a little creativity we could still test our software on ARM64!

One way is to use hardware emulation (Docker+QEMU) — I have explained this already here. Pros: Free! Cons: 1) might be slow; 2) some syscalls are not implemented in QEMU and it might not work for you.

Going one step further is to use real ARM64 hardware at your favorite cloud provider! In this article I am going to explain how to do it at HuaweiCloud.

The idea is in your Github Actions workflow to enable QEMU for ARM64, checkout the source code, store it in a fresh Docker ARM64 image, push the image in some Docker registry and finally execute the Docker image at HuaweiCloud’s Kubernetes. It is not that complicated! Follow me along!

Step 0: Enable QEMU/binfmt

- name: Enable QEMU/binfmt
  run: |
    docker run --rm --privileged multiarch/qemu-user-static --credential yes --persistent yes

This step is needed so we can create a new Docker image for ARM64 architecture on a x86_64 host!

Step 1: Checkout the project’s source code

- uses: actions/checkout@v2
  with:
    path: my-project

This step checks out my-project’s code into folder ./my-project.

Step 2: Store the code in a new Docker image

Dockerfile

FROM arm64v8/ubuntu:focal

COPY my-project /my-project
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
    apt-get install -y openjdk-8-jdk

ENTRYPOINT [ "/my-project/.github/scripts/build-and-test.sh" ]

The idea here is that we extend from an existing/base ARM64 image, copy the source code inside it, install any prerequisites and finally execute the build. If the build is more complex than a one-liner then we can extract it into a Shell script like I did. If you have a lot of prerequisites then it would be a good idea to create a base image with them so they are not downloaded and installed again and again. It will save you few minutes for each build.

Step 3: Build the new image

- name: Build new image
  run: |
    docker build --pull --tag ghcr.io/${{ github.repository }}/my-project-aarch64:${{ github.sha }} \
     --cache-from ghcr.io/${{ github.repository }}/my-project-aarch64 \
     --file my-project/.github/dockerfiles/Dockerfile \
     .

In the tag name we specify the Docker Registry where we want to push it — here I use Github Container Registry. The important part here is that either the Docker Registry should be readable from everyone or later you will have to use Kubernetes Secret to be able to reach it!

Step 4: Login to the Docker Image Registry and push the image

- name: Docker login
  uses: docker/login-action@v1
  with:
    registry: ghcr.io
    username: ${{ github.actor }}
    password: ${{ github.token }}
- name: Push the image
  run: docker push ghcr.io/${{ github.repository }}/my-project-aarch64:${{ github.sha }}

To be able to push to the registry you need to login first! github.actor and github.token variables are provided by Github Actions.

Step 5: Install kubectl

- name: "Install kubectl"
  uses: azure/setup-kubectl@v1

kubectl is the client tool that communicates with Kubernetes clusters.

Step 6: Setup KUBECONFIG environment variable

- name: Setup Huawei Cloud K8S environment
  uses: huaweicloud/cce-cluster-credentials@v1
  with:
    region: "${{ secrets.REGION }}"
    access-key-id: "${{ secrets.ACCESS_KEY_ID }}"
    access-key-secret: "${{ secrets.SECRET_ACCESS_KEY }}"
    project-id: "${{ secrets.PROJECT_ID }}"
    cluster-id: "${{ secrets.CLUSTER_ID }}"

This step is the single one specific to the cloud provider you use. HuaweiCloud developers kindly provide a Github Action that depending on the input variables configured in your Github project settings generates a proper Kubernetes config and exports it into the special environment variable named KUBECONFIG. In the next step kubectl will use it to connect to your Kubernetes cluster.

The values could be get from HuaweiCloud’s console. For more information consult with Cloud Container Engine’s documentation.

More examples how to use different cloud providers can be found here.

Step 7: Run the Docker image at Kubernetes

- name: Run the Docker image
  run: |-
    kubectl config use-context external # HuaweiCloud specific !
    kubectl run my-project-on-arm64-${{ github.run_id }} --rm -i --image ghcr.io/${{ github.repository }}/my-project-aarch64:${{ github.sha }}  --restart=Never --tty=false

Note: HuaweiCloud’s Kubernetes config has two contexts — internal and external. The first is used for communication between pods inside the Kubernetes cluster. The latter is to communicate with the Kubernetes resources from outside of the Kubernetes cluster.

This will run the Docker container on a fresh Kubernetes pod created dynamically just for this Github Actions workflow. Depending on the pod’s exit status the job will either succeed or fail.

Almost the same Github Actions workflow could be used for any other Kubernetes cluster. You just need to update step 6 to create KUBECONFIG for it.

A complete demo project could be found here.