Native integration between Apache Spark and Volcano Kubernetes scheduler
Series of related articles
What is Volcano ?
Volcano is a Kubernetes native system for high-performance workloads, which has been accepted by Cloud Native Computing Foundation (CNCF) as its first and only official container batch scheduling project. Volcano supports popular computing frameworks such as Spark, TensorFlow, PyTorch, Flink, Argo, MindSpore, and PaddlePaddle. Volcano also supports scheduling of computing resources on different architecture, such as x86, Arm, and Kunpeng.
What does Volcano provides ?
- Gang scheduling — a.k.a. all or nothing. Volcano will try to reserve resources so it is possible to execute all tasks at once, or postpone the execution of the tasks until the required resources are available. It is used for scenarios that require multi-process collaboration.
- Queue scheduling — hierarchical pools with min/max resources quotas
- Fair-share scheduling — a scheduling algorithm that makes sure that unrelated queues with same weight have equal access to the underlying resources
- Preemption scheduling — resource preemption between jobs in a queue , or between tasks in a job
- Topology-based scheduling — an algorithm that computes the priority of tasks and nodes based on the affinity and anti-affinity configuration between tasks within a job
- Reclaim — an algorithm that reclaims the resources for a queue when a new task is added to the queue and the allocated resources for it are no more enough
- Backfill — an algorithm that re-schedules (smaller) jobs as soon as the required amount of resources are made available / freed. It helps for better throughput
That is, depending on your applications’ requirements you could use Volcano to reserve specific resources for them, to prioritize them, to prioritize a specific task(s) over others, etc.
Spark + Volcano integration
Before Spark 3.3.0 the only way to use Spark with Volcano scheduler was the Spark on Kubernetes operator — a Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes. The idea behind it is that you install Kubernetes CRD (Custom Resource Definition) that glues Spark with Volcano.
With Spark 3.3.0 you can take advantage of Volcano’s facilities by adding few additional configuration properties to your
- The first step you have to do is to install Volcano on your Kubernetes cluster:
$ kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.5.1/installer/volcano-development.yaml
$ kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/v1.5.1/installer/volcano-development-arm64.yaml
Note: version 1.5.1 is the minimum required for Spark 3.3.0, but any newer version is also fine!
2. The next step is to add the new Spark config properties:
This property instructs Spark to set
volcano as a value for
spec.schedulerName for the pods, e.g. their YAML would contain:
Note: the unrelated properties are removed!
3. The next properties we need to set are:
VolcanoFeatureStep is responsible to add a metadata annotation to all pods (driver and executor(s)) so that Volcano recognizes them as manageable. See metadata > annotations > scheduling.k8s.io/group-name above.
In addition this feature step will create a PodGroup, so that all the pods are managed together (Gang scheduling). The name of this PodGroup will be
my-application-id-podgroup and it will be added to the same Kubernetes namespace as all other Spark resources (property
spark.kubernetes.namespace) but it won’t set resource quotas!
$ kubectl get podgroup -n spark-on-k8s
spark-eb75b0d69737441c9b3d0e784692baee-podgroup 32m$ kubectl get podgroup/spark-eb75b0d69737441c9b3d0e784692baee-podgroup -n spark-on-k8s -oyaml
- apiVersion: v1
- lastTransitionTime: "2022-03-25T07:49:37Z"
reason: tasks in gang are ready to be scheduled
To make use of Volcano’s features you should provide a PodGroup template with:
The template might contain any valid Volcano PodGroup settings.
The PriorityClass and the Queue are cluster-wide Kubernetes resources, so they have to be pre-created by a user with the respective permissions.
$ kubectl apply -f priority-classes.yaml
$ kubectl apply -f my-queue.yaml
Congratulations! You have successfully ran your first Spark job with Volcano!