Ray K8s cluster deployment

Ray K8s cluster deployment

        • 1. Introduction
        • 2. Ray on Kubernetes
          • 2.1 Get Started
          • 2.2User Guides
            • 2.2.1 Managed K8s services
            • 2.2.2 RayCluster Configuration
            • 2.2.3 Kuberay Autoscaling
            • 2.2.4 Logging
            • 2.2.5 Using GPUs
            • 2.2.6 Experimental Features
            • 2.2.7 (Advanced) Deploying a static Ray Cluster without Kuberay
          • 2.3 Examples
            • 2.3.1 Ray AIR XGBoost Trainer on K8s
            • 2.3.2 ML training with GPUs on k8s
        • 3. Ray Job

1. Introduction

  • A ray cluster is a set of worker nodes connected to a common Ray head node.
    • A Ray cluster is worker nodes connected to the same master node. Ray clusters can remain fixed in size or scale automatically
  • Ray cluster supports building on k8s cluster through Kuberay project
  • Key Concept
    • Ray Cluster
      • A ray cluster consists of a master node and any number of worker nodes
      • On a k8s cluster, a ray node is equivalent to a pod
      • Users can execute tasks on the Ray cluster by submitting Jobs, or connect to the master node through ray.init and respond interactively
    • Head Node
      • Except that the master node also independently runs the Autoscaler and Ray Driver Process processes, it is no different from other worker nodes
    • Worker Node
      • The working node does not run any management process similar to the master node, and is only used to execute the code of tasks such as user tasks and actors
      • They also participate in distributed scheduling, while storing and distributing Ray Objects in the cluster memory
    • Autoscaling
      • It is an independent process running on the master node, or a sidecar container in the k8s head pod
      • Responsible for increasing or decreasing the number of worker nodes in the cluster based on certain conditions
    • Ray Jobs
      • A Ray job is a single application: it is the collection of Ray tasks, objects, and actors that originate from the same script. The worker that runs the Python script is known as the driver of the job.
      • A Ray Job is similar to a single application: it is a collection of Ray tasks, objects, actors from the same project code. The worker process that initially executed the project code is also marked as the driver process (driver process)
      • There are three ways to execute a Ray Job in a Ray cluster:
        • 1) Recommended way: Submit a job to the cluster through the Ray Jobs API
        • 2) Run directly on one of the nodes in the Ray cluster
        • 3) For experts only: use Ray Client and a Ray Script to remotely connect to the Ray cluster
          Three ways to run Ray Jobs

2. Ray on Kubernetes

2.1 Get Started
  • Using the Kuberay Operator is the officially recommended way. The operator provides a k8s native way to manage ray clusters

  • In this section, we can learn:

    • 1) How to install and configure Ray cluster on k8s cluster
    • 2) Deploy and monitor Ray applications
    • 3) Integrate Ray applications with Kubernetes networking
  • 1) Installation and deployment

    • Note: I may have used an earlier version of Kuberay operator when I deployed Ray before

    • 1) Pre-work: Install the latest version of Ray release on the local machine, install kubectl, and be able to access the k8s cluster

      • pip install -U “ray[default]”, the purpose is to be able to interact with the Ray cluster by submitting the Ray Job
      • Install kubectl, in order to be able to interact with the k8s cluster
      • Install or configure k8s cluster: for subsequent deployment of ray cluster
    • 2) First deploy the kuberay operator: a pod used to create and manage the ray cluster

      • a) Deploy the kuberay operator according to the configuration file

        • # Note that create cannot be replaced by apply, the issue has been raised
          # It can also be deployed by downloading the kuberay git repository and finding the configuration file of the kuberay operator
          # The Namespace.yaml and kustomization files in the default folder can customize the name of the namespace in K8s, and the default is ray-system
          ! kubectl create -k "github.com/ray-project/kuberay/ray-operator/config/default?ref=v0.4.0 &timeout=90s"
          
          # Check that kuberay-operator has run successfully
          ! kubectl -n ray-system get pod --selector=app.kubernetes.io/component=kuberay-operator
          
    • 3) Deploy Ray cluster

      • The kuberay operator has been successfully executed, and now a simple ray cluster can be deployed through it

        # Deploy a sample Ray Cluster CR from the KubeRay repo:
        # This configuration file can also be found in the kuberay git repository
        ! kubectl -n ray-system apply -f https://raw.githubusercontent.com/ray-project/kuberay/master/ray-operator/config/samples/ray-cluster.autoscaler.yaml
        
        # This Ray cluster is named `raycluster-autoscaler` because it has optional Ray Autoscaler support enabled.
        
    • 4) Run the Ray application on the Ray cluster:

      • a) Execute directly in the pod using the kubect exec command

      • # Substitute your output from the last cell in place of "raycluster-autoscaler-head-xxxxx"
        
        ! kubectl exec raycluster-autoscaler-head-xxxxx -it -c ray-head -- python -c "import ray; ray.init()"
        # 2022-08-10 11:23:17,093 INFO worker.py:1312 -- Connecting to existing Ray cluster at address: <IP address>:6379...
        # 2022-08-10 11:23:17,097 INFO worker.py:1490 -- Connected to Ray cluster.
        
        • ray.init() Initialize the links included in the detailed analysis later
      • b) Through Ray Job Submission

        # First forward the port of the master node
        # Execute this in a separate shell.
        !kubectl port-forward service/raycluster-autoscaler-head-svc 8265:8265
        
        # Then submit the ray code where you have the ray release locally
        # The following job's logs will show the Ray cluster's total resource capacity, including 3 CPUs.
        
        !ray job submit --address http://localhost:8265 -- python -c "import ray; ray.init(); print(ray.cluster_resources())"
        
    • 5) Cleanup work: uninstall Ray cluster, kubera operator and related resources

      • Cleanup works as follows:

        # Delete by reference to the RayCluster custom resource
        ! kubectl delete raycluster raycluster-autoscaler -n ray-system
        
        !kubectl delete -k "github.com/ray-project/kuberay/ray-operator/config/default?ref=v0.4.0 &timeout=90s" -n ray-system
        
2.2 User Guides
2.2.1 Managed K8s services
  • This step is to teach how to quickly build k8s clusters on platforms such as Google Cloud, Amazon Cloud, and Microsoft Cloud
2.2.2 RayCluster Configuration
  • This part of the guide explains what is included in deploying a ray cluster on k8s
  • Pod configuration: headGroupSpec and workerGroupSpecs
    • Pod templates
      • resources
        • Note: It is recommended to use fewer Ray Pods with larger resources instead of more Ray Pods with smaller resources. The advantages are:
          • more efficient use of each Ray pod’s shared memory object store
          • Reduced communication overhead between Ray pods
          • Reduced redundancy of per-pod Ray control structures such as Raylets
        • ? If it is used in the serverless field, how to solve the problem of the expansion speed of Ray Pods?
2.2.3 Kuberay Autoscaling
  • Pod expansion process:
    [External link picture transfer failed, the source site may have an anti-leeching mechanism, it is recommended to save the picture directly Upload (img-2dY6UlEc-1684819698231)(C:\Users\14112\AppData\Roaming\Typora\typora-user-images\image-20230404105020184.png)]

    • Worker pod upscaling occurs through the following sequence of events:
      1. The user submits a Ray workload.
      2. Workload resource requirements are aggregated by the Ray head container and communicated to the Ray autoscaler sidecar.
      3. The autoscaler determines that a Ray worker pod must be added to satisfy the workload’s resource requirement.
      4. The autoscaler requests an additional worker pod by incrementing the RayCluster CR’s replicas field.
      5. The KubeRay operator creates a Ray worker pod to match the new replicas specification.
      6. The Ray scheduler places the user’s workload on the new worker pod.
  • suggestion:

    • In practical applications, Ray developers recommend setting the Ray Pod resource size as large as the k8s node
    • ? But does this Ray Pod result in lower k8s resource usage? Will there be security and isolation issues?
  • Scaling speed:

    • The speed of scaling can be controlled through the autoscalerOptions field in RayCluster CR, which contains two subfields
      • upscalingMode:
        • Conservative: upscaling is rate-limited, the number of pending worker pods is at most the number of worker pods connected to the Ray cluster
        • Default: upscaling is not rate-limited
        • Aggressive: an alias for Default, upscaling is not rate-limited
      • idleTimeoutSeconds: default 60s, 60s without any tasks, actors, objects tasks can shrink
  • Ray Autoscaler vs. Horizontal Pod Autoscaler

    • There is a certain difference between the extender provided by RAY and the HPA provided by K8s
  • Relationship between Ray Autoscaler and K8s Cluster Autoscaler

  • Relationship to Vertical Pod Autoscaler VPA

2.2.4 Logging
  • See here for now
2.2.5 Using GPUs
2.2.6 Experimental Features
2.2.7 (Advanced) Deploying a static Ray Cluster without Kuberay
2.3 Examples
2.3.1 Ray AIR XGBoostTrainer on K8s
2.3.2 ML training with GPUs on k8s

3. Ray Job

  • The local scripts project can be submitted locally to the Ray cluster for execution through Ray job submit

    $ ray job submit --working-dir your_working_directory -- python script.py
    
    • This command will run the script on the Ray Cluster and wait until the job has finished. Note that it also streams the stdout of the job back to the client (hello world in this case). Ray will also make the contents of the directory passed as --working-dir available to the Ray job by downloading the directory to all nodes in your cluster.
  • Warning:

    • When using the Ray Jobs API, the runtime environment should be specified only in the Jobs API (e.g. in ray job submit --runtime-env=... or JobSubmissionClient.submit_job(runtime_env= ...)), not via ray.init(runtime_env=...) in the driver script.