[Experimental] Background Mode Jobs

Jobs Background Mode requires Ray 1.7.0 or higher

Normally, when you create a Job through a Python script in Anyscale, the driver of your Job will run on the local machine while tasks and actors will run remotely on the Anyscale Cluster. If the driver on the local machine exits or disconnects, the Job and all associated tasks and actors on the Cluster will be killed.

In contrast, Jobs Background Mode runs the driver on the Anyscale Cluster, enabling the local machine to disconnect without ending your job. This enables you to run jobs in the background (e.g., to run a job overnight).

When using Background Mode, the logs for the job are streamed to the local machine and are also viewable in the Anyscale UI.

There are two supported APIs for running jobs in Background Mode.

Bash

# Set the RAY_ADDRESS environment variable to select an anyscale cluster
export RAY_ADDRESS="anyscale://cluster"

# Note: myscript.py is a python script that called ray.init()

# Interactive: this will run with the driver on the laptop
python myscript.py

# This will run with the driver on the head node
anyscale run -- python myscript.py
anyscale run -- rllib train -f train.yml

# You can also specify the runtime environment of your driver
# See https://docs.ray.io/en/latest/advanced.html#runtime-environments 
# To see the format of supported runtime environments
anyscale run --env env.yaml -- python myscript.py

# When you disconnect from your laptop, you should see a message printed:
...
This job will continue to run in background mode on Anyscale.
Job URL: https://anyscale.dev/jobs/job_VbYvaeTQ31gvm1ftyB8VR49Y

To kill this job, run the following in your terminal.

`anyscale kill job_VbYvaeTQ31gvm1ftyB8VR49Y`

# Kill the job
anyscale kill job_VbYvaeTQ31gvm1ftyB8VR49Y

Python

from anyscale.background import run

# bg_job is a BackgroundJob object
bg_job = run("python myscript.py", address="anyscale://cluster-1")

# synchronously wait for the job
bg_job.wait()

# kill the job
bg_job.kill()

# await the job in an async function
await bg_job

To use anyscale run from the CLI, RAY_ADDRESS must be set to an anyscale:// address.