Running RAFT on the cloud

Overview

RAFT supports an experimental Bring Your Own Cloud (BYOC) execution path on GCP Batch through the RAFT cloud launcher.

Warning

The BYOC launcher is experimental. Expect interface and execution details to change, and validate runs carefully before relying on it for routine production use.

Use the web launcher to:

  • upload or generate your manifest/config in the browser

  • write those files directly to your own GCS bucket

  • submit a Batch job directly in your own GCP project

  • keep infrastructure ownership in your own account

Launch the BYOC interface.

How to use the experimental BYOC path

  1. Open the BYOC launcher in your browser.

  2. Provide your GCP project, bucket, region, and runner image information.

  3. Choose a workflow.

  4. Provide your own data or use the available dataset path your workflow expects.

  5. Generate or upload your manifest and configuration in the browser.

  6. Submit the Batch job in your own cloud environment.

The launcher is intended to keep cloud ownership in your own account rather than relying on centrally managed RAFT infrastructure.

Before you begin, review the relevant cloud prerequisites in Installing RAFT.

Runner container

The Batch jobs launched by the BYOC UI use the RAFT runner container defined in:

  • containers/raft-runner/

See containers/raft-runner/README.md for build details.

Advanced CLI cloud mode

RAFT also exposes a --bucket option on raft run for advanced use cases where you want to parameterize a run against bucket-backed paths directly from the CLI.

Example:

$ raft run \
    --project-id my-project \
    --workflow lens \
    --version v1.9-dev \
    --bucket gs://my-bucket/raft

For this CLI path, RAFT uses the Python Google Cloud Storage client for GCS object checks and uploads before the workflow starts, authenticated through ADC from your local environment.

For bucket-backed CLI runs, RAFT also applies default cloud resourceLabels through a temporary Nextflow config overlay. The defaults are provider-aware:

  • s3:// infers AWS-oriented labels

  • gs:// infers GCP-oriented labels

You can override them per run with repeated --resource-label KEY=VALUE flags, or customize workspace defaults in .raft.cfg under cloud.resource_labels.

If you use the raft_scaling profile, RAFT will try to generate sample_multipliers.csv before launching the workflow by reading the manifest and the matching local or bucket-backed FASTQ sizes. If RAFT cannot resolve those FASTQ sizes, scaling falls back to the default multiplier of 1.0.

For additional setup and execution flags, see Helpful RAFT options.