Running RAFT with local/cloud hybrid

In hybrid mode, you install and run RAFT on your local machine but offload pipeline execution and data storage to cloud infrastructure using the --cloud flag. This is the recommended way to run RAFT on AWS.

How it works

When you pass --cloud <bucket-url> to raft run, RAFT replaces local directory paths with cloud bucket URLs in the Nextflow command. This means:

  • Input data (FASTQs, references, metadata) is uploaded from your local filesystem to the specified bucket before the run starts.

  • Pipeline execution happens on cloud compute resources via your Nextflow profile (e.g., an AWS Batch profile).

  • Output data is written directly to the cloud bucket.

RAFT supports both s3:// (AWS) and gs:// (GCP) bucket URLs.

Prerequisites

  • RAFT installed locally (see Running RAFT locally)

  • An AWS account with a configured S3 bucket

  • The aws CLI installed and authenticated

  • A Nextflow profile configured for AWS Batch in your ~/.nextflow/config (see the Nextflow documentation for AWS Batch configuration)

  • Docker or Singularity/Apptainer available on the compute nodes

Run a project in hybrid mode

Create and run a new project, pointing --cloud at your S3 bucket:

$ raft run \
    --project-id my-project \
    --workflow lens \
    --version v1.9-dev \
    --manifest my_manifest.tsv \
    --cloud s3://my-bucket/raft \
    --profile awsbatch

If a project already exists, re-run it with:

$ raft run --project-id my-project --cloud s3://my-bucket/raft --profile awsbatch

Note

The --cloud flag replaces local paths (projects, references, inputs, work directory) with their bucket equivalents in the Nextflow command. It also uploads references, FASTQs, and metadata from your local filesystem to the bucket before launching the pipeline.

Note

Do not repeat setup-only flags such as --workflow, --version, --manifest, or --setup-only when running an existing project.

Using GCS in hybrid mode

While the BYOC cloud launcher (see Running RAFT on the cloud) is the primary path for GCP users, hybrid mode also works with GCS buckets:

$ raft run \
    --project-id my-project \
    --workflow lens \
    --version v1.9-dev \
    --manifest my_manifest.tsv \
    --cloud gs://my-bucket/raft \
    --profile google

For additional setup and execution flags, see Helpful RAFT options. For common errors and next-step commands, see Troubleshooting.