Running RAFT with local/cloud hybrid

In hybrid mode, you install and run RAFT on your local machine but offload pipeline execution and data storage to cloud infrastructure using the --bucket flag or the more specific --project-bucket, --references-bucket, --shared-bucket, --fastqs-bucket, and --metadata-bucket options. This is the recommended way to run RAFT on AWS.

How it works

When you pass --bucket <bucket-url> to raft run, RAFT replaces local directory paths with cloud bucket URLs in the Nextflow command. This means:

  • Input data (FASTQs, references, metadata) is uploaded from your local filesystem to the specified bucket before the run starts.

  • Pipeline execution happens on cloud compute resources via your Nextflow profile (e.g., an AWS Batch profile).

  • Output data is written directly to the cloud bucket.

RAFT supports both s3:// (AWS) and gs:// (GCP) bucket URLs.

For bucket-backed runs, RAFT also applies provider-aware Nextflow resourceLabels defaults so cloud resources can be tagged consistently for cost tracking and governance. RAFT infers the provider from the bucket scheme:

  • s3:// routes use AWS-oriented defaults

  • gs:// routes use GCP-oriented defaults

RAFT also supports a split-routing model where different bucket parameters control different classes of data:

  • --project-bucket for project, work, FASTQ, and metadata paths

  • --references-bucket for params.ref_dir

  • --shared-bucket for params.shared_dir

  • --fastqs-bucket for discovering already-staged FASTQs

  • --metadata-bucket for discovering already-staged metadata

Prerequisites

  • RAFT installed locally (see Running RAFT locally)

  • For s3:// targets: an AWS account with a configured S3 bucket

  • For s3:// targets: the aws CLI installed and authenticated

  • For gs:// targets: a GCS bucket you control

  • For gs:// targets: Google Application Default Credentials (ADC) configured locally, for example with gcloud auth application-default login

  • A Nextflow profile configured for AWS Batch in your ~/.nextflow/config (see the Nextflow documentation for AWS Batch configuration)

  • Docker or Singularity/Apptainer available on the compute nodes

Note

RAFT uses the Python Google Cloud Storage client for gs:// uploads, not the gsutil or gcloud storage CLIs. You still typically use the system gcloud install once to create ADC credentials with:

$ gcloud auth application-default login

Run a project in hybrid mode

Create and run a new project, pointing --bucket at your S3 bucket:

$ raft run \
    --project-id my-project \
    --workflow lens \
    --version v1.9-dev \
    --manifest my_manifest.tsv \
    --bucket s3://my-bucket/raft \
    --profile awsbatch

If a project already exists, re-run it with:

$ raft run --project-id my-project --bucket s3://my-bucket/raft --profile awsbatch

Note

The --bucket flag replaces local paths (projects, references, inputs, work directory) with their bucket equivalents in the Nextflow command. It also uploads references, FASTQs, and metadata from your local filesystem to the bucket before launching the pipeline. RAFT sets params.clean_intermediates to '' by default. Pass --delete-intermediates if you want cleanup enabled for the run.

Note

Use --references-bucket to point RAFT at a specific bucket or prefix for references. RAFT stores reference files under references/ there. If you do not also pass --shared-bucket, RAFT uses that same bucket root for params.shared_dir under shared/ as well. --references-bucket cannot be combined with --bucket, but it can be combined with --project-bucket.

Note

Use --project-bucket when you want a specific bucket or prefix for project, work, FASTQ, and metadata paths without using the all-in-one --bucket flag.

Note

When you use --bucket or any of the split --*-bucket options, RAFT enables cloud resource labels by default. Edit .raft.cfg under cloud.resource_labels to change the defaults for your workspace, or add one or more --resource-label KEY=VALUE flags on the command line to override labels for a specific run.

Note

Do not repeat setup-only flags such as --workflow, --version, --manifest, or --setup-only when running an existing project.

Use multiple bucket parameters

Use the split bucket-routing options when you do not want one bucket or prefix to own every part of the run.

This exists for a few practical reasons:

  • to keep reusable references and shared assets in a central cache while still writing project-specific work and outputs somewhere else

  • to reuse FASTQs or metadata that are already staged in another bucket or prefix, instead of copying everything into one monolithic bucket layout first

  • to separate buckets by permissions, ownership, retention policy, or cost

  • to make cloud runs fit existing storage layouts rather than forcing all data into the all-in-one --bucket structure

Example: keep project/work data in one location, but references and shared files in separate caches:

$ raft run \
    --project-id my-project \
    --workflow lens \
    --version v1.9-dev \
    --manifest my_manifest.tsv \
    --project-bucket s3://my-bucket/project-root \
    --references-bucket s3://my-bucket/reference-cache \
    --shared-bucket s3://my-bucket/shared-cache \
    --profile awsbatch

Example: keep project/work data in the project bucket, but discover FASTQs and metadata from separate staged locations:

$ raft run \
    --project-id my-project \
    --workflow lens \
    --version v1.9-dev \
    --manifest my_manifest.tsv \
    --project-bucket s3://my-bucket/project-root \
    --fastqs-bucket s3://my-bucket/fastq-cache \
    --metadata-bucket s3://my-bucket/metadata-cache \
    --profile awsbatch

Using GCS in hybrid mode

While the experimental BYOC cloud launcher (see Running RAFT on the cloud) is available for GCP users, hybrid mode also works with GCS buckets:

$ raft run \
    --project-id my-project \
    --workflow lens \
    --version v1.9-dev \
    --manifest my_manifest.tsv \
    --bucket gs://my-bucket/raft \
    --profile google

For gs:// hybrid runs, RAFT uses the Python Google Cloud Storage client to check and upload objects before launching Nextflow. Those operations use Application Default Credentials from your local environment.

For additional setup and execution flags, see Helpful RAFT options. For common errors and next-step commands, see Troubleshooting.