Preparing your samples
======================

.. The dataset status table below relies on raw HTML/JavaScript to fetch and
   render live status from the GitLab API. That section will be empty or broken
   in non-HTML output formats (PDF, LaTeX, man pages) and requires internet
   access to display.

RAFT supports both user-provided samples and off-the-shelf immuno-oncology
datasets.

Making your files available to RAFT
-----------------------------------

RAFT starts each workflow using sample-associated data. These data could be
FASTQs, BAMs, VCFs, or RNA count data. Regardless of the type of input data,
they all need to be made discoverable by RAFT. Users can make input files
discoverable by RAFT by copying or symlinking them into the RAFT ``inputs/``
directories, for example ``/path/to/raft/inputs/fastqs``. Grouping input files
by dataset, such as
``/path/to/raft/inputs/fastqs/my_favorite_dataset/samp1_1.fq.gz``, is
recommended but not required.

.. note::
   Input data for demonstration workflows and off-the-shelf datasets are
   downloaded automatically.


The RAFT manifest
-----------------

A manifest describes the samples in a dataset and tells RAFT:

- how the samples should be named
- how the samples are related
- what input files are associated with each sample

For user-provided datasets, you supply this manifest directly. For supported
off-the-shelf datasets, RAFT can generate the run manifest automatically as
part of dataset preparation.


Using an off-the-shelf dataset
------------------------------

The table below lists the currently supported off-the-shelf immuno-oncology
datasets. Click a dataset name to view its publication, abstract, and README.

.. raw:: html

   <style>
     .raft-ds-status {
       display: grid;
       gap: 1.5rem;
       margin-bottom: 2rem;
     }
     .raft-ds-status__meta {
       color: #4b5563;
       font-size: 0.95rem;
     }
     .raft-ds-status__error {
       border-left: 4px solid #b91c1c;
       background: #fef2f2;
       color: #7f1d1d;
       padding: 0.9rem 1rem;
     }
     .raft-ds-status__section {
       border: 1px solid #d1d5db;
       border-radius: 0.5rem;
       overflow: hidden;
       background: #ffffff;
     }
     .raft-ds-status__section-header {
       display: flex;
       justify-content: space-between;
       gap: 1rem;
       align-items: baseline;
       padding: 0.9rem 1rem;
       background: #f9fafb;
       border-bottom: 1px solid #e5e7eb;
     }
     .raft-ds-status__section-header h2 {
       margin: 0;
       font-size: 1.1rem;
     }
     .raft-ds-status__summary {
       color: #374151;
       font-size: 0.95rem;
       text-align: right;
     }
     .raft-ds-status__table-wrap {
       overflow-x: auto;
     }
     .raft-ds-status table {
       width: 100%;
       border-collapse: collapse;
     }
     .raft-ds-status th,
     .raft-ds-status td {
       padding: 0.7rem 1rem;
       text-align: left;
       border-bottom: 1px solid #e5e7eb;
       vertical-align: middle;
     }
     .raft-ds-status th {
       background: #ffffff;
       font-weight: 700;
     }
     .raft-ds-status td:last-child,
     .raft-ds-status th:last-child {
       text-align: center;
     }
     .raft-ds-status tbody tr:hover {
       background: #f9fafb;
     }
     .raft-ds-status__badge {
       display: inline-block;
       min-width: 6.5rem;
       padding: 0.22rem 0.55rem;
       border-radius: 999px;
       font-size: 0.84rem;
       font-weight: 700;
       text-align: center;
       white-space: nowrap;
     }
     .raft-ds-status__badge--pass     { background: #dcfce7; color: #166534; }
     .raft-ds-status__badge--fail     { background: #fee2e2; color: #991b1b; }
     .raft-ds-status__badge--progress { background: #fef3c7; color: #92400e; }
     .raft-ds-status__badge--none     { background: #e5e7eb; color: #374151; }
     .raft-ds-status__muted { color: #6b7280; }
   </style>

   <div id="raft-dataset-status" class="raft-ds-status">
     <div id="raft-dataset-status-meta" class="raft-ds-status__meta">Loading datasets from GitLab...</div>
   </div>

   <script>
   (function () {
     var root = document.getElementById("raft-dataset-status");
     var meta = document.getElementById("raft-dataset-status-meta");
     if (!root || !meta) return;

     var apiBase = "https://gitlab.com/api/v4";
     var datasetGroup = "reproducible-analyses-framework-and-tools%2Fmodules%2Fdataset-prep";

     var tumorTypes = {
       /* dataset-name: "Tumor Type" */
     };

     function escapeHtml(value) {
       return String(value).replace(/[&<>"']/g, function (ch) {
         return { "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" }[ch];
       });
     }

     function statusMeta(raw) {
       if (raw === "success") return { label: "Pass", cls: "raft-ds-status__badge--pass" };
       if (raw === "failed" || raw === "canceled") return { label: "Fail", cls: "raft-ds-status__badge--fail" };
       if (raw === "no-pipeline") return { label: "No pipeline", cls: "raft-ds-status__badge--none" };
       if (String(raw).startsWith("http-") || raw === "unavailable") return { label: "Unavailable", cls: "raft-ds-status__badge--none" };
       return { label: "In progress", cls: "raft-ds-status__badge--progress" };
     }

     function compareStatus(a, b) {
       var rank = function (s) {
         if (s === "success") return 0;
         if (s === "failed" || s === "canceled") return 1;
         if (["running", "pending", "created", "preparing", "waiting_for_resource", "manual"].indexOf(s) !== -1) return 2;
         if (s === "no-pipeline") return 3;
         return 4;
       };
       return rank(a.pipelineStatus) - rank(b.pipelineStatus) || a.name.localeCompare(b.name);
     }

     function fetchJson(url) {
       return fetch(url, { headers: { Accept: "application/json" } }).then(function (r) {
         if (!r.ok) throw new Error("HTTP " + r.status + " for " + url);
         return r.json();
       });
     }

     function fetchAllProjects() {
       var projects = [];
       var next = function (page) {
         var url = apiBase + "/groups/" + datasetGroup + "/projects?simple=true&per_page=100&page=" + page;
         return fetchJson(url).then(function (data) {
           if (!Array.isArray(data) || data.length === 0) return projects;
           projects = projects.concat(data);
           return data.length < 100 ? projects : next(page + 1);
         });
       };
       return next(1);
     }

     function fetchPipeline(project) {
       var url = apiBase + "/projects/" + encodeURIComponent(project.id) + "/pipelines?per_page=1";
       return fetchJson(url).then(function (data) {
         if (!Array.isArray(data) || data.length === 0) return { status: "no-pipeline", ref: "" };
         return { status: data[0].status || "unknown", ref: data[0].ref || "" };
       }).catch(function (err) {
         var m = /HTTP (\d+)/.exec(String(err && err.message || ""));
         return { status: m ? "http-" + m[1] : "unavailable", ref: "" };
       });
     }

     function mapWithConcurrency(items, limit, worker) {
       var results = new Array(items.length);
       var index = 0;
       function run() {
         var current = index++;
         if (current >= items.length) return Promise.resolve();
         return Promise.resolve(worker(items[current], current)).then(function (result) {
           results[current] = result;
           return run();
         });
       }
       var workers = [];
       var n = Math.min(limit, items.length || 1);
       for (var i = 0; i < n; i++) workers.push(run());
       return Promise.all(workers).then(function () { return results; });
     }

     function renderTable(rows) {
       var counts = { Pass: 0, Fail: 0, "In progress": 0, "No pipeline": 0, Unavailable: 0 };
       rows.forEach(function (r) { counts[statusMeta(r.pipelineStatus).label] += 1; });
       var total = rows.length;
       var summary = Object.keys(counts)
         .filter(function (k) { return counts[k] > 0; })
         .map(function (k) {
           return k + ": " + counts[k] + " (" + ((counts[k] / total) * 100).toFixed(0) + "%)";
         }).join(", ") || "No repositories found";

       var tbody = rows.map(function (row) {
         var badge = statusMeta(row.pipelineStatus);
         var detailUrl = "dataset-detail.html?dataset=" + encodeURIComponent(row.name);
         var tumorType = tumorTypes[row.name] || "\u2014";
         return "<tr>" +
           '<td><a href="' + escapeHtml(detailUrl) + '">' + escapeHtml(row.name) + "</a></td>" +
           "<td>" + escapeHtml(tumorType) + "</td>" +
           '<td><span class="raft-ds-status__badge ' + badge.cls + '">' + escapeHtml(badge.label) + "</span>" +
           (row.pipelineRef ? ' <span class="raft-ds-status__muted">(' + escapeHtml(row.pipelineRef) + ")</span>" : "") +
           "</td>" +
           '<td><a href="' + escapeHtml(row.webUrl + "/-/issues") + '">Report an issue</a></td>' +
           "</tr>";
       }).join("");

       return '<section class="raft-ds-status__section">' +
         '<div class="raft-ds-status__section-header">' +
           "<h2>Datasets</h2>" +
           '<div class="raft-ds-status__summary">' + escapeHtml(summary) + "</div>" +
         "</div>" +
         '<div class="raft-ds-status__table-wrap">' +
           "<table>" +
             "<thead><tr><th>Dataset</th><th>Tumor Type</th><th>CI Status</th><th></th></tr></thead>" +
             "<tbody>" + tbody + "</tbody>" +
           "</table>" +
         "</div>" +
       "</section>";
     }

     function load() {
       meta.textContent = "Loading datasets from GitLab...";
       fetchAllProjects().then(function (projects) {
         meta.textContent = "Loading pipeline status...";
         return mapWithConcurrency(projects, 8, function (project) {
           return fetchPipeline(project).then(function (pipeline) {
             return { name: project.path, webUrl: project.web_url, pipelineStatus: pipeline.status, pipelineRef: pipeline.ref };
           });
         });
       }).then(function (rows) {
         rows.sort(compareStatus);
         var now = new Date();
         root.innerHTML =
           '<div class="raft-ds-status__meta">Last loaded from GitLab: ' + escapeHtml(now.toLocaleString()) + "</div>" +
           renderTable(rows);
       }).catch(function (err) {
         root.innerHTML = '<div class="raft-ds-status__error">' +
           "Unable to load dataset status from GitLab. This can happen because of network restrictions, " +
           "GitLab rate limiting, or browser CORS policy. " +
           "Error: " + escapeHtml(String(err && err.message || err)) +
           "</div>";
       });
     }

     load();
   })();
   </script>

.. toctree::
   :hidden:

   dataset-detail


Specifying an off-the-shelf dataset with RAFT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To create a new project from an off-the-shelf dataset, simply pass
``--dataset`` and the dataset's identifier to RAFT:

.. code-block:: console

  $ raft run \
      --project-id my-project \
      --workflow lens \
      --dataset <DATASET_NAME> \
      --version v1.9-dev \

RAFT will clone the dataset-prep module, download the required FASTQs from
ENA/EBI, and stage them before execution. No user-provided manifest is required;
the run manifest is generated automatically.

Using your own samples
----------------------

Running your own samples through RAFT requires creating a manifest for your
dataset. RAFT provides a `web-based interface <https://spvensko.github.io/raft-ui/raft-manifest-generator.html>`_ to help generate that manifest.
This interface becomes available when you execute ``raft run`` on a new project
for the first time.

.. figure:: _static/manifest-generator.png
   :alt: RAFT manifest generator interface

   The RAFT manifest generator interface.

If you prefer to create the manifest separately and then provide it with
``raft run ... --manifest <MANIFEST_FILE_NAME.tsv>``, you can use the hosted
manifest generator `here <https://spvensko.github.io/raft-ui/raft-manifest-generator.html>`_.

More information about the manifest format is available in :doc:`manifest`.

Specifying your own samples with RAFT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you already have a manifest, create a new project by passing ``--manifest``
and the manifest filename.

.. note::
   Manifest TSVs must be in your RAFT's ``inputs/metadata`` directory to be
   discovered by RAFT.

.. code-block:: console

  $ raft run \
      --project-id my-project \
      --workflow lens \
      --version v1.9-dev \
      --manifest <MANIFEST_NAME.TSV>

Alternatively, if you want to make your manifest as part of running RAFT, then
simply run RAFT without ``--manifest``:

.. code-block:: console

  $ raft run \
      --project-id my-project \
      --workflow lens \
      --version v1.9-dev \