Skip to content
Learning Outcomes
  • Understanding how to define connection profiles for different environments.
  • Step-by-step process to configure host, Amazon, and Google profiles.
  • Knowledge of executor settings and resource allocation for optimal performance.

Run Environments

In most cases, Foundry is already configured for your run environment and this step is not required. However, if you're submitting jobs to an HPC/cluster environment, you might need to create a run environment. In this case, a template configuration for your environment should be available. After selecting the template run environment, you can enter your user name for the connection.

Configure Run Environment

This section allows you to define connection profiles through the Create Environment button.

Nextflow Configuration

By default, Runs use executor settings established in your Run Environment. You can establish overrides at the Pipeline level or at the Process level in any Run on the Advanced section of your run page.

Executor of Nextflow

This setting controls how Via Foundry initiates Nextflow. Here you can establish the defaults for the underlying Nextflow executor.

Recommended configuration Executor of Nextflow

Supported Nextflow Executors

  • Local
  • SGE
  • SLURM
  • LSF

Recommended Parameters

  • Long: 8GB memory
  • 1 CPU
  • Time range: 5000-8000 minutes

Executor of Nextflow Jobs

This setting applies if no parameters are manually set in the Advanced section of your run page. If an option other than Local, AWS-Batch or GCP-Batch is selected, you must input values for Queue, Memory (GB), CPU, and Time (min).

Recommended configuration Executor of Nextflow Jobs

Supported Nextflow Executors

  • Local
  • SGE
  • SLURM
  • LSF
  • AWS-Batch
  • GCP-Batch

Resource Requirements

  • Short: 20GB memory
  • 1 CPU
  • 240 minutes of execution time

Defining Host Profiles

To configure a Host profile, use the following fields:

  • Username/Hostname: Enter your username and the hostname separately. For example, if your run is submitted to us2r@yourHostname, input us2r for the username and yourHostname for the hostname.

  • SSH Port (optional): The default SSH port is TCP port 22. Specify a different port number if required.

  • SSH Keys: SSH keys are stored in the SSH Keys tab and utilized for authentication during SSH connections.

  • Nextflow Path (optional): If the Nextflow executable is not in your $PATH, provide the path here, e.g.:

    /project/foundry/bin

  • Singularity Cache Folder: This is the directory where Singularity images from remote hosts are stored. By default, it is your home directory.

  • Profile Variables: Facilitate usage of genome reference and index files in your pipelines by specifying a download directory, preferably a shared path for multiple users. For example:

    params.DOWNDIR="/share/viafoundry"

  • Environment Variables: Set BASH environment variables here. Use newlines to separate multiple variables instead of spaces.

  • Executor Settings:

    • Within Via Foundry, there are four sections to control executor settings:

      1. Executor of Nextflow: Under Profile -> Run Environments, select the executor type (Local, SGE, SLURM, LSF) as needed.

      2. Executor of Nextflow Jobs: Default settings for submitted jobs when parameters in the Advanced section are not set.

      3. Executor Settings for All Processes: Overwrites settings defined for Executor of Nextflow Jobs and sets default parameters for all Nextflow jobs.

      4. Executor Settings for Each Process: Allows overriding of default settings on a per-process basis.

Warning

If non-standard resources or settings are needed, specify these in the Other Options box. For example, to submit an SGE job with 3 CPUs using parallel environments, enter -pe orte 3 (MPI) or -pe smp 3 (OpenMP) in the Other Options box.

Defining AWS Batch Profiles

To configure an AWS Batch profile, select the Host option and provide the following information:

  • Follow the same fields as in the Host profile above with the following modifications:

  • Executor of Nextflow: Choose the Local Nextflow executor.

  • Executor Settings for Nextflow: Enter 10GB in the Memory field and 1 in the CPU field.

  • Executor of Nextflow Jobs: Set this to AWS Batch.

  • Queue, Memory, CPU, and Other Options: Enter the queue name and default memory and CPU you plan to allocate to each job. Example: 10GB memory and 1 CPU.

  • Amazon Keys: In most cases, Foundry has built-in configuration for AWS access and this credential is not required.

  • Default Working Directory: Specify the default execution directory on the host machine (e.g., /opt/viafoundry/runs).

  • Default Bucket Location for Publishing: Define where run reports will be published (e.g., s3://bucket/viafoundry).

Defining Google Profiles

To configure Google profiles, include the following:

  • SSH Keys: Stored in the Credentials tab for authentication.

  • Google Keys: In most cases, Foundry has built-in configuration for GCP access and this credential is not required.

  • Region: Specify the Google region for computation (e.g., us-east1-b).

  • Project Id: In most cases, this section is automatically configured. If you selected a Google key above, you can leave this field blank. Otherwise, enter your Google Cloud Project ID. To find it, open the Google Cloud Console, go to the Dashboard, locate the Project info card, and copy the Project ID (for example, foundry-193616).

  • Nextflow Path (optional): Specify if not in $PATH.

  • Profile Variables: Similar to previous sections, set commonly used pipeline variables appropriately.

  • Executor of Nextflow/Executor of Nextflow Jobs: Similar to AWS, select Local for Executor of Nextflow and Google-Batch for Executor of Nextflow Jobs.

  • Default Working Directory: Specify the default execution directory on the host machine (e.g., /opt/viafoundry/runs).

  • Default Bucket Location for Publishing: Define where run reports will be published (e.g., gs://bucket/viafoundry).