Nextflow training

Enter this command in your terminal:

Or, if you prefer curl:

Then ensure that the downloaded binary is executable:

AND put the nextflow executable into your $PATH (e.g. /usr/local/bin)

Download the training material by copy & pasting the following command in the terminal:

Check the correct installation of nextflow by running the following command:

Click the following URL, which opens the course:

training.seqera.io/

To run Gitpod:

gitpod.io/#https://github.com/seqeralabs/nf-training-public

(which is our Github repository URL, prefixed with gitpod.io/#).

OR

Install the Gitpod browser extension, following the instructions here. This will add a green Gitpod button to each Git repository for easy access to the environment. Then go to the git repository for the training (github.com/seqeralabs/nf-training-public), and click the green button.

You may need to close the window at each stage and click the gitpod link again.

For BitBucket or GitLab, you may have to sign into Gitpod and change your integration on the following page: gitpod.io/integrations.

Once you have signed in, gitpod will load (skip prebuild, if asked):

Gitpod gives you 50 hours per month to run the environment, with up to 4 parellel workspaces at a time. So feel free to come back at any time and try the course at your own pace.

It is useful to save the URL that is now your Gitpod environment. Later, we can use this to return to the same closed environment (so please take note of it now).

You should see something similar to the following:

The sidebar allows you to customise your Gitpod environment and perform basic tasks (Copy/Paste, Open files, search, git, etc.) Click the Explorer button to see which files are in this repository.

The terminal allows you to run all the programs in the repository, for example nextflow and docker are installed.

The main window allows you to view and edit files. Clicking on a file in the explorer will open it within the main window. We can also see the nf-training material in this window, but if you find the space too small you can open it separately in another browser window (training.seqera.io/). Finally, if you accidently close the Gitpod training window, you can reopen it by entering gp preview training.seqera.io.

To save your files, choose your file of interest, then either use the left/top hand side bar (File/Save As…​) or use you’re keyboard shortcut to save as (On Mac: CMD, shift, s), then choose Show local. This will open up a explorer to choose where to save your file on your local machine.

Any running workspace will be automatically stopped after 30 minutes. You can open the environment again by going to gitpod.io/workspaces and finding your previous environment, then clicking the three dot button to the right, and selecting Open.

If you save the URL from your previous Gitpod environment, you can just paste this into your browser to open the previous environment. Environments are saved for up to two weeks, but don’t rely on their existance, download any important files you want for posterity.

Alternatively, you can start a new workspace by clicking the green gitpod button, or following the Gitpod URL: gitpod.io/#https://github.com/seqeralabs/nf-training-public

This tutorial provides all the scripts, so don’t worry if you have lost your environment. In the nf-training and nf-training/scripts directories, you can find the main scripts and individual snippets used in the tutorial.

If you want to change git provider (between GitHub, GitLab and BitBucket), go to gitpod.io/integrations. Then you will need to login and deactive the current provider.

In the terminal section, you can type the following:

This should come up with the Nextflow information from this environment. This tells us that the environment is working. All the training material and scripts are in this environment.

You should see the following (or similar):

In practice, a Nextflow pipeline is made by joining together different processes. Each process can be written in any scripting language that can be executed by the Linux platform (Bash, Perl, Ruby, Python, etc.).

Processes are executed independently and are isolated from each other, i.e. they do not share a common (writable) state. The only way they can communicate is via asynchronous first-in, first-out (FIFO) queues, called channels in Nextflow.

Any process can define one or more channels as input and output. The interaction between these processes, and ultimately the pipeline execution flow itself, is implicitly defined by these input and output declarations.

While a process defines what command or script has to be executed, the executor determines how that script is actually run in the target platform.

If not otherwise specified, processes are executed on the local computer. The local executor is very useful for pipeline development and testing purposes, but for real world computational pipelines, an HPC or cloud platform is often required.

In other words, Nextflow provides an abstraction between the pipeline’s functional logic and the underlying execution system (or runtime). Thus it is possible to write a pipeline once and to seamlessly run it on your computer, a cluster, or the cloud, without modifying it, by simply defining the target execution platform in the configuration file.

Nextflow implements a declarative domain specific language (DSL) that simplifies the writing of complex data analysis workflows as an extension of a general purpose programming language.

This approach makes Nextflow very flexible, because it allows in the same computing environment the benefits of a concise DSL allowing the handling of recurrent use cases with ease and the flexibility and power of a general purpose programming language to handle corner cases, which may be difficult to implement using a purely declarative approach.

In practical terms, Nextflow scripting is an extension of the Groovy programming language, which in turn is a super-set of the Java programming language. Groovy can be considered as Python for Java in that it simplifies the writing of code and is more approachable.

Please now copy the following example into your favourite text editor and save it to a file named hello.nf.

Execute the script by entering the following command in your terminal:

The output will look similar to the text shown below:

Where the standard output shows (line by line):

It’s worth noting that the process convertToUpper is executed in parallel, so there’s no guarantee that the instance processing the first split (the chunk 'Hello ') will be executed before the one processing the second split (the chunk 'world!').

Thus, it is perfectly possible that you will get the final result printed out in a different order:

To better understand how Nextflow is dealing with the data in this pipeline, we share below a DAG-like figure to visualise all the inputs, outputs, channels and processes.

A container can be run using the following command:

Try for example the following publically available container (if you have docker installed):

The pull command allows you to download a Docker image without running it. For example:

The above command download a Debian Linux image. You can check it exists by using:

Launching a BASH shell in the container allows you to operate in an interactive mode in the containerised operating system. For example:

Once launched, you will notice that it is running as root (!). Use the usual commands to navigate in the file system. This is useful to check if the expected programs are present within a container.

To exit from the container, stop the BASH session with the exit command.

Docker images are created by using a so-called Dockerfile, which is a simple text file containing a list of commands to assemble and configure the image with the software packages required.

In this step you will create a Docker image containing the Salmon tool.

Then use your favourite editor (e.g. vim or nano) to create a file named Dockerfile and copy the following content:

Build the Dockerfile image by using the following command:

Where "my-image" is the user specified name tag for the Dockerfile, present in the current directory.

You can try your new container by running this command:

Add the Salmon package to the Docker image by adding to the Dockerfile the following snippet:

Save the file and build the image again with the same command as before:

You will notice that it creates a new Docker image with the same name but with a different image ID.

Check that Salmon is running correctly in the container as shown below:

You can even launch a container in an interactive mode by using the following command:

Use the exit command to terminate the interactive session.

Create a genome index file by running Salmon in the container.

Try to run Salmon in the container with the following command:

The above command fails because Salmon cannot access the input file.

This happens because the container runs in a completely separate file system and it cannot access the hosting file system by default.

You will need to use the --volume command line option to mount the input file(s) e.g.

or set a folder you want to mount as an environmental variable, called DATA:

Now check the content of the transcript-index folder entering the command:

Use the option -u $(id -u):$(id -g) to allow Docker to create files with the right permission.

Publish your container in the Docker Hub to share it with other people.

Create an account in the hub.docker.com web site. Then from your shell terminal run the following command, entering the user name and password you specified registering in the Hub:

Tag the image with your Docker user name account:

Finally push it to the Docker Hub:

After that anyone will be able to download it by using the command:

Note how after a pull and push operation, Docker prints the container digest number e.g.

This is a unique and immutable identifier that can be used to reference a container image in a univocally manner. For example:

The simplest way to run a Nextflow script with a Docker image is using the -with-docker command line option:

We’ll see later how to configure in the Nextflow config file which container to use instead of having to specify every time as a command line argument.

Singularity images are created using a Singularity file in similar manner to Docker, though using a different syntax.

Once you have save the Singularity file. Create the image with these commands:

Note: the build command requires sudo permissions. A common workaround consists to build the image on a local workstation and then deploy in the cluster just copying the image file.

Once done, you can run your container with the following command

By using the shell command you can enter in the container in interactive mode. For example:

Once in the container instance run the following commands:

An easier way to create Singularity container without requiring sudo permission and boosting the containers interoperability is to import a Docker container image pulling it directly from a Docker registry. For example:

The above command automatically downloads the Debian Docker image and converts it to a Singularity image store in the current directory with the name debian-jessie.simg.

Nextflow allows the transparent usage of Singularity containers as easy as with Docker ones.

It only requires to enable the use of Singularity engine in place of Docker in the Nextflow configuration file using the -with-singularity command line option:

As before the Singularity container can also be provided in the Nextflow config file. We’ll see later how to do it.

The authors of Singularity, SyLabs have their own repository of Singularity containers.

In the same way that we can push docker images to Docker Hub, we can upload Singularity images to the Singularity Library.

A Conda environment is defined using a YAML file, which lists the required software packages. For example:

Given the recipe file, the environment is created using the command shown below:

You can check the environment was created successfully with the command shown below:

This should look something like this:

To enable the environment you can use the activate command:

Nextflow is able to manage the activation of a Conda environment when the its directory is specified using the -with-conda option (using the same path shown in the list function. For example:

This makes easier to manage different environments for the processes in the workflow script.

See the docs for details.

Another way to build conda-like environments is through a Dockerfile and micromamba.

micromamba is fast and robust package for building small conda-based environments.

This saves having to build a conda environment each time you want to use it (as outlined in previous sections).

To do this, you simply require a Dockerfile, such as the one in this public repo:

The above Dockerfile takes the parent image 'mambaorg/micromamba', then installs a conda environment using micromamba, and installs salmon, fastqc and multiqc.

Try executing the RNA-Seq pipeline from earlier (script7.nf), but start by building your own micromamba Dockerfile (from above), save to your docker hub repo and direct Nextflow to run from this container (changing your nexflow.config).

A queue channel is a asynchronous unidirectional FIFO queue which connects two processes or operators.

A queue channel is implicitly created by process output definitions or using channel factories such as Channel.from or Channel.fromPath.

Try the following snippets:

Try to execute this snippet, it will produce an error message.

A value channel (a.k.a. singleton channel) by definition is bound to a single value and it can be read unlimited times without consuming its contents.

It prints:

A value channel is created using the value factory method or by operators returning a single value, such us first, last, collect, count, min, max, reduce, sum.

The value factory method is used to create a value channel. An optional not null argument can be specified to bind the channel to a specific value. For example:

The factory Channel.from allows the creation of a queue channel with the values specified as argument.

The first line in this example creates a variable ch which holds a channel object. This channel emits the values specified as a parameter in the from method. Thus the second line will print the following:

The method Channel.of works in a similar manner to Channel.from, though it fixes some inconsistent behavior of the latter and provides a better handling for range of values. For example:

The method Channel.fromList creates a channel emitting the elements provided by a list objects specified as argument:

The fromPath factory method create a queue channel emitting one or more files matching the specified glob pattern.

This example creates a channel and emits as many items as there are files with csv extension in the /data/meta folder. Each element is a file object implementing the Path interface.

Learn more about the glob patterns syntax at this link.

Use the Channel.fromPath method to create a channel emitting all files with the suffix .fq in the data/ggal/ directory and any subdirectory, in addition to hidden files. Then print the file names.

The fromFilePairs method creates a channel emitting the file pairs matching a glob pattern provided by the user. The matching files are emitted as tuples in which the first element is the grouping key of the matching pair and the second element is the list of files (sorted in lexicographical order).

It will produce an output similar to the following:

Use the fromFilePairs method to create a channel emitting all pairs of fastq read in the data/ggal/ directory and print them. Then use the flat:true option and compare the output with the previous execution.

The Channel.fromSRA method makes it possible to query the NCBI SRA archive and returns a channel emitting the FASTQ files matching the specified selection criteria.

The query can be project ID or accession number(s) supported by the NCBI ESearch API.

For example the following snippet will print the contents of an NCBI project ID:

This should print:

Multiple accession IDs can be specified using a list object:

It’s straightforward to use this channel as an input using the usual Nextflow syntax. The code below creates a channel containing 2 samples from a public SRA study and runs FASTQC on the resulting files. See:

The splitText operator allows you to split multi-line strings or text file items, emitted by a source channel into chunks containing n lines, which will be emitted by the resulting channel. See:

You can define the number of lines in each chunk by using the parameter by, as shown in the following example:

An optional closure can be specified in order to transform the text chunks produced by the operator. The following example shows how to split text files into chunks of 10 lines and transform them into capital letters:

You can also make counts for each line:

Finally, you can also use the operator on plain files (outside of the channel context), as so:

The splitCsv operator allows you to parse text items emitted by a channel, that are formatted using the CSV format.

It then splits them into records or groups them into a list of records with a specified length.

In the simplest case, just apply the splitCsv operator to a channel emitting a CSV formatted text files or text entries, to view only the first and fourth columns. For example:

When the CSV begins with a header line defining the column names, you can specify the parameter header: true which allows you to reference each value by its name, as shown in the following example:

Alternatively you can provide custom header names by specifying a the list of strings in the header parameter as shown below:

You can also process multiple csv files at the same time:

Finally, you can also operate on csv files outside the channel context, as so:

Try inputting fastq reads to the RNA-Seq workflow from earlier using .splitCSV.

Parsing tsv files works in a similar way, just adding the sep:'\t' option in the splitCsv context:

Try using the tab separation technique on the file "data/meta/regions.tsv", but print just the first column, and remove the header.

We can also easily parse the JSON file format using the following groovy schema:

In a similar way, this is a way to parse YAML files:

The best way to store parser scripts is to keep them in a nextflow module file.

This follows the DSL2 way of working.

See the following nextflow script:

To get this script to work, first we need to create a file called parsers.nf, and store it in the modules folder in the current directory.

This file should have the parseJsonFile function present, then Nextflow will use this as a custom function within the workflow scope.

Script parameters (params) can be defined dynamically using variable values, like this:

This can be tricky when the script uses many Bash variables. A possible alternative is to use a script string delimited by single-quote characters

However, this blocks the usage of Nextflow variables in the command script.

Another alternative is to use a shell statement instead of script which uses a different syntax for Nextflow variables: !{..}. This allows the use of both Nextflow and Bash variables in the same script.

The process script can also be defined in a completely dynamic manner using an if statement or any other expression evaluating to a string value. For example:

The val qualifier allows you to receive data of any type as input. It can be accessed in the process script by using the specified input name, as shown in the following example:

In the above example the process is executed three times, each time a value is received from the channel num and used to process the script. Thus, it results in an output similar to the one shown below:

The file qualifier allows the handling of file values in the process execution context. This means that Nextflow will stage it in the process execution directory, and it can be access in the script by using the name specified in the input declaration.

The input file name can also be defined using a variable reference as shown below:

The same syntax it’s also able to handle more than one input file in the same execution. Only requiring a change in the channel composition.

Consider the following snippet:

The above code creates a temporary file named input.1 with the string data/ggal/transcriptome.fa as content. That likely is not what you wanted to do.

As of version 19.10.0, Nextflow introduced a new path input qualifier that simplifies the handling of cases such as the one shown above. In a nutshell, the input path automatically handles string values as file objects. The following example works as expected:

Write a script that creates a channel containing all read files matching the pattern data/ggal/*_1.fq followed by a process that concatenates them into a single file and prints the first 20 lines.

A key feature of processes is the ability to handle inputs from multiple channels. However it’s important to understands how channel content and their semantics affect the execution of a process.

Consider the following example:

Both channels emit three values, therefore the process is executed three times, each time with a different pair:

What is happening is that the process waits until there’s a complete input configuration i.e. it receives an input value from all the channels declared as input.

When this condition is verified, it consumes the input values coming from the respective channels, and spawns a task execution, then repeat the same logic until one or more channels have no more content.

This means channel values are consumed serially one after another and the first empty channel cause the process execution to stop even if there are other values in other channels.

So what happens when channels do not have the same cardinality (i.e. they emit a different number of elements)?

For example:

In the above example the process is executed only two times, because when a channel has no more data to be processed it stops the process execution.

However, what happens if you replace value x with a value channel?

Compare the previous example with the following one :

This is because value channels can be consumed multiple times, so it doesn’t affect process termination.

Write a process that is executed for each read file matching the pattern data/ggal/*_1.fq and use the same data/ggal/transcriptome.fa in each execution.

The each qualifier allows you to repeat the execution of a process for each item in a collection, every time new data is received. For example:

In the above example every time a file of sequences is received as input by the process, it executes three tasks, each running a different alignment method, set as a mode variable. This is useful when you need to repeat the same task for a given set of parameters.

Extend the previous example so a task is executed for each read file matching the pattern data/ggal/*_1.fq and repeat the same task both with salmon and kallisto.

The val qualifier specifies a defined value output in the script context. In a common usage scenario, this is a value, which has been defined in the input declaration block, as shown in the following example:

The file qualifier specifies one or more files as output, produced by the process, into the specified channel.

In the above example the process randomNum creates a file named result.txt containing a random number.

Since a file parameter using the same name is declared in the output block, when the task is completed that file is sent over the numbers channel. A downstream process declaring the same channel as input will be able to receive it.

When an output file name contains a * or ? wildcard character it is interpreted as a glob path matcher. This allows us to capture multiple files into a list object and output them as a sole emission. For example:

it prints:

Some caveats on glob pattern behavior:

Remove the flatMap operator and see out the output change. The documentation for the flatMap operator is available at this link.

When an output file name needs to be expressed dynamically, it is possible to define it using a dynamic evaluated string, which references values defined in the input declaration block or in the script global context. For example:

In the above example, each time the process is executed an alignment file is produced whose name depends on the actual value of the x input.

So far we have seen how to declare multiple input and output channels, but each channel was handling only one value at time. However Nextflow can handle a tuple of values.

When using a channel emitting a tuple of values, the corresponding input declaration must be declared with a tuple qualifier followed by definition of each element in the tuple.

In the same manner, output channels emitting a tuple of values can be declared using the tuple qualifier following by the definition of each tuple element.

Modify the script of the previous exercise so that the bam file is named as the given sample_id.

Given each process is being executed in separate temporary work/ folders (e.g. work/f1/850698…​; work/g3/239712…​; etc.), we may want to save important, non-intermediary, final files into a results folder.

To store our workflow result files, we need to be explicitly mark them using the directive publishDir in the process that’s creating these file. For example:

The above example will copy all blast script files created by the blastSeq task in the directory path my-results.

You can use more then one publishDir to keep different outputs in separate directory. For example:

The above example will create an output structure in the directory my-results, which contains a separate sub-directory for each given sample ID each of which contain the folders counts and outlooks.

The view operator prints the items emitted by a channel to the console standard output, appending a new line character to each item. For example:

It prints:

An optional closure parameter can be specified to customize how items are printed. For example:

It prints:

The map operator applies a function of your choosing to every item emitted by a channel, and returns the items obtained as a new channel. The function applied is called the mapping function and is expressed with a closure as shown in the example below:

A map can associate to each element a generic tuple containing any data as needed.

Use fromPath to create a channel emitting the fastq files matching the pattern data/ggal/*.fq, then chain with a map to return a pair containing the file name and the path itself. Finally print the resulting channel.

The into operator connects a source channel to two or more target channels in such a way the values emitted by the source channel are copied to the target channels. For example:

The mix operator combines the items emitted by two (or more) channels into a single channel.

The flatten operator transforms a channel in such a way that every tuple is flattened so that each entry is emitted as a sole element by the resulting channel.

The above snippet prints:

The collect operator collects all the items emitted by a channel to a list and returns the resulting object as a sole emission.

It prints a single value:

The groupTuple operator collects tuples (or lists) of values emitted by the source channel, grouping together the elements that share the same key. Finally it emits a new tuple object for each distinct key collected.

Try the following example:

It shows:

This operator is useful to process together all elements for which there is a common property or grouping key.

Use fromPath to create a channel emitting all the files in the folder data/meta/, then use a map to associate to each file the baseName prefix. Finally group together all files having the same common prefix.

The join operator creates a channel that joins together the items emitted by two channels for which there exists a matching key. The key is defined, by default, as the first element in each item emitted.

The resulting channel emits:

The branch operator allows you to forward the items emitted by a source channel to one or more output channels, choosing one of them at a time.

The selection criteria is defined by specifying a closure that provides one or more boolean expressions, each of which is identified by a unique label. On the first expression that evaluates to a true value, the current item is bound to a named channel as the label identifier. For example:

DSL2 separates the definition of a process from its invocation. The process definition follows the usual syntax as described in the process documentation. The only difference is that the from and into channel declarations now have to be omitted. By removing the preset channels from and into, we allow the process to accept any value channel without needing to specify its channel name.

To run this process in DSL2, we have to invoke it in the workflow scope, passing the expected input channels as parameters as it if were a custom function. For example:

Next, we can define additional processes within the script and add another call to the workflow` directive, thereby separating the processes into 'submodules'.

In the example above, we can propagate the output of one process to another using the .out attribute. In this way, we now have a simpler workflow scope to follow, that creates custom functions from processes defined earlier in the script.

If a process defines two or more output channels, each of them can be accessed by indexing the .out attribute e.g. .out[0], .out[1], etc.

The process output definition also allows the use of the emit statment to define a named identifier that can be used to reference the channel in the external scope. For example:

The workflow scope allows the definition of components that define the invocation of one or more processes and operators:

For example, the snippet above defines a workflow named my_pipeline, that can be invoked via another workflow definition.

A workflow component can access any variable and parameter defined in the outer scope. In the running example, we can also access params.greeting directly within the workflow definition.

A workflow component can declare one or more input channels using the take statement. For example:

The input for the workflow can then be specified as an argument as shown below:

A workflow can declare one or more output channels using the emit statement. For example:

As a result, we can use the my_pipeline.out notation to access the outputs of my_pipeline in the invoking workflow.

We can also declare named outputs within the emit block.

The result of the above snippet can then be accessed using my_pipeline.out.my_data.

Components defined in the module script can be imported into other Nextflow scripts using the include statement. This allows you to store these components in separate file(s) so that they can be re-used in multiple workflows.

Using the hello.nf example, we can acheive this by:

If a Nextflow module script contains multiple process definitions these can also be imported using a single include statement as shown in the example below:

When including a module component it is possible to specify a name alias using the as declaration. This allows the inclusion and the invocation of the same component multiple times in your script using different names. For example:

A module script can define one or more parameters or custom functions using the same syntax as with any other Nextflow script. Using the minimal examples below:

Running main.nf should print:

As highlighted above, the script will print Hola mundo! instead of Hello world! because parameters are inherited from the including context.

The addParams option can be used to extend the module parameters without affecting the external scope. For example:

Executing the main script above should print:

Create an account and login into Tower.

1. Create a new token

You can access your tokens from the The Settings drop-down menu :

2. Name your token

3. Save your token safely

4. Export your token

Once your token has been created, open a terminal and type:

Where eyxxxxxxxxxxxxxxxQ1ZTE= is the token you have just created.

To submit a pipeline to a Workspace using the Nextflow command line tool, add the workspace ID to your environment. For example

The workspace ID can be found on the organisation’s Workspaces overview page.

5. Run Nextflow with tower

Run your Nextflow workflows as usual with the addition of the -with-tower command:

You will see and be able to monitor your Nextflow jobs in Tower.

To configure and execute Nextflow jobs in Cloud environments, visit the Compute environments section.

Run the RNA-Seq script7.nf using the -with-tower flag, after correctly completing the token settings outlined above.

To run using the GUI, there are three main steps:

1. Create an account and login into Tower, available free of charge, at tower.nf.

2. Create and configure a new compute environment.

3. Start launching pipelines.

To learn more about using the Tower API, visit to the API section in this documentation.

You can create your own organisation and participant workspace by following the docs at tower.

Tower allows creation of multiple organizations, each of which can contain multiple workspaces with shared users and resources. This allows any organization to customize and organize the usage of resources while maintaining an access control layer for users associated with a workspace.

Any user can be added or removed from a particular organization or a workspace and can be allocated a specific access role within that workspace.

The Teams feature provides a way for the organizations to group various users and participants together into teams, for example workflow-developers or analysts, and apply access control to all the users within this team as a whole.

For further information, please refer the User Management section.

A Nextflow configuration file is a simple text file containing a set of properties defined using the syntax:

Configuration properties can be used as variables in the configuration file itself, by using the usual $propertyName or ${expression} syntax.

Configuration files use the same conventions for comments used in the Nextflow script:

Configuration settings can be organized in different scopes by dot prefixing the property names with a scope identifier or grouping the properties in the same scope using the curly brackets notation. This is shown in the following example:

The scope params allows the definition of workflow parameters that overrides the values defined in the main workflow script.

This is useful to consolidate one or more execution parameters in a separate file.

Save the first snippet as nextflow.config and the second one as params.nf. Then run:

Execute is again specifying the foo parameter on the command line:

Compare the result of the two executions.

The env scope allows the definition of one or more variables that will be exported into the environment where the workflow tasks will be executed.

Save the above snippet a file named my-env.config. The save the snippet below in a file named foo.nf:

Finally executed the following command:

process directives allow the specification of specific settings for the task execution such as cpus, memory, container and other resources in the pipeline script.

This is useful specially when prototyping a small workflow script.

However it’s always a good practice to decouple the workflow execution logic from the process configuration settings, i.e. it’s strongly suggested to define the process settings in the workflow configuration file instead of the workflow script.

The process configuration scope allows the setting of any process directives in the Nextflow configuration file. For example:

The above config snippet defines the cpus, memory and container directives for all processes in your workflow script.

The process selector can be used to apply the configuration to a specific process or group of processes (discussed later).

The syntax for setting process directives in the configuration file requires = (i.e. assignment operator), whereas it should not be used when setting the process directives within the workflow script.

This is important, especially when you want to define a config setting using a dynamic expression using a closure. For example:

Directives that requires more than one value, e.g. pod, in the configuration file need to be expressed as a map object.

Finally directives that allows to be repeated in the process definition, in the configuration files need to be defined as a list object. For example:

The container image to be used for the process execution can be specified in the nextflow.config file:

The use of unique "SHA256" docker image IDs guarantees that the image content does not change over time, for example:

To run a workflow execution with Singularity, a container image file path is required in the Nextflow config file using the container directive:

The following protocols are supported:

The above configuration instructs Nextflow to use the Singularity engine to run your script processes. The container is pulled from the Docker registry and cached in the current directory to be used for further runs.

Alternatively if you have a Singularity image file, its absolute path location can be specified as the container name either using the -with-singularity option or the process.container setting in the config file.

Try to run the script as shown below, changing the nextflow.config file to the one above using singularity:

The use of a Conda environment can also be provided in the configuration file adding the following setting in the nextflow.config file:

You can either specify the path of an existing Conda environment directory or the path of Conda environment YAML file.

Use the scope process to define the resource requirements for all processes in your workflow applications. For example:

In real world application different tasks need different amount of computing resources. It is possible to define the resources for a specific task using the select withName: followed by the process name:

Run the RNA-Seq script (script7.nf) from earlier, but specify that the quantification process cpu reuirements to 2 cpu and with 5 GB of memory, withon the nextflow.config file.

When a workflow application is composed by many processes listing all process names in the configuration file can be difficult to choose resources for each of them.

A better strategy consists of annotating the processes with a label directive. Then specify the resources in the configuration file used for all processes having the same label.

The workflow script:

The configuration file:

It is possible to use a different container for each process in your workflow. You can define their continers in a config file as shown below:

Read more about config process selector at this link.