---
title: "systemPipeR"
author: "Author: Thomas Girke (thomas.girke@ucr.edu)"
date: "July 22, 2015" 
output:
  ioslides_presentation:
    keep_md: yes
    logo: logo.png
    toc: yes
    toc_depth: 2
    widescreen: yes
subtitle: "NGS workflow and report generation environment" 
bibliography: bibtex.bib
---

<!---
- Latex to markdown conversion with pandoc
  pandoc -s systemPipeRslides.tex -o systemPipeRslides.text

- rmarkdown manuals/movies: 
  http://rmarkdown.rstudio.com/ioslides_presentation_format.html
	http://blog.rstudio.org/2014/06/18/r-markdown-v2/

- Compile from R console like this
library(rmarkdown)
render('systemPipeRslides.Rmd', ioslides_presentation(logo="logo.png"))

- Compile from command-line
echo "library(rmarkdown); library(knitr); render('systemPipeRslides.Rmd', ioslides_presentation(logo='logo.png')); knit('systemPipeRslides.Rmd', tangle=TRUE)" | R -slave
-->

Outline
=======================
- <div class="white">__Introduction__</div>
- Motivation
- Design
- Templates
- Getting started

## Introduction 

-   [_systemPipeR_](http://www.bioconductor.org/packages/devel/bioc/html/systemPipeR.html) is an R package for building *end-to-end* analysis pipelines with automated report generation for next generation NGS applications [@Girke2014-oy].
-   Important features:
    - Support for R and command-line software, such as NGS aligners, peak callers, variant callers, etc.
    - Runs on single machines and compute clusters with schedulers
    - Uniform sample handling and annotation

Outline
=======================
- Introduction
- <div class="white">__Motivation__</div>
- Design
- Templates
- Getting started

## Motivation

- Many NGS applications share several analysis routines, such as: 
    - Read QC and preprocessing
    - Alignments
    - Quantification
    - Feature annotations
    - Enrichment analysis
- Thus, a common workflow environment has many advantages for improving efficiency, standardization and reproducibility


## Advantages of _`systemPipeR`_

- Facilitates design of complex NGS workflows involving multiple R/Bioconductor packages [@Huber2015-ag].
- Makes NGS analysis with Bioconductor utilities more accessible to new users
- Simplifies usage of command-line software from within R
- Reduces complexity of using compute clusters for R and command-line software
- Accelerates runtime of workflows via parallelization on computer systems with mutiple CPU cores and/or multiple compute nodes
- Automates generation of analysis reports to improve reproducibility

Outline
=======================
- Introduction
- Motivation
- <div class="white">__Design__</div>
- Templates
- Getting started

## Workflow design in _`systemPipeR`_ {.flexbox .vcenter .smaller}

<img src="./images/SystemPipeR_Workflow.png" alt="Drawing" style="width: 800px;"/>
</div><br></br>

- Workflow steps with input/output file operations are controlled by _`SYSargs`_ objects.

- Each _`SYSargs`_ instance is constructed from a _`targets`_ file and a _`param`_ file.

- Only input provided by user is initial _`targets`_ file. Subsequent _`targets`_ instances are created automatically. 

- Any number of predefined or custom workflow steps are supported.

Outline
=======================
- Introduction
- Motivation
- Design
- <div class="white">__Templates__</div>
- Getting started

## _`systemPipeRdata`_: template workflows

- Helper package to generate with a single command NGS workflow templates for _`systemPipeR`_. 
- Includes sample data for testing.
- User can create new workflows or change and extend existing ones.

## RNA-Seq workflow template

1. Read preprocessing
    + Quality filtering (trimming)
    + FASTQ quality report
2. Alignments: _`rsubread`_, _`Bowtie2/Tophat2`_
    + <span style="font-size: 24px;">Available soon: alignment free approaches with [Kallisto](http://pachterlab.github.io/kallisto/) via [artemis](https://github.com/RamsinghLab/artemis) or [sleuth](https://github.com/pachterlab/sleuth)</span>
3. Alignment statistics
4. Read counting per annotation
5. Sample-wise correlation analysis 
5. DEG analysis with _`edgeR`_ or _`DESeq2`_
6. Enrichment analysis of GO terms or other annotation types
7. Gene-wise cluster analysis

## VAR-Seq workflow template

1. Read preprocessing
    + Quality filtering (trimming)
    + FASTQ quality report
2. Alignments: _`gsnap`_, _`bwa`_
3. Alignment statistics
4. Variant calling: _`VariantTools`_, _`GATK`_, _`BCFtools`_
5. Variant filtering: _`VariantTools`_ and _`VariantAnnotation`_
6. Variant annotation: _`VariantAnnotation`_
7. Combine results from many samples
8. Summary statistics of samples

## ChIP-Seq workflow template

1. Read preprocessing
    + Quality filtering and/or trimming
    + FASTQ quality report
2. Alignments: _`rsubread`_, _`Bowtie2`_
3. Alignment statistics
4. Genome-wide coverage statistics
5. Peak calling: _`MACS2`_, _`BayesPeak`_
6. Peak annotation with genomic context
7. Differential binding analysis
8. Enrichment analysis of GO terms or other annotation types
9. Motif analysis

## Ribo-Seq workflow template {.smaller}

1. Read preprocessing
    + Adaptor trimming and quality filtering
    + FASTQ quality report
2. Alignments: _`Tophat2`_ (or any other RNA-Seq aligner)
3. Alignment stats
4. Compute read distribution across genomic features
5. Adding custom features to workflow (e.g. uORFs)
6. Genomic read coverage along transcripts
7. Read counting 
8. Sample-wise correlation analysis
9. Analysis of differentially expressed genes (DEGs)
10. GO term enrichment analysis
11. Gene-wise clustering
12. Differential ribosome binding (translational efficiency)

## Coming soon

Workflow templates for:

- miRNA-Seq
- BS-Seq

Outline
=======================
- Introduction
- Motivation
- Design
- Templates
- <div class="white">__Getting Started__</div>

## Install and load packages {.smaller}

Install required packages
```{r install, eval=FALSE}
if (!requireNamespace("BiocManager", quietly=TRUE))
    install.packages("BiocManager")
BiocManager::install("systemPipeR") # Install systemPipeR from Bioconductor
BiocManager::install("tgirke/systemPipeRdata", build_vignettes=TRUE, dependencies=TRUE) # From github
```

Load packages and accessing help
```{r load_systemPipeR_hidden, eval=TRUE, include=FALSE, messages=FALSE, warnings=FALSE, cache=TRUE}
library("systemPipeR"); library("systemPipeRdata")
```
```{r load_systemPipeR_print, eval=FALSE, messages=FALSE, warnings=FALSE, cache=TRUE}
library("systemPipeR")
library("systemPipeRdata")
```
Access help
```{r accessing_help, eval=FALSE, messages=FALSE, warnings=FALSE, cache=TRUE}
library(help="systemPipeR")
vignette("systemPipeR")
```

## _`Targets`_ file organizes samples {.smaller}

Structure of _`targets`_ file for single-end (SE) library
```{r show_targetsSE, eval=TRUE, messages=FALSE, warnings=FALSE, cache=TRUE}
targetspath <- system.file("extdata", "targets.txt", package="systemPipeR")
read.delim(targetspath, comment.char = "#")[1:3,1:5]
```

Structure of _`targets`_ file for paired-end (PE) library
```{r show_targetsPE, eval=TRUE, messages=FALSE, warnings=FALSE, cache=TRUE}
targetspath <- system.file("extdata", "targetsPE.txt", package="systemPipeR")
read.delim(targetspath, comment.char = "#")[1:3,1:4]
```

## _`SYSargs`_: _`targets`_ & _`param`_ {.smaller}
_`SYSargs`_ instances are constructed from a _`targets`_ file and a _`param`_ file. The  _`param`_ file contains the settings for running command-line software.
```{r sysargs_instance, eval=TRUE, messages=FALSE, warnings=FALSE, cache=TRUE}
parampath <- system.file("extdata", "tophat.param", package="systemPipeR")
(args <- suppressWarnings(systemArgs(sysma=parampath, mytargets=targetspath)))
```
Slots and accessor functions have the same names 
```{r sysargs_names, eval=TRUE, messages=FALSE, warnings=FALSE, cache=TRUE}
names(args)[c(5,8,13)]
```
Return command-line arguments for given software, here _`Tophat2`_ for 1st sample.
```{r sysargs_args, eval=FALSE, messages=FALSE, warnings=FALSE, cache=TRUE}
sysargs(args)[1]
```
<pre>## tophat -p 4 -o SRR446027_1.fastq.tophat tair10.fasta SRR446027_1.fastq .SRR446027_2.fastq</pre>


## Run on single machines or clusters

Run command-line tool, here _`Tophat2`_, on single machine. Command-line tool needs to be installed for this.
```{r run_args_single, eval=FALSE, messages=FALSE, warnings=FALSE, cache=TRUE}
runCommandline(args)
```

Submit command-line or R processes to a computer cluster with a queueing system.
```{r run_args_cluster, eval=FALSE, messages=FALSE, warnings=FALSE, cache=TRUE}
clusterRun(args, ...) 
```
The last step requires additional resource allocation arguments. For details please visit the main manual [here](http://www.bioconductor.org/packages/devel/bioc/vignettes/systemPipeR/inst/doc/systemPipeR.html#alignment-with-tophat2).


## Workflow templates 

Generate workflow template, _e.g._ "rnaseq", "varseq" or "chipseq"
```{r generate_workenvir, eval=FALSE, cache=TRUE}
### <b>
genWorkenvir(workflow="varseq", mydirname=NULL)
### </b>
setwd("varseq")
```
<br></br>
Command-line alternative for generating workflow environments
```{.sh generate_workenvir_from_shell, eval=FALSE, cache=TRUE}
$ echo 'library(systemPipeRdata); 
        genWorkenvir(workflow="varseq", mydirname=NULL)' | R --slave
```

## Workflow template structure

The workflow templates generated by _`genWorkenvir`_ contain the following preconfigured directory structure:
<br></br>
```{r workflow_template_structure, eval=FALSE}
### <b>
workflow_name/            # *.Rnw/*.Rmd scripts, targets file, etc.
                param/    # parameter files for command-line software 
                data/     # inputs e.g. FASTQ, reference, annotations
                results/  # analysis result files
### </b>
```
<br></br>
The above structure can be customized as needed, but for first-time users it is easier to keep changes to a minimum.

## Run workflows 

- Next, run from within R the chosen sample workflow by executing the code provided in the corresponding _`*.Rnw`_ template file (or _`*.Rmd`_ or _`*.R`_ versions).
- Alternatively, one can run an entire workflow from start to finish with a single command by executing from the command-line:
```{.sh run_make, eval=FALSE}
$ make -B
```
- Analysis reports in PDF or HTML format are autogenerated when running a workflow using standard R resources for scientific report generation including _`knitr`_ and _`rmarkdown`_, respectively.

- Integration of [ReportingTools](http://www.bioconductor.org/packages/devel/bioc/html/ReportingTools.html) is also straightforward.

## Continue here  
<br></br><br></br><br></br>
<center>
[Overview Vignette](http://www.bioconductor.org/packages/devel/bioc/vignettes/systemPipeR/inst/doc/systemPipeR.html) 
</center>

## Future development

- Workflow templates with support for both PDF (.Rnw) and HTML (.Rmd) reports
- Workflow templates for additional NGS applications (see [here](#14))
- [docopt](https://github.com/docopt/docopt.R) support for generating _`.param`_ files
- Additional visualization functions
- Streamline support of very complex experimental designs

## References {.smaller}


