Bistro: a library to build large-scale workflows in computational biology (OCaml)
Computational pipelines for analyzing high-throughput genomics datasets typically consist of tens to hundreds of shell commands, generating thousands of files and running for days or weeks. While becoming rather complex pieces of software, they are most of the time still programmed using rudimentary tools like shell scripts, which offer very little help to develop large and reusable programs. In addition to being error-prone, implementing computational pipelines using shell scripts leaves lots of tedious aspects to the programmer, diverting her/his attention from data analysis considerations. In this work, I propose to leverage a modern, statically typed programming language to implement as a simple library a comfortable environment to develop bioinformatics pipelines. This library is named bistro and is written in the OCaml language. Among other features, it provides dependency tracking, parallel execution, resume-on-failure, automatic naming of intermediate files, easy deployment of pipelines using Docker or Singularity for enhanced reproducibility. Thanks to the compiler type checker, errors on file formats or typos in command arguments are detected at compile-time, that is even before running the pipeline. I'll show various benefits of embedding a pipeline development framework in a generalist language. Among other things, it becomes very easy to integrate a pipeline into a web server, or write extensible libraries of highly configurable pipelines.

Philippe Veber - LBBE, UMR 5558 Lyon