GNU Parallel

Introduction

GNU Parallel is a tool for running many sequential tasks at the same time on one or more nodes. It is useful for running a large number of sequential tasks, especially if they are short or of variable durations, as well as when doing a parameter sweep. We will only cover the basic options here; for more advanced usage, please see the official documentation.

GNU parallel is Perl script based tool, developed to run several serial jobs simultaneously. Each serial job is executed on different processor core. This tool is very versatile and particular useful in running parametric analysis. GNU-Parallel is simple and can be used by prepending 'parallel' keyword before the command that needs to be executed in the bash script or terminal. By default, this launches as many serial sub-jobs as the number of core assigned by the scheduler. Once a job completes, parallel executes the next serial job in the queue on the available processor. This process continues until all the serial jobs have been launched.