Processing Linux Commands in Parallel

Processing the commands in parallel can ease your work while executing multiple commands in Linux. As a Linux administrator, you can process multiple commands in parallel to perform various tasks, such as restarting services, installing system patches, and installing applications.

Combining two or more commands in the command line is called command chaining. You can use command chaining to execute multiple commands in parallel.

You can easily chain different commands at once to simplify the process. So in this long- guide, we will explain every process you can try to process Linux commands in parallel.

Processing Linux Commands in Parallel

In Linux, you can run multiple commands in parallel using multiple operators and commands. Now we will see how to run multiple commands in parallel through different operators under several conditions.

Using the & Operator

In Linux, you can run multiple commands simultaneously using the '&' sign after one command. Using this built-in bash ampersand or operator causes the shell to run the next command without waiting for the currently running one, and the commands are run in parallel.

:~$ command1 & command2…

For example, we run the ls and cat commands together using the & operator.

:~$ ls & cat Linux.txt
[1] 6178
Don't erase the existing data. 
Desktop    file.sh      Myfile     prateek.pcap       Templates

You can see that running both the above commands together displays the output. The cat command displays the contents of the 'Linux.txt' file, while the ls command shows all the files from the current directory.

Using Wait with & (& :)

You can execute multiple commands simultaneously using wait (:) with the & operator. The & operator works similarly, but the 'wait (:)' stops all background tasks to complete before returning control to the shell. With this approach, you can run batches of operations, and its syntax would be something like this −

:~$ command1 & : command2
.
.
.
command3

To better understand this method, let's run the bash file and the ls command together using & operator. When these commands run backward, we display the details of the current time by running the uptime command.

:~$ ./file.sh & : ls
[1] 3650
:~$  Desktop    file.sh     Myfile    prateek.pcap      Templates 
  Documents 
uptime
19:06:59 up 1 min, 1 user, load average: 1.31, 0.55, 0.20
[1]+ Done            ./file.sh

One of the disadvantages of this method is that you can only start new processes once all batch processes are completed. To solve this, you can use the xargs command.

Using Xargs

The xargs command in Linux specifies the number of processes you want to run simultaneously. You can execute multiple input arguments in parallel in Linux using the xargs command. Here we piped the following xargs command with the ls command so that it uses 3 processes in parallel on each file from the output of ls.

:~$ ls | xargs -P 3 -n 2
Desktop Documents
Downloads example.txt
File.sh Linux.txt
Pictures Points
Prateek.pcap Public
Snap source.sh
Templates

Xargs starts another process as soon as the previous one is completed. In the above example, we specified the number of parallel tasks using the -P and -n arguments to call the number of arguments per call.

Using Semicolon (;) Operator

Through the semicolon (;) operator, you can run various commands in succession, regardless of whether your previous command was successful.

:~$ command 1; command2;.....

Here we will run all three commands pwd, ls, and uptime, by separating them with semicolons (;) in the terminal as follows −

:~$ pwd; ls; uptime
/home/prateek
Desktop    file.sh     Myfile    prateek.pcap      Templates 
Documents 
12:36:51 up 1 min, 1 user, load average: 3.14, 1.09, 0.39

The [pwd] locates your current directory, [ls] displays a list of the current directory, and [uptime] provides information related to time, such as current time, load average, number of online users, etc. Running these three commands together in the terminal will immediately tell you all the above information. Remember, there should not be any space between the commands and semicolons.

Using | (Pipe) Operator

You can execute multiple commands in parallel using the pipe operator. Through this operator, you can combine and execute two or more commands sequentially.

:~$ command 1 | command 2 | command 3….

For example, here we will first read the content of the text file named 'Linux1.txt' and, in the second command, search for the keyword named 'erasing' in it.

:~$ cat Linux.txt | grep erasing
Don't erase the existing data.

Here, we displayed the content of the 'Linux.txt' file with the cat command. Through the grep command, we searched the keyword 'erasing' from this file, highlighted in red.

Using GNU Parallel

GNU is a command line tool in Linux by which you can execute Linux commands in parallel on a remote server or local machine. It lets you simultaneously run multiple command instances that take input from a standard or file input.

This tool is not pre-installed in Linux distros but is in the repositories. Hence, you can install it through the following command −

sudo apt install parallel -y (for Debian/Ubuntu-based distros)
yum install parallel (for RedHat-based distros)

Using Bash Script

Through the bash script, you can run multiple commands in parallel. However, you have to create a bash file first. You can enter all the commands you want to run in this bash file. After that, you will just run that bash script, and then you will get the result of all the commands together.

Let's create a bash file named 'file.sh' through the following command and enter all the commands in it −

:~$ nano file.sh
#! /bin/bash 
touch myfile.txt
echo "tutorials Point" > myfile.txt
ls && cat myfile.txt

• In this script, we first created a file named 'myfile' with the touch command.

• After that, enter the text 'Tutorials Point' in this file with the echo command.

• Finally, with the ls command, locate the current directory, and then with the cat command, see the content of the file 'myfile.'

Press Ctrl+O to save this file, and exit by pressing Ctrl+X. Furthermore, we can make the file executable through the following chmod command −

:~$ chmod +x file.sh

Let's run the above-created bash file so that whatever commands we have entered will run parallel and show the output.

:~$ ./file.sh
Desktop    file.sh     Myfile    prateek.pcap      Templates 
Documents 
Tutorials Point

From the above output, the system has executed all the commands mentioned in the script.

Using && (AND) Operator

You can separate multiple commands and run them sequentially using the '&&' operator. In this operator, the system will execute the second command after the successful execution of the first one.

:~$ command 1 && command 2…

For example, if we have to create a directory named 'MyFile' and later change to that directory, we can run both these processes successfully through the && operator.

:~$ mkdir MyFile && cd MyFile
:~/MyFile$

You can see that the above command is successfully run, and first, your directory is created, and then you are in it.

Using (||) Operator

In some conditions, when you run two or more commands together, you want the following command to be executed when the first one fails, or there is an error. You can use two vertical bars (||) or a logical OR operator.

The 'OR' operator will execute the following command only if the first one fails. If the first command runs successfully, the next command will not execute.

:~$ command 1 || command 2…

Let's check whether the system contains a 'Myfile' directory; if it is not available, then the system must create it. We will use the following command using the logical 'OR' operator −

:~$ [ -d ~/Myfile ] || mkdir ~/Myfile

In this example, the first command [ -d ~/Myfile ] failed because no directory was named 'Myfile,' so the system's second command runs [mkdir ~/Myfile] to create a MyFile directory.

Combining Multiple Operators

You can combine the above operators and run them simultaneously per your requirement. Let's combine AND (&&) and OR (||) operators here. For example, we will check the availability of the 'myfile.txt' file in the system, but if it is not present, the system will create or else print the message 'File exists' −

:~$ [ -f ~/myfile.txt ] && echo "File exists." || touch ~/myfile.txt
File exists

You can see that the file is already in the file system, so it prints 'File exists.'

Conclusion

Command chaining is helpful when you want to run many commands simultaneously. In this article, we have explained various methods to process Linux commands in parallel using multiple commands and operators. This tutorial includes almost all possible methods such as (&) (||), (;), (|), operators, bash approach, GNU parallel approach, and many more commands and operators are used.