Redirect output of process to a file and streams?

Output redirection allows us to send the output of processes to files and standard streams (stdout and stderr) simultaneously. This is essential for logging, debugging, and monitoring system activities in Unix-like operating systems.

The tee Command

The tee command is a fundamental Linux utility that reads from standard input and writes to both standard output and one or more files simultaneously. It acts like a T-junction in plumbing, splitting the data flow into multiple directions.

Redirect stdout

Here's a simple example redirecting the output of the ls command to both stdout and a file:

$ ls -C | tee /tmp/out.log
bin   dev  home mnt  proc  run srv  tmp  var
boot  etc  lib64    opt  root  sbin sys  usr

We can verify that the file contains the same output:

$ cat /tmp/out.log
bin   dev  home mnt  proc  run srv  tmp  var
boot  etc  lib64    opt  root  sbin sys  usr

By default, tee overwrites the target file. Use the -a (append) option to add new content to existing files instead of replacing them.

Redirect stdout and stderr to the Same File

To capture both standard output and error messages in one file, we redirect stderr to stdout using 2>&1 before piping to tee:

$ (ls -C; cmd_with_err) 2>&1 | tee /tmp/out.log
bin   dev  home mnt  proc  run srv  tmp  var
boot  etc  lib64    opt  root  sbin sys  usr
bash: cmd_with_err: command not found

Alternatively, use the shorthand |& syntax which is equivalent to 2>&1|:

$ (ls -C; cmd_with_err) |& tee /tmp/out.log

Redirect stdout and stderr to Separate Files

To redirect stdout and stderr to different files while still displaying both on the terminal, we use process substitution:

$ ((ls -C; cmd_with_err) 1> >(tee /tmp/out.log)) 2> >(tee /tmp/err.log 2>&2)
bin   dev  home mnt  proc  run srv  tmp  var
boot  etc  lib64    opt  root  sbin sys  usr
bash: cmd_with_err: command not found

This command uses the template: fd> >(tee filename fd>&fd) where fd represents file descriptors (0=stdin, 1=stdout, 2=stderr).

Verification shows the outputs are correctly separated:

$ cat /tmp/out.log
bin   dev  home mnt  proc  run srv  tmp  var
boot  etc  lib64    opt  root  sbin sys  usr
$ cat /tmp/err.log
bash: cmd_with_err: command not found

Redirection Delays

When using tee with programs that produce continuous output, you might encounter buffering delays where output appears in chunks rather than real-time.

Example Scenario

Consider this Python script that prints timestamps every second:

#!/usr/bin/python
from datetime import datetime
import time
import sys
from sys import stdout
while True:
   sys.stdout.write(datetime.today().strftime("%H:%M:%S %p<br>"))
   time.sleep(1)

Running directly shows immediate output:

$ ./time.py
9:29:48 PM
9:29:49 PM
9:29:50 PM

However, when piped through tee, output may be delayed due to output buffering:

$ ./time.py | tee time.out
# Long delay, then output appears in chunks

Mitigation Strategies

Method 1: Flush Output Buffer - Modify the program to flush stdout explicitly:

while True:
   sys.stdout.write(datetime.today().strftime("%H:%M:%S %p<br>"))
   sys.stdout.flush()  # Force immediate output
   time.sleep(1)

Method 2: Use unbuffer Command - For programs you cannot modify, use the unbuffer utility:

$ unbuffer ./time.py | tee time.out
21:52:22 PM
21:52:23 PM
# Real-time output without delays

Key Points

• tee splits output streams to multiple destinations simultaneously

• Use 2>&1 or |& to combine stdout and stderr before teeing

• Process substitution >(command) enables complex redirection patterns

• Output buffering can cause delays in real-time applications

• unbuffer command solves buffering issues for external programs

Conclusion

Output redirection with tee provides flexible ways to capture process output while maintaining visibility on the terminal. Understanding buffering behavior and mitigation techniques ensures reliable real-time output redirection for monitoring and logging purposes.