BCC – Dynamic Tracing Tools for Linux Performance Monitoring, Networking and More

If you are a Linux user or administrator, you might have heard of term "BCC tools" or "BPF Compiler Collection." BCC is a powerful set of dynamic tracing tools that provides a simple yet effective way to monitor system performance, networking, and much more. In this article, we will discuss what BCC tools are, their benefits, and how to use them with examples.

What are BCC Tools?

BCC (BPF Compiler Collection) is a set of dynamic tracing tools built on top of eBPF (extended Berkeley Packet Filter) technology in Linux kernel. eBPF is a virtual machine that runs inside kernel and allows for efficient and flexible tracing of system events, without need for kernel modifications or recompilation.

BCC tools are designed to provide a simple, user-friendly interface for using eBPF to trace and analyze various system events. They are written in Python and C, and can be used for a wide range of tasks, including system performance monitoring, network analysis, security, and more.

Benefits of BCC Tools

BCC tools offer a number of benefits for Linux users and administrators. These include −

Low overhead

BCC tools are designed to have minimal impact on system performance. They use eBPF technology to trace events directly inside kernel, which reduces need for context switching and other overhead associated with traditional system monitoring tools.

Flexibility

BCC tools can be used for a wide range of tasks, from monitoring system performance to network analysis and more. They are highly flexible and customizable, making them a powerful tool for Linux users and administrators.

User-friendly Interface

BCC tools provide a simple, user-friendly interface for using eBPF to trace system events. They are easy to use and require no specialized knowledge of kernel internals or programming.

Active Development Community

BCC tools are actively developed and maintained by a large community of developers. This means that new features and improvements are constantly being added, and bugs are quickly addressed.

How to Use BCC Tools

BCC tools can be used for a wide range of tasks, including system performance monitoring, network analysis, security, and more. In this section, we will discuss how to use some of most commonly used BCC tools with examples.

BPFtrace

BPFtrace is a high-level tracing language for eBPF, designed to make it easy to write and read eBPF programs. It provides a simple, user-friendly interface for tracing system events and analyzing performance.

To use BPFtrace, you first need to install it on your system. You can do this using package manager for your distribution. For example, on Ubuntu, you can install BPFtrace by running following command −

sudo apt-get install bpftrace

Once you have installed BPFtrace, you can use it to write eBPF programs and trace system events. For example, following BPFtrace program will print a message every time a process is started −

tracepoint:process:process_start
{
   printf("process %s started
", args->comm);
}

You can save this program to a file (e.g., "process_start.bt") and run it using following command −

sudo bpftrace process_start.bt

When you run this command, BPFtrace will start tracing process events and print a message every time a process is started.

BCC Tools

BCC tools provide a wide range of tracing and monitoring capabilities for Linux systems. Some of most commonly used BCC tools include −

Execsnoop

The execsnoop tool traces new process execution on system. It can be used to monitor which processes are running and when they were started. To use execsnoop, simply run following command −

sudo execsnoop

This will start tracing process execution events and print information about new processes as they are started.

Opensnoop

The opensnoop tool traces file system events, including file opens, closes, and reads. It can be used to monitor file system activity and track down performance bottlenecks. To use opensnoop, run following command −

sudo opensnoop

This will start tracing file system events and print information about file activity as it occurs.

Tcptracer

The tcptracer tool traces TCP connections on system, including connections to remote hosts and their associated ports. It can be used to monitor network activity and diagnose network-related problems. To use tcptracer, run following command −

sudo tcptracer

This will start tracing TCP connections and print information about connection events as they occur.

BCC Script Examples

BCC tools can also be used to write more complex scripts for monitoring and analyzing system performance. In this section, we will provide some examples of BCC scripts that can be used to monitor CPU usage, disk I/O, and network activity.

CPU Usage

The following BCC script can be used to monitor CPU usage on system −

#!/usr/bin/python

from bcc import BPF

# load BPF program
bpf_text = """
#include <uapi/linux/ptrace.h>

int count_sched(struct pt_regs *ctx) {
   u64 ts = bpf_ktime_get_ns();
   bpf_trace_printk("Hello, world! %d\
", ts);
   return 0;
}
"""

# initialize BPF program
bpf = BPF(text=bpf_text)

# attach BPF program to sched_switch tracepoint
bpf.attach_tracepoint(tp="sched:sched_switch", fn_name="count_sched")

# print trace messages as they occur
bpf.trace_print()

This script will print a message every time CPU scheduler switches tasks on system.

Disk I/O

The following BCC script can be used to monitor disk I/O activity on system −

#!/usr/bin/python

from bcc import BPF

# load BPF program
bpf_text = """
#include <uapi/linux/ptrace.h>

int count_disk_io(struct pt_regs *ctx, const char *rwflag) {
   u64 ts = bpf_ktime_get_ns();
   bpf_trace_printk("Disk I/O %s %d\
", rwflag, ts);
   return 0;
}
"""

# initialize BPF program
bpf = BPF(text=bpf_text)

# attach BPF program to blk_account_io_done tracepoint
bpf.attach_tracepoint(tp="block:block_rq_complete", fn_name="count_disk_io", ctx="R")

# print trace messages as they occur
bpf.trace_print()

This script will print a message every time a disk I/O operation is completed on system.

Network Activity

The following BCC script can be used to monitor network activity on system −

#!/usr/bin/python

from bcc import BPF

# load BPF program
bpf_text = """
#include <uapi/linux/ptrace.h>

int count_network(struct pt_regs *ctx, int protocol) {
   u64 ts = bpf_ktime_get_ns();
   bpf_trace_printk("Network %d %d\
", protocol, ts);
   return 0;
}
"""

# initialize BPF program
bpf = BPF(text=bpf_text)

# attach BPF program to tcp_v{4,6}_connect trace points
bpf.attach_tracepoint(tp="tcp_v4_connect", fn_name="count_network", ctx="R")
bpf.attach_tracepoint(tp="tcp_v6_connect", fn_name="count_network", ctx="R")


print trace messages as they occur
bpf.trace_print()

This script will print a message every time a TCP connection is established on system.

In addition to examples we have provided, BCC tools have many other use cases. For example, BCC tools can be used to monitor system calls, trace user-level events, and diagnose kernel-level issues. Some of other BCC tools that can be useful include −

• csysdig − A tool that provides a graphical interface for analyzing system activity using eBPF.

• funccount − A tool that counts number of times a specified function is called.

• tcpconnect − A tool that traces TCP connections on system.

• biosnoop − A tool that traces block I/O operations at BIOS level.

• syncsnoop − A tool that traces sync events on system.

BCC tools can also be used in conjunction with other system monitoring tools, such as prometheus, grafana, and nagios, to provide a more comprehensive view of system performance.

It is also worth noting that BCC tools are constantly being developed and improved by community, so new features and capabilities are added regularly. As such, it is important to keep your BCC tools up to date to take advantage of latest improvements.

Conclusion

In conclusion, BCC tools provide a powerful set of dynamic tracing tools for monitoring and analyzing system performance, networking, and more on Linux systems. They offer a user-friendly interface, low overhead, and high flexibility, making them an essential tool for Linux users and administrators. Whether you are a system administrator, developer, or security analyst, BCC tools can help you gain insights into your system and diagnose performance issues quickly and efficiently. So, if you haven't already, give BCC tools a try and see how they can benefit you and your Linux system.