Operating Systems 2019W: Tutorial 7

In this tutorial you will be working with bcc-tools, a set of tools that is part of the BPF Compiler Collection (BCC). BCC programs are a combination of C and Python. The C code is compiled at runtime into eBPF bytecode; this code is then uploaded to the kernel and interfaced with by a Python program.

Installation

To install bcc-tools on an Ubuntu 18.04 system (such as your openstack VM), do the following:

# become root, do the rest of the commands below as root
sudo su -

# update your system
apt update
apt dist-upgrade
apt clean
apt autoremove

# import the key for the iovisor repository
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 4052245BD4284CDD

# add the iovisor repository to apt
echo "deb https://repo.iovisor.org/apt/$(lsb_release -cs) $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/iovisor.list

# update the package list and install the necessary packages
apt update
apt install bcc-tools libbcc-examples linux-headers-$(uname -r)

# add the bcc tools to root's path.  Log out and log back in again as root to enable
# for example, type exit (to become a regular user) and then run "sudo su -"
echo "export PATH=/usr/share/bcc/tools:\$PATH" >> /root/.profile

There is an older version of BCC the the standard Ubuntu repositories in the bpfcc-tools package. Please do not install this as it seems to have significant bugs. If you've previously uninstalled it, you'll need to uninstall before following the steps above. (Distributions newer than Ubuntu 18.04 may have a more up-to-date version of BCC, check your repositories.)

If you wish to install bcc-tools on a system running something other than Ubuntu 18.04, follow these directions.

UNIX command review

Don't forget about these commands!

• which
• tty
• ps aux (shows tty's for processes)

Tasks

You'll need root access to upload eBPF scripts to the kernel. Thus, run all of the following as root, either using "sudo" or a root shell you got via "sudo su -".

opensnoop

opensnoop lets you see every every open on the system.

• Run "opensnoop". If you have the x2go server running you may see a lot of activity. Try running "service x2goserver stop" to turn it off (assuming you are not using x2go!).
• Run "opensnoop -x". Run some commands and see how many failed opens there are. Why do you think there are so many?
• Look at the code of opensnoop. What kernel events is it monitoring? Hint: look at the calls to attach_kprobe and attach_kretprobe.

bashreadline

bashreadline lets you see what commands are typed in to bash on any process on the system. It does so by intercepting calls to readline.

• Run "bashreadline" and observe the commands that are entered in other terminals (you'll have to have multiple shells running on your VM).
• Can you trivially modify bashreadline to observe what is entered in other programs that use readline, such as bc and gdb? You may need to specify the location of the readline shared library (remember ldd!).

ttysnoop

ttysnoop allows you to observe what is being entered on other tty's.

• How do you figure out the tty associated with another ssh connection or window? How can you find out all of the pseudo tty's that have been allocated on the system?
• Notice how the output of some programs (e.g., top) are truncated. Looking at the code of ttysnoop and doing an strace of the program, can you figure out why output is being truncated?
• Can you (easily) reduce or eliminate the truncation? Why or why not?

killsnoop

killsnoop reports on calls to the kill system call.

• Send a process a signal using "kill". Try sending the TERM, STOP, and KILL signals. Are these all picked up by killsnoop?
• Type Ctrl-C at the command line. Note that sometimes it generates a signal, sometimes it doesn't. What happens when your own program gets Ctrl-C? Can you change whether or not typing Ctrl-C generates a signal?
• Put a process into the background using Ctrl-Z and bring it back to the foreground using fg. Were there any calls to kill?
• strace a process. Do you see any kill system calls?

syscount

syscount counts system calls

• What are the frequent system calls when the system is idle (you aren't doing anything on the command line)?
• What programs are generating the most system calls?
• What system calls are slow (have high latency)?
• Are there many system calls failing?

Note that you may want to connect to your VM using x2go for this part as then you'll have a desktop's worth of processes to observe.

trace

trace allows you to write simple eBPF scripts on the command line.

• Run the man page example of trace that imitates opensnoop.
• Run the man page example of trace that imitates bashreadline.
• Get the user stack when intercepting calls to readline in bash.
• Get the kernel stack when intercepting calls to readline in bash.
• Use trace to implement a version of killsnoop.