Operating Systems 2021F: Tutorial 4

In this tutorial you will use a variety of bpftrace scripts in order to observe 3000shell and other running programs.

ptrace, eBPF, and bpftrace

Processes are normally isolated from each other, in that code and data in one cannot be accessed by another. However, in the past tutorials we've used tools to observe process behaviour: strace, ltrace, and gdb. It turns out these programs use a special system call, ptrace, to observe and control another process.

The ptrace system call is designed for debugging and is very intrusive: running ptrace on a process can change how that process behaves even when you don't want it to. As a result, you don't want to use ptrace on anything in a production environment.

But, what if you need to observe what is happening in a production system? How can you do so safely? Well, Linux has a technology called eBPF for doing exactly this. We will later discuss more about how eBPF works. In this tutorial, though, we will just use some scripts that make use of bpftrace, a relatively easy to use yet powerful front end to eBPF.

Because eBPF can monitor and even change almost anything in a running system, only root can use eBPF. In contrast, ptrace-based tools like strace can be used safely by anyone because they can only affect one process at a time.

Getting set up (Running the right kernel)

bpftrace is already installed on the class virtual machines. If you are using a different version of Linux, you'll need to install bpftrace yourself. You may be able to use a binary package; however, bpftrace's functionality is very closely tied to the running version of the Linux kernel, thus scripts that work on one system may not work on another. To minimize problems, then, you should use the COMP 3000 VM if at all possible.

Having said this, there is one problem with the original class VMs: they are running the wrong kernel! The Ubuntu kvm kernel for some reason doesn't have all of the eBPF-related configuration options enabled even though the generic Ubuntu kernel does. So, you'll need to change the kernel that you're running in your VM.

There are two solutions:

Make a new VM with an updated image COMP3000A-2021F-2. This VM has the right kernel installed. (If you do this, please delete your old VM.)
Follow the steps below to install the right kernel and make it the default.

Installing a new kernel in the (old) class VM

Run the following commands: (Note the # indicate comments:)

 sudo -i                      #  become root
 apt update                   #  update the package database
 apt -y dist-upgrade          #  upgrade all packages
 apt install linux-virtual    #  install linux-virtual, which will
                              #    install a generic linux kernel
 apt clean; apt -y autoremove #  clean up
 grub-reboot 1\>4             #  on next reboot, select the 2nd
                              #    menu item and then the 5th one
 reboot                       #  reboot!

Note the grub-reboot command assumes that you have two kvm kernels installed plus the new generic one. Your installed packages should look something like this:

 student@comp3000:~$ dpkg --list | grep "linux-image-5" | col2
 linux-image-5.11.0-1015-kvm
 linux-image-5.11.0-1017-kvm
 linux-image-5.11.0-37-generic

If it doesn't, you'll need to change the 4 in the 1\>4 to the right number. Every kernel generates two entries with the second of each being a recovery image; thus, the number we want is the base-0 index times two. In other words, to run each of the above kernels we would put in 0, 2, or 4, respectively.

(Note this is all easier if you can see the grub menu when the kernel boots. We will later show you how to navigate this menu; for now the above should get you running the correct kernel.)

Checking bpftrace

If you are running the right kernel, the following command should produce a long list:

 sudo bpftrace -l

(How can you keep this command from filling your screen?)

If this command stops with an error, you aren't running the right kernel.

bpftrace scripts

The bpftrace scripts we want to run are all in

 /usr/local/share/bpftrace/tools/

You can run them by typing

 sudo bpftrace <file>

So, to run bashreadline.bt, type

 sudo bpftrace /usr/local/share/bpftrace/tools/bashreadline.bt

Copy/paste, up arrow, and Ctrl-K and Ctrl-Y are your friends! Also, you can cd to this directory and run them from there.

Tasks & Questions

Run opensnoop.bt:
1. What files are being opened when you run ls? Are these what you expect?
2. What files are being opened when you run top? What directory are most of them in? Why?
3. Compare the files being opened by bash and static-sh. Is there a significant difference? Why?
4. What files does 3000shell open? Can you change what files it opens by changing how you create the 3000shell binary?
Run execsnoop.bt:
1. Does every typed command result in one execve system call?
2. If you type in a non-existent command, does it produce any execve's in bash? What about 3000shell?
3. When you ssh into the VM you should see many execve calls. Where are they coming from?
Run killsnoop.bt:
1. Are kill system calls happening when you do nothing? Who is sending them?
2. When you interrupt a command using Ctrl-C or Ctrl-S, is a kill system call generated?
3. If you hit Ctrl-C at a bash prompt, does it generate a kill system call?
4. Are all signals being sent via kill system calls? How do you know?
5. What is a 0 signal used for? Do you see any process sending these signals when you log in or out?
Run syscount.bt:
1. What programs are generating lots of system calls?
2. What calls are the most common? Use /usr/include/x86_64-linux-gnu/asm/unistd_64.h for the names associated with system call numbers on the class VM.
Run bashreadline.bt:
1. What does this program do?
2. Look at the code for this script. How do you think it works?
3. If bash is already running, this script won't report the first command that you type. Why?