https://homeostasis.scs.carleton.ca/wiki/index.php?title=Operating_Systems_2021F_Lecture_8&feed=atom&action=historyOperating Systems 2021F Lecture 8 - Revision history2024-03-28T19:43:20ZRevision history for this page on the wikiMediaWiki 1.37.1https://homeostasis.scs.carleton.ca/wiki/index.php?title=Operating_Systems_2021F_Lecture_8&diff=23369&oldid=prevSoma: Created page with "==Video== Video from the lecture given on October 5, 2021 is now available: * [https://homeostasis.scs.carleton.ca/~soma/os-2021f/lectures/comp3000-2021f-lec08-20211005.m4v v..."2021-10-05T17:29:49Z<p>Created page with "==Video== Video from the lecture given on October 5, 2021 is now available: * [https://homeostasis.scs.carleton.ca/~soma/os-2021f/lectures/comp3000-2021f-lec08-20211005.m4v v..."</p>
<p><b>New page</b></p><div>==Video==<br />
<br />
Video from the lecture given on October 5, 2021 is now available:<br />
* [https://homeostasis.scs.carleton.ca/~soma/os-2021f/lectures/comp3000-2021f-lec08-20211005.m4v video]<br />
* [https://homeostasis.scs.carleton.ca/~soma/os-2021f/lectures/comp3000-2021f-lec08-20211005.cc.vtt auto-generated captions]<br />
Video is also available through Brightspace (Resources->Class zoom meetings->Cloud Recordings tab)<br />
<br />
==Notes==<br />
<br />
<pre><br />
Lecture 8<br />
---------<br />
* Assignment 1 solutions released (will discuss at end)<br />
* Tutorial 3 & gdb<br />
* Tutorial 4<br />
<br />
Using gdb<br />
---------<br />
- To allow attaching to processes that aren't gdb's children, do the following:<br />
<br />
sudo -i<br />
echo 0 > /proc/sys/kernel/yama/ptrace_scope<br />
exit # to become a regular user again<br />
<br />
(if you try doing the attach without doing this, you'll get an error<br />
in gdb telling you about this file)<br />
<br />
- Compile with -g (to get debugging symbols) (keep -O)<br />
- connect in two windows<br />
- run the program you want to watch in one window<br />
- in the other, find out its pid (eg using ps aux | grep)<br />
- run gdb on the binary, then attach the PID ("attach <PID>")<br />
- set a breakpoint (probably at a function) so execution stops<br />
at a point of interest<br />
- do "tui enable" to get a litle text-mode interface that shows you code<br />
- note gdb will only follow one process at a time<br />
- so you have to decide whether you want to follow the parent or child<br />
on fork<br />
- by default, follows the parent<br />
- "set follow-fork-mode child" to follow child<br />
- remember that gdb has extensive help and command completion<br />
- tab is your friend!<br />
- n = next statement<br />
c = continue until next breakpoint/signal/program termination<br />
s = next statement, but going into functions<br />
print = view state of variables<br />
x = examine memory<br />
b = breakpoint (by line or function name)<br />
catch syscall = see every system call entered and exited (like strace but<br />
slower)<br />
<br />
- you can't run the program backwards<br />
(there are cool things that can, but not standard tools)<br />
<br />
GDB is a powerful tool, lots to play with and master<br />
- For this class I don't care about you learning gdb per se<br />
- rather, it is a tool for you to understand how<br />
processes work<br />
<br />
<br />
Question: how does gdb actually work?<br />
- aren't processes separate?<br />
- they each have their own address space<br />
- how is one process controlling another?<br />
- how can ltrace and strace watch another process?<br />
- ONLY WAY: ask the kernel for help<br />
- they use ptrace<br />
- ptrace can only follow one process at a time<br />
- it is also very intrusive, can change program behavior<br />
- you don't want to use it when someone cares about the program<br />
continuing to work<br />
- you use ptrace-based tools to debug programs<br />
- but what if you want to debug in production?<br />
<br />
- traditionally, to debug in production you'd just look at logs & crash dumps<br />
- but now we have something better: eBPF<br />
- "enhanced Berkeley Packet Filter" (name is almost meaningless now)<br />
- allows us to add code to the kernel safely to interact with the system<br />
<br />
- if your vm does not have /usr/local/share/bpftrace/tools, you're running the wrong VM (it should be the 2021 os VM)<br />
- this is the one I created for the class<br />
<br />
- VM is all set for bpftrace except for one thing<br />
- WRONG KERNEL<br />
- kvm kernels (for some strange reason) don't have full eBPF support<br />
- error in its configuration<br />
- so you need to install the generic kernel<br />
- instructions in Tutorial 4<br />
- if you have problems please let me know!<br />
- get the generic kernel running before doing Tutorial 4<br />
- check /proc/version that it says something like<br />
Linux version 5.11.0-37-generic<br />
NOT -kvm<br />
<br />
<br />
- unlike strace-based tools, eBPF-based ones must run as root<br />
- they can SEE ALL, so it makes sense<br />
<br />
- what's great about bpftrace is it lets you see what is happening anywhere on<br />
the system<br />
- so can watch specific system calls, who are making them and when<br />
- but can also watch function in userspace & kernelspace<br />
<br />
- yes in an attacker's hands this is potentially very bad<br />
- that's why only root can do it<br />
- lots of other ways for root to get this kind of info,<br />
this is just crazy easy<br />
- I will add to the tutorial the header file with the system call numbers<br />
- so you can interpret the output of syscalls.bt<br />
<br />
<br />
bpftrace works by attaching "probes" to specific tracepoints<br />
- events that can be monitored<br />
- a probe runs when the event happens<br />
- you can see a list of possible events with bpftrace -l<br />
- but you can also do uprobes of arbitrary<br />
userspace functions<br />
- run "sudo bpftrace -l | wc" to see how many, I see 50K+<br />
- use grep to search!<br />
- to see what probes are being used, run bpftrace -v<br />
(verbose)<br />
<br />
I don't expect you to understand how bpftrace works<br />
- it is pretty magical<br />
<br />
But I do expect you to get an understanding of what it is showing you<br />
- files being opened, programs being run, signals being sent<br />
- perspective on everything we've covered up to now<br />
<br />
eBPF is a hot technology in the cloud today<br />
- major companies use all kinds of eBPF-based tools to monitor their<br />
infrastructure, track down bugs, and even secure systems<br />
- look up cilium.io to see the kinds of things being enabled with eBPF<br />
<br />
(bpftrace is just one eBPF-based tool by the way)<br />
<br />
Later you'll try writing your own bpftrace scripts<br />
- but for now, if there is something you'd like to see, ask for<br />
it on Teams, I can try putting something together<br />
<br />
by default, a bpftrace scripts watches the whole system<br />
- you have to add logic to limit what you see<br />
<br />
If you have time to spend learning bpftrace, go ahead, but it won't be covered directly on the midterm<br />
- it is its own language, not fully documented<br />
- I want you to understand the output of the bpftrace scripts asked about<br />
in Tutorial 4<br />
<br />
Other cool things in eBPF:<br />
- bcc, the eBPF compiler collection (python + C)<br />
- cilium (cloud monitoring)<br />
- bpfcontain (William Findlay, my PhD student, doing container security)<br />
<br />
<br />
We're going to use eBPF to learn how the kernel works later<br />
(after the midterm)<br />
<br />
eBPF is (a) tool you use to find out the overhead of other tools<br />
<br />
(Try running "gdb 3000shell" and then type "run" at the gdb prompt. See how well things work)<br />
<br />
Midterm is not proctored, but I will do randomized & selected interviews after<br />
- online proctoring is ridiculous<br />
- you'll submit a text file via brightspace<br />
- open book, open note, open internet, just NO COLLABORATION<br />
(you only have 80 minutes so collaboration would mean cheating,<br />
don't do that)<br />
- you may volunteer for interviews<br />
(good way to make sure you got all the points you should)<br />
- I'll post a schedule once midterms are graded<br />
<br />
A2 will be due by class time on the 14th, along with tutorials 3 & 4.<br />
</pre></div>Soma