https://homeostasis.scs.carleton.ca/wiki/index.php?title=Operating_Systems_2021F_Lecture_18&feed=atom&action=historyOperating Systems 2021F Lecture 18 - Revision history2024-03-29T06:44:15ZRevision history for this page on the wikiMediaWiki 1.37.1https://homeostasis.scs.carleton.ca/wiki/index.php?title=Operating_Systems_2021F_Lecture_18&diff=23506&oldid=prevSoma at 21:24, 18 November 20212021-11-18T21:24:12Z<p></p>
<table style="background-color: #fff; color: #202122;" data-mw="interface">
<col class="diff-marker" />
<col class="diff-content" />
<col class="diff-marker" />
<col class="diff-content" />
<tr class="diff-title" lang="en">
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">← Older revision</td>
<td colspan="2" style="background-color: #fff; color: #202122; text-align: center;">Revision as of 17:24, 18 November 2021</td>
</tr><tr><td colspan="2" class="diff-lineno" id="mw-diff-left-l1">Line 1:</td>
<td colspan="2" class="diff-lineno">Line 1:</td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">==Video==</ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;"></ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">Video from the lecture given on November 18, 2021 is now available:</ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">* [https://homeostasis.scs.carleton.ca/~soma/os-2021f/lectures/comp3000-2021f-lec18-20211118.m4v video]</ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">* [https://homeostasis.scs.carleton.ca/~soma/os-2021f/lectures/comp3000-2021f-lec18-20211118.cc.vtt auto-generated captions]</ins></div></td></tr>
<tr><td colspan="2" class="diff-side-deleted"></td><td class="diff-marker" data-marker="+"></td><td style="color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div><ins style="font-weight: bold; text-decoration: none;">Video is also available through Brightspace (Resources->Class zoom meetings->Cloud Recordings tab)</ins></div></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><br/></td></tr>
<tr><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>==Notes==</div></td><td class="diff-marker"></td><td style="background-color: #f8f9fa; color: #202122; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #eaecf0; vertical-align: top; white-space: pre-wrap;"><div>==Notes==</div></td></tr>
</table>Somahttps://homeostasis.scs.carleton.ca/wiki/index.php?title=Operating_Systems_2021F_Lecture_18&diff=23505&oldid=prevSoma: Created page with " ==Notes== <pre> Lecture 18 ---------- - Restoring from access - ssh-agent - chmod - see man page on online explanations - supervisor vs user mode - what the kernel..."2021-11-18T16:56:32Z<p>Created page with " ==Notes== <pre> Lecture 18 ---------- - Restoring from access - ssh-agent - chmod - see man page on online explanations - supervisor vs user mode - what the kernel..."</p>
<p><b>New page</b></p><div><br />
==Notes==<br />
<br />
<pre><br />
Lecture 18<br />
----------<br />
- Restoring from access<br />
- ssh-agent<br />
- chmod<br />
- see man page on online explanations<br />
<br />
- supervisor vs user mode<br />
- what the kernel does<br />
- virtual memory<br />
- mmap<br />
- kernel modules<br />
- vs eBPF<br />
- filesystems<br />
- VFS<br />
- superblocks & inodes<br />
- character devices<br />
- creating, how they work<br />
<br />
Restoring from access<br />
- look in .bash_aliases for original<br />
- run this to restore:<br />
rsync -a -v --delete --force<br />
<scs-username>@access.scs.carleton.ca:COMP3000VM-backup/<br />
/home/student/<br />
- note this will delete whatever you have in /home/student currently, in favor of the backup!<br />
- specify a new directory if you don't want this, like /home/student/Restore<br />
- only /home/student gets backed up<br />
- you can do other rsync commands to backup other things<br />
- but note those other files will probably be owned by other<br />
users<br />
- add a "-n" flag when you first run a new rsync command to have it simulate what it would do otherwise<br />
<br />
<br />
ssh-agent<br />
- this saves your unlocked private key locally, answers remote requests<br />
- can forward through multiple ssh connections<br />
- should really run locally<br />
- and it is running by default in most desktop environments<br />
(yes even windows and macOS)<br />
- so only need one ssh agent, all ssh connections you make will ask it for authentication info<br />
- if you are chaining ssh commands (ssh'ing to one machine, then from there to another) you may have to add the -A flag to forward the agent connection<br />
<br />
- note you can ssh to access from anywhere, so if you use -J with access you don't need a VPN to connect to the openstack instances<br />
<br />
<br />
<br />
Supervisor vs user mode<br />
- CPU has at least two modes<br />
- user mode (ring 3 on x86)<br />
- supervisor mode (ring 0 on x86)<br />
- for ring 1 and 2, look up MULTICS<br />
- kernel runs in supervisor mode<br />
- processes run in user mode<br />
- system calls invoke the kernel by causing the CPU to switch<br />
to supervisor mode<br />
<br />
Switching between CPU modes is expensive<br />
- because CPUs have lots of caches and registers,<br />
these have to be saved/restored/flushed on each switch<br />
<br />
When in supervisor mode, you can do things that you can't do in user mode<br />
- let's discuss!<br />
<br />
The kernel's job is to abstract and allocate resources to other programs<br />
- running in processes<br />
- so, implement the process abstraction<br />
<br />
- other hardware (than CPU & RAM) => character or block devices<br />
- block devices => files<br />
- one CPU => many CPUs <br />
- share CPUs between processes<br />
- when a process runs, it gets a core to use<br />
- so it has to be kicked off periodically so other<br />
processes can use the same core<br />
- process scheduling: sharing CPUs<br />
(Lots to talk about scheduling, we'll do that later)<br />
- physical memory => virtual memory<br />
<br />
Recall that when we looked at the address space of a process, memory locations were roughly the same for different programs<br />
- and could run same program with same addresses in parallel<br />
<br />
Processes access memory using virtual addresses<br />
- a pointer contains a virtual address<br />
<br />
Virtual memory is good because we don't have to care about where other processes put things in their own address space<br />
- different process, different set of addresses (different address space)<br />
<br />
But at the end of the day we need to use RAM, and RAM is accessed by address<br />
- but these are physical addresses<br />
<br />
So the kernel has to maintain a virtual->physical map for every process<br />
- this map is called a page table<br />
- every process has its own page table<br />
<br />
On every memory access, the CPU has to take the virtual address and figure out the corresponding physical address<br />
- if you had to consult a big data structure every time, would be<br />
too slow<br />
- but CPU maintains a cache of virtual->physical mappings called the TLB<br />
(Translation Lookaside Buffer). Don't ask what it means, just call<br />
it a TLB and blame some old IBM engineers<br />
<br />
Why is this mapping called a page table?<br />
- because it maps pages to pages<br />
- a page is a fixed amount of RAM, power of 2<br />
- 4K or 8K, but can be multiple megabytes<br />
- different CPUs support different page sizes<br />
- basically equivalent to a block, but blocks are for disks<br />
and pages are for RAM<br />
<br />
If you want to learn how CPUs and computers work at a low level and what makes them fast, learn computer architecture<br />
<br />
Great book:<br />
Computer Architecture: A Quantitative Approach 5th Edition<br />
by John L. Hennessy (Author), David A. Patterson (Author)<br />
<br />
<br />
virtual-physical memory mappings aren't done on an address by address basis<br />
- they are done on a page-by-page basis<br />
- so in 4K => 4K chunks<br />
- we divide virtual addresses into a page number and an offset<br />
- high bits are page number, lower bits are offset within page<br />
<br />
So all virtual memory refers to is this maintaining of separate virtual address spaces per process and how these map to physical addresses (dynamically)<br />
<br />
So why bother with all this machinery of virtual memory?<br />
- allows for easy sharing of RAM<br />
- can give a process pages from anywhere, don't<br />
need continuous ranges of memory<br />
- same memory used by multiple processes can map to same physical memory<br />
<br />
So virtual memory is a great way to save on RAM usage<br />
- dynamic linking takes advantage of it, only need one copy of<br />
a library in memory and every process using it can share it<br />
(many virtual mappings but one range of physical RAM being used)<br />
<br />
Remember all those calls to mmap?<br />
- tutorial 6, to access the file on disk<br />
- made by malloc back in tutorial 2<br />
- we see them whenever a dynamically linked process loads<br />
<br />
mmap maps a file into the virtual address space<br />
- so logically, the file has been completely loaded into memory<br />
- but in reality, parts are loaded on demand<br />
<br />
Different ranges of memory can have different permissions<br />
- read, write, execute<br />
- important for security, but also for performance<br />
- if memory is not writable and it came from disk,<br />
we don't have to worry about saving it if we need space,<br />
we can just kick it out of RAM<br />
<br />
Think about fork<br />
- logically, we copy all of a process's memory<br />
- in practice, we just duplicate the virtual address mappings<br />
and mark everything read only<br />
- when one process writes, we duplicate that memory so the<br />
two processes have distinct copies<br />
<br />
<br />
mmap can also be used to make "anonymous mmap's", so allocate RAM but without a corresponding file<br />
- so like sbrk, but can be anywhere<br />
<br />
Can also mark memory regions as being private or shared<br />
- if private, then always make copies<br />
- if shared, then can be accessed by more than one thread/process<br />
<br />
multithreaded processes is really just multiple process-like things (threads) that share virtual memory<br />
<br />
If you see an mmap that is anonymous and uses a file descriptor of -1<br />
- it is allocating memory rather than loading a file<br />
<br />
So fork/execve isn't as waseful as you'd think<br />
- fork, duplicate the process (but not much memory is actually being copied)<br />
- execve, throw out memory map and create a new one based on an<br />
executable on disk<br />
<br />
<br />
Because of how much memory gets shared between processes, it is actually hard to say how much RAM a process is taking up<br />
<br />
Many similarities & connections between files and process memory<br />
- just remember that files are normally persistent, and process memory is volatile<br />
<br />
How does the Linux kernel make sure it doesn't allocate too much RAM?<br />
- it doesn't! It just says yes and hopes things work out<br />
(it is "optimistic")<br />
<br />
It has a few strategies to help out<br />
- program code loaded from disk can always be kicked out of RAM<br />
- data/modified pages in RAM can be written to a swap or page<br />
file/partition<br />
- but eventually we'll run out, and then the out of memory killer<br />
(OOM) gets to work<br />
- it will kill processes until there's enough physical memory<br />
- what it kills can be kind of random...and in general<br />
won't be what you want<br />
- because of the former mechanisms, normally the computer will slow<br />
to a crawl before actually needing to kill processes<br />
- because it will keep trying to free up physical memory,<br />
and eventually will spend most of its time doing that, rather<br />
than useful work<br />
<br />
managing all of this is the Linux kernel's job.<br />
<br />
<br />
<br />
Assume we have a process that uses 2G of virtual address space, 500M resident in RAM.<br />
- system has 8G of RAM total, 1G free<br />
<br />
On such a system, that process can fork with no problem<br />
- won't end up using much more RAM because of sharing between<br />
parent and child<br />
<br />
But what if the child now starts doing lots of work and uses that 2G of virtual address space, using everything<br />
- now we could easily use up that 1G of free RAM<br />
- and even get to the point where the kernel is looking for 500M<br />
<br />
Linux is optimistic when allowing memory allocations because it allows full use of resources<br />
- if you want to put limits, you can impose quotas with cgroups<br />
<br />
By the way, phones run the same basic kernels as desktops and servers nowadays<br />
- but they normally don't have swap<br />
- so, they have to be very aggressive in how they manage RAM<br />
- will proactively kill processes that aren't needed<br />
<br />
Swap is where you "swap" a process to disk when we don't have enough<br />
room for it in physical memory<br />
- used to be we'd have to move the entire process to disk,<br />
hence swapping<br />
- but now we can just move some of it (some pages, so moving to disk=><br />
paging to disk)<br />
<br />
We don't have swap on phones to<br />
- save power<br />
- save wear and tear on flash memory<br />
(flash can only be written to so much)<br />
<br />
When you develop for a phone, your processes have to be ready to be<br />
killed at any time<br />
- you tell the OS how to save necessary state so you can resume<br />
</pre></div>Soma