Soma-notes - User contributions [en]

Operating Systems 2017F Lecture 23

2017-12-07T19:10:57Z

Rquaium: /* Additional Notes */

== Additional Notes ==
Written solutions for midterm exam are on the course webpage 
Assignment 4 
Q2: SSH keygen generates the secret key and the public key file. The private key is stored in the private key file: .ssh/id_rsa 
Q4: Both lines because first you start from 1 and then increment from there. 
Q11: Only bs because it is write. Would be ibs and bs if it was read. 
Q12: Local kernel forwards the write system call but doesn't actually make the system call. Kernels don't make system calls. 
Lecture 23 Prof Notes 
-----------------
How can you tell when a process has been compromised. 
- from outside the process 
Use signatures 
- is it running "bad code" 
- is it doing "bad things" 
-- For example: password program start modifying files other than etc/password. You could specify rules to prevent this. 
-- bad system calls 
For a process to do damage, it has to make "bad" system calls 
I want to be lazy 
- complex rules are a pain 
- and they don't work well either 
Make the computer solve this problem 
- Use Machine Learning 
I can't teach good versus bad if I don't know what is bad 
But I know how systems behave normally 
How about teaching the system to differentiate normal from abnormal 
- normal is good 
- abnormal may be bad 
abnormal but not bad => false positive 
False positives can be a big issue because they may cause ppl to not trust the machine's detection capabilities 
 
How can we detect abnormal system calls? 
Learn normal patterns of system calls over time 
Once you've learned enough, watch for abnormal system calls 
Since I'm lazy, I want to learn it as it runs 
- and automatically decide when it has learned enough 
 
Could I do the learning in a process (or set of processes)? 
you could, but all data would have to come from the kernel 
Want something fast and simple 
- implement in the kernel 
How simple could it be? 
 
First assumption: ignore arguments 
Second assumption: look at ordering of system calls on a per-thread, per-process basis 
Third Assumption: characterize processes based on the executable they are running 
model per executable, each trained on multiple processes 
How to model the trace of system calls coming from a procesd 
* frequency analysis? 
- on a system call basis 
- high variance 
* what system calls are made (and not made)? 
* short sequence of system calls? 6-10 calls

Operating Systems 2017F Lecture 23

2017-12-07T18:58:52Z

Rquaium: /* Additional Notes */

== Additional Notes ==
Written solutions for midterm exam are on the course webpage 
Assignment 4 
Q2: SSH keygen generates the secret key and the public key file. The private key is stored in the private key file: .ssh/id_rsa 
Q4: Both lines because first you start from 1 and then increment from there. 
Q11: Only bs because it is write. Would be ibs and bs if it was read. 
Q12: Local kernel forwards the write system call but doesn't actually make the system call. Kernels don't make system calls. 
Lecture 23 Prof Notes 
-----------------
How can you tell when a process has been compromised. 
- from outside the process 
Use signatures 
- is it running "bad code" 
- is it doing "bad things" 
-- For example: password program start modifying files other than etc/password. You could specify rules to prevent this. 
-- bad system calls 
For a process to do damage, it has to make "bad" system calls 
I want to be lazy 
- complex rules are a pain 
- and they don't work well either 
Make the computer solve this problem 
- Use Machine Learning 
I can't teach good versus bad if I don't know what is bad 
But I know how systems behave normally 
How about teaching the system tod ifferentiate normal from abnormal 
- normal is good 
- abnormal may be bad 
abnormal but not bad => false positive 
False positives can be a big issue because they may cause ppl to not trust the machine's detection capabilities 
 
How can we detect abnormal system calls? 
Learn normal patterns of system calls over time 
Once you've learned enough, watch for abnormal system calls 
Since I'm lazy, I want to learn it as it runs 
- and automatically decide when it has learned enough 
 
Could I do the learning in a process (or set of processes)? 
you could, but all data would have to come from the kernel 
Want something fast and simple 
- implement in the kernel 
How simple could it be? 
 
First assumption: ignore arguments 
Second assumption: look at ordering of system calls on a per-thread, per-process basis 
Third Assumption: characterize processes based on the executable they are running 
model per executable, each trained on multiple processes

Operating Systems 2017F Lecture 23

2017-12-07T18:39:56Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 23

2017-12-07T18:34:48Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 23

2017-12-07T18:29:34Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 23

2017-12-07T18:20:54Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 23

2017-12-07T18:16:24Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 23

2017-12-07T18:09:01Z

Rquaium: /* Additional Notes */

== Additional Notes ==
Solutions for midterm exam are on the course webpage

Operating Systems 2017F Lecture 23

2017-12-07T18:07:25Z

Rquaium: Created page with " == Additional Notes =="

== Additional Notes ==

Operating Systems 2017F Lecture 20

2017-11-28T19:20:08Z

Rquaium: /* Additional Notes */

== Additional Notes ==
Assignment 4 will be autograded and in fill in the blank form. This will not be like the final exam. 
You should think about who is doing what? Otherwise FUSE won't make much sense 
 
Inode numbers completely changed when you use ssh. It starts with 1 from "." (root) and increments from there. 
Inodes come from filesystem you are acessing 
He used df to find filesystem info 
The used Commands: mount | grep ubuntu and mount | grep vda1 to find type of filesystem 
In the SSH there is the filesystem type "fuse.sshfs" 
When you run strace on ls on the two terminals (ssh connection and the local one) you see similar output 
When we run "ls" on an ssh-mounted filesystem 
* ls makes system calls (open, getdents) to the local Linux kernel 
* the local kernel sees the filesystem is FUSE, calls DUSE routines for open, getdents (via vfs abstraction) 
* FUSE calls the sshfs process that mounted the filesystem 
* sshfs process sends request to remote system 
- via socket system calls 
* remote sshd process receives request (via system calls) 
* remote sshd process receives requests (via system calls) 
- makes open, getdents sytem calls 
- rmeote kernel checks vfs, calls ext4 routines to access data 
* remote sshd process responds to requests (via system calls) 
* local sshfs process receives response (via system calls) 
* local sshfs process responds to FUSE request (via system calls) 
* FUSE passes data back to vfs layer, then back to requesting process 
 If you look as lsmod | less you see a ton of modules. Some of the modules are not necessary to mount the root filesystem. 
ls /lib/modules there are directories that store the kernel modules 
You have to have a filesystem that you mount initially that has a bunch of modules in it. Thus, there is an initial root file system that is necessary to load everything else 
This is the initial RAM disk. This filesystem loads the modules needed for the real filesystem. 
To remove a file I need to remove the hardlink from the directory where the hardlink exists 
The password files maps usernames to user id's (Linux does not care about your username) 
Normal file access permission check br>
- compare uid, gid of file with uid, gid of process 
But really: 
- compares it with fsuid, fsgid of process 
- which is normally same as euid, egid of process

Operating Systems 2017F Lecture 20

2017-11-28T19:17:55Z

Rquaium:

== Additional Notes ==
Assignment 4 will be autograded and in fill in the blank form. This will not be like the final exam. 
You should think about who is doing what? Otherwise FUSE won't make much sense 
 
Inode numbers completely changed when you use ssh. It starts with 1 from "." (root) and increments from there. 
Inodes come from filesystem you are acessing 
He used df to find filesystem info 
The used Commands: mount | grep ubuntu and mount | grep vda1 to find type of filesystem 
In the SSH there is the filesystem type "fuse.sshfs" 
When you run strace on ls on the two terminals (ssh connection and the local one) you see similar output 
When we run "ls" on an ssh-mounted filesystem 
* ls makes system calls (open, getdents) to the local Linux kernel 
* the local kernel sees the filesystem is FUSE, calls DUSE routines for open, getdents (via vfs abstraction) 
* FUSE calls the sshfs process that mounted the filesystem 
* sshfs process sends request to remote system 
- via socket system calls 
* remote sshd process receives request (via system calls) 
* remote sshd process receives requests (via system calls) 
- makes open, getdents sytem calls 
- rmeote kernel checks vfs, calls ext4 routines to access data 
* remote sshd process responds to requests (via system calls) 
* local sshfs process receives response (via system calls) 
* local sshfs process responds to FUSE request (via system calls) 
* FUSE passes data back to vfs layer, then back to requesting process 
 If you look as lsmod | less you see a ton of modules. Some of the modules are not necessary to mount the root filesystem. 
ls /lib/modules there are directories that store the kernel modules 
You have to have a filesystem that you mount initially that has a bunch of modules in it. Thus, there is an initial root file system that is necessary to load everything else 
This is the initial RAM disk. This filesystem loads the modules needed for the real filesystem. 
To remove a file I need to remove the hardlink from the directory where the hardlink exists 
The password files maps usernames to user id's (Linux does nto care about your username) 
Normal file access permission check br>
- compare uid, gid of file with uid, gid of process 
But really: 
- compares it with fsuid, fsgid of process 
- which is normally same as euid, egid of process

Operating Systems 2017F Lecture 20

2017-11-28T19:16:02Z

Rquaium:

<nowiki>Insert non-formatted text here</nowiki>== Additional Notes ==
Assignment 4 will be autograded and in fill in the blank form. This will not be like the final exam. 
You should think about who is doing what? Otherwise FUSE won't make much sense 
 
Inode numbers completely changed when you use ssh. It starts with 1 from "." (root) and increments from there. 
Inodes come from filesystem you are acessing 
He used df to find filesystem info 
The used Commands: mount | grep ubuntu and mount | grep vda1 to find type of filesystem 
In the SSH there is the filesystem type "fuse.sshfs" 
When you run strace on ls on the two terminals (ssh connection and the local one) you see similar output 
When we run "ls" on an ssh-mounted filesystem 
* ls makes system calls (open, getdents) to the local Linux kernel 
* the local kernel sees the filesystem is FUSE, calls DUSE routines for open, getdents (via vfs abstraction) 
* FUSE calls the sshfs process that mounted the filesystem 
* sshfs process sends request to remote system 
- via socket system calls 
* remote sshd process receives request (via system calls) 
* remote sshd process receives requests (via system calls) 
- makes open, getdents sytem calls 
- rmeote kernel checks vfs, calls ext4 routines to access data 
* remote sshd process responds to requests (via system calls) 
* local sshfs process receives response (via system calls) 
* local sshfs process responds to FUSE request (via system calls) 
* FUSE passes data back to vfs layer, then back to requesting process 
 If you look as lsmod | less you see a ton of modules. Some of the modules are not necessary to mount the root filesystem. 
ls /lib/modules there are directories that store the kernel modules 
You have to have a filesystem that you mount initially that has a bunch of modules in it. Thus, there is an initial root file system that is necessary to load everything else 
This is the initial RAM disk. This filesystem loads the modules needed for the real filesystem. 
To remove a file I need to remove the hardlink from the directory where the hardlink exists 
The password files maps usernames to user id's (Linux does nto care about your username) 
Normal file access permission check br>
- compare uid, gid of file with uid, gid of process 
But really: 
- compares it with fsuid, fsgid of process 
- which is normally same as euid, egid of process

Operating Systems 2017F Lecture 20

2017-11-28T19:14:22Z

Rquaium:

Operating Systems 2017F Lecture 20

2017-11-28T19:05:32Z

Rquaium:

<nowiki>Insert non-formatted text here</nowiki>== Additional Notes ==
Assignment 4 will be autograded and in fill in the blank form. This will not be like the final exam. 
You should think about who is doing what? Otherwise FUSE won't make much sense 
 
Inode numbers completely changed when you use ssh. It starts with 1 from "." (root) and increments from there. 
Inodes come from filesystem you are acessing 
He used df to find filesystem info 
The used Commands: mount | grep ubuntu and mount | grep vda1 to find type of filesystem 
In the SSH there is the filesystem type "fuse.sshfs" 
When you run strace on ls on the two terminals (ssh connection and the local one) you see similar output 
Prof Notes:
When we run "ls" on an ssh-mounted filesystem 
* ls makes system calls (open, getdents) to the local Linux kernel 
* the local kernel sees the filesystem is FUSE, calls DUSE routines for open, getdents (via vfs abstraction) 
* FUSE calls the sshfs process that mounted the filesystem 
* sshfs process sends request to remote system 
- via socket system calls 
* remote sshd process receives request (via system calls) 
* remote sshd process receives requests (via system calls) 
- makes open, getdents sytem calls 
- rmeote kernel checks vfs, calls ext4 routines to access data 
* remote sshd process responds to requests (via system calls) 
* local sshfs process receives response (via system calls) 
* local sshfs process responds to FUSE request (via system calls) 
* FUSE passes data back to vfs layer, then back to requesting process 
End of prof notes 
 If you look as lsmod | less you see a ton of modules. Some of the modules are not necessary to mount the root filesystem. 
ls /lib/modules there are directories that store the kernel modules 
You have to have a filesystem that you mount initially that has a bunch of modules in it. Thus, there is an initial root file system that is necessary to load everything else 
This is the initial RAM disk. This filesystem loads the modules needed for the real filesystem. 
To remove a file I need to remove the hardlink from the directory where the hardlink exists

Operating Systems 2017F Lecture 20

2017-11-28T18:56:07Z

Rquaium:

<nowiki>Insert non-formatted text here</nowiki>== Additional Notes ==
Assignment 4 will be autograded and in fill in the blank form. This will not be like the final exam. 
You should think about who is doing what? Otherwise FUSE won't make much sense 
 
Inode numbers completely changed when you use ssh. It starts with 1 from "." (root) and increments from there. 
Inodes come from filesystem you are acessing 
He used df to find filesystem info 
The used Commands: mount | grep ubuntu and mount | grep vda1 to find type of filesystem 
In the SSH there is the filesystem type "fuse.sshfs" 
When you run strace on ls on the two terminals (ssh connection and the local one) you see similar output 
Prof Notes:
When we run "ls" on an ssh-mounted filesystem 
* ls makes system calls (open, getdents) to the local Linux kernel 
* the local kernel sees the filesystem is FUSE, calls DUSE routines for open, getdents (via vfs abstraction) 
* FUSE calls the sshfs process that mounted the filesystem 
* sshfs process sends request to remote system 
- via socket system calls 
* remote sshd process receives request (via system calls) 
* remote sshd process receives requests (via system calls) 
- makes open, getdents sytem calls 
- rmeote kernel checks vfs, calls ext4 routines to access data 
* remote sshd process responds to requests (via system calls) 
* local sshfs process receives response (via system calls) 
* local sshfs process responds to FUSE request (via system calls) 
* FUSE passes data back to vfs layer, then back to requesting process 
End of prof notes 
 If you look as lsmod | less you see a ton of modules. Some of the modules are not necessary to mount the root filesystem. 
ls /lib/modules there are directories that store the kernel modules 
You have to have a filesystem that you mount initially that has a bunch of modules in it. Thus, there is an initial root file system that is necessary to load everything else 
This is the initial RAM disk. This filesystem loads the modules needed for the real filesystem.

Operating Systems 2017F Lecture 20

2017-11-28T18:51:30Z

Rquaium:

<nowiki>Insert non-formatted text here</nowiki>== Additional Notes ==
Assignment 4 will be autograded and in fill in the blank form. This will not be like the final exam. 
You should think about who is doing what? Otherwise FUSE won't make much sense 
 
Inode numbers completely changed when you use ssh. It starts with 1 from "." (root) and increments from there. 
Inodes come from filesystem you are acessing 
He used df to find filesystem info 
The used Commands: mount | grep ubuntu and mount | grep vda1 to find type of filesystem 
In the SSH there is the filesystem type "fuse.sshfs" 
When you run strace on ls on the two terminals (ssh connection and the local one) you see similar output 
Prof Notes:
When we run "ls" on an ssh-mounted filesystem 
* ls makes system calls (open, getdents) to the local Linux kernel 
* the local kernel sees the filesystem is FUSE, calls DUSE routines for open, getdents (via vfs abstraction) 
* FUSE calls the sshfs process that mounted the filesystem 
* sshfs process sends request to remote system 
- via socket system calls 
* remote sshd process receives request (via system calls) 
* remote sshd process receives requests (via system calls) 
- makes open, getdents sytem calls 
- rmeote kernel checks vfs, calls ext4 routines to access data 
* remote sshd process responds to requests (via system calls) 
* local sshfs process receives response (via system calls) 
* local sshfs process responds to FUSE request (via system calls) 
* FUSE passes data back to vfs layer, then back to requesting process 
End of prof notes 
 If you look as lsmod | less you see a ton of modules. Some of the modules are not necessary to mount the root filesystem. 
ls /lib/modules there are directories that store the kernel modules 
You have to have a filesystem that you mount initially that has a bunch of modules in it. Thus, there is an initial root file system that is necessary to load everything else 
This is the initial RAM disk. This filesystem loads the modules needed for the real filesystem.

Operating Systems 2017F Lecture 20

2017-11-28T18:49:19Z

Rquaium: /* Additional Notes */

<nowiki>Insert non-formatted text here</nowiki>== Additional Notes ==
Assignment 4 will be autograded and in fill in the blank form. This will not be like the final exam. 
You should think about who is doing what? Otherwise FUSE won't make much sense 
 
Inode numbers completely changed when you use ssh. It starts with 1 from "." (root) and increments from there. 
Inodes come from filesystem you are acessing 
He used df to find filesystem info 
The used Commands: mount | grep ubuntu and mount | grep vda1 to find type of filesystem 
In the SSH there is the filesystem type "fuse.sshfs" 
When you run strace on ls on the two terminals (ssh connection and the local one) you see similar output 
Prof Notes:
When we run "ls" on an ssh-mounted filesystem 
* ls makes system calls (open, getdents) to the local Linux kernel 
* the local kernel sees the filesystem is FUSE, calls DUSE routines for open, getdents (via vfs abstraction) 
* FUSE calls the sshfs process that mounted the filesystem 
* sshfs process sends request to remote system 
- via socket system calls 
* remote sshd process receives request (via system calls) 
* remote sshd process receives requests (via system calls) 
- makes open, getdents sytem calls 
- rmeote kernel checks vfs, calls ext4 routines to access data 
* remote sshd process responds to requests (via system calls) 
* local sshfs process receives response (via system calls) 
* local sshfs process responds to FUSE request (via system calls) 
* FUSE passes data back to vfs layer, then back to requesting process 
End of prof notes 
 If you look as lsmod | less you see a ton of modules. Some of the modules are not necessary to mount the root filesystem. 
ls /lib/modules there are directories that store the kernel modules 
You have to have a filesystem that you mount initially that has a bunch of modules in it. Thus, there is an initial root file system that is necessary to load everything else

Operating Systems 2017F Lecture 20

2017-11-28T18:46:38Z

Rquaium: /* Additional Notes */

== Additional Notes ==
Assignment 4 will be autograded and in fill in the blank form. This will not be like the final exam. 
You should think about who is doing what? Otherwise FUSE won't make much sense 
 
Inode numbers completely changed when you use ssh. It starts with 1 from "." (root) and increments from there. 
Inodes come from filesystem you are acessing 
He used df to find filesystem info 
The used Commands: mount | grep ubuntu and mount | grep vda1 to find type of filesystem 
In the SSH there is the filesystem type "fuse.sshfs" 
When you run strace on ls on the two terminals (ssh connection and the local one) you see similar output 
Prof Notes:
When we run "ls" on an ssh-mounted filesystem 
* ls makes system calls (open, getdents) to the local Linux kernel 
* the local kernel sees the filesystem is FUSE, calls DUSE routines for open, getdents (via vfs abstraction) 
* FUSE calls the sshfs process that mounted the filesystem 
* sshfs process sends request to remote system 
- via socket system calls 
* remote sshd process receives request (via system calls) 
* remote sshd process receives requests (via system calls) 
- makes open, getdents sytem calls 
- rmeote kernel checks vfs, calls ext4 routines to access data 
* remote sshd process responds to requests (via system calls) 
* local sshfs process receives response (via system calls) 
* local sshfs process responds to FUSE request (via system calls) 
* FUSE passes data back to vfs layer, then back to requesting process 
End of prof notes 
 If you look as lsmod | less you see a ton of modules. Some of the modules are not necessary to mount the root filesystem. 
ls /lib/modules there are directories that store the kernel modules 
You have to have a filesystem that you mount initially that has a bunch of modules in it. Thus, there is an initial root file system that is necessary to load everything else

Operating Systems 2017F Lecture 20

2017-11-28T18:39:12Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 20

2017-11-28T18:38:43Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 20

2017-11-28T18:34:33Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 20

2017-11-28T18:23:09Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 20

2017-11-28T18:20:41Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 20

2017-11-28T18:19:15Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 20

2017-11-28T18:14:30Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 20

2017-11-28T18:07:47Z

Rquaium: Created page with " == Additional Notes == What's Left?"

== Additional Notes ==
What's Left?

Operating Systems 2017F Lecture 18

2017-11-16T19:10:46Z

Rquaium: /* Additional Notes */

== Additional Notes ==
Lec 18 
* More on filesystems 
* How can you recover a fs and how do you delete a file? 
 
A filesystem is a: 
* Persistent data structure 
* Stored in fixed size blocks (at least 512 bytes in size) 
* Maps hierarchical filenames to file contents 
* Has metadata about files somehow 
 
What's in a filesystem 
* data blocks 
* metadata blocks, you need someway to find the blocks 
 
How do you organize metadata? 
 
First identify basic characteristics of the filesystem 
 
You need a "superblock" which is a "summary" block that tells you about everything else 
 
Normally the superblock is the first block of the filesystem 
 
In the superblock 
* Type of filesystem 
** What filesystem magic number is there 
** file command to know file type 
* Size of the filesystem 
* How the filesystem is organized 
* Where can I find the rest of the metadata 
He opened a .jpg as a binary file to show us the magic number in a file, first several bytes identify type of file. Kernel does not care about file extension. Userspace programs may care about the extension. 
 
POSIX is a standard for UNIX 
For POSIX filesystems 
* File metadata is stored in INODES 
* most have pre-reserved inodes 
Usenet is a worldwide distribute discussion system that is deprecated now because it could not handle the spam people uploaded into it, lol. Format for usenet was every message stored in individual file. You will have lots of files! 
So we have: 
* superblock 
* inode blocks 
* data blocks 
** data blocks for directories 
** data blocks for files 
 
How do you recover from damage? 
* filesystems never "reboot", must remain correct over the course of years 
* Errors will happen: bitrot, accidental corruption, computer failiure/memory corruption/hard reboot 
 
To make filesystems fast, data and meta-data is cached in RAM 
* Bad things happen if this data hasn't been writen to disk and you reboot 
* Even worse things happen if your RAM is bad and corrupts the data 
* FSCK is like scandisk in Windows 98
 
Also bad...what if you lose the superblock? 
* You could lose EVERYTHING 
* node trunc dd command blew away first bytes of the file system so you could not mount it because you corrupted the superblock. However, fsck fixed this because we have backup superblocks :D 
 
Old scandisk/fsck was slow because we they had to scan all filesystem metadata 
* Not to recover data, but to fix metadata 
* lost+found might have some files that you might recover. It is a part of the filesystem for fsck to use. 
 
Nowadays fsck is very fast and we rarely lose data due to losing power 
* What this means is we must be writing to disk all the time 
* But isn't writing slow? All writes aren't slow particularly on conventional hard disks. 
 
On Magnetic Hard Disks (not SSD's) 
* sequential oeprations are fast 
* random access is slow 
** we have to move the read/write head 
 
So, on modern systems we update metadata (and sometimes data) by writing sequentially to disk...and then later writing randomly 
* Sequential writes go to the journal 
 
On fsck on a journaled filesystem
* just check the journal for pending operations (replaying the journal) 
There exists filesytems for optimizing writes that are pure journal 
* log-based filesystem 
 
Logs and journal inherently create multiple copies of data and metadata that are hard to track. This makes deletion nearly impossible (at least to guarantee) 
 
Only way to garauntee...encrypt everything 
* if every file has its own key, you can delete the key and this "delete" the data 
 
SSD's use log-structured sotrage at a level below blocks 
* Writes are coarse-grained (you have to write a lot at once) 
* You don't want to write to the same cells too often, they will die 
** You have to do "wear-leveling"

Operating Systems 2017F Lecture 18

2017-11-16T19:02:47Z

Rquaium: /* Additional Notes */

== Additional Notes ==
Lec 18 
* More on filesystems 
* How can you recover a fs and how do you delete a file? 
 
A filesystem is a: 
* Persistent data structure 
* Stored in fixed size blocks (at least 512 bytes in size) 
* Maps hierarchical filenames to file contents 
* Has metadata about files somehow 
 
What's in a filesystem 
* data blocks 
* metadata blocks, you need someway to find the blocks 
 
How do you organize metadata? 
 
First identify basic characteristics of the filesystem 
 
You need a "superblock" which is a "summary" block that tells you about everything else 
 
Normally the superblock is the first block of the filesystem 
 
In the superblock 
* Type of filesystem 
** What filesystem magic number is there 
** file command to know file type 
* Size of the filesystem 
* How the filesystem is organized 
* Where can I find the rest of the metadata 
He opened a .jpg as a binary file to show us the magic number in a file, first several bytes identify type of file. Kernel does not care about file extension. Userspace programs may care about the extension. 
 
POSIX is a standard for UNIX 
For POSIX filesystems 
* File metadata is stored in INODES 
* most have pre-reserved inodes 
Usenet is a worldwide distribute discussion system that is deprecated now because it could not handle the spam people uploaded into it, lol. Format for usenet was every message stored in individual file. You will have lots of files! 
So we have: 
* superblock 
* inode blocks 
* data blocks 
** data blocks for directories 
** data blocks for files 
 
How do you recover from damage? 
* filesystems never "reboot", must remain correct over the course of years 
* Errors will happen: bitrot, accidental corruption, computer failiure/memory corruption/hard reboot 
 
To make filesystems fast, data and meta-data is cached in RAM 
* Bad things happen if this data hasn't been writen to disk and you reboot 
* Even worse things happen if your RAM is bad and corrupts the data 
* FSCK is like scandisk in Windows 98
 
Also bad...what if you lose the superblock? 
* You could lose EVERYTHING 
* node trunc dd command blew away first bytes of the file system so you could not mount it because you corrupted the superblock. However, fsck fixed this because we have backup superblocks :D 
 
Old scandisk/fsck was slow because we they had to scan all filesystem metadata 
* Not to recover data, but to fix metadata 
* lost+found might have some files that you might recover. It is a part of the filesystem for fsck to use. 
 
Nowadays fsck is very fast and we rarely lose data due to losing power 
* What this means is we must be writing to disk all the time 
* But isn't writing slow? All writes aren't slow particularly on conventional hard disks. 
 
On Magnetic Hard Disks (not SSD's) 
* sequential oeprations are fast 
* random access is slow 
** we have to move the read/write head 
 
So, on modern systems we update metadata (and sometimes data) by writing sequentially to disk...and then later writing randomly 
* Sequential writes go to the journal 
 
On fsck on a journaled filesystem
* just check the journal for pending operations (replaying the journal) 
There exists filesytems for optimizing writes that are pure journal 
* log-based filesystem 
 
Logs and journal inherently create multiple copies of data and metadata that are hard to track. This makes deletion nearly impossible (at least to garauntee) 
 
Only way to garauntee...encrypt everything 
* if every file has its own key, you can delete the key and this "delete" the data

Operating Systems 2017F Lecture 18

2017-11-16T18:55:38Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 18

2017-11-16T18:53:16Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 18

2017-11-16T18:47:55Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 18

2017-11-16T18:44:32Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 18

2017-11-16T18:43:13Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 18

2017-11-16T18:40:00Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 18

2017-11-16T18:36:42Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 18

2017-11-16T18:32:40Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 18

2017-11-16T18:28:44Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 18

2017-11-16T18:25:13Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 18

2017-11-16T18:24:31Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 18

2017-11-16T18:16:26Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 18

2017-11-16T18:12:36Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 18

2017-11-16T18:10:21Z

Rquaium: Created page with " == Additional Notes == Lec 18 * More on filesystems * How can you recover a fs and how do you delete a file? "

== Additional Notes ==
Lec 18 
* More on filesystems 
* How can you recover a fs and how do you delete a file?

Operating Systems 2017F Lecture 17

2017-11-15T03:19:14Z

Rquaium: /* Additional Notes */

==Video==

The video from the lecture given on Nov. 14, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec17-14Nov2017.mp4 is now available].

==Notes==

===In Class===

<pre>
Lecture 17
----------

How do you do kernel hacking?

* Be humble
- you don't know how everything works, and nobody else does

* Verify your assumptions as you go
- perform lots of experiments
- incremental development, compile and run very often

* Check for errors ALWAYS
- will save you from later errors
- kernel has to "live cleanly"

* Find another part of the kernel that is close to what you want to do.
Read that code, use their approach where possible
- You don't understand all the abstractions and assumptions, so
"pattern match" to minimize trouble
- Realize that the assumptions behind code you find don't necessarily match
your own.

* Understand the "flow of control" inside the program
- architecture
- division of responsibilites
- purpose of data structures/objects

Scheduling
----------
How does the kernel maintain control?
- CPU is given entirely to userspace processes to run most of the time

Kernel needs to run when it needs to run
- and no process should be able to stop it

*The interrupt table*
- pointers to code that the CPU runs when it receives different hardware
or software interrupts
- since the kernel runs first, it gets to set the interrupt table
- only supervisor code can change the table, not user code

So when the ethernet card receives data...
...the ethernet card sends an interrupt to the kernel
...the CPU calls the kernel code for handling ethernet data

When the clock generates a timer interrupt...
...the CPU calls the kernel code for handling timer interrupts

In order to process an interrupt, an CPU core has to be taken over
- that core was probably running a userspace process

Scheduling is all about what to do after having kicked a userspace process
off of a core

Normally on a core
* userspace process is running
* interrupt happens
* core swiches to supervisor mode, runs kernel code
* last part of kernel code is scheduler, chooses which userspace code to run
* goto top :-)

Kernel is entered via interrupts, exited via the scheduler

Entry and exit to the kernel has to do low-level tasks like changing CPU modes,
manage CPU registers
- must be written in assembly

Most OS kernels minimize this low-level code

What criteria should the scheduler use in deciding what to run?
- "fairness"
- prevent starvation
- absent other conditions, equal allocation of resources
- bias resources towards "foreground" tasks in interactive systems
- never biased enough
- always hacks, heuristics

Memory overcommitment
- it is possible to allocate way more memory than can be ever used
- goes into "memory debt"
</pre>

===Midterm Review===
1. Execve and no it does not create a new process

2. Dynamically linked programs map in dynamically linked libraries

3. Hard links maps filenames to inodes

4. Sleep until the consumer wakes it

5. Using type conversion, read returns number of bytes written. We must convert back to the number of unsigned long, minus one because maximum index of array is one less

6. uses of signals:
* kill process
* find out when child process is terminated
* start/stop a process
* detect when a program has made a ptr error

7. Pointers in C contain virtual memory addresses because the memory of a process is a virtual address space. Each process thinks it has its own stretch of memory.

8. processes make a sys call to allocate memory. execve -> then mmap and/or sbreak to allocate memory

9. shell opens the file output.txt.

10. mmap contents of file can allocate the contents of it into ram.
* comparing 1 mb of both files at a time will be less expensive
* only makes sense to mmap if you will access files multiple times

11. producer and consumer now have no way of talking to each other because they do not share the same memory.

12. no. there is a race condition. while loop loops while sem < 0.
someone else could access and modify the variable between the while and the sem--;.
* you need special instructions to test and set the variable at the same time.
* also, int sem parameter isn't a ptr so it's a local variable.

===Additional Notes===

How do you do kernel hacking? Some tips for low level kernel code:
Be humble; You cannot know how everything works 
Verify your assumptions as you go 
Perform lots of experiments 
Incremental Development, compile and run often 
 
Check for errors ALWAYS or they will cascade and hurt you later 
*This will save you from later errors 
*Kernel has to live cleanly 
*This principle applies to any large scale coding
 
Find another part of the kernel that is close to what you want to do (Search online! This is a good link: http://elixir.free-electrons.com/linux/v4.4.83/source 
You do not understand all abstractions and assumptions, so "pattern match" to minimize trouble 
*Functionality that you want to implement might already exist in the kernel, look at how they do things and follow the pattern. 
*Refine understanding as you go, the other implementation might be for something else. 
 
Realize that the assumptions behind code you don't necessarily match to your own. Spend the time reading other people's code. 
Understand the "flow of control" inside the program 
*Architecture 
*Division of responsibilities 
*Purpose of data structures/objects 

For the tutorial 6 he gave us a link for a reason. If you click on it you can search getpid and then dig deeper! For example cut and paste the line inside getuid function. 
 
Scheduling 
Scheduler decides when what process should run at a high level 
Questions: How does the scheduler work/kernel control when a program runs/how can you make sure one program does not take over CPU from anyone else? 
CPU is given entirely to userspace processes to run most of the time 
Kernel needs to run when it needs to run and no process should be able to stop it 
* The interrupt table: pointers to code that the CPU runs when it receives different hardware or software interrupts. Since the kernel runs first it gets to set the interrupt table. Only supervisor code can change the table not user code.
 
When the Ethernet card receives data: 
1) The Ethernet card sends an interrupt to the kernel 
2) The CPU calls the kernel code for handling Ethernet data 
 
When the clock generates a timer interrupt: 
1) The CPU calls the kernel code for handling timer interrupts 
 
2) In order to process an interrupt, a CPU core has to be taken over 
3) That core was probably running a userspace process 
4) Scheduling is all about what to do after having kicked a userspace process off of a core 
 
Normally on a core: 
* Userspace is happening 
* Interrupt happens 
* core switches to supervisor mode, runs kernel code 
* last part of kernel code is scheduler, chooses which userspace code to run. Repeat the process 
Kernel is entered via interrupts, exited via the scheduler 
Scheduler needs to be efficient because interrupts are frequent and we do not want to waste resources that could be put towards actual computing 
Entry and exit to kernel has to do low-level tasks like changing CPU modes, manage CPU registers. This is Assembly code.
 
Go to http://elixir.free-electrons.com/linux/v4.4.83/source 
Go into arch: code specific to CPU architecture. Two most commons are arm and x86 
Go up one level 
Drivers directory have drivers for all kinds of things 
Linux Kernel designed for x86 but it now abstracted to support different architectures.
Go into architecture -> x86 -> entry 
Open entry.s: Large assembly language file. This is the code that runs before system call specific code. 
shced.h -> Contains definition of task_struct. Data structure that keeps track of a process. 
 
What criteria should the scheduler use in deciding what to run? 
*Fairness: Starvation is when a program does not get the CPU 
**Everyone should get a turn
**CPU is passed around
** Nobody should be left that they never get the CPU
**Prevent Starvation 
**Absent other conditions, equal allocation of resources
*Bias resources towards "foreground" tasks in interactive systems
**never Biased enough
**always hacks, heuristics
 
Memory overcommitment 
Memory does not get allocated when you mmap. It is loaded "lazily" in the Linux kernel. It is possible to allocate way more memory than can ever be used. 
Goes into "memory debt"

Operating Systems 2017F Lecture 17

2017-11-15T03:14:24Z

Rquaium: /* Midterm Review */

==Video==

The video from the lecture given on Nov. 14, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec17-14Nov2017.mp4 is now available].

==Notes==

===In Class===

<pre>
Lecture 17
----------

How do you do kernel hacking?

* Be humble
- you don't know how everything works, and nobody else does

* Verify your assumptions as you go
- perform lots of experiments
- incremental development, compile and run very often

* Check for errors ALWAYS
- will save you from later errors
- kernel has to "live cleanly"

* Find another part of the kernel that is close to what you want to do.
Read that code, use their approach where possible
- You don't understand all the abstractions and assumptions, so
"pattern match" to minimize trouble
- Realize that the assumptions behind code you find don't necessarily match
your own.

* Understand the "flow of control" inside the program
- architecture
- division of responsibilites
- purpose of data structures/objects

Scheduling
----------
How does the kernel maintain control?
- CPU is given entirely to userspace processes to run most of the time

Kernel needs to run when it needs to run
- and no process should be able to stop it

*The interrupt table*
- pointers to code that the CPU runs when it receives different hardware
or software interrupts
- since the kernel runs first, it gets to set the interrupt table
- only supervisor code can change the table, not user code

So when the ethernet card receives data...
...the ethernet card sends an interrupt to the kernel
...the CPU calls the kernel code for handling ethernet data

When the clock generates a timer interrupt...
...the CPU calls the kernel code for handling timer interrupts

In order to process an interrupt, an CPU core has to be taken over
- that core was probably running a userspace process

Scheduling is all about what to do after having kicked a userspace process
off of a core

Normally on a core
* userspace process is running
* interrupt happens
* core swiches to supervisor mode, runs kernel code
* last part of kernel code is scheduler, chooses which userspace code to run
* goto top :-)

Kernel is entered via interrupts, exited via the scheduler

Entry and exit to the kernel has to do low-level tasks like changing CPU modes,
manage CPU registers
- must be written in assembly

Most OS kernels minimize this low-level code

What criteria should the scheduler use in deciding what to run?
- "fairness"
- prevent starvation
- absent other conditions, equal allocation of resources
- bias resources towards "foreground" tasks in interactive systems
- never biased enough
- always hacks, heuristics

Memory overcommitment
- it is possible to allocate way more memory than can be ever used
- goes into "memory debt"
</pre>

===Midterm Review===
1. Execve and no it does not create a new process

2. Dynamically linked programs map in dynamically linked libraries

3. Hard links maps filenames to inodes

4. Sleep until the consumer wakes it

5. Using type conversion, read returns number of bytes written. We must convert back to the number of unsigned long, minus one because maximum index of array is one less

6. uses of signals:
* kill process
* find out when child process is terminated
* start/stop a process
* detect when a program has made a ptr error

7. Pointers in C contain virtual memory addresses because the memory of a process is a virtual address space. Each process thinks it has its own stretch of memory.

8. processes make a sys call to allocate memory. execve -> then mmap and/or sbreak to allocate memory

9. shell opens the file output.txt.

10. mmap contents of file can allocate the contents of it into ram.
* comparing 1 mb of both files at a time will be less expensive
* only makes sense to mmap if you will access files multiple times

11. producer and consumer now have no way of talking to each other because they do not share the same memory.

12. no. there is a race condition. while loop loops while sem < 0.
someone else could access and modify the variable between the while and the sem--;.
* you need special instructions to test and set the variable at the same time.
* also, int sem parameter isn't a ptr so it's a local variable.

===Additional Notes===

How do you do kernel hacking? Some tips for low level kernel code:
Be humble; You cannot know how everything works 
Verify your assumptions as you go 
Perform lots of experiments 
Incremental Development, compile and run often 
 
Check for errors ALWAYS or they will cascade and hurt you later 
*This will save you from later errors 
*Kernel has to live cleanly 
*This principle applies to any large scale coding
 
Find another part of the kernel that is close to what you want to do (Search online! This is a good link: http://elixir.free-electrons.com/linux/v4.4.83/source 
You do not understand all abstractions and assumptions, so "pattern match" to minimize trouble 
*Functionality that you want to implement might already exist in the kernel, look at how they do things and follow the pattern. 
*Refine understanding as you go, the other implementation might be for something else. 
 
Realize that the assumptions behind code you don't necessarily match to your own. Spend the time reading other people's code. 
Understand the "flow of control" inside the program 
*Architecture 
*Division of responsibilities 
*Purpose of data structures/objects 

For the tutorial 6 he gave us a link for a reason. If you click on it you can search getpid and then dig deeper! For example cut and paste the line inside getuid function. 
 
Scheduling 
Scheduler decides when what process should run at a high level 
Questions: How does the scheduler work/kernel control when a program runs/how can you make sure one program does not take over CPU from anyone else? 
CPU is given entirely to userspace processes to run most of the time 
Kernel needs to run when it needs to run and no process should be able to stop it 
* The interrupt table: pointers to code that the CPU runs when it receives different hardware or software interrupts. Since the kernel runs first it gets to set the interrupt table. Only supervisor code can change the table not user code.
 
When the Ethernet card receives data: 
1) The Ethernet card sends an interrupt to the kernel 
2) The CPU calls the kernel code for handling Ethernet data 
 
When the clock generates a timer interrupt: 
1) The CPU calls the kernel code for handling timer interrupts 
 
2) In order to process an interrupt, a CPU core has to be taken over 
3) That core was probably running a userspace process 
4) Scheduling is all about what to do after having kicked a userspace process off of a core 
 
Normally on a core: 
* Userspace is happening 
* Interrupt happens 
* core switches to supervisor mode, runs kernel code 
* last part of kernel code is scheduler, chooses which userspace code to run. Repeat the process 
Kernel is entered via interrupts, exited via the scheduler 
Scheduler needs to be efficient because interrupts are frequent and we do not want to waste resources that could be put towards actual computing 
Entry and exit to kernel has to do low-level tasks like changing CPU modes, manage CPU registers. This is Assembly code.
 
Go to http://elixir.free-electrons.com/linux/v4.4.83/source 
Go into arch: code specific to CPU architecture. Two most commons are arm and x86 
Go up one level 
Drivers directory have drivers for all kinds of things 
Linux Kernel designed for x86 but it now abstracted to support different architectures.
Go into architecture -> x86 -> entry 
Open entry.s: Large assembly language file. This is the code that runs before system call specific code. 
shced.h -> Contains definition of task_struct. Data structure that keeps track of a process. 
 
What criteria should the scheduler use in deciding what to run? 
*Fairness: Starvation is when a program does not get the CPU 
**Everyone should get a turn
**CPU is passed around
** Nobody should be left that they never get the CPU
**Prevent Starvation 
**Absent other conditions, equal allocation of resources
*Bias resources towards "foreground" tasks in interactive systems
**never Biased enough
**always hacks, heuristics
 
Memory overcommitment 
Memory does not get allocated when you mmap. It is loaded "lazily" int he Linux kernel. It is possible to allocate way more memory than can ever be used. 
Goes into "memory debt"

Operating Systems 2017F Lecture 17

2017-11-14T19:24:01Z

Rquaium: /* Additional Notes */

== Midterm Review ==
1. Execve and no it does not create a new process

6. uses of signals:
* kill process
* find out when child process is terminated
* start/stop a process
* detect when a program has made a ptr error

7. Pointers in C contain virtual memory addresses because the memory of a process is a virtual address space. Each process thinks it has its own stretch of memory.

8. processes make a sys call to allocate memory. execve -> then mmap and/or sbreak to allocate memory

9. shell opens the file output.txt.

10. mmap contents of file can allocate the contents of it into ram.
* comparing 1 mb of both files at a time will be less expensive
* only makes sense to mmap if you will access files multiple times

11. producer and consumer now have no way of talking to each other because they do not share the same memory.

12. no. there is a race condition. while loop loops while sem < 0.
someone else could access and modify the variable between the while and the sem--;.
* you need special instructions to test and set the variable at the same time.
* also, int sem parameter isn't a ptr so it's a local variable.

== Additional Notes ==

How do you do kernel hacking? Some tips for low level kernel code:
Be humble; You cannot know how everything works 
Verify your assumptions as you go 
Perform lots of experiments 
Incremental Development, compile and run often 
 
Check for errors ALWAYS or they will cascade and hurt you later 
This will save you from later errors 
Kernel has to live cleanly 
 
Find another part of the kernel that is close to what you want to do (Search online! This is a good link: http://elixir.free-electrons.com/linux/v4.4.83/source 
You do not understand all abstractions and assumptions, so "pattern match" to minimize trouble 
 
Realize that the assumptions behind code you don't necessarily match to your own. Spend the time reading other people's code. 
Understand the "flow of control" inside the program 
*Architecture 
*Division of responsibilities 
*Purpose of data structures/objects 

For the tutorial 6 he gave us a link for a reason. If you click on it you can search getpid and then dig deeper! For example cut and paste the line inside getuid function. 
 
Scheduling 
Scheduler decides when what process should run at a high level 
Questions: How does the scheduler work/kernel control when a program runs/how can you make sure one program does not take over CPU from anyone else? 
CPU is given entirely to userspace processes to run most of the time 
Kernel needs to run when it needs to run and no process should be able to stop it 
* The interrupt table: pointers to code that the CPU runs when it receives different hardware or software interrupts. Since the kernel runs first it gets to set the interrupt table. Only supervisor code can change the table not user code.
 
When the Ethernet card receives data: 
1) The Ethernet card sends an interrupt to the kernel 
2) The CPU calls the kernel code for handling Ethernet data 
 
When the clock generates a timer interrupt: 
1) The CPU calls the kernel code for handling timer interrupts 
 
2) In order to process an interrupt, a CPU core has to be taken over 
3) That core was probably running a userspace process 
4) Scheduling is all about what to do after having kicked a userspace process off of a core 
 
Normally on a core: 
* Userspace is happening 
* Interrupt happens 
* core switches to supervisor mode, runs kernel code 
* last part of kernel code is scheduler, chooses which userspace code to run. Repeat the process 
Kernel is entered via interrupts, exited via the scheduler 
Scheduler needs to be efficient because interrupts are frequent and we do not want to waste resources that could be put towards actual computing 
Entry and exit to kernel has to do low-level tasks like changing CPU modes, manage CPU registers. This is Assembly code.
 
Go to http://elixir.free-electrons.com/linux/v4.4.83/source 
Go into arch: code specific to CPU architecture. Two most commons are arm and x86 
Go up one level 
Drivers directory have drivers for all kinds of things 
Linux Kernel designed for x86 but it now abstracted to support different architectures.
Go into architecture -> x86 -> entry 
Open entry.s: Large assembly language file. This is the code that runs before system call specific code. 
shced.h -> Contains definition of task_struct. Data structure that keeps track of a process. 
 
What criteria should the scheduler use in deciding what to run? 
*Fairness: Starvation is when a program does not get the CPU 
**Prevent Starvation 
**Absent other conditions, equal allocation of resources
*Bias resources towards "foreground" tasks in interactive systems
**never Biased enough
**always hacks, heuristics
 
Memory overcommitment 
Memory does not get allocated when you mmap. It is loaded "lazily" int he Linux kernel. It is possible to allocate way more memory than can ever be used. 
Goes into "memory debt"

Operating Systems 2017F Lecture 17

2017-11-14T19:20:06Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 17

2017-11-14T19:19:07Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 17

2017-11-14T19:17:14Z

Rquaium: /* Additional Notes */

Operating Systems 2017F Lecture 17

2017-11-14T19:16:51Z

Rquaium: /* Additional Notes */

== Midterm Review ==
1. Execve and no it does not create a new process

6. uses of signals:
* kill process
* find out when child process is terminated
* start/stop a process
* detect when a program has made a ptr error

7. Pointers in C contain virtual memory addresses because the memory of a process is a virtual address space. Each process thinks it has its own stretch of memory.

8. processes make a sys call to allocate memory. execve -> then mmap and/or sbreak to allocate memory

9. shell opens the file output.txt.

10. mmap contents of file can allocate the contents of it into ram.
* comparing 1 mb of both files at a time will be less expensive
* only makes sense to mmap if you will access files multiple times

11. producer and consumer now have no way of talking to each other because they do not share the same memory.

12. no. there is a race condition. while loop loops while sem < 0.
someone else could access and modify the variable between the while and the sem--;.
* you need special instructions to test and set the variable at the same time.
* also, int sem parameter isn't a ptr so it's a local variable.

== Additional Notes ==

How do you do kernel hacking? Some tips for low level kernel code:
Be humble; You cannot know how everything works 
Verify your assumptions as you go 
Perform lots of experiments 
Incremental Development, compile and run often 
 
Check for errors ALWAYS or they will cascade and hurt you later 
This will save you from later errors 
Kernel has to live cleanly 
 
Find another part of the kernel that is close to what you want to do (Search online! This is a good link: http://elixir.free-electrons.com/linux/v4.4.83/source 
You do not understand all abstractions and assumptions, so "pattern match" to minimize trouble 
 
Realize that the assumptions behind code you don't necessarily match to your own. Spend the time reading other people's code. 
Understand the "flow of control" inside the program 
*Architecture 
*Division of responsibilities 
*Purpose of data structures/objects 

For the tutorial 6 he gave us a link for a reason. If you click on it you can search getpid and then dig deeper! For example cut and paste the line inside getuid function. 
 
Scheduling 
Scheduler decides when what process should run at a high level 
Questions: How does the scheduler work/kernel control when a program runs/how can you make sure one program does not take over CPU from anyone else? 
CPU is given entirely to userspace processes to run most of the time 
Kernel needs to run when it needs to run and no process should be able to stop it 
* The interrupt table: pointers to code that the CPU runs when it receives different hardware or software interrupts. Since the kernel runs first it gets to set the interrupt table. Only supervisor code can change the table not user code.
 
When the Ethernet card receives data: 
1) The Ethernet card sends an interrupt to the kernel 
2) The CPU calls the kernel code for handling Ethernet data 
 
When the clock generates a timer interrupt: 
1) The CPU calls the kernel code for handling timer interrupts 
 
2) In order to process an interrupt, a CPU core has to be taken over 
3) That core was probably running a userspace process 
4) Scheduling is all about what to do after having kicked a userspace process off of a core 
 
Normally on a core: 
* Userspace is happening 
* Interrupt happens 
* core switches to supervisor mode, runs kernel code 
* last part of kernel code is scheduler, chooses which userspace code to run. Repeat the process 
Kernel is entered via interrupts, exited via the scheduler 
Scheduler needs to be efficient because interrupts are frequent and we do not want to waste resources that could be put towards actual computing 
Entry and exit to kernel has to do low-level tasks like changing CPU modes, manage CPU registers. This is Assembly code.
 
Go to http://elixir.free-electrons.com/linux/v4.4.83/source 
Go into arch: code specific to CPU architecture. Two most commons are arm and x86 
Go up one level 
Drivers directory have drivers for all kinds of things 
Linux Kernel designed for x86 but it now abstracted to support different architectures.
Go into architecture -> x86 -> entry 
Open entry.s: Large assembly language file. This is the code that runs before system call specific code. 
shced.h -> Contains definition of task_struct. Data structure that keeps track of a process. 
 
What criteria should the scheduler use in deciding what to run? 
*Fairness: Starvation is when a program does not get the CPU 
*Prevent Starvation 
*Absent other conditions, equal allocation of resources

Operating Systems 2017F Lecture 17

2017-11-14T19:15:03Z

Rquaium: /* Additional Notes */