Soma-notes - User contributions [en]

Operating Systems 2017F Lecture 23

2017-12-12T04:20:02Z

HebaJallad: /* In Class */

==Video==

[http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec23-07Dec2017.mp4 Lecture 23 Video]

==Notes==

===In Class===
<pre>
Lecture 23
----------

How can you tell when a process has been compromised?
- from outside the process

Classic: signatures
- is it running "bad code"
- is it doing "bad things"
- bad system calls

For a process to do damage, it has to make "bad" system calls

How can I tell if a process is making bad system calls?

I want to be lazy
- complex rules are a pain
- and, they don't work well either

Make the computer solve the problem for me of determining what is good and bad
- use machine learning

But I can't teach good versus bad because I don't know bad very well

But...I do know how systems "normally" behave

How about teaching the system to differentiate normal from abnormal?
- normal is "good"
- abnormal may be bad

abnormal but not bad => false positive

How do we detect abnormal system calls?

Learn normal patterns of system calls over time
once you've learned enough, watch for abnormal system calls

Since I'm lazy, I want it to learn as it runs
- and automatically decide when it has learned enough

Could I do the learning in a process (or set of processes)?
- you could, but all data would have to come from the kernel

Want something fast and simple
- implement in the kernel

How simple could it be?

First assumption: ignore arguments

Second assumption: look at ordering of systems calls on a per-thread, per-process basis

Third assumption: characterize processes based on the executable they are running

- model per executable, each trained on multiple processes

How to model the trace of system calls coming from a process?

* frequency analysis?
- on a system call basis
- high variance

* what system calls are made (and not made)?

* short sequences of system calls?
6-10
</pre>
Addiional notes :

Assignment 4 Review:

11) Dd write things in blocks so each write is a system call
If it is read then both bs and ibs are correct
You can also dd and then run strace on it and see how many bytes you write
12) kernel doesn’t make a system call , processes do. System call transition from kernel to userspace. You can do function system
*********************************************************************************
Someone messed up a process, how can you identify has been compromised from outside the process?
- > use classes, signatures, pattern matching, running bad code,bad strings etc.
o Process doing bad things, such as system calls, this may damage the system
o Ex: password programs starts modifying file other than the password
- Refine the question, how can I identify if a process is making bad system calls?
o Inspecting this while the program is running is inefficient and expensive
o Be lazy and don’t make rules as a scale
o Make the computer solve this problem and determine if it is bad or good by using machine learning
 However , I cant tell it the difference between good and bad but I don’t know bad very well
 I know how the computer behaves
- Refine the question again: How about teaching the system differentiate normal and abnormal?
o Normal is good
o Abnormal may be bad
- False positives are bad(especially in security)
- Refine the question, how do we detect abnormal system call?
o By learning normal patterns of system call over time then watch for abnormal system calls.
- System calls are complicates, some have arguments like execve and some don’t like fork. To learn all of their complexity
- Learn as it runs:
o To automatically decide when it has learned enough
- You can do the learning in a process but you want it to be fast and simple and implement in the kernel.
- You want something really simple and run it as you go, but how simple could it be?
o Do strace xclock
1) Ignore arguments
2) Analyze the ordering of system calls on per-thread, per-process basis
3) Characterize system calls behavior , according to the executable they are running
a. Example: xclock cant be based of ls
b. Model per executable (can do frequency analysis)
-Refine, how to model the trace of system, calls coming from process?
-> Frequency analysis
- ex : Is xclock behaving weird,
- run strace of different programs and watch the variation in a the pattern of sequences and it detects if a program has been compromised
-what system calls are made and not made?
- Short sequences
- 6 – 10

===Additional Notes===
Written solutions for midterm exam are on the course webpage 
Assignment 4 
Q2: SSH keygen generates the secret key and the public key file. The private key is stored in the private key file: .ssh/id_rsa 
Q4: Both lines because first you start from 1 and then increment from there. 
Q11: Only bs because it is write. Would be ibs and bs if it was read. 
Q12: Local kernel forwards the write system call but doesn't actually make the system call. Kernels don't make system calls. 
Lecture 23 Prof Notes 
-----------------
How can you tell when a process has been compromised. 
- from outside the process 
Use signatures 
- is it running "bad code" 
- is it doing "bad things" 
-- For example: password program start modifying files other than etc/password. You could specify rules to prevent this. 
-- bad system calls 
For a process to do damage, it has to make "bad" system calls 
I want to be lazy 
- complex rules are a pain 
- and they don't work well either 
Make the computer solve this problem 
- Use Machine Learning 
I can't teach good versus bad if I don't know what is bad 
But I know how systems behave normally 
How about teaching the system to differentiate normal from abnormal 
- normal is good 
- abnormal may be bad 
abnormal but not bad => false positive 
False positives can be a big issue because they may cause ppl to not trust the machine's detection capabilities 
 
How can we detect abnormal system calls? 
Learn normal patterns of system calls over time 
Once you've learned enough, watch for abnormal system calls 
Since I'm lazy, I want to learn it as it runs 
- and automatically decide when it has learned enough 
 
Could I do the learning in a process (or set of processes)? 
you could, but all data would have to come from the kernel 
Want something fast and simple 
- implement in the kernel 
How simple could it be? 
 
First assumption: ignore arguments 
Second assumption: look at ordering of system calls on a per-thread, per-process basis 
Third Assumption: characterize processes based on the executable they are running 
model per executable, each trained on multiple processes 
How to model the trace of system calls coming from a procesd 
* frequency analysis? 
- on a system call basis 
- high variance 
* what system calls are made (and not made)? 
* short sequence of system calls? 6-10 calls 

Lecture 23

How can you tell a process has been compromised (i.e. from outside the process)?
* The process is working on behalf of an attacker
 
Classic way to do this:
* Pattern matching -> signatures
::* is the process running bad code?
::* is the process doing bad things?
:::* i.e. /etc/passwd -> password prog. should only be able to access it
:::* if a process is going to do bad things, it's going to make "bad" system calls
 
So, how can we tell if a process is making bad system calls?
 
Don't want to sit and write complex rules to determine:
* Which programs should make which system calls, etc.
::* i.e. policy based systems and sandboxing of processes
 
Therefore, we want the computer to determine what call is good/bad.
* i.e. use machine learning
 
The issue is, we have to demonstrate, not just "good", but also "bad" 
We have lots of examples of "bad", but is not necessarily representative of "bad" 
Difficult to enumerate all possible occurences of "bad" 
 
However, we know how systems "normally" behave
 
How about teaching the system to differentiate normal from abnormal?
* Assume:
::* normal is "good"
::* abnormal may be bad
:::* there is no guarantee that abnormal is bad, however, if it's bad, but not abnormal... we're in trouble
:::* false positives are bad (i.e. abnormal but not good)
 
How do we detect abnormal system calls?
* a machine learning problem
* the system should learn as it runs and decide when it has learned "enough"
::* learn normal patterns of system calls over time
:::* once learned enough, watch for abnormal system calls
 
Could we do the learning within processes?
* possible, but all data would have to come from the kernel
 
Want something fast and simple, so it can be implemented in the kernel
* you're right at ground-level, where decisions are being made
::* i.e. if bad system call being made -> can stop it immediatelly

* don't want to be training a neural network to do this -> too complicated, too much overhead
 
''Thu 7 Dec 2017 13:53:01 EST -> Video of observing system calls, ls vs. xclock''
 
=== First assumption: ===
ignore the arguments system calls are making -> look at the calls themselves
* but, different processes invoke different calls -> how to compare them?
* even multi-threaded processes will mirror the structure of the code in the calls it makes

=== Second assumption: ===
look at the ordering of system calls on a per-thread, per-process basis
* doesn't make sense to think of 'ls' system calls in the context of 'xclock' system calls
 
Therefore, any profiling will be based on the code being executed

=== Third assumption: ===
characterize processes based on the executable they are running
* model per executable, with each one trained on multiple processes
 
How do we model the trace of system calls coming from a process?
* How often do different system calls hapen? -> frequency analysis
* high variance -> the calls change frequently
::* i.e. ls of a large dir vs. small dir

::* What system calls does a process makes or doesn't make?

::* Rather than examining if a process does or doesn't make a particular system call, instead look at short sequences of system calls being made.
:::* What is the variation in the pattern of sequences of calls being made? A compromised program will be detectable.

:::* Table lookup of sequences made by a program and compare against new sequences
 
How short is a short sequence of system calls? -> 6 to 10

When a program is running, the short sequences define the control flow path of the program 
The short sequences together represent the control flow 
When a program is exploited, an abnormal control flow, an uncommon path, is being used 
 
Try the simple hack first, rather than designing/engineering a complex solution
* the simple hack will often present valuable insights

Operating Systems 2017F Lecture 19

2017-12-07T16:19:08Z

HebaJallad: /* In Class */

==Video==

[http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec19-21Nov2017.mp4]

==Notes==

===In Class===

<pre>
Lecture 19
----------

Where's main?

* lots of programs have "main" functions - a function that runs first and controls the execution of the program
* Do these have "main" functions? In what sense?
- Linux kernel modules
- FUSE applications?
- the Linux kernel?
- node web applications?

In many systems, "main" just sets up event handlers
- the event loop can be implicit or explicit
- or there may be no loop at all, just handlers and "interrupts" of some kind
- event loops poll (check) to see when there are new events

OS kernels are essentially the same thing
</pre>

Additional Notes:

Important notes:
How can you recover a filesystem?
How do you delete a file?
What is a filesystem?:
*persistent data structure
* stored in fixed-sized blocks (at least 512 bytes in size)
*maps hierarchical filenames to file contents
*has metadata about files (somwhow)
What is in a filesystem?
*data blocks
*metadata blocks
How do you organize metdata:
1) First you must identify characteristics of the file system
Superblock : summary y block which tells you about the other blocks you have and it depends on which file system you have. It’s usually the first block of a file system.
In the superblock? :
1) What kind of file system is this? By checking what is the magic number it has
2) How big is the file system?
3) How is it organized?
4) Where can I find the rest of the metadata?

*How can you identify which file system it is from looking at the super class
-> google “magic number of a file”
-> ex: jpg ctr^c ctr^c : switched the pictures into a binary file
-> look at the beginning of the file you will see JFIF: first several bytes in general that identifies the type of the file (magic number)
File extension :
 what is it ?
 is it important
 the kernel does not know and not care about it

For POSIX file systems:
-.> file metadata is stored in inodes
-.> most have pre-reserved inodes
-> the only way you can run out of inodes if you keep creating small files

Usenet : al the things you use to post messages thro social media, email, etc. Those were made using Usenet. Like email but Local Usenet server. But it died over time. Every message is stored in an individual file.

Important commands:
File * : to identify the kind of file system
1. As : Run dumpe2fs foo. What does the output of this command mean?
 Does this give you info about the file system?
 File bar : bar is the file name and cp comp3000-midterm-2017.pdf bar
 Evince bar : opens up the pdf file

=== Additional Notes ===

Where's main?
* lots of program shave "main" functions - a function that runs first and controls the execution of the program 
* Do these have "main" functions? 
** Linux kernel modules 
** FUSE applications? 
** the linux kernel? 
** node web applications? 

In many systems, "main" just sets up even handlers 
* the event loop can be implicit or explicit 
** or there may be no loop at all, just handlers and "interrupts" some kind 
* event loops poll (check) to see when there are new events 
* what are event loops for node app? 
** where are interrupts for node apps? 
***Incoming network requests, it's an event 

Code run differently in the kernel : 

1)functions runs on the bhealf of insmod, unles sit is Independence context 

2)codes that run on the bhelaf o the process 

3)after an interrupt: no process , it is an interrupt cotext 

4) file names : regular programs but the square brackets, execution context + address space. they share the kernel's address space, they are called kernel threads which are independently scheduling . You can not kill them but you can change their scheduling , maybe their priority but not 100%. 

does it create a proces? no , but it can create a kernel thread (is it a process? virtual adress space, . 
multi- threaded: maintains multiple address processes , ex: fire fox. 

ps -elF | less "number" : displays threads.

top : displays all the processes 

ls time : shows you the time .

sys: how much time in the kernel space
real: how much time
user : how much time in user space

process : can't manipulate its own memory map directly, it has an address space, but cant change it. Process: is limited but the kernel is not and the kernel can change it's own address and in charge of its self. 

Kernel tasks : are threads, when a process makes a system call , thi sis schedules in the process priority. 

When running 
"time ls" 
real = realtime it took to run
user = the user space time
sys = kernel time

What's flow of control in tutorial 7? 
What is the connection? 
To exit the program, we must unmount the filesystem, run "sudo umount mnt" 

OS kernels are essentially the same thing 

what is the flow of control ? what connection between things we are doing in the new terminal and the old one 

-> these programs are communicating to each other, it will be invoked when we use mnt (mount) the kernel knows it is a filesystem , process runs system calls, then kernel talks to out original terminal. How? you can use strace to know, it is waiting to be invoked, to receive and responds to messages. events will be passed off to another process. switching between one process to another. it has potential security benefits. 

Key to understand this tutorial : 

-> understand net flow control. 

-> how do processes communicate? 

-> how does it take a directory int and creates a filesystem from it?: 

***sub tree starting at mount is delegated to this process. 
***permissions are limited 

How to kill it? 

-> ctrl c , no , you can but the kernel will be unhappy. 
-> unmount the file system when you are done using it 
-> umount / you have to do it as root 

Node web applications 
* Theres not a dedicated main function but its the first thing that runs 
* Every line in the node application terminates 
* If you start a web server, the function call starting it will terminate 
* Whats running? The main has finished. Is the program doing anything actively if theres no external input? 
* In many systems main just sets up event handlers 
* The event loop can be implicit or explicit 
* Its possible to have no loop at all just handlers and interrupts of some kind 
* Event loops check to see when there are new events 
* Os kernels behave the same way as node applications 
* Waiting for events 
 

What happens when you use insmod? 
* In newgetpid: Init and exit are called when the insmod program makes system calls to load the module 
* Init is run on behalf on insmod 
* Kernel code fits into 3 categories 
** Code that runs on behalf of a process 
** Code that runs after an interrupt 
** Kernel threads, has lots of functionality 
 
Why is a kernel thread not a process? 
* The kernel maintains its own address space 
* Has its own virtual address map 
* The kernel always has just 1 address space regardless of how many threads there are 
* Processes cannot manipulate its own memory map directly, needs to ask kernel first 
* Processes are limited, the kernel is not, has control over itself 
 
* Kernel tasks = kernel threads 
* Independently scheduled 
* Once we call insmod everything happens in user space 
* Strace uses a system call called ptrace 
 
Tutorial 7: Fuse 
* Memoryll.py program 
* Always unmount a filesystem when youre done using it 

*No class Thursday! no office hours on Wednesday

Operating Systems 2017F Lecture 22

2017-12-07T16:17:29Z

HebaJallad: /* In Class */

==Video==

[http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec22-05Dec2017.mp4 Lecture 22 Video]

==Notes==

===In Class===

<pre>
Lecture 22
----------

What's left?

* scheduling
* device drivers
* virtual memory
- page replacement algorithms
- predict the future (optimal)
- least recently used
- one-handed, two-handed clocks
* power management
* security
- hardening processes so coding errors don't lead to vulnerabilities
(machine code injection, e.g. buffer overflow attacks)

* virtualization
- hardware-level (run multiple kernels) <-- vmware, openstack
- OS-level (run multiple userspaces) <-- containers, web hosting
- application level (run programs on simulated machines)
- JVM
- JavaScript runtime in browsers/node

* distributed operating systems
</pre>

ADDITIONAL NOTES :

Comp 3000
Premissions on this directory, readable writable and executable
Execute permission on a Regular file : you can execute
Execute permission on a directory : follow the links on the directory
Can’t make any changes to the directory if you can’t write
There are exceptions :
Less/etc/passwd: doesn’t actually store the password
if you want to change this file, you must have a way to allow limited editing to this.
 You can have a process running as root and send it signals and tell it to update the entry in the password file.
o Starting up a process which has more privileges which I can do , ex: EUID
Ls –la /sbin | grep rws : execve EUID will be set to whatever it is from the file . equal to the uid
Ls –la /sbin | grep r-s: s is a sticky bit, if you need extra premissions
You want your stcky bit to be a regular user
Euid = uid yes
Cd /tmp : directory in which everyone can write
This allows binaries run as users
Set uid and get guid :
Myid has euid now
Change the ownership
- > chown root : root myid
- >chown root : root mytouch
Ls –la : 3rd column identifies the ownership of each file on the file system
You can overwrite any file on the fille system using mytouch binary
Question : why can you remove file owned by root?
- > to change the context of the directory , the permissions of a file don’t matter but the permissions and privileges of the directory only matter
o Someone putting a directory in ur directory is hard to remove
- Ssh to a remote serve :
- 2 public keys involved: identity key, private key pair: one in the known host file (connecting to the machine).
- If you rm _known host and do ssh , a question will ask you to add the key to ur host file
- What happens if a person tries to personate your machine (same IP address)?
o It will identity it is a fake person from the host
First line is a Hashed versionof an IP address : cat .ssh/known_host
Ssh demon : running in the background and must have a public key to identify its self. process that runs in the background that doesn’t run in the background(connects 1 file system to another)
- > connects sockets and listens to connect. Doesn’t interact with user
Thursday: written version of the solutions for the midterm and we will talk about assignment 4
3000 class content
We didn’t discuss scheduling much :
Virtual memory: similar to scheduling since, If you don’t have enough memory , you delete the page that you may want to need at last . Choosing which pages you replace : one-handed and two-handed clocks
Power management
Security
Virtualization : not one thing , vm ware, system which run multiple of kernels.
SSH question student asked, how can they know that they have the private key belongs to the pubkey it belongs to: sends a public key or a hash of th Pubkey , then an exchange : yes I have a secret key which can be inverted by the pubkey. Private key must be corresponding. encrypts with thr private key and sends it back

Operating Systems 2017F Lecture 21

2017-12-07T16:15:14Z

HebaJallad: /* In Class */

==Video==

[http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec21-30Nov2017.mp4 Lecture 21 video]

==Notes==

===In Class===

[http://homeostasis.scs.carleton.ca/~soma/os-2017f/code/lec21/assign3-sol.txt Assignment 3 solutions]

<pre>
Lecture 21
----------

ssh part of a long tradition in remote access to UNIX-like systems
(and other systems with command lines)

* hard link via a serial line to a terminal
* modem-mediated serial connection to a terminal
* telnet - connect in a serial-like fashion over the internet
- have a program listening for remote access on a fixed port
- when there's a connection, connect remote program to login
- login takes username, password
- if correct, starts command shell for the user
* Problems with telnet
- annoying to type in username and password every time
- username and password (and everything else) goes in the clear, so
subject to eavesdropping, modification
* rsh - remote shell
- idea: allow login without passwords and running of one-off commands
- how? check IP address/hostname of source
- but...IP addresses are not a secure authenticator
- also...trusting configuration of remote system
- and...all communication again in the clear

* ssh was designed as a drop-in replacement for rsh
- but with better security
- and also with better features, specifically X11 forwarding

X11 is a system for displaying graphics and handing input
- windows, keyboard, mice
- developed in late 1980's, early 1990's
- designed to allow remote connections
- primarily a network protocol
- X11 connections went in the clear
- and were a pain to set up

public and private keys
- asymmetric cryptography
- used for key establishment and signatures (authentication)

symmetric crypto
- encrypt with a key, decrypt with the same key
- modern form: block ciphers (e.g., AES)
- fast and very secure

asymmetric crypto is based on "one way" functions
- easy one way
- hard to invert (unless you know extra info)

To send someone a secret message
- get their public key
- encrypt message with their public key
- really, encrypt random symmetric key (session key) with public key and
encrypt message with symmetric key

To receive a secret message
- decrypt session key with your private key
- use session key to decrypt message

NEVER IMPLEMENT CRYPTO ON YOUR OWN
PLAN TO FIX CRYPTO CODE/PROTOCOLS OFTEN
</pre>

ADDITIONAL NOTES

Remote access : are processes accessed through command lines aka terminal.
- > hard link via serial line : first thing you should think about
- How to access the computer else where from somewhere else(remote access ) this is called a modem mediated serial connection to terminal .
Network: computer talk to each other
What is wrong with telnet :
1) Insecure, everything you type in including your password and user name is watched (and modifiable)
2) Annoying since it requires you to constantly log in

RSH : is a remote shell . the ability to run a command by giving it arguments simply
How does it allow you to do so without password? Since it has your IP address and hostname of source.
*however this is not secure enough since the IP address can be faked.
Ssh : developed to replace rsh with better improved security and features such as x11 forwarding.
When killing ssh , the x clock & program ended since it used that to mediate through.
X11: windows focused, remote desktops are different. system which displays graphics, it is weird in a way that it is used be useful but not anymore. It is a network protocol and which the program will display something on screen will talk to the computer (ask?) .
-Example : The server is the display and the terminal is client.
-disadvantages : difficult to set up .
-How do I talk to the current display and connect to it?
-Using an environment variable. (display environment variable)
- Env | grep DISPLAY . if it is :=0 like our case then we have none.
Goal : use a local computer like running remote computers , use graphic access remotely
Clone of ssh syntax
*example : su –soma, rsh it then do ls
Xterm command after ssh? :
Xclock & : where is this program running? It is not running locally. It is running a program remotely but it is displaying here.
-x option to ssh : setting the display environment variable to the remote system
--ford their own their own things magically
-x , can you run the x clock & without –x : NO unless you do things on command lines
Ssh : can manage redirection from a local port to another machine (ssh tunnel).
Known_hosts:
Public VS private keys: asymmetric cryptography: when you encrypt a file with a password, you must use the same password to decrypt it.
-used for key establishments and signatures (key authentication)
- easy to go one way (hard to go the other way and invert )

Modern for : fixed size units , block ciphers
Symmetric cryptography : is much more efficient and harder to break than asymmetric.
- Fast and very secure
- Disadvantages: required a way over network to obtain a key to connect
To send someone privately :
-obtain their public key (in directories ex)
-then use it to encrypt symmetric
-send both , attach the key with the message
*to receive a message :
-use the session key to decrypt it : assymetric algorithms are slow and less secure , why RSA isn’t used alone
- NEVER IMPLEMENT CRYPTO ON YOUR OWN because you cant implement it in the same algorithm, too many numbers of attacks and unreliable but if you do use it in your system , use it with libraries but expect them to be changed.
Cryptography is the base of ssh
The lock icon : secure connection , uses same thing as ssh. Info about the public key.]
-Symantec : dangerous because anyone can sign.
Fire fox : has public keys of certificates so you are able to send secure message and anyone can do the inverse operation using the public keys.
Ssh : stores on first use .
Rm .ssh/known_hosts :
Less known_hosts
Cd /etc/ssh : host keys and configurations.
What happens if you change those keys? : it will give you a warning
Transfer the public key to the remote system and make sure I don’t share my private one
We are stuck using passwords because of usability and dangerous

Operating Systems 2017F Lecture 20

2017-11-29T03:27:31Z

HebaJallad:

==Video==

[http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec20-28Nov2017.mp4 Class Video]

==Notes==

===In Class===

<pre>
Lecture 20
----------

When we run "ls" on an sshfs-mounted filesystem
* ls makes system calls (open, getdents) to the local Linux kernel
* the local kernel sees filesystem is FUSE, calls FUSE routines for open, getdents (via vfs abstraction)
* FUSE calls the sshfs process that mounted the filesystem
* sshfs process sends request to remote system
- via socket system calls
* remote sshd process receives request (via system calls)
* remote sshd process accesses local filesystem
- makes open, getdents system calls
- remote kernel checks vfs, calls ext4 routines to access data
* remote sshd process responds to request (via system calls)
* local sshfs process receives response (via system calls)
* local sshfs process responds to FUSE request
* FUSE passes data back to vfs layer, then back to requesting process

Normal file access permission check
- compare uid, gid of file with uid, gid of process

But really...
- compares it with fsuid, fsgid of process
- which is normally same as euid, egid of process
</pre>

===Additional Notes===

Assignment 4 will be autograded and in fill in the blank form. This will not be like the final exam. 
You should think about who is doing what? Otherwise FUSE won't make much sense 
 
Inode numbers completely changed when you use ssh. It starts with 1 from "." (root) and increments from there. 
Inodes come from filesystem you are acessing 
He used df to find filesystem info 
The used Commands: mount | grep ubuntu and mount | grep vda1 to find type of filesystem 
In the SSH there is the filesystem type "fuse.sshfs" 
When you run strace on ls on the two terminals (ssh connection and the local one) you see similar output 
When we run "ls" on an ssh-mounted filesystem 
* ls makes system calls (open, getdents) to the local Linux kernel 
* the local kernel sees the filesystem is FUSE, calls DUSE routines for open, getdents (via vfs abstraction) 
* FUSE calls the sshfs process that mounted the filesystem 
* sshfs process sends request to remote system 
- via socket system calls 
* remote sshd process receives request (via system calls) 
* remote sshd process receives requests (via system calls) 
- makes open, getdents sytem calls 
- rmeote kernel checks vfs, calls ext4 routines to access data 
* remote sshd process responds to requests (via system calls) 
* local sshfs process receives response (via system calls) 
* local sshfs process responds to FUSE request (via system calls) 
* FUSE passes data back to vfs layer, then back to requesting process 
 If you look as lsmod | less you see a ton of modules. Some of the modules are not necessary to mount the root filesystem. 
ls /lib/modules there are directories that store the kernel modules 
You have to have a filesystem that you mount initially that has a bunch of modules in it. Thus, there is an initial root file system that is necessary to load everything else 
This is the initial RAM disk. This filesystem loads the modules needed for the real filesystem. 
To remove a file I need to remove the hardlink from the directory where the hardlink exists 
The password files maps usernames to user id's (Linux does not care about your username) 
Normal file access permission check br>
- compare uid, gid of file with uid, gid of process 
But really: 
- compares it with fsuid, fsgid of process 
- which is normally same as euid, egid of process 
 
Sshfs 
* Inode values are different in remote vm vs local vm 
* Values increment from 1 in local vm 
* Who's supplying the inode values? 
** Ext4 filesystem in local vm 
** Fuse.sshfs filesystem in remote vm 
* Filesystem determines the interpretation of inode values 
* Inodes have no meaning outside its filesystem 
* Local to its filesystem => hardlinks are different too 
* "strace ls" in both output the same thing 
* Vfs not built into original unix filesystems 
* Try "strace"ing the sshfs call, see how its interacts with fuse 
* Don't need to know exact system calls, but you should know when it has to make system calls and why 
** Needs to access files over the network 
* Understand why you can't do this with regular library calls 
* Removing the root filesystem will trigger a kernel panic 
** Kernel will prevent you from unmounting filesystems that contain other filesystems 
* Kernel modules located in "/lib/modules/" 
* Initial filesystem is loaded into kernel 
** Fake filesystem, throw away after loaded, not persistent 
** Ram disk = filesystem stored in ram 
** Gets the system up to the point where the kernel can load the real filesystem 
** Bootloader has to load both kernel and the initial ram disk into ram 
** Ram disks located in "/boot = initrd.img<...>" 
** Kernel version specific 
** Generated as part of installation of kernel 
* Cpio: copies files to and from archives 
 
* What determines whether you can access a file? 
** The filesystem 
* How to map from user id to username? 
** Recall that 
- Kernel knows nothing about usernames 
- This is a userspace process 
- Looks in local password file 
* What determines whether an operation is allowed? 
** Checks to see who owns the file 
** Compare uid, gid of file with uid, gid of process 
* In remote system, sftp does file access checks 
********************************************************************
ADDITIONAL NOTES:

Fuse: makes you think who is doing what?
 How it is connected in the kernel .
 Fuse.sshfs : is what chose the inode numbers?
o Can they be identical to the other inode numbers?
 Yes but why?
 What the process of getting the directories?
The inodes come from the filesystem which you are accessing

Who chose the inode number: the filesystem determine the intereprtation of an inode number, Inode is local to a filesystem.
How is the kernel obtaining information? :
Fuse : has three kinds
1) Fuseblk
2) Fuse
3) Fusectl
Summary of sshfs:
Sshfs process that has to make system call to talk to the network to the remote system.
Similar to a web request , but the difference separate from that we have a different system calls which is the ls process. To do this, the kernel must call a process.
Does sshfs create 2 processes? Yes , it actually creates multiple of process
The remote kernel prespective : file accessing files and sending data back to another system. Remote data is not doing anything special.
*To verify : ps aux | grep ssh
To learn system call sshfs doing ,running local strace as much as you can.
You can mount a file system which would not allow you to mount any further
When you update the kernel : the initial ram disk will not be update since it is redundant, it will generate it.
Can we mount a filesystem in a file?
*
What is suspicious? A lot of repetitive inodes
3rd column : amount of hard links.
Avoid multiple of binaries if you want a small system, you just need the basics.
After mounting it it loads the linux system.
Permissions: the owner can change these permissions, but you can’t take away your own privileges.
1) Removing a file : unlinking the file, removing a hardlink which modifies the directory in which a hardlink exists. Why do I have the permission to remove it?
2) Create a file : ouch afile, ls –la afile,
Premissions:
1) Other
2) User
3) Group
4) Files system has EUID( very important because processes can run as a user but have permissions of another user ) AND GID: permissions what you have and fs is what ur system sees
Minix system which manages your CPU.
Busy Box : One binary to pretend it is different programs and we also have a program which pretends to be many. Give it hard link to different names.
Soma vs anilclass: anil class is special account. The current directory is owned by soma so how did anilclass was able to create a file? Filesystem understands those primission checks? (prof is not sure about this fact and will check it)
Map a user ID to your user name? it’s a user space problem, kernel doesn’t care about the username. By looking at the password file : grep 1000 /etc/passwd.
Where are the permissions done in sshfs? (in the steps
*sftp search process which will determine the permissions. remote file system and the remote kernel (ext4) will check via sftp. All the file access (grabbing the files)
Verify : ps aux | grep sftp
Who started this sftp: the sshd did.

Commands:
Mount | grep “name”
Strace ls : directories look the same although they are mounted since ls can not tell , ls is not doing anything differe just opening ur directory and getting data there
Man sshd : run it and give it aport like 222. Easiest way is using 2 machines on open stack and terminal. It is a good exercise
Initial Ram disk : to get the real root system to be mounted , similar to memory ll.
Sudo mount initrd.img-4.10.0-40-generic /mnt : to mount a file in a file system but it didn’t not work and there is an alternative way of doing this
Anther way : man CPIO : copy files to an form archives . ex : TAR
CPI - - help : to figure it out
Ls –a bin | less
Cd \ boot : RAM disk
Du –s –h : in this case (121mb) to see the size of the busy box but the modules takes up half of that space.
Ls –lai | head
Chmod a-w : you are unable to write to this directory anymore
Chmod o+w : granted permission to other….
Ls –la /bin/fusermount
Who gives us premissions: just does a normal open and read , the remote kernel enforces these premissions.

Operating Systems 2017F Lecture 19

2017-11-21T19:14:51Z

HebaJallad:

== [Notes] ==

Sample
sample 

=== Additional Notes ===

Where's main?
* lots of program shave "main" functions - a function that runs first and controls the execution of the program 
* Do these have "main" functions? 
** Linux kernel modules 
** FUSE applications? 
** the linux kernel? 
** node web applications? 

In many systems, "main" just sets up even handlers 
* the event loop can be implicit or explicit 
** or there may be no loop at all, just handlers and "interrupts" some kind 
* event loops poll (check) to see when there are new events 
* what are event loops for node app? 
** where are interrupts for node apps? 
***Incoming network requests, it's an event 

Code run differently in the kernel : 

1)functions runs on the bhealf of insmod, unles sit is Independence context 

2)codes that run on the bhelaf o the process 

3)after an interrupt: no process , it is an interrupt cotext 

4) file names : regular programs but the square brackets, execution context + address space. they share the kernel's address space, they are called kernel threads which are independently scheduling . You can not kill them but you can change their scheduling , maybe their priority but not 100%. 

does it create a proces? no , but it can create a kernel thread (is it a process? virtual adress space, . 
multi- threaded: maintains multiple address processes , ex: fire fox. 

ps -elF | less "number" : displays threads.

top : displays all the processes 

ls time : shows you the time .

sys: how much time in the kernel space
real: how much time
user : how much time in user space

process : can't manipulate its own memory map directly, it has an address space, but cant change it. Process: is limited but the kernel is not and the kernel can change it's own address and in charge of its self. 

Kernel tasks : are threads, when a process makes a system call , thi sis schedules in the process priority. 

When running 
"time ls" 
real = realtime it took to run
user = the user space time
sys = kernel time

What's flow of control in tutorial 7? 
What is the connection? 
To exit the program, we must unmount the filesystem, run "sudo umount mnt" 

OS kernels are essentially the same thing 

what is the flow of control ? what connection between things we are doing in the new terminal and the old one 

-> these programs are communicating to each other, it will be invoked when we use mnt (mount) the kernel knows it is a filesystem , process runs system calls, then kernel talks to out original terminal. How? you can use strace to know, it is waiting to be invoked, to receive and responds to messages. events will be passed off to another process. switching between one process to another. it has potential security benefits. 

Key to understand this tutorial : 

-> understand net flow control. 

-> how do processes communicate? 

-> how does it take a directory int and creates a filesystem from it?: 

***sub tree starting at mount is delegated to this process. 
***permissions are limited 

How to kill it? 

-> ctrl c , no , you can but the kernel will be unhappy. 
-> unmount the file system when you are done using it 
-> umount / you have to do it as root 

*No class Thursday! no office hours on Wednesday

Operating Systems 2017F Lecture 19

2017-11-21T18:56:50Z

HebaJallad:

Operating Systems 2017F Lecture 19

2017-11-21T18:50:28Z

HebaJallad: /* [Notes] */

== [Notes] ==

Sample 
sample 

=== Additional Notes ===

Where's main?
* lots of program shave "main" functions - a function that runs first and controls the execution of the program
* Do these have "main" functions?
** Linux kernel modules
** FUSE applications?
** the linux kernel?
** node web applications?

In many systems, "main" just sets up even handlers 
* the event loop can be implicit or explicit 
** or there may be no loop at all, just handlers and "interrupts" some kind 
* event loops poll (check) to see when there are new events 
* what are event loops for node app? 
** where are interrupts for node apps? 
***Incoming network requests, it's an event 

Code run differently in the kernel : 

1)functions runs on the bhealf of insmod, unles sit is Independence context 

2)codes that run on the bhelaf o the process 

3)after an interrupt: no process , it is an interrupt cotext 

4) file names : regular programs but the square brackets, execution context + address space. they share the kernel's address space, they are called kernel threads which are independently scheduling . You can not kill them but you can change their scheduling , maybe their priority but not 100%. 

does it create a proces? no , but it can create a kernel thread (is it a process? virtual adress space, . 
multi- threaded: maintains multiple address processes , ex: fire fox. 

ps -elF | less "number" : displayes threads.

OS kernels are essentially the same thing

Operating Systems 2017F Lecture 19

2017-11-21T18:08:53Z

HebaJallad:

== [Notes] ==

Sample 
sample

Operating Systems 2017F Lecture 19

2017-11-21T18:08:44Z

HebaJallad: Created page with "[Notes] Sample sample "

[Notes]
Sample 
sample

Operating Systems 2017F Lecture 18

2017-11-16T20:41:15Z

HebaJallad: /* In Class */

==Video==

The video from the lecture given on Nov. 16, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec18-16Nov2017.mp4 is now available].

==Notes==

===In Class===
<pre>
Lecture 18: Filesystems and such
--------------------------------
* How can you recover a filesystem?
* How do you delete a file?

A filesystem is
* persistent data structure
* stored in fixed-sized blocks (at least 512 bytes in size)
* maps hierarchical filenames to file contents
* has metadata about files (somehow)

What's in a filesystem?
* data blocks
* metadata blocks

How do you organize metadata?

First job: identify basic characteristics of the filesystem

You need a "summary" block that tells you about everything else
=> this is the "superblock"

Normally the superblock is the first block of the filesystem

In the superblock
- what kind of filesystem is this?
- what filesystem magic number is there
- how big is the filesystem?
- how is it organized?
- where can I find the rest of the metadata?

for POSIX filesystems
- file metadata is stored in...inodes
- most have pre-reserved inodes

So we have
- superblock
- inode blocks
- data blocks
- data blocks for directories
- data blocks for files

How do you recover from damage?
- filesystems never "reboot", must remain correct over
the course of years
- but errors will happen
- bitrot
- "accidental" corruption
- computer failure/memory corruption/hard reboot

To make filesystems fast, data & metadata is cached in RAM
- bad things happen if this data hasn't been written to disk and you reboot
- even worse things happen if your RAM is bad and corrupts the data

Also bad...what if we lose the superblock?
- you could lose EVERYTHING
- so we have backup superblocks

Old scandisk/fsck was slow because they had to scan all filesystem metadata
- not to recover data, but to fix metadata

Nowadays fsck is very fast and we rarely lose data due to losing power
- we must be writing data to disk all the time
- but isn't writing all the time slow?

On magnetic hard disks (not SSDs)
- sequential operations are fast
- random access is slow
- we have to move the read/write head

So, on modern systems we update metadata (and sometimes data) by writing
sequentially to disk...and then later writing randomly
- sequential writes go to the "journal"

On fsck on a journaled filesystem
- just check the journal for pending operations (replay the journal)

There exist filesystems that are pure journal
- log-based filesystem

logs and journal inherently create multiple copies of data and metadata that are hard to track. This makes deletion nearly impossible (at least to guarantee)

Only way to guarantee...encrypt everything
- if every file has its own key, you can delete the key and thus "delete" the data

Solid State Disks (SSD) use log-structured storage at a level below blocks.
- writes are coarse-grained (you have to write a lot at once)
- you don't want to write to the same cells too often, they'll die
- have to do "wear-leveling"
</pre>

Additional notes :

Lecture 18, november 17

Comp 3000:
Midterm review:
2)mmap is called in dynamically linked libraries
Work on 2404 assignment

8) yes , using mmap
9) shell did , because it has to open a new file
10) mmap allocates the entire file ,
11) after the fork , the memory won’t be shared, no communication will happen.
12) no, race condition, busy wait , spin lock in the kernel. Some other user can modify the code which we are waiting. Before we decrement someone else will modify it. Someone can change with the semantics.
How do you do kernel hacking?
1) Be humble,
o you don’t necessary know everything, everyone is retarded in their own way
2) Verify you assumptions
o By experiments
o Compile and run
3) Check for errors!
o Saves time
o Kernel has to live “ cleanly”
4) Find another part of the kernel that is close to what you want to do
o Use their ideas to apply yours by analyzing their code.
o Follow their pattern “pattern match” to avoid problems since you may not understand all the abstractions and assumptions
o Realize if their assumptions match yours
5) Understand the “flow of control” in the program
o Architecture
o Division of responsibilities
o Division Why does this matter?
o Possible to make a module and run in the background! Anything is possible! All you have to do is be creative. In fact you have to do a kernel thread
 When does the Ethernet card receives data?
o The Ethernet card sends an interrupt to the kernel
o The CPU calls the kernel code for handling Ethernet data
 When does the kernel run?
o The kernel gets woken up for those events
o The clock generates a timer interrupt
 Interrupt requires a CPU score to be taken over
o Core was probably running a user space process and this is scheduling is about
 Schedule: what to do after having kicked a user space process off a core.
o Can it be a complex CPU algorithm : no since it always sending interrupts?
o Determines what is the next task to do?
 Normally on a core:
o A running userspace process
o Interrupt happens
o Core switches to supervisor mode, runs kernel code
o Last part of the kernel code is the scheduler, chooses which userspace code to run
o Goto top
 Kernel is entered via interrupts, exited via scheduler
 Entry and exit the kernel has to do low-level tasks
o Uses assembly code
 Limits because it is hard to manage
On the website:
 Arch : arch culture specific and the driver specific code,
 Entry_64.5 : before the system call it calls this. Don’t mess up with it. It take cares of dispatching the system calls.
 Shced.h: what the kernel uses to keep track of processes , go through it
 :1 means bitfields in c
 What criteria should the scheduler use?
o “fairness” : everyone should get a turn, everyone gets to share the CPU.
o Starvation : a term when a program does not get the CPU
o Prevent starvation!
o Equal share of resources.
o Why would not you want your scheduler being fair? To avoid “foreground” tasks in the interactive systems.
o Never enough biased towards “ foreground” tasks
o Series of hacks and heuristics

 Memory is allocated lazily is the kernel :
o which means it is possible to allocate way more memory that can be ever used.
o This will cause “memory debt”
o Out of memory killer : killing process when you exceed the amount of memory(ex: shoots whoever deposits the money)

Continuation of lecture 18:

Important notes:
How can you recover a filesystem?
How do you delete a file?
What is a filesystem?:
*persistent data structure
* stored in fixed-sized blocks (at least 512 bytes in size)
*maps hierarchical filenames to file contents
*has metadata about files (somwhow)
What is in a filesystem?
*data blocks
*metadata blocks
How do you organize metdata:
1) First you must identify characteristics of the file system
Superblock : summary y block which tells you about the other blocks you have and it depends on which file system you have. It’s usually the first block of a file system.
In the superblock? :
1) What kind of file system is this? By checking what is the magic number it has
2) How big is the file system?
3) How is it organized?
4) Where can I find the rest of the metadata?

*How can you identify which file system it is from looking at the super class
-> google “magic number of a file”
-> ex: jpg ctr^c ctr^c : switched the pictures into a binary file
-> look at the beginning of the file you will see JFIF: first several bytes in general that identifies the type of the file (magic number)
File extension :
 what is it ?
 is it important
 the kernel does not know and not care about it

For POSIX file systems:
-.> file metadata is stored in inodes
-.> most have pre-reserved inodes
-> the only way you can run out of inodes if you keep creating small files

Usenet : al the things you use to post messages thro social media, email, etc. Those were made using Usenet. Like email but Local Usenet server. But it died over time. Every message is stored in an individual file.

Important commands:
File * : to identify the kind of file system
1. As : Run dumpe2fs foo. What does the output of this command mean?
 Does this give you info about the file system?
 File bar : bar is the file name and cp comp3000-midterm-2017.pdf bar
 Evince bar : opens up the pdf file

===Additional Notes===
Lec 18 
* More on filesystems 
* How can you recover a fs and how do you delete a file? 
 
A filesystem is a: 
* Persistent data structure 
* Stored in fixed size blocks (at least 512 bytes in size) 
* Maps hierarchical filenames to file contents 
* Has metadata about files somehow 
 
What's in a filesystem 
* data blocks (stores file content) 
* metadata blocks, you need someway to find the blocks 
 
How do you organize metadata? 
 
First identify basic characteristics of the filesystem 
- How big is the filesystem? 
- What is the block size? 
 
How do we differentiate between this and other filesystems?
 
You need a "superblock" which is a "summary" block that tells you about everything else 
 
- Format depends on filesystem 
- Normally the superblock is the first block of the filesystem 
 
- Think of it almost like the root of a binary tree 
In the superblock 
* Type of filesystem 
** What filesystem magic number is there (lets us identify one filesystem from another just by looking at the first block) 
** File command to know file type 
* Size of the filesystem 
* How the filesystem is organized (different filesystems organize their data differently) 
* Where can I find the rest of the metadata 
He opened a .jpg as a binary file to show us the magic number in a file, first several bytes identify type of file. 
- Kernel does not care about file extension but userspace programs may care about the extension. 
- File extensions are only really useful for the people looking at them
- Typical for binary file formats to have a set of bytes that identify the type of file
 
POSIX is a standard for maintaining compatibility between operating systems 
- QNX, UNIX, MacOS are all POSIX compliant
- Others comply on a varying scale
For POSIX filesystems 
* File metadata is stored in INODES 
* When you create a filesystem, certain blocks are dedicated to being INODES 
- Possible to have space in your filesystem without being able to store things if you run out of INODES 
 
What is usenet?
- A worldwide distributed discussion system (stone age version of reddit) 
- Deprecated now because it could not handle the spam people uploaded into it, lol 
- Format for usenet was every message stored in its own file => lots of small files 
- Everyone has a local usenet server, access to read posts on the forum 
- To post a message, send to local server which replicates it and sends it to all the other servers 
So we have: 
* superblock 
* inode blocks 
* data blocks 
** data blocks for directories 
** data blocks for files 
 
How do you recover from damage? 
* Filesystems never "reboot", must remain correct over time 
* Errors will happen: bitrot (when bits change), accidental corruption, computer failiure/memory corruption/hard reboot 
 
To make filesystems fast, data and metadata are cached in RAM 
* Bad things happen if this data hasn't been writen to disk and you reboot 
* Even worse things happen if your RAM is bad and corrupts the data 
* FSCK is like scandisk in Windows 98 (this only happens when you do a hard reset)
 
What happens if you lose the superblock? 
* You could lose EVERYTHING 
* Node trunc dd command blew away first bytes of the file system so you could not mount it because you corrupted the superblock. However, fsck fixed this because we have backup superblocks :D 
- Most filesystems keep copies of the superblock in random locations throughout which takes up some unusable amount of data 
- But this is an impractical way to deal with data blocks 
 
Old scandisk/fsck was slow because we they had to scan all filesystem metadata 
* Not to recover data, but to fix metadata 
* lost+found might have some files that you might recover. It is a part of the filesystem for fsck to use. 
 
Nowadays fsck is very fast and we rarely lose data due to losing power 
* What this means is we must be writing to disk all the time 
* But isn't writing slow? All writes aren't slow particularly on conventional hard disks. 
 
On Magnetic Hard Disks (not SSD's) 
* sequential oeprations are fast 
* random access is slow 
** we have to move the read/write head 
 
So, on modern systems we update metadata (and sometimes data) by writing sequentially to disk...and then later writing randomly 
* Sequential writes go to the journal 
 
On fsck on a journaled filesystem
* just check the journal for pending operations (replaying the journal) 
There exists filesytems for optimizing writes that are pure journal 
* log-based filesystem 
 
Logs and journal inherently create multiple copies of data and metadata that are hard to track. This makes deletion nearly impossible (at least to guarantee) 
 
Only way to garauntee...encrypt everything 
* if every file has its own key, you can delete the key and this "delete" the data 
 
SSD's use log-structured sotrage at a level below blocks 
* Writes are coarse-grained (you have to write a lot at once) 
* You don't want to write to the same cells too often, they will die 
** You have to do "wear-leveling"

Operating Systems 2017F Lecture 17

2017-11-16T20:39:44Z

HebaJallad: /* In Class */

==Video==

The video from the lecture given on Nov. 14, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec17-14Nov2017.mp4 is now available].

==Notes==

===In Class===

<pre>
Lecture 17
----------

How do you do kernel hacking?

* Be humble
- you don't know how everything works, and nobody else does

* Verify your assumptions as you go
- perform lots of experiments
- incremental development, compile and run very often

* Check for errors ALWAYS
- will save you from later errors
- kernel has to "live cleanly"

* Find another part of the kernel that is close to what you want to do.
Read that code, use their approach where possible
- You don't understand all the abstractions and assumptions, so
"pattern match" to minimize trouble
- Realize that the assumptions behind code you find don't necessarily match
your own.

Additional Notes :
Lecture 17:
COMP 3000
Important notes:

Install kernel module on the remote vm
Edit them locally, mount the file locally
Sshfs : What determines what files I can or can’t create, it is file operations, useful tools to edit modules
Cd . : what does the . do ? nothing
. = is the current directory
.. = parent directory
It can link to itself
Compiling and loading all in the vm : inorder to build ur own modules , you must associate the header to a specific kernel.
Something is wrong with the metadata ? the files are in the soma not anil class?

Module Professor is walking through , ones:
1) Extend its functionality , so it can access info about the process which has made the system call.
2) Output your process ID is blocked
-> how to make these changes:
1) change the dev 1 name , newgetpid, change the make file
-> 2) #define dev name = newgetpid (renaming the device)
-> 3) get the pid of the user:
 where is get pid? /kernel/sys.c. SYS: not a c code, defining a system call that get pid with 0 arguments .
 Get pid returns?
Line 37 ) newgetpid_read: these changes are close but it kept repeating itself
 Typy char *message = “your PID is unknown!\n”;
o Msglen = 21;
 Change the for loop
o For (I =0; I < msglen; i++){
o Put_user(message[i], buf++);
o }
*offset : long int , and it was equal to 0 so we had to update the offset
-1: error
0 : no more bytes to read
 To figure out if we return the right PID , code a printk
 Convert int to a string presentation , in order to output how?
o Line 39 : create a buffer called message char message [100];
o Snprintf : like printf
 Why is the number is the pid increasing ? since the cat spawns a process
 Why not use printf instead of snprintf ? since we don’t have STDOUT, it prints to a buffer.

Printk : goeas to kernel log
 Reading from a file is just an API
 REST API is similar to the os : since we can read and write anything
 With a device file : ?? //ask the prof
 To implement a new function, get the major and minor and device name , creat a function by registering a device file
 Every device as their own read and write functions , they have to be custome defined
 They are special, and you can do this to any file
 Fuse : file system user space , to connect
 Why do we use goto? : we should not use , can only in the same functions , raising an exception essential in kernel code since the kernel has to resource management and allocating resources and in error condition you don’t .
 Failed_devreg : deallocating
 Printf : its not defined in the kernel code, it is defined in the c code and we don’t have stdout


Textbook : No more readings for further lectures
 Active learning rather than passing

Important Commands:
Ls vm : to display the directories in the vm
Ls –la vm ?:
Sshfs:
Get pid: process ID
Rm ones*: removing the ones
Sudo rmmod newgetpid :
Sudo insmode newgetpid.ko: to load the modules?
Sudo insmode|grep new
Man getpid : information to get pif of a process
Ls –la : for directories and links
Mkdir : making directory.
Uname –r?:
First number on the left : 21 hardlinks : every directory has a sub directory to it.
Additional Notes : snprintf is essentially a function that redirects the output of printf to a buffer. This is particularly useful for avoiding repetition of a formatted string.

* Understand the "flow of control" inside the program
- architecture
- division of responsibilites
- purpose of data structures/objects

Scheduling
----------
How does the kernel maintain control?
- CPU is given entirely to userspace processes to run most of the time

Kernel needs to run when it needs to run
- and no process should be able to stop it

*The interrupt table*
- pointers to code that the CPU runs when it receives different hardware
or software interrupts
- since the kernel runs first, it gets to set the interrupt table
- only supervisor code can change the table, not user code

So when the ethernet card receives data...
...the ethernet card sends an interrupt to the kernel
...the CPU calls the kernel code for handling ethernet data

When the clock generates a timer interrupt...
...the CPU calls the kernel code for handling timer interrupts

In order to process an interrupt, an CPU core has to be taken over
- that core was probably running a userspace process

Scheduling is all about what to do after having kicked a userspace process
off of a core

Normally on a core
* userspace process is running
* interrupt happens
* core swiches to supervisor mode, runs kernel code
* last part of kernel code is scheduler, chooses which userspace code to run
* goto top :-)

Kernel is entered via interrupts, exited via the scheduler

Entry and exit to the kernel has to do low-level tasks like changing CPU modes,
manage CPU registers
- must be written in assembly

Most OS kernels minimize this low-level code

What criteria should the scheduler use in deciding what to run?
- "fairness"
- prevent starvation
- absent other conditions, equal allocation of resources
- bias resources towards "foreground" tasks in interactive systems
- never biased enough
- always hacks, heuristics

Memory overcommitment
- it is possible to allocate way more memory than can be ever used
- goes into "memory debt"
</pre>

===Midterm Review===
1. Execve and no it does not create a new process

2. Dynamically linked programs map in dynamically linked libraries

3. Hard links maps filenames to inodes

4. Sleep until the consumer wakes it

5. Using type conversion, read returns number of bytes written. We must convert back to the number of unsigned long, minus one because maximum index of array is one less

6. uses of signals:
* kill process
* find out when child process is terminated
* start/stop a process
* detect when a program has made a ptr error

7. Pointers in C contain virtual memory addresses because the memory of a process is a virtual address space. Each process thinks it has its own stretch of memory.

8. processes make a sys call to allocate memory. execve -> then mmap and/or sbreak to allocate memory

9. shell opens the file output.txt.

10. mmap contents of file can allocate the contents of it into ram.
* comparing 1 mb of both files at a time will be less expensive
* only makes sense to mmap if you will access files multiple times

11. producer and consumer now have no way of talking to each other because they do not share the same memory.

12. no. there is a race condition. while loop loops while sem < 0.
someone else could access and modify the variable between the while and the sem--;.
* you need special instructions to test and set the variable at the same time.
* also, int sem parameter isn't a ptr so it's a local variable.

===Additional Notes===

How do you do kernel hacking? Some tips for low level kernel code:
Be humble; You cannot know how everything works 
Verify your assumptions as you go 
Perform lots of experiments 
Incremental Development, compile and run often 
 
Check for errors ALWAYS or they will cascade and hurt you later 
*This will save you from later errors 
*Kernel has to live cleanly 
*This principle applies to any large scale coding
 
Find another part of the kernel that is close to what you want to do (Search online! This is a good link: http://elixir.free-electrons.com/linux/v4.4.83/source 
You do not understand all abstractions and assumptions, so "pattern match" to minimize trouble 
*Functionality that you want to implement might already exist in the kernel, look at how they do things and follow the pattern. 
*Refine understanding as you go, the other implementation might be for something else. 
 
Realize that the assumptions behind code you don't necessarily match to your own. Spend the time reading other people's code. 
Understand the "flow of control" inside the program 
*Architecture 
*Division of responsibilities 
*Purpose of data structures/objects 

For the tutorial 6 he gave us a link for a reason. If you click on it you can search getpid and then dig deeper! For example cut and paste the line inside getuid function. 
 
Scheduling 
Scheduler decides when what process should run at a high level 
Questions: How does the scheduler work/kernel control when a program runs/how can you make sure one program does not take over CPU from anyone else? 
CPU is given entirely to userspace processes to run most of the time 
Kernel needs to run when it needs to run and no process should be able to stop it 
* The interrupt table: pointers to code that the CPU runs when it receives different hardware or software interrupts. Since the kernel runs first it gets to set the interrupt table. Only supervisor code can change the table not user code.
 
When the Ethernet card receives data: 
1) The Ethernet card sends an interrupt to the kernel 
2) The CPU calls the kernel code for handling Ethernet data 
 
When the clock generates a timer interrupt: 
1) The CPU calls the kernel code for handling timer interrupts 
 
2) In order to process an interrupt, a CPU core has to be taken over 
3) That core was probably running a userspace process 
4) Scheduling is all about what to do after having kicked a userspace process off of a core 
 
Normally on a core: 
* Userspace is happening 
* Interrupt happens 
* core switches to supervisor mode, runs kernel code 
* last part of kernel code is scheduler, chooses which userspace code to run. Repeat the process 
Kernel is entered via interrupts, exited via the scheduler 
Scheduler needs to be efficient because interrupts are frequent and we do not want to waste resources that could be put towards actual computing 
Entry and exit to kernel has to do low-level tasks like changing CPU modes, manage CPU registers. This is Assembly code.
 
Go to http://elixir.free-electrons.com/linux/v4.4.83/source 
Go into arch: code specific to CPU architecture. Two most commons are arm and x86 
Go up one level 
Drivers directory have drivers for all kinds of things 
Linux Kernel designed for x86 but it now abstracted to support different architectures.
Go into architecture -> x86 -> entry 
Open entry.s: Large assembly language file. This is the code that runs before system call specific code. 
shced.h -> Contains definition of task_struct. Data structure that keeps track of a process. 
 
What criteria should the scheduler use in deciding what to run? 
*Fairness: Starvation is when a program does not get the CPU 
**Everyone should get a turn
**CPU is passed around
** Nobody should be left that they never get the CPU
**Prevent Starvation 
**Absent other conditions, equal allocation of resources
*Bias resources towards "foreground" tasks in interactive systems
**never Biased enough
**always hacks, heuristics
 
Memory overcommitment 
Memory does not get allocated when you mmap. It is loaded "lazily" in the Linux kernel. It is possible to allocate way more memory than can ever be used. 
Goes into "memory debt"

Operating Systems 2017F Lecture 16

2017-11-07T19:08:49Z

HebaJallad:

In Class
Comp 3000
Lecture 16
Important notes: Tutorial 5:
File system :
 persistent data structure organized around blocks (which are fixed allocation units)
 maps hierarchal names (keys) to values
 provides a file-like API like open, read, write, close,etc
What does it mean to “make” a file system?
 Initializing a data structure.
 “formatting” a disk
Physical vs Logical : logical size of a file: the size your program see when accessing the file (bytes in a file)
Physical : How much space it takes up on disk , in terms of blocks , fixed units of storage allocation

Physical :
 By default or for multiple of files it is 1K blocks
 Example : Ext4 has 4k blocks

Kernel Programing :
 Warning:
o If you use linux base , you may crash your whole system, just backup before you do so using “rsync”.
Open Stack : log in through the terminal using your instance’s Ip address , but it failed to work . when you ssh to it you must write ssh “Address” –l Ubuntu
 You are required to use sude to add a user name, so u can play around in root.
What is a Kernel module ?
 A way of splitting up kernel functionality so everything does not have to load at boot.
 Modifies a kernel functionality
 Runs in kernel space , is the key thing to think about
o It is more powerful than root and it can do anything
o Access to all kernel memory
o And you can modify everything
 If you miss anything in the kernel development your system will crash
 Kernel machine provides you with a floppy by default which explains why it still exists in Anil’s terminal
 Once you install a module , the module is unstrained
Why do we use modules? Why don’t we load processes instead?
 No new mechanisms
 Increased security (restricted access)
 Makes the kernel less smaller, microkernel design,
o Putting in the functions that are supposed to be in the kernel into processes
o Process do IPC rather than code talking in supervisor mode
 Examples :
• Filesystems
• Drivers
• Networking
• Minix, QNX, GNU, hurd,
 Why is Linux “monolithics” kernel ?
o Switching between contexts are expensive (context switch)
o How to make microkernels fast can be adopted by monolithics kernels to make them even faster
o Unreal security benefits :
 if you control the file system process, you can control everything

Rebuilding and changing the kernel:
 1)Type “make” : more compilacted than 2401
o Kernel built
 2)Make modules
 3)Sudo make install
 4) sudo make-modules install
 5) Sudo shutdown –r now : for the vm to reboot
 Which configuration would you use to build your own kernel ?
o Don’t do configurations from scratch
o Copy the configurations and use them
o Make localmodconfig : output for ls mod and uses that for configuring your kernel
o Requires time and effort
 Why less /dev/ones doesn’t exist anymore?
o Since reboot occurred
o You must load the modules again
o Head –c 100 /dev/ones to be able to use it again
 Implementing the device file of dev 1 ?

o Implement the file API required
o Teach the kernel what it means to do operations like read, etc
Code from the tutorial ones.c:
 Open ones_read code: file descriptor, file , buffer, amount of bytes to read and offset
o Offset : position in the file
o Fills the buffer with ones
o Why don’t we just set it to 1 instead of putting put_user?
 Char *buf : Pointer for a user space process, in order for the kernel to write to user spacer safely
o Line 46: Why use printk and not printf? Since printf is not defined because the c library is not available in the kernel, how can you can c library when the c library depends on the kernel? Kernel is independent, does not depend on any libraries.

Commands:
Man ls : to see different ls commands
 Ls –las block
 Cat /dev/ones |less : it is like dev u random , but instead of generating random number, it instead generates infinite of number 1
 Ls –mod : displays all the moduls which are currently loaded on the virtual machine
 IBM ps/2 : series of computers created to control PC, developed the interfaces to have a mouse and keyboard.
 Less readme : to check instructions of how to do a make
 Make menuconfig : options of kernel configurations
 Cat/pro
 Less .config : bad idea to go directly in it , use “make menuconfig “
 /boot : where the kernel got installed.
o Ls –lah : to see the size
 Less/ etc/modules
 Modul init: what function should be called when loaded and when it is unloaded
 Modul exit:
 Creating a device file : defining a file which has special semantics, define a struct and functions which should be called to explain each file operation, open , read, release(like closing but not really)
 What happens if you start running to the file ? permission are read only, not writing
o Override that? Still your permission is denied, you can only read since we didn’t write a function in the struct to write.

Operating Systems 2017F Lecture 14

2017-11-04T04:59:49Z

HebaJallad: /* Review Notes */

==Video==

The video from the lecture given on Oct. 31, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec14-31Oct2017.mp4 is now available].

==Review Notes==

Definitions:

- OS: A resource Manager that abstracts the hardware so that applications can get easy access to it. The OS facilitates resource sharing among concurrent programs.
- Kernel: The first program to launch when the computer boots. The kernel has absolute power over the hardware; it manages and controls access to these resources.
- Process: A running program. It has one or more threads and address space.
- Thread: A function that runs in the same memory space as other functions.
- Memory: A contiguous array of bytes. Modern systems virtualize memory, mapping it to physical memory so it can be shared among processes.
- Interrupts: A mechanism used by the kernel that interrupts the CPU when other tasks are higher priority.
- Signals: Inter-process communications. A mechanism used by the kernel and processes to communicate with one another.
- System Call: Specialized functions used by programs to request more resources from the kernel.
- File: A hierarchical name with a value mapping (Key/Value pair)
- Shell: Another program that is used to send system calls to the kernel or launch other programs.
- File system: A set of files grouped together with a common hierarchical root.
- Fragmentation: Variable sized memory allocation.
- Internal Fragmentation: Space wasted in an allocation.
- Concurrency: More than one thing happening at a time.
- Atomic Variables: Variables that guarantee a strict ordering of operations.
- Segmentation: Managing memory without paging.
- Segments: A variable sized block of memory that can be placed in different parts of the address space.
- Segfault: When a virtual address does not have a corresponding physical address.
- mmap: The virtual address range corresponds to the contents of a file or RAM.
- Concurrency: More than one thing is happening at the same time.

**********************************************************
Professor's explanations of assignment 1 and 2 :

What does stat really return?
Inode number, directry , number if hard links
If you have a file, you basically hard link, and it is basically a reference count.
 Manual page for stat() , man stat(2)
o Ctime: modification made by the inode,
o Everytime you read a file, you write to a file system , and it is expensive and not good for you device
 Device files are not real file, they are name spaces to access other things, like terminal, screen video.
o Ex: dev random, is not really a file, it just produces bits
o You have to compare which device they are pointing to and if they return the same major and minor ID’s then they are equal
Q10 Review :
 Only if it is waiting you wake it up and set the flag into 1
o Affect = kicks you out
o Generating a lot of signals to the consumer
 A signal should happen occasionally
o Kill -1 , you kill every process . don’t get it ask ?
o Maybe you should set the consumer to 0 and see what happens , in 3000pc
 Why does the program deadlocks when request a large amount of numbers ?
o signals aren’t always delivered , they sometimes get lost, but generlay they get there unless you send various amount of signals
 theory about time:
o gettimeofdat: man gettimeofday

Creating a process:
* fork() creates a process and execve() loads the binary to the new process.
** returns a process id
*** pid > 0, parent process
*** pid < 0, error
*** pid = 0, child process

* When a parent does not deal with its child processes they become zombies
* If a parent dies before its children they become orphans that are re-parented by the OS

* wait()
** suspends the current process until one, some, or all of its children have terminated
** wait() can be called anytime to check on the status of children

IO redirection:
* >, direct to stdout
* <, direct to stdin
* |, connect the output of one process to the input of another process
* when a process starts it is given file descriptors to stdout, stdin, and stderr

Inodes:
* Contain information about the blocks corresponding to the contents of a file on disk, its owner, and the file access permissions.
* A filename points to an inode
* The inode count represents all of the files and folders on disk
* Hard links create another name to the same inode number
** Only point to files within the same file system
* Symbolic links are files with their own inode
** Can point to files in other partitions or file systems

Virtual Memory:
* Every process receives a memory map with virtual addresses that correspond to physical addresses.

Memory Allocation Challenges:
* Need to have a contiguous address range
* Data Structures can not be moved once allocated
* Memory is constantly allocated and deallocated

Solution to these challenges:
* Use fixed size chunks that can be stored anywhere
** Called a page in memory or a block on disk
** Sizes are always a power of 2 because addresses are indexed in bits
** The benefit: it removes the limitations on array sizes.
** Use a table to map virtual memory to physical memory -- called a page table
** In a 32 bit system: the top 20 bits are the page locations and the lower 12 bits are the page offset

Concurrency:
* Challenge: keeping the shared state consistent; want deterministic behavior from concurrent computations
* Solution: Need mechanisms that bypass the memory hierarchy.
** Use filesystem locking primitives -> atomic variables -> semaphores
** Use signals to synchronize the shared state

Segmentation:
* The physical address can be found by adding the segment register to the offset.

Operating Systems 2017F Lecture 13

2017-11-04T04:58:24Z

HebaJallad: /* Note */

==Video==

Video from the lecture given on October 19, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec13-19Oct2017.mp4 is now available].

==Code==

Files from the lecture [http://homeostasis.scs.carleton.ca/~soma/os-2017f/code/lec13/ are available].

==Note==

===Questions:===
Mutex and semaphore?

Mutex is a binary semaphore, it is a semaphore that take a value that is zero or one, a classic semaphore can count.

Mutex stands for ‘mutual exclusion’: I am exclusive access, no one else does.

More general semaphore is used when you need to keep track of numbers.

Most classic semaphore: read write semaphore. which will be seen in the kernel. The idea is multiple reader and only one writer.

Most of the time we use mutex.

=== dinner philosopher problem ===

Wiki page: https://en.wikipedia.org/wiki/Dining_philosophers_problem

You have five philosophers sitting around the table, these philosophers will talk, and they will talk so much, they don’t really want to focus on eating. And for some reason there is not enough of forks at the table. There is one fork at every position between the plates. In order to eat anything, you need two forks.(Maybe better to think about using chopsticks instead of forks). There are five people there and only five forks.

******************************************************************************************************************************************************************************************************
Important notes, 3000
Lecture 13
//can be represented with chopsticks which makes more sense.
Illustration of the dining philosophers problems: an example the professor represented, in which philosophers need 2 forks to have dinner but they only have 1 so they will share at random timing but will end up in a mess and an error . This error illustrates deadlock. We can have them to have a delay then pick it up but then they will never eat, we can also have them to do that at randomization which will work but then end up in an error which I explained earlier.
Code used in class is tutorial 4 code and the professor is modifying it to explain.
Shared state: when they get both of locks (2 chopsticks) they get a message. Modify the code. The message will state “eat it” in the code.
In the code presented in class: 5 philosopher and 5 chopsticks. Initilized each semaphore, each Copstick has a lock.
Create a processes for each philosopher by forking each one and each will have an ID.
Exit the for loop so each child will continue to fork
Wait time : they are concurrent , the real time is 1 sec. Philosophers are running concurrently

 Dead lock: the computation cant proceed any further, since the concurrent processes can’t access the other resources. Enough resources you never have to share, deadlock won’t happen. But if you have shared sources you may occur with a deadlock.
> Release the locks: ( put a chopstick down) using post in the code
Does is matter which one will be released first?
 Running the code: someone only has one chopstick and someone did not get it. The code got stuck
o How so we solve this? :
 Man Sem_wait() to see if it is available to solve this problem. Put sem_wait() and sem_trywait() to check out the problem.
 Create a new function called get chopstick , and it failed so thr prof added a sleep()
 There is no chopstick 5 = have a bug, #1 failed and no one else is eating except #1. They tried to eat 50 times
 No deadlock since we have enough sources, since they exit.
 If one exits there is enough chopsticks
 The prof tried while (1) which is an infinite loop.
 Our code never locks, no dead lock now. Why is that? Nvm some of them are deadlocked such as 4.
 Why don’t we access the CHPstick in a fixed order? (Ask what he means)
• Which one do we get first?
 Solution : If you get 1 only and not 2, you must let go of it and then try to get both.
 Release in the opposite order you acquire them
 Practice on semaphores, and creating processes
 After the midterm : kernel hacking
 Linux cross reference : source code into file system (fs) data_sem and others are semaphores
 Have to enforce mutual exclusion and that why we use semaphores down = lock, up = release.
 Have to release the semaphores so someone accesses the inode
To ask :
Why is there more chopsticks than philosophers

*Senario 1: they all pick the left one at the same time, they all now have one fork.- can any of them eat? no, they suppose to use two forks to eat.
*Senario 2: Immediately put a fork down, wait for five seconds, and pick a fork. (have a delay, and pick it up) -they will never eat, because they always get one fork.
*Senario 3: randomize it? When they get one fork and couldn’t get the other one, they put it back down, wait a random amount of time and get the next one. -Yes this will work on average, except sometime randomly they will pick up a fork at the same time and starve to death. (unless you have a timeout)

This illustrate the concept of '''deadlock''': you get to the point where the computation cannot proceed any further. Because the concurrence processes cannot get all the resource. Some of them can access the resources and some of them do not have the resources, so everyone stuck.

Wiki page: https://en.wikipedia.org/wiki/Deadlock

When your computer logs up, a lot of time it is because of the deadlock. Something get locked and never get unlocked at the driver.

One way to avoid deadlock: if you have enough resources you never have to share, deadlock never happen.
If you have shared resources, you can’t have deadlock.

After the midterm we will do more kernel hacking.