Soma-notes - User contributions [en]

Operating Systems 2017F Lecture 20

2017-11-29T02:17:52Z

Krithikasaravanan:

==Video==

[http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec20-28Nov2017.mp4 Class Video]

==Notes==

===In Class===

<pre>
Lecture 20
----------

When we run "ls" on an sshfs-mounted filesystem
* ls makes system calls (open, getdents) to the local Linux kernel
* the local kernel sees filesystem is FUSE, calls FUSE routines for open, getdents (via vfs abstraction)
* FUSE calls the sshfs process that mounted the filesystem
* sshfs process sends request to remote system
- via socket system calls
* remote sshd process receives request (via system calls)
* remote sshd process accesses local filesystem
- makes open, getdents system calls
- remote kernel checks vfs, calls ext4 routines to access data
* remote sshd process responds to request (via system calls)
* local sshfs process receives response (via system calls)
* local sshfs process responds to FUSE request
* FUSE passes data back to vfs layer, then back to requesting process

Normal file access permission check
- compare uid, gid of file with uid, gid of process

But really...
- compares it with fsuid, fsgid of process
- which is normally same as euid, egid of process
</pre>

===Additional Notes===

Assignment 4 will be autograded and in fill in the blank form. This will not be like the final exam. 
You should think about who is doing what? Otherwise FUSE won't make much sense 
 
Inode numbers completely changed when you use ssh. It starts with 1 from "." (root) and increments from there. 
Inodes come from filesystem you are acessing 
He used df to find filesystem info 
The used Commands: mount | grep ubuntu and mount | grep vda1 to find type of filesystem 
In the SSH there is the filesystem type "fuse.sshfs" 
When you run strace on ls on the two terminals (ssh connection and the local one) you see similar output 
When we run "ls" on an ssh-mounted filesystem 
* ls makes system calls (open, getdents) to the local Linux kernel 
* the local kernel sees the filesystem is FUSE, calls DUSE routines for open, getdents (via vfs abstraction) 
* FUSE calls the sshfs process that mounted the filesystem 
* sshfs process sends request to remote system 
- via socket system calls 
* remote sshd process receives request (via system calls) 
* remote sshd process receives requests (via system calls) 
- makes open, getdents sytem calls 
- rmeote kernel checks vfs, calls ext4 routines to access data 
* remote sshd process responds to requests (via system calls) 
* local sshfs process receives response (via system calls) 
* local sshfs process responds to FUSE request (via system calls) 
* FUSE passes data back to vfs layer, then back to requesting process 
 If you look as lsmod | less you see a ton of modules. Some of the modules are not necessary to mount the root filesystem. 
ls /lib/modules there are directories that store the kernel modules 
You have to have a filesystem that you mount initially that has a bunch of modules in it. Thus, there is an initial root file system that is necessary to load everything else 
This is the initial RAM disk. This filesystem loads the modules needed for the real filesystem. 
To remove a file I need to remove the hardlink from the directory where the hardlink exists 
The password files maps usernames to user id's (Linux does not care about your username) 
Normal file access permission check br>
- compare uid, gid of file with uid, gid of process 
But really: 
- compares it with fsuid, fsgid of process 
- which is normally same as euid, egid of process 
 
Sshfs 
* Inode values are different in remote vm vs local vm 
* Values increment from 1 in local vm 
* Who's supplying the inode values? 
** Ext4 filesystem in local vm 
** Fuse.sshfs filesystem in remote vm 
* Filesystem determines the interpretation of inode values 
* Inodes have no meaning outside its filesystem 
* Local to its filesystem => hardlinks are different too 
* "strace ls" in both output the same thing 
* Vfs not built into original unix filesystems 
* Try "strace"ing the sshfs call, see how its interacts with fuse 
* Don't need to know exact system calls, but you should know when it has to make system calls and why 
** Needs to access files over the network 
* Understand why you can't do this with regular library calls 
* Removing the root filesystem will trigger a kernel panic 
** Kernel will prevent you from unmounting filesystems that contain other filesystems 
* Kernel modules located in "/lib/modules/" 
* Initial filesystem is loaded into kernel 
** Fake filesystem, throw away after loaded, not persistent 
** Ram disk = filesystem stored in ram 
** Gets the system up to the point where the kernel can load the real filesystem 
** Bootloader has to load both kernel and the initial ram disk into ram 
** Ram disks located in "/boot = initrd.img<...>" 
** Kernel version specific 
** Generated as part of installation of kernel 
* Cpio: copies files to and from archives 
 
* What determines whether you can access a file? 
** The filesystem 
* How to map from user id to username? 
** Recall that 
- Kernel knows nothing about usernames 
- This is a userspace process 
- Looks in local password file 
* What determines whether an operation is allowed? 
** Checks to see who owns the file 
** Compare uid, gid of file with uid, gid of process 
* In remote system, sftp does file access checks

Operating Systems 2017F Lecture 20

2017-11-29T02:17:08Z

Krithikasaravanan:

==Video==

[http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec20-28Nov2017.mp4 Class Video]

==Notes==

===In Class===

<pre>
Lecture 20
----------

When we run "ls" on an sshfs-mounted filesystem
* ls makes system calls (open, getdents) to the local Linux kernel
* the local kernel sees filesystem is FUSE, calls FUSE routines for open, getdents (via vfs abstraction)
* FUSE calls the sshfs process that mounted the filesystem
* sshfs process sends request to remote system
- via socket system calls
* remote sshd process receives request (via system calls)
* remote sshd process accesses local filesystem
- makes open, getdents system calls
- remote kernel checks vfs, calls ext4 routines to access data
* remote sshd process responds to request (via system calls)
* local sshfs process receives response (via system calls)
* local sshfs process responds to FUSE request
* FUSE passes data back to vfs layer, then back to requesting process

Normal file access permission check
- compare uid, gid of file with uid, gid of process

But really...
- compares it with fsuid, fsgid of process
- which is normally same as euid, egid of process
</pre>

===Additional Notes===

Assignment 4 will be autograded and in fill in the blank form. This will not be like the final exam. 
You should think about who is doing what? Otherwise FUSE won't make much sense 
 
Inode numbers completely changed when you use ssh. It starts with 1 from "." (root) and increments from there. 
Inodes come from filesystem you are acessing 
He used df to find filesystem info 
The used Commands: mount | grep ubuntu and mount | grep vda1 to find type of filesystem 
In the SSH there is the filesystem type "fuse.sshfs" 
When you run strace on ls on the two terminals (ssh connection and the local one) you see similar output 
When we run "ls" on an ssh-mounted filesystem 
* ls makes system calls (open, getdents) to the local Linux kernel 
* the local kernel sees the filesystem is FUSE, calls DUSE routines for open, getdents (via vfs abstraction) 
* FUSE calls the sshfs process that mounted the filesystem 
* sshfs process sends request to remote system 
- via socket system calls 
* remote sshd process receives request (via system calls) 
* remote sshd process receives requests (via system calls) 
- makes open, getdents sytem calls 
- rmeote kernel checks vfs, calls ext4 routines to access data 
* remote sshd process responds to requests (via system calls) 
* local sshfs process receives response (via system calls) 
* local sshfs process responds to FUSE request (via system calls) 
* FUSE passes data back to vfs layer, then back to requesting process 
 If you look as lsmod | less you see a ton of modules. Some of the modules are not necessary to mount the root filesystem. 
ls /lib/modules there are directories that store the kernel modules 
You have to have a filesystem that you mount initially that has a bunch of modules in it. Thus, there is an initial root file system that is necessary to load everything else 
This is the initial RAM disk. This filesystem loads the modules needed for the real filesystem. 
To remove a file I need to remove the hardlink from the directory where the hardlink exists 
The password files maps usernames to user id's (Linux does not care about your username) 
Normal file access permission check br>
- compare uid, gid of file with uid, gid of process 
But really: 
- compares it with fsuid, fsgid of process 
- which is normally same as euid, egid of process 
 
Sshfs 
* Inode values are different in remote vm vs local vm 
* Values increment from 1 in local vm 
* Who's supplying the inode values? 
** Ext4 filesystem in local vm 
** Fuse.sshfs filesystem in remote vm 
* Filesystem determines the interpretation of inode values 
* Inodes have no meaning outside its filesystem 
* Local to its filesystem => hardlinks are different too 
* "strace ls" in both output the same thing 
* Vfs not built into original unix filesystems 
* Try "strace"ing the sshfs call, see how its interacts with fuse 
* Don't need to know exact system calls, but you should know when it has to make system calls and why 
** Needs to access files over the network 
* Understand why you can't do this with regular library calls 
* Removing the root filesystem will trigger a kernel panic 
** Kernel will prevent you from unmounting filesystems that contain other filesystems 
* Kernel modules located in "/lib/modules/" 
* Initial filesystem is loaded into kernel 
** Fake filesystem, throw away after loaded, not persistent 
** Ram disk = filesystem stored in ram 
** Gets the system up to the point where the kernel can load the real filesystem 
** Bootloader has to load both kernel and the initial ram disk into ram 
** Ram disks located in "/boot = initrd.img<...>" 
** Kernel version specific 
** Generated as part of installation of kernel 
* Cpio: copies files to and from archives 
 
* What determines whether you can access a file? 
** The filesystem 
* How to map from user id to username? 
** Recall that 
* Kernel knows nothing about usernames 
* This is a userspace process 
* Looks in local password file 
* What determines whether an operation is allowed? 
** Checks to see who owns the file 
** Compare uid, gid of file with uid, gid of process 
* In remote system, sftp does file access checks

Operating Systems 2017F Lecture 16

2017-11-24T08:38:56Z

Krithikasaravanan:

==Video==

The video from the lecture given on Nov. 7, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec16-09Nov2017.mp4 is now available].

==Code==

[http://homeostasis.scs.carleton.ca/~soma/os-2017f/code/lec16/newgetpid.zip Code for download]

==newgetpid.c==

<source lang="C" line>
/* Code derived from:
https://appusajeev.wordpress.com/2011/06/18/writing-a-linux-character-device-driver/
and
http://pete.akeo.ie/2011/08/writing-linux-device-driver-for-kernels.html
*/

#include <linux/module.h>
#include <linux/string.h>
#include <linux/fs.h>
#include <linux/device.h>
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/sched.h>
#include <asm/uaccess.h>

#define dbg(format, arg...) do { if (debug) pr_info(CLASS_NAME ": %s: " format, __FUNCTION__, ## arg); } while (0)
#define err(format, arg...) pr_err(CLASS_NAME ": " format, ## arg)
#define info(format, arg...) pr_info(CLASS_NAME ": " format, ## arg)
#define warn(format, arg...) pr_warn(CLASS_NAME ": " format, ## arg)

#define DEVICE_NAME "newgetpid"
#define CLASS_NAME "comp3000"

static struct class* newgetpid_class = NULL;
static struct device* newgetpid_device = NULL;
static int newgetpid_major;

static int newgetpid_open(struct inode *the_inode, struct file *f)
{
return 0;
}

static ssize_t newgetpid_read(struct file *f, char *buf, size_t len, loff_t *offset)
{
size_t i, msglen;
pid_t thepid;

char message[100];

if (*offset > 0) {
return 0;
}

thepid = task_tgid_vnr(current);

snprintf(message, 100, "Your PID is %d!\n", thepid);

msglen = strlen(message);

if (len < msglen) {
msglen = len;
}

for (i = 0; i < msglen; i++) {
put_user(message[i], buf++);
}

*offset = i;

return i;
}

static int newgetpid_release(struct inode *the_inode, struct file *f)
{
printk(KERN_ALERT "Newgetpid device closed\n");
return 0;
}

static struct file_operations newgetpid_fops = {
.open = newgetpid_open,
.read = newgetpid_read,
.release = newgetpid_release,
};

static char *newgetpid_devnode(struct device *dev, umode_t *mode)
{
if (mode)
*mode = 0444;
return NULL;
}

static int __init newgetpid_init(void)
{
int retval;

newgetpid_major = register_chrdev(0, DEVICE_NAME, &newgetpid_fops);
if (newgetpid_major < 0) {
err("failed to register device: error %d\n", newgetpid_major);
retval = newgetpid_major;
goto failed_chrdevreg;
}

newgetpid_class = class_create(THIS_MODULE, CLASS_NAME);
if (IS_ERR(newgetpid_class)) {
err("failed to register device class '%s'\n", CLASS_NAME);
retval = PTR_ERR(newgetpid_class);
goto failed_classreg;
}

newgetpid_class->devnode = newgetpid_devnode;

newgetpid_device = device_create(newgetpid_class, NULL, MKDEV(newgetpid_major, 0),
NULL, DEVICE_NAME);

if (IS_ERR(newgetpid_device)) {
err("failed to create device '%s'\n", DEVICE_NAME);
retval = PTR_ERR(newgetpid_device);
goto failed_devreg;
}

info("Newgetpid device registered using major %d.\n", newgetpid_major);

return 0;

failed_devreg:
class_unregister(newgetpid_class);
class_destroy(newgetpid_class);
failed_classreg:
unregister_chrdev(newgetpid_major, DEVICE_NAME);
failed_chrdevreg:
return -1;
}

static void __exit newgetpid_exit(void)
{
device_destroy(newgetpid_class, MKDEV(newgetpid_major, 0));
class_unregister(newgetpid_class);
class_destroy(newgetpid_class);
unregister_chrdev(newgetpid_major, "newgetpid");
info("Unloading Newgetpid module.\n");
return;
}

module_init(newgetpid_init);
module_exit(newgetpid_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Anil Somayaji <soma@scs.carleton.ca>");
MODULE_DESCRIPTION("A write newgetpid character device module");
</source>

==Makefile==

<source lang=make line>
obj-m := newgetpid.o
KDIR := /lib/modules/$(shell uname -r)/build
PWD := $(shell pwd)
default:
$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
</source>

=== Additional Notes ===

What determines what files you can and cannot create? 
* Ssh privileges 
 
* Anything that you can do as an ssh user you can do in the vm, just doing file operations 
* Sshfs means when you do read and write system calls for programs in a directory it also does it on the remote machine 
* Ssh is a good tool to access remote files locally, we will use it to edit modules 
 
Ones/newgetpid program: 
* We want to extend its functionality in a specific way 
* We want to access info about the process that made the system call 
* Let's get current processes id 
** Normally we would use getpid 
** But we can't make system calls in kernel space 
** But we can call the function that the system call uses or just copy the functionality 
* Code for this is in "kernel/sys.c" 
 
Syscall_define0: 
* Macro that expands 
* Defines system calls with "getpid" and takes no arguments 
 
* Can use the code inside the function but not the function itself in the kernel 
* Getpid returns a pid_t 
* Instead of get_ones returning all those ones we want it to return the pid 
* Lets try to get it to output a basic string with the pid 
 
How does printk work? 
* Printk sends its output to the kernel log 
* We changed the name to "newgetpid" 
* How do we convert int to string to print the pid? 
** Make a buffer, let's call it "message" 
 
Why does the pid keep incrementing each time we call "cat /dev/newgetpid"? 
* "Cat" spawns a process so every time we do a fork we get a new pid 
 
Why can we use snprintf but not printf? 
* We include "linux/kernel.h" which defines snprintf but not printf 
* Printf assumes we have a standard output to print to 
* Snprintf only needs character arrays for it to work 
 
* Read functionality uses an API 
* Adding new functionality like "write" is easy, just look at the standard API and original kernel source 
* All device files have their own custom read and write functions 
 
Why do we use goto's? 
* C has no exception handling functionality so we implement our own 
* Jumps to error paths: failed_devreg, failed_classreg, failed_chrdevreg (very important in the kernel) 
* Kernel needs to be able to handle it's own errors 
* Needs to free up allocated resources, "undo" everything 
 
* "." is the current directory 
* ".." is the parent directory 
** Introduces another hardlink 
 
* In order to build kernel modules, you need to have the headers associated with the current kernel you're running
* Modules are specified to a particular version of the kernel

Operating Systems 2017F Lecture 16

2017-11-24T08:37:53Z

Krithikasaravanan:

==Video==

The video from the lecture given on Nov. 7, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec16-09Nov2017.mp4 is now available].

==Code==

[http://homeostasis.scs.carleton.ca/~soma/os-2017f/code/lec16/newgetpid.zip Code for download]

==newgetpid.c==

<source lang="C" line>
/* Code derived from:
https://appusajeev.wordpress.com/2011/06/18/writing-a-linux-character-device-driver/
and
http://pete.akeo.ie/2011/08/writing-linux-device-driver-for-kernels.html
*/

#include <linux/module.h>
#include <linux/string.h>
#include <linux/fs.h>
#include <linux/device.h>
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/sched.h>
#include <asm/uaccess.h>

#define dbg(format, arg...) do { if (debug) pr_info(CLASS_NAME ": %s: " format, __FUNCTION__, ## arg); } while (0)
#define err(format, arg...) pr_err(CLASS_NAME ": " format, ## arg)
#define info(format, arg...) pr_info(CLASS_NAME ": " format, ## arg)
#define warn(format, arg...) pr_warn(CLASS_NAME ": " format, ## arg)

#define DEVICE_NAME "newgetpid"
#define CLASS_NAME "comp3000"

static struct class* newgetpid_class = NULL;
static struct device* newgetpid_device = NULL;
static int newgetpid_major;

static int newgetpid_open(struct inode *the_inode, struct file *f)
{
return 0;
}

static ssize_t newgetpid_read(struct file *f, char *buf, size_t len, loff_t *offset)
{
size_t i, msglen;
pid_t thepid;

char message[100];

if (*offset > 0) {
return 0;
}

thepid = task_tgid_vnr(current);

snprintf(message, 100, "Your PID is %d!\n", thepid);

msglen = strlen(message);

if (len < msglen) {
msglen = len;
}

for (i = 0; i < msglen; i++) {
put_user(message[i], buf++);
}

*offset = i;

return i;
}

static int newgetpid_release(struct inode *the_inode, struct file *f)
{
printk(KERN_ALERT "Newgetpid device closed\n");
return 0;
}

static struct file_operations newgetpid_fops = {
.open = newgetpid_open,
.read = newgetpid_read,
.release = newgetpid_release,
};

static char *newgetpid_devnode(struct device *dev, umode_t *mode)
{
if (mode)
*mode = 0444;
return NULL;
}

static int __init newgetpid_init(void)
{
int retval;

newgetpid_major = register_chrdev(0, DEVICE_NAME, &newgetpid_fops);
if (newgetpid_major < 0) {
err("failed to register device: error %d\n", newgetpid_major);
retval = newgetpid_major;
goto failed_chrdevreg;
}

newgetpid_class = class_create(THIS_MODULE, CLASS_NAME);
if (IS_ERR(newgetpid_class)) {
err("failed to register device class '%s'\n", CLASS_NAME);
retval = PTR_ERR(newgetpid_class);
goto failed_classreg;
}

newgetpid_class->devnode = newgetpid_devnode;

newgetpid_device = device_create(newgetpid_class, NULL, MKDEV(newgetpid_major, 0),
NULL, DEVICE_NAME);

if (IS_ERR(newgetpid_device)) {
err("failed to create device '%s'\n", DEVICE_NAME);
retval = PTR_ERR(newgetpid_device);
goto failed_devreg;
}

info("Newgetpid device registered using major %d.\n", newgetpid_major);

return 0;

failed_devreg:
class_unregister(newgetpid_class);
class_destroy(newgetpid_class);
failed_classreg:
unregister_chrdev(newgetpid_major, DEVICE_NAME);
failed_chrdevreg:
return -1;
}

static void __exit newgetpid_exit(void)
{
device_destroy(newgetpid_class, MKDEV(newgetpid_major, 0));
class_unregister(newgetpid_class);
class_destroy(newgetpid_class);
unregister_chrdev(newgetpid_major, "newgetpid");
info("Unloading Newgetpid module.\n");
return;
}

module_init(newgetpid_init);
module_exit(newgetpid_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Anil Somayaji <soma@scs.carleton.ca>");
MODULE_DESCRIPTION("A write newgetpid character device module");
</source>

==Makefile==

<source lang=make line>
obj-m := newgetpid.o
KDIR := /lib/modules/$(shell uname -r)/build
PWD := $(shell pwd)
default:
$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
</source>

=== Additional Notes ===

What determines what files you can and cannot create? -> Ssh privileges 

* Anything that you can do as an ssh user you can do in the vm, just doing file operations 
* Sshfs means when you do read and write system calls for programs in a directory it also does it on the remote machine 
* Ssh is a good tool to access remote files locally, we will use it to edit modules 
 
Ones/newgetpid program: 
* We want to extend its functionality in a specific way 
* We want to access info about the process that made the system call 
* Let's get current processes id 
** Normally we would use getpid 
** But we can't make system calls in kernel space 
** But we can call the function that the system call uses or just copy the functionality 
* Code for this is in "kernel/sys.c" 
 
Syscall_define0: 
* Macro that expands 
* Defines system calls with "getpid" and takes no arguments 
 
* Can use the code inside the function but not the function itself in the kernel 
* Getpid returns a pid_t 
* Instead of get_ones returning all those ones we want it to return the pid 
* Lets try to get it to output a basic string with the pid 
 
How does printk work? 
* Printk sends its output to the kernel log 
* We changed the name to "newgetpid" 
* How do we convert int to string to print the pid? 
** Make a buffer, let's call it "message" 
 
Why does the pid keep incrementing each time we call "cat /dev/newgetpid"? 
* "Cat" spawns a process so every time we do a fork we get a new pid 
 
Why can we use snprintf but not printf? 
* We include "linux/kernel.h" which defines snprintf but not printf 
* Printf assumes we have a standard output to print to 
* Snprintf only needs character arrays for it to work 
 
* Read functionality uses an API 
* Adding new functionality like "write" is easy, just look at the standard API and original kernel source 
* All device files have their own custom read and write functions 
 
Why do we use goto's? 
* C has no exception handling functionality so we implement our own 
* Jumps to error paths: failed_devreg, failed_classreg, failed_chrdevreg (very important in the kernel) 
* Kernel needs to be able to handle it's own errors 
* Needs to free up allocated resources, "undo" everything 
 
* "." is the current directory 
* ".." is the parent directory 
** Introduces another hardlink 
 
* In order to build kernel modules, you need to have the headers associated with the current kernel you're running
* Modules are specified to a particular version of the kernel

Operating Systems 2017F Lecture 19

2017-11-21T19:21:30Z

Krithikasaravanan:

== [Notes] ==

Sample
sample 

=== Additional Notes ===

Where's main?
* lots of program shave "main" functions - a function that runs first and controls the execution of the program 
* Do these have "main" functions? 
** Linux kernel modules 
** FUSE applications? 
** the linux kernel? 
** node web applications? 

In many systems, "main" just sets up even handlers 
* the event loop can be implicit or explicit 
** or there may be no loop at all, just handlers and "interrupts" some kind 
* event loops poll (check) to see when there are new events 
* what are event loops for node app? 
** where are interrupts for node apps? 
***Incoming network requests, it's an event 

Code run differently in the kernel : 

1)functions runs on the bhealf of insmod, unles sit is Independence context 

2)codes that run on the bhelaf o the process 

3)after an interrupt: no process , it is an interrupt cotext 

4) file names : regular programs but the square brackets, execution context + address space. they share the kernel's address space, they are called kernel threads which are independently scheduling . You can not kill them but you can change their scheduling , maybe their priority but not 100%. 

does it create a proces? no , but it can create a kernel thread (is it a process? virtual adress space, . 
multi- threaded: maintains multiple address processes , ex: fire fox. 

ps -elF | less "number" : displays threads.

top : displays all the processes 

ls time : shows you the time .

sys: how much time in the kernel space
real: how much time
user : how much time in user space

process : can't manipulate its own memory map directly, it has an address space, but cant change it. Process: is limited but the kernel is not and the kernel can change it's own address and in charge of its self. 

Kernel tasks : are threads, when a process makes a system call , thi sis schedules in the process priority. 

When running 
"time ls" 
real = realtime it took to run
user = the user space time
sys = kernel time

What's flow of control in tutorial 7? 
What is the connection? 
To exit the program, we must unmount the filesystem, run "sudo umount mnt" 

OS kernels are essentially the same thing 

what is the flow of control ? what connection between things we are doing in the new terminal and the old one 

-> these programs are communicating to each other, it will be invoked when we use mnt (mount) the kernel knows it is a filesystem , process runs system calls, then kernel talks to out original terminal. How? you can use strace to know, it is waiting to be invoked, to receive and responds to messages. events will be passed off to another process. switching between one process to another. it has potential security benefits. 

Key to understand this tutorial : 

-> understand net flow control. 

-> how do processes communicate? 

-> how does it take a directory int and creates a filesystem from it?: 

***sub tree starting at mount is delegated to this process. 
***permissions are limited 

How to kill it? 

-> ctrl c , no , you can but the kernel will be unhappy. 
-> unmount the file system when you are done using it 
-> umount / you have to do it as root 

Node web applications 
* Theres not a dedicated main function but its the first thing that runs 
* Every line in the node application terminates 
* If you start a web server, the function call starting it will terminate 
* Whats running? The main has finished. Is the program doing anything actively if theres no external input? 
* In many systems main just sets up event handlers 
* The event loop can be implicit or explicit 
* Its possible to have no loop at all just handlers and interrupts of some kind 
* Event loops check to see when there are new events 
* Os kernels behave the same way as node applications 
* Waiting for events 
 

What happens when you use insmod? 
* In newgetpid: Init and exit are called when the insmod program makes system calls to load the module 
* Init is run on behalf on insmod 
* Kernel code fits into 3 categories 
** Code that runs on behalf of a process 
** Code that runs after an interrupt 
** Kernel threads, has lots of functionality 
 
Why is a kernel thread not a process? 
* The kernel maintains its own address space 
* Has its own virtual address map 
* The kernel always has just 1 address space regardless of how many threads there are 
* Processes cannot manipulate its own memory map directly, needs to ask kernel first 
* Processes are limited, the kernel is not, has control over itself 
 
* Kernel tasks = kernel threads 
* Independently scheduled 
* Once we call insmod everything happens in user space 
* Strace uses a system call called ptrace 
 
Tutorial 7: Fuse 
* Memoryll.py program 
* Always unmount a filesystem when youre done using it 

*No class Thursday! no office hours on Wednesday

Operating Systems 2017F Lecture 18

2017-11-16T21:05:20Z

Krithikasaravanan:

==Video==

The video from the lecture given on Nov. 16, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec18-16Nov2017.mp4 is now available].

==Notes==

===In Class===
<pre>
Lecture 18: Filesystems and such
--------------------------------
* How can you recover a filesystem?
* How do you delete a file?

A filesystem is
* persistent data structure
* stored in fixed-sized blocks (at least 512 bytes in size)
* maps hierarchical filenames to file contents
* has metadata about files (somehow)

What's in a filesystem?
* data blocks
* metadata blocks

How do you organize metadata?

First job: identify basic characteristics of the filesystem

You need a "summary" block that tells you about everything else
=> this is the "superblock"

Normally the superblock is the first block of the filesystem

In the superblock
- what kind of filesystem is this?
- what filesystem magic number is there
- how big is the filesystem?
- how is it organized?
- where can I find the rest of the metadata?

for POSIX filesystems
- file metadata is stored in...inodes
- most have pre-reserved inodes

So we have
- superblock
- inode blocks
- data blocks
- data blocks for directories
- data blocks for files

How do you recover from damage?
- filesystems never "reboot", must remain correct over
the course of years
- but errors will happen
- bitrot
- "accidental" corruption
- computer failure/memory corruption/hard reboot

To make filesystems fast, data & metadata is cached in RAM
- bad things happen if this data hasn't been written to disk and you reboot
- even worse things happen if your RAM is bad and corrupts the data

Also bad...what if we lose the superblock?
- you could lose EVERYTHING
- so we have backup superblocks

Old scandisk/fsck was slow because they had to scan all filesystem metadata
- not to recover data, but to fix metadata

Nowadays fsck is very fast and we rarely lose data due to losing power
- we must be writing data to disk all the time
- but isn't writing all the time slow?

On magnetic hard disks (not SSDs)
- sequential operations are fast
- random access is slow
- we have to move the read/write head

So, on modern systems we update metadata (and sometimes data) by writing
sequentially to disk...and then later writing randomly
- sequential writes go to the "journal"

On fsck on a journaled filesystem
- just check the journal for pending operations (replay the journal)

There exist filesystems that are pure journal
- log-based filesystem

logs and journal inherently create multiple copies of data and metadata that are hard to track. This makes deletion nearly impossible (at least to guarantee)

Only way to guarantee...encrypt everything
- if every file has its own key, you can delete the key and thus "delete" the data

Solid State Disks (SSD) use log-structured storage at a level below blocks.
- writes are coarse-grained (you have to write a lot at once)
- you don't want to write to the same cells too often, they'll die
- have to do "wear-leveling"
</pre>

Additional notes :

Lecture 18, november 17

Comp 3000:
Midterm review:
2)mmap is called in dynamically linked libraries
Work on 2404 assignment

8) yes , using mmap
9) shell did , because it has to open a new file
10) mmap allocates the entire file ,
11) after the fork , the memory won’t be shared, no communication will happen.
12) no, race condition, busy wait , spin lock in the kernel. Some other user can modify the code which we are waiting. Before we decrement someone else will modify it. Someone can change with the semantics.
How do you do kernel hacking?
1) Be humble,
o you don’t necessary know everything, everyone is retarded in their own way
2) Verify you assumptions
o By experiments
o Compile and run
3) Check for errors!
o Saves time
o Kernel has to live “ cleanly”
4) Find another part of the kernel that is close to what you want to do
o Use their ideas to apply yours by analyzing their code.
o Follow their pattern “pattern match” to avoid problems since you may not understand all the abstractions and assumptions
o Realize if their assumptions match yours
5) Understand the “flow of control” in the program
o Architecture
o Division of responsibilities
o Division Why does this matter?
o Possible to make a module and run in the background! Anything is possible! All you have to do is be creative. In fact you have to do a kernel thread
 When does the Ethernet card receives data?
o The Ethernet card sends an interrupt to the kernel
o The CPU calls the kernel code for handling Ethernet data
 When does the kernel run?
o The kernel gets woken up for those events
o The clock generates a timer interrupt
 Interrupt requires a CPU score to be taken over
o Core was probably running a user space process and this is scheduling is about
 Schedule: what to do after having kicked a user space process off a core.
o Can it be a complex CPU algorithm : no since it always sending interrupts?
o Determines what is the next task to do?
 Normally on a core:
o A running userspace process
o Interrupt happens
o Core switches to supervisor mode, runs kernel code
o Last part of the kernel code is the scheduler, chooses which userspace code to run
o Goto top
 Kernel is entered via interrupts, exited via scheduler
 Entry and exit the kernel has to do low-level tasks
o Uses assembly code
 Limits because it is hard to manage
On the website:
 Arch : arch culture specific and the driver specific code,
 Entry_64.5 : before the system call it calls this. Don’t mess up with it. It take cares of dispatching the system calls.
 Shced.h: what the kernel uses to keep track of processes , go through it
 :1 means bitfields in c
 What criteria should the scheduler use?
o “fairness” : everyone should get a turn, everyone gets to share the CPU.
o Starvation : a term when a program does not get the CPU
o Prevent starvation!
o Equal share of resources.
o Why would not you want your scheduler being fair? To avoid “foreground” tasks in the interactive systems.
o Never enough biased towards “ foreground” tasks
o Series of hacks and heuristics

 Memory is allocated lazily is the kernel :
o which means it is possible to allocate way more memory that can be ever used.
o This will cause “memory debt”
o Out of memory killer : killing process when you exceed the amount of memory(ex: shoots whoever deposits the money)

Continuation of lecture 18:

Important notes:
How can you recover a filesystem?
How do you delete a file?
What is a filesystem?:
*persistent data structure
* stored in fixed-sized blocks (at least 512 bytes in size)
*maps hierarchical filenames to file contents
*has metadata about files (somwhow)
What is in a filesystem?
*data blocks
*metadata blocks
How do you organize metdata:
1) First you must identify characteristics of the file system
Superblock : summary y block which tells you about the other blocks you have and it depends on which file system you have. It’s usually the first block of a file system.
In the superblock? :
1) What kind of file system is this? By checking what is the magic number it has
2) How big is the file system?
3) How is it organized?
4) Where can I find the rest of the metadata?

*How can you identify which file system it is from looking at the super class
-> google “magic number of a file”
-> ex: jpg ctr^c ctr^c : switched the pictures into a binary file
-> look at the beginning of the file you will see JFIF: first several bytes in general that identifies the type of the file (magic number)
File extension :
 what is it ?
 is it important
 the kernel does not know and not care about it

For POSIX file systems:
-.> file metadata is stored in inodes
-.> most have pre-reserved inodes
-> the only way you can run out of inodes if you keep creating small files

Usenet : al the things you use to post messages thro social media, email, etc. Those were made using Usenet. Like email but Local Usenet server. But it died over time. Every message is stored in an individual file.

Important commands:
File * : to identify the kind of file system
1. As : Run dumpe2fs foo. What does the output of this command mean?
 Does this give you info about the file system?
 File bar : bar is the file name and cp comp3000-midterm-2017.pdf bar
 Evince bar : opens up the pdf file

===Additional Notes===
Lec 18 
* More on filesystems 
* How can you recover a fs and how do you delete a file? 
 
A filesystem is a: 
* Persistent data structure 
* Stored in fixed size blocks (at least 512 bytes in size) 
* Maps hierarchical filenames to file contents 
* Has metadata about files somehow 
 
What's in a filesystem 
* data blocks (stores file content) 
* metadata blocks, you need someway to find the blocks 
 
How do you organize metadata? 
 
First identify basic characteristics of the filesystem 
- How big is the filesystem? 
- What is the block size? 
 
How do we differentiate between this and other filesystems?
 
You need a "superblock" which is a "summary" block that tells you about everything else 
 
- Format depends on filesystem 
- Normally the superblock is the first block of the filesystem 
 
- Think of it almost like the root of a binary tree 
In the superblock 
* Type of filesystem 
** What filesystem magic number is there (lets us identify one filesystem from another just by looking at the first block) 
** File command to know file type 
* Size of the filesystem 
* How the filesystem is organized (different filesystems organize their data differently) 
* Where can I find the rest of the metadata 
He opened a .jpg as a binary file to show us the magic number in a file, first several bytes identify type of file. 
- Kernel does not care about file extension but userspace programs may care about the extension. 
- File extensions are only really useful for the people looking at them
- Typical for binary file formats to have a set of bytes that identify the type of file
 
POSIX is a standard for maintaining compatibility between operating systems 
- QNX, UNIX, MacOS are all POSIX compliant
- Others comply on a varying scale
For POSIX filesystems 
* File metadata is stored in INODES 
* When you create a filesystem, certain blocks are dedicated to being INODES 
- Possible to have space in your filesystem without being able to store things if you run out of INODES 
 
What is usenet?
- A worldwide distributed discussion system (stone age version of reddit) 
- Deprecated now because it could not handle the spam people uploaded into it, lol 
- Format for usenet was every message stored in its own file => lots of small files 
- Everyone has a local usenet server, access to read posts on the forum 
- To post a message, send to local server which replicates it and sends it to all the other servers 
So we have: 
* superblock 
* inode blocks 
* data blocks 
** data blocks for directories 
** data blocks for files 
 
How do you recover from damage? 
* Filesystems never "reboot", must remain correct over time 
* Errors will happen: bitrot (when bits change), accidental corruption, computer failiure/memory corruption/hard reboot 
 
To make filesystems fast, data and metadata are cached in RAM 
* Bad things happen if this data hasn't been writen to disk and you reboot 
* Even worse things happen if your RAM is bad and corrupts the data 
* FSCK is like scandisk in Windows 98 (this only happens when you do a hard reset)
 
What happens if you lose the superblock? 
* You could lose EVERYTHING 
* Node trunc dd command blew away first bytes of the file system so you could not mount it because you corrupted the superblock. However, fsck fixed this because we have backup superblocks :D 
- Most filesystems keep copies of the superblock in random locations throughout which takes up some unusable amount of data 
- But this is an impractical way to deal with data blocks 
 
Old scandisk/fsck was slow because it had to scan all filesystem metadata, this is bad since we may lose power before it finished running 
* Not to recover data, but to fix metadata 
* lost+found directory might have some files that you can recover 
 
What is lost+found? 
- Part of the filesystem for fsck to use, dedicated directory 
- Can find nodes that appear to be allocated but has no associated filename, no hardlinks, is inaccessible 
- If you run fsck and it returns error messages you may be able to recover them by looking in lost+found 
- Almost useless for modern filesystems 
 
Nowadays fsck is very fast and we rarely lose data due to losing power 
* What this means is we must be writing to disk all the time 
* But isn't writing slow? => Not necessarily, all writes aren't the same 
 
On Magnetic Hard Disks (not SSD's) 
* sequential oeprations are fast 
* random access is slow 
** we have to move the read/write head 
 
So, on modern systems we update metadata (and sometimes data) by writing sequentially to disk...and then later writing randomly (which means were actually writing twice) 
* Sequential writes go to the journal 
 
On fsck on a journaled filesystem
* Just check the journal for pending operations (replaying the journal) 
* There exists filesystems for optimizing writes that are pure journal 
* Log based filesystem 
 
How to delete things? 
* Compact it, we don't really know how many copies are stored in the filesystem
 
Logs and journal inherently create multiple copies of data and metadata that are hard to track. This makes deletion nearly impossible (at least to guarantee) 
 
Only way to guarantee...encrypt everything 
* If every file has its own key, you can delete the key and this technically deletes the data 
* Only way to recover data is to break encryption scheme (this is nearly impossible)
 
SSDs use log-structured storage at a layer below the regular filesystem 
* Writes are coarse-grained (efficient for writing large amounts at once) 
* You don't want to write to the same cells too often, they will die 
** Instead spread out where you write data => "wear-leveling" 
 
* All modern Intel cpus have a management chip that manages power and such 
** Runs a small operating system called minux (this can also be compromised)

Operating Systems 2017F Lecture 18

2017-11-16T20:29:30Z

Krithikasaravanan:

Operating Systems 2017F Lecture 15

2017-11-08T15:13:26Z

Krithikasaravanan: /* Additional */

==Video==

The video from the lecture given on Nov. 7, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec15-07Nov2017.mp4 is now available]. Unfortunately, the video cut out halfway through; [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec15-07Nov2017-audio.m4a audio is also available] however.

==Notes==

===In Class===

<pre>
Lecture 15
----------

What's a filesystem?
- persistent data structure organized around fixed allocation units (blocks)
- maps hierarchical names (keys) to values
- provide a file-like API (open, read, write, close, etc)

What does it mean to "make" a filesystem?
- initialize the data structure
- "formatting" a disk

Physical versus logical size of a file
- logical: the "size" your program sees when accessing the file
- physical: how much space the file takes up on disk

Physical is in terms of blocks - fixed units of storage allocation
- ext4 has 4k blocks
- default for many command line utilities is 1k blocks

Kernel programming
- you may destroy the system you are working on AT ANY TIME
- HAVE GOOD BACKUPS
- rsync is your friend

Kernel modules
- way of splitting up kernel functionality so everything doesn't have to load
at boot
- code loaded as part of the initial boot image is hard to get rid of at
runtime
- why do we need modules? why not processes?
- no new mechanisms
- increased security (restricted access)
- "microkernel" approach
- instead of code talking in supervisor mode, processes do IPC
- filesystems
- drivers
- networking
- minix, QNX, GNU hurd
- Linux is a "monolithic" kernel. Why?
- performance: context switches are expensive
- techniques to make microkernels fast can be adopted by
monolithic kernels to make them even faster
- security benefit is illusory
- if you control the filesystem process, you already own everything
</pre>

===Additional===

--> Core kernel functionality is implemented via modules
--> use ls mod to see modules that are loaded

'''What is a monolithic kernel?'''
--> a type of OS architecture where the entire OS is working in kernel space
--> can dynamically load/unload modules at runtime

'''make localmodconfig:'''
--> takes output of ls mod and configures your kernel

'''ones.c program:'''

'''/dev/ones:'''
--> permissions are read only

'''file_operations ones_fops():'''
--> defines what happens when you open a file, read from it, release tells you what happens when you're done with it (not the same thing as close)

'''ones_read():'''
--> len is the number of bytes to read
--> offset tells you where you are in the file
--> put_user() takes care of whatever needs to be done to write into that process properly

'''ones_release:'''

'''Why are we using printk instead of printf?'''
--> printf is not yet defined (ie. C library is not available in the kernel)
--> kernel doesn't depend on any libraries, all code belongs to the kernel itself
--> printk is the kernel's own implementation of printf (outputs to the kernel log --> /var/log/kern.log)

vfs = virtual filesystem layer

'''How do we limit access to user space processes?'''
--> Do a permission check

--> kernels need to be updated regularly to correct bugs that make the kernel vulnerable to programs trying to gain access to important user space processes
--> unlikely() = tells you that this branch is not likely to be taken, optimize the current path

'''vfs_read:'''

'''file->f_op->read:'''
--> this is how our read function will be called

Operating Systems 2017F Lecture 15

2017-11-08T15:12:28Z

Krithikasaravanan: /* Additional */

==Video==

The video from the lecture given on Nov. 7, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec15-07Nov2017.mp4 is now available]. Unfortunately, the video cut out halfway through; [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec15-07Nov2017-audio.m4a audio is also available] however.

==Notes==

===In Class===

<pre>
Lecture 15
----------

What's a filesystem?
- persistent data structure organized around fixed allocation units (blocks)
- maps hierarchical names (keys) to values
- provide a file-like API (open, read, write, close, etc)

What does it mean to "make" a filesystem?
- initialize the data structure
- "formatting" a disk

Physical versus logical size of a file
- logical: the "size" your program sees when accessing the file
- physical: how much space the file takes up on disk

Physical is in terms of blocks - fixed units of storage allocation
- ext4 has 4k blocks
- default for many command line utilities is 1k blocks

Kernel programming
- you may destroy the system you are working on AT ANY TIME
- HAVE GOOD BACKUPS
- rsync is your friend

Kernel modules
- way of splitting up kernel functionality so everything doesn't have to load
at boot
- code loaded as part of the initial boot image is hard to get rid of at
runtime
- why do we need modules? why not processes?
- no new mechanisms
- increased security (restricted access)
- "microkernel" approach
- instead of code talking in supervisor mode, processes do IPC
- filesystems
- drivers
- networking
- minix, QNX, GNU hurd
- Linux is a "monolithic" kernel. Why?
- performance: context switches are expensive
- techniques to make microkernels fast can be adopted by
monolithic kernels to make them even faster
- security benefit is illusory
- if you control the filesystem process, you already own everything
</pre>

===Additional===

--> Core kernel functionality is implemented via modules
--> use ls mod to see modules that are loaded

'''What is a monolithic kernel?'''
--> a type of OS architecture where the entire OS is working in kernel space
--> can dynamically load/unload modules at runtime

make localmodconfig:
--> takes output of ls mod and configures your kernel

ones.c program:

/dev/ones:
--> permissions are read only

file_operations ones_fops():
--> defines what happens when you open a file, read from it, release tells you what happens when you're done with it (not the same thing as close)

ones_read():
--> len is the number of bytes to read
--> offset tells you where you are in the file
--> put_user() takes care of whatever needs to be done to write into that process properly

ones_release:

Why are we using printk instead of printf?
--> printf is not yet defined (ie. C library is not available in the kernel)
--> kernel doesn't depend on any libraries, all code belongs to the kernel itself
--> printk is the kernel's own implementation of printf (outputs to the kernel log --> /var/log/kern.log)

vfs = virtual filesystem layer

How do we limit access to user space processes?
--> Do a permission check

--> kernels need to be updated regularly to correct bugs that make the kernel vulnerable to programs trying to gain access to important user space processes
--> unlikely() = tells you that this branch is not likely to be taken, optimize the current path

vfs_read:

file->f_op->read:
--> this is how our read function will be called

Operating Systems 2017F Lecture 15

2017-11-08T15:11:47Z

Krithikasaravanan:

==Video==

The video from the lecture given on Nov. 7, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec15-07Nov2017.mp4 is now available]. Unfortunately, the video cut out halfway through; [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec15-07Nov2017-audio.m4a audio is also available] however.

==Notes==

===In Class===

<pre>
Lecture 15
----------

What's a filesystem?
- persistent data structure organized around fixed allocation units (blocks)
- maps hierarchical names (keys) to values
- provide a file-like API (open, read, write, close, etc)

What does it mean to "make" a filesystem?
- initialize the data structure
- "formatting" a disk

Physical versus logical size of a file
- logical: the "size" your program sees when accessing the file
- physical: how much space the file takes up on disk

Physical is in terms of blocks - fixed units of storage allocation
- ext4 has 4k blocks
- default for many command line utilities is 1k blocks

Kernel programming
- you may destroy the system you are working on AT ANY TIME
- HAVE GOOD BACKUPS
- rsync is your friend

Kernel modules
- way of splitting up kernel functionality so everything doesn't have to load
at boot
- code loaded as part of the initial boot image is hard to get rid of at
runtime
- why do we need modules? why not processes?
- no new mechanisms
- increased security (restricted access)
- "microkernel" approach
- instead of code talking in supervisor mode, processes do IPC
- filesystems
- drivers
- networking
- minix, QNX, GNU hurd
- Linux is a "monolithic" kernel. Why?
- performance: context switches are expensive
- techniques to make microkernels fast can be adopted by
monolithic kernels to make them even faster
- security benefit is illusory
- if you control the filesystem process, you already own everything
</pre>

===Additional===

--> Core kernel functionality is implemented via modules
--> use ls mod to see modules that are loaded

What is a monolithic kernel?
--> a type of OS architecture where the entire OS is working in kernel space
--> can dynamically load/unload modules at runtime

make localmodconfig:
--> takes output of ls mod and configures your kernel

ones.c program:

/dev/ones:
--> permissions are read only

file_operations ones_fops():
--> defines what happens when you open a file, read from it, release tells you what happens when you're done with it (not the same thing as close)

ones_read():
--> len is the number of bytes to read
--> offset tells you where you are in the file
--> put_user() takes care of whatever needs to be done to write into that process properly

ones_release:

Why are we using printk instead of printf?
--> printf is not yet defined (ie. C library is not available in the kernel)
--> kernel doesn't depend on any libraries, all code belongs to the kernel itself
--> printk is the kernel's own implementation of printf (outputs to the kernel log --> /var/log/kern.log)

vfs = virtual filesystem layer

How do we limit access to user space processes?
--> Do a permission check

--> kernels need to be updated regularly to correct bugs that make the kernel vulnerable to programs trying to gain access to important user space processes
--> unlikely() = tells you that this branch is not likely to be taken, optimize the current path

vfs_read:

file->f_op->read:
--> this is how our read function will be called

Operating Systems 2017F Lecture 16

2017-11-07T19:40:20Z

Krithikasaravanan:

In Class
Comp 3000
Lecture 16
Important notes: Tutorial 5:
File system :
 persistent data structure organized around blocks (which are fixed allocation units)
 maps hierarchal names (keys) to values
 provides a file-like API like open, read, write, close,etc
What does it mean to “make” a file system?
 Initializing a data structure.
 “formatting” a disk
Physical vs Logical : logical size of a file: the size your program see when accessing the file (bytes in a file)
Physical : How much space it takes up on disk , in terms of blocks , fixed units of storage allocation

Physical :
 By default or for multiple of files it is 1K blocks
 Example : Ext4 has 4k blocks

Kernel Programing :
 Warning:
o If you use linux base , you may crash your whole system, just backup before you do so using “rsync”.
Open Stack : log in through the terminal using your instance’s Ip address , but it failed to work . when you ssh to it you must write ssh “Address” –l Ubuntu
 You are required to use sude to add a user name, so u can play around in root.
What is a Kernel module ?
 A way of splitting up kernel functionality so everything does not have to load at boot.
 Modifies a kernel functionality
 Runs in kernel space , is the key thing to think about
o It is more powerful than root and it can do anything
o Access to all kernel memory
o And you can modify everything
 If you miss anything in the kernel development your system will crash
 Kernel machine provides you with a floppy by default which explains why it still exists in Anil’s terminal
 Once you install a module , the module is unstrained
Why do we use modules? Why don’t we load processes instead?
 No new mechanisms
 Increased security (restricted access)
 Makes the kernel less smaller, microkernel design,
o Putting in the functions that are supposed to be in the kernel into processes
o Process do IPC rather than code talking in supervisor mode
 Examples :
• Filesystems
• Drivers
• Networking
• Minix, QNX, GNU, hurd,
 Why is Linux “monolithics” kernel ?
o Switching between contexts are expensive (context switch)
o How to make microkernels fast can be adopted by monolithics kernels to make them even faster
o Unreal security benefits :
 if you control the file system process, you can control everything

Rebuilding and changing the kernel:
 1)Type “make” : more compilacted than 2401
o Kernel built
 2)Make modules
 3)Sudo make install
 4) sudo make-modules install
 5) Sudo shutdown –r now : for the vm to reboot
 Which configuration would you use to build your own kernel ?
o Don’t do configurations from scratch
o Copy the configurations and use them
o Make localmodconfig : output for ls mod and uses that for configuring your kernel
o Requires time and effort
 Why less /dev/ones doesn’t exist anymore?
o Since reboot occurred
o You must load the modules again
o Head –c 100 /dev/ones to be able to use it again
 Implementing the device file of dev 1 ?

o Implement the file API required
o Teach the kernel what it means to do operations like read, etc
Code from the tutorial ones.c:
 Open ones_read code: file descriptor, file , buffer, amount of bytes to read and offset
o Offset : position in the file
o Fills the buffer with ones
o Why don’t we just set it to 1 instead of putting put_user?
 Char *buf : Pointer for a user space process, in order for the kernel to write to user spacer safely
o Line 46: Why use printk and not printf? Since printf is not defined because the c library is not available in the kernel, how can you can c library when the c library depends on the kernel? Kernel is independent, does not depend on any libraries.

Commands:
Man ls : to see different ls commands
 Ls –las block
 Cat /dev/ones |less : it is like dev u random , but instead of generating random number, it instead generates infinite of number 1
 Ls –mod : displays all the moduls which are currently loaded on the virtual machine
 IBM ps/2 : series of computers created to control PC, developed the interfaces to have a mouse and keyboard.
 Less readme : to check instructions of how to do a make
 Make menuconfig : options of kernel configurations
 Cat/pro
 Less .config : bad idea to go directly in it , use “make menuconfig “
 /boot : where the kernel got installed.
o Ls –lah : to see the size
 Less/ etc/modules
 Modul init: what function should be called when loaded and when it is unloaded
 Modul exit:
 Creating a device file : defining a file which has special semantics, define a struct and functions which should be called to explain each file operation, open , read, release(like closing but not really)
 What happens if you start running to the file ? permission are read only, not writing
o Override that? Still your permission is denied, you can only read since we didn’t write a function in the struct to write.

'''Additional notes:'''
--> Warning: it's possible to destroy your entire system with 1 command. Solution: have good backups
--> Core kernel functionality is implemented via modules
--> use ls mod to see modules that are loaded
--> In practice, you load modules all at once

'''Why do we need to load code into the kernel anyways?'''
--> more secure, ability to restrict access

--> examples of microkernel: minix (predecessor to Linux), QNX, GNU hurd
--> once you install a module, it's unrestrained

'''What is a monolithic kernel?'''
--> a type of OS architecture where the entire OS is working in kernel space
--> can dynamically load/unload modules at runtime

'''make localmodconfig:'''
--> takes output of ls mod and configures your kernel

'''ones.c program:'''

'''/dev/ones:'''
--> permissions are read only

'''file_operations ones_fops():'''
--> define what happens when you open a file, read from it, release tells you what happens when you're done with it (not the same thing as close)

'''ones_read():'''
--> len = number of bytes to read
--> offset tells you where you are in the file
--> put_user(): takes care of whatever needs to be done to write into that process properly

'''ones_release:'''

'''Why are we using printk instead of printf?'''
--> printf is not yet defined (ie. C library is not available in the kernel)
--> kernel doesn't depend on any libraries, all code belongs to the kernel itself
--> printk is the kernel's own implementation of printf (outputs to the kernel log --> /var/log/kern.log)

vfs = virtual filesystem layer

'''How do we limit access to user space processes?'''
--> Do a permission check

--> kernels need to be updated regularly to correct bugs that make the kernel vulnerable to programs trying to gain access to important user space processes
--> unlikely() = tells you that this branch is not likely to be taken, optimize the current path

'''vfs_read:'''

'''file->f_op->read:'''
--> this is how our read function will be called

Operating Systems 2017F Lecture 16

2017-11-07T19:39:58Z

Krithikasaravanan:

In Class
Comp 3000
Lecture 16
Important notes: Tutorial 5:
File system :
 persistent data structure organized around blocks (which are fixed allocation units)
 maps hierarchal names (keys) to values
 provides a file-like API like open, read, write, close,etc
What does it mean to “make” a file system?
 Initializing a data structure.
 “formatting” a disk
Physical vs Logical : logical size of a file: the size your program see when accessing the file (bytes in a file)
Physical : How much space it takes up on disk , in terms of blocks , fixed units of storage allocation

Physical :
 By default or for multiple of files it is 1K blocks
 Example : Ext4 has 4k blocks

Kernel Programing :
 Warning:
o If you use linux base , you may crash your whole system, just backup before you do so using “rsync”.
Open Stack : log in through the terminal using your instance’s Ip address , but it failed to work . when you ssh to it you must write ssh “Address” –l Ubuntu
 You are required to use sude to add a user name, so u can play around in root.
What is a Kernel module ?
 A way of splitting up kernel functionality so everything does not have to load at boot.
 Modifies a kernel functionality
 Runs in kernel space , is the key thing to think about
o It is more powerful than root and it can do anything
o Access to all kernel memory
o And you can modify everything
 If you miss anything in the kernel development your system will crash
 Kernel machine provides you with a floppy by default which explains why it still exists in Anil’s terminal
 Once you install a module , the module is unstrained
Why do we use modules? Why don’t we load processes instead?
 No new mechanisms
 Increased security (restricted access)
 Makes the kernel less smaller, microkernel design,
o Putting in the functions that are supposed to be in the kernel into processes
o Process do IPC rather than code talking in supervisor mode
 Examples :
• Filesystems
• Drivers
• Networking
• Minix, QNX, GNU, hurd,
 Why is Linux “monolithics” kernel ?
o Switching between contexts are expensive (context switch)
o How to make microkernels fast can be adopted by monolithics kernels to make them even faster
o Unreal security benefits :
 if you control the file system process, you can control everything

Rebuilding and changing the kernel:
 1)Type “make” : more compilacted than 2401
o Kernel built
 2)Make modules
 3)Sudo make install
 4) sudo make-modules install
 5) Sudo shutdown –r now : for the vm to reboot
 Which configuration would you use to build your own kernel ?
o Don’t do configurations from scratch
o Copy the configurations and use them
o Make localmodconfig : output for ls mod and uses that for configuring your kernel
o Requires time and effort
 Why less /dev/ones doesn’t exist anymore?
o Since reboot occurred
o You must load the modules again
o Head –c 100 /dev/ones to be able to use it again
 Implementing the device file of dev 1 ?

o Implement the file API required
o Teach the kernel what it means to do operations like read, etc
Code from the tutorial ones.c:
 Open ones_read code: file descriptor, file , buffer, amount of bytes to read and offset
o Offset : position in the file
o Fills the buffer with ones
o Why don’t we just set it to 1 instead of putting put_user?
 Char *buf : Pointer for a user space process, in order for the kernel to write to user spacer safely
o Line 46: Why use printk and not printf? Since printf is not defined because the c library is not available in the kernel, how can you can c library when the c library depends on the kernel? Kernel is independent, does not depend on any libraries.

Commands:
Man ls : to see different ls commands
 Ls –las block
 Cat /dev/ones |less : it is like dev u random , but instead of generating random number, it instead generates infinite of number 1
 Ls –mod : displays all the moduls which are currently loaded on the virtual machine
 IBM ps/2 : series of computers created to control PC, developed the interfaces to have a mouse and keyboard.
 Less readme : to check instructions of how to do a make
 Make menuconfig : options of kernel configurations
 Cat/pro
 Less .config : bad idea to go directly in it , use “make menuconfig “
 /boot : where the kernel got installed.
o Ls –lah : to see the size
 Less/ etc/modules
 Modul init: what function should be called when loaded and when it is unloaded
 Modul exit:
 Creating a device file : defining a file which has special semantics, define a struct and functions which should be called to explain each file operation, open , read, release(like closing but not really)
 What happens if you start running to the file ? permission are read only, not writing
o Override that? Still your permission is denied, you can only read since we didn’t write a function in the struct to write.

Additional notes:
--> Warning: it's possible to destroy your entire system with 1 command. Solution: have good backups
--> Core kernel functionality is implemented via modules
--> use ls mod to see modules that are loaded
--> In practice, you load modules all at once

'''Why do we need to load code into the kernel anyways?'''
--> more secure, ability to restrict access

--> examples of microkernel: minix (predecessor to Linux), QNX, GNU hurd
--> once you install a module, it's unrestrained

'''What is a monolithic kernel?'''
--> a type of OS architecture where the entire OS is working in kernel space
--> can dynamically load/unload modules at runtime

'''make localmodconfig:'''
--> takes output of ls mod and configures your kernel

'''ones.c program:'''

'''/dev/ones:'''
--> permissions are read only

'''file_operations ones_fops():'''
--> define what happens when you open a file, read from it, release tells you what happens when you're done with it (not the same thing as close)

'''ones_read():'''
--> len = number of bytes to read
--> offset tells you where you are in the file
--> put_user(): takes care of whatever needs to be done to write into that process properly

'''ones_release:'''

'''Why are we using printk instead of printf?'''
--> printf is not yet defined (ie. C library is not available in the kernel)
--> kernel doesn't depend on any libraries, all code belongs to the kernel itself
--> printk is the kernel's own implementation of printf (outputs to the kernel log --> /var/log/kern.log)

vfs = virtual filesystem layer

'''How do we limit access to user space processes?'''
--> Do a permission check

--> kernels need to be updated regularly to correct bugs that make the kernel vulnerable to programs trying to gain access to important user space processes
--> unlikely() = tells you that this branch is not likely to be taken, optimize the current path

'''vfs_read:'''

'''file->f_op->read:'''
--> this is how our read function will be called

Operating Systems 2017F Lecture 16

2017-11-07T18:03:19Z

Krithikasaravanan: Created page with "In Class"

In Class