Soma-notes - User contributions [en]

Operating Systems 2017F Lecture 15

2017-11-12T18:57:35Z

Ssarazin:

==Video==

The video from the lecture given on Nov. 7, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec15-07Nov2017.mp4 is now available]. Unfortunately, the video cut out halfway through; [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec15-07Nov2017-audio.m4a audio is also available] however.

==Notes==

===In Class===

<pre>
Lecture 15
----------

What's a filesystem?
- persistent data structure organized around fixed allocation units (blocks)
- maps hierarchical names (keys) to values
- provide a file-like API (open, read, write, close, etc)

What does it mean to "make" a filesystem?
- initialize the data structure
- "formatting" a disk

Physical versus logical size of a file
- logical: the "size" your program sees when accessing the file
- physical: how much space the file takes up on disk

Physical is in terms of blocks - fixed units of storage allocation
- ext4 has 4k blocks
- default for many command line utilities is 1k blocks

Kernel programming
- you may destroy the system you are working on AT ANY TIME
- HAVE GOOD BACKUPS
- rsync is your friend

Kernel modules
- way of splitting up kernel functionality so everything doesn't have to load
at boot
- code loaded as part of the initial boot image is hard to get rid of at
runtime
- why do we need modules? why not processes?
- no new mechanisms
- increased security (restricted access)
- "microkernel" approach
- instead of code talking in supervisor mode, processes do IPC
- filesystems
- drivers
- networking
- minix, QNX, GNU hurd
- Linux is a "monolithic" kernel. Why?
- performance: context switches are expensive
- techniques to make microkernels fast can be adopted by
monolithic kernels to make them even faster
- security benefit is illusory
- if you control the filesystem process, you already own everything
</pre>

===Additional===

--> Core kernel functionality is implemented via modules
--> use ls mod to see modules that are loaded

'''What is a monolithic kernel?'''
--> a type of OS architecture where the entire OS is working in kernel space
--> can dynamically load/unload modules at runtime

'''make localmodconfig:'''
--> takes output of ls mod and configures your kernel

'''ones.c program:'''

'''/dev/ones:'''
--> permissions are read only

'''file_operations ones_fops():'''
--> defines what happens when you open a file, read from it, release tells you what happens when you're done with it (not the same thing as close)

'''ones_read():'''
--> len is the number of bytes to read
--> offset tells you where you are in the file
--> put_user() takes care of whatever needs to be done to write into that process properly

'''ones_release:'''

'''Why are we using printk instead of printf?'''
--> printf is not yet defined (ie. C library is not available in the kernel)
--> kernel doesn't depend on any libraries, all code belongs to the kernel itself
--> printk is the kernel's own implementation of printf (outputs to the kernel log --> /var/log/kern.log)

vfs = virtual filesystem layer

'''How do we limit access to user space processes?'''
--> Do a permission check

--> kernels need to be updated regularly to correct bugs that make the kernel vulnerable to programs trying to gain access to important user space processes
--> unlikely() = tells you that this branch is not likely to be taken, optimize the current path

'''vfs_read:'''

'''file->f_op->read:'''
--> this is how our read function will be called

=== From the Text Book (Filesystem Implementation) ===

VSFS
- Illustrates the a typical Unix file system using vsfs (Very Simple File System).
- Introduces some of the basic on-disk structures, access methods, and various policies

The mental model of File Systems
*Want to answer these questions:
**What on-disk structures store the file system's data and metadata?
**What happens when a process opens a file?
**Which on-disk structures are accessed during a read or write?

*The file system is pure software that implements:
**a data-structure to organize its data and metadata.
**access methods - how it maps the calls made by a process (open(), read(), write()) onto its structures

Data structure
*A file system divides the hard disk into blocks, a common sized block is 4KB.
*The majority of the blocks hold user data and their inodes
*Two bitmaps, one of the inode table and the other for the user data blocks track usage
*Finally, a superblock contains information about the filesystem -- how many inodes, datablocks, and type of filesystem

Mounting the filesystem
*The operating system reads the superblock and then attaches the volume to the filesystem tree

File Organization
*inode (index node) number indexes into the inode table to find the desired block of inodes. Once the inode is retrieved all of the corresponding file information is known. Some metadata include: uid (who owns it), size (how many bytes are in the file), gid (which group does the file belong to), blocks ( how many blocks have been allocated to this file) etc…
*inode's refer to data blocks using a multi-level index. Direct pointers and indirect pointers. Direct pointers reference data blocks and indirect pointers reference pointers to data blocks. Adding levels of pointers allows the inode to reference very large files

Directory Organization
*A directory is a list of pairs (entry name, inode number). The entry name is the string representing the directory.
*'dot' is the current directory and 'dot-dot' is the parent directory
*Directories are stored in a file system the same way regular files are - they have an inode. Only directories are marked as such in the inode type field.

Free Space Management
*When a file is created the file system searches the inode bitmap and allocates a free inode to the new file.
*The filesystem uses a pre-allocation policy to allocate data blocks for a new file. It tries to contiguous blocks thus improving performance.

Reading and Writing a file
*Open: the filesystem traverses the pathname and locates the inode for the file. It starts at the root directory (this is known to the filesystem when it is mounted) and traverses the pathname for the file's inode. Once the permissions have been verified a file descriptor to the file is returned to the user. Note that several reads happen during open in order to find the correct inode number.
*Read: The filesystem starts at file offset 0 which it gets from the inode. It updates memory with the file contents and moves the file offset such that the next read knows where to begin. Disk IO is accessed as needed.
*Close: The file descriptor is deallocated.
*Writing: Each write to a file generates five I/O's: one to read the data bitmap, one to write the bitmap, two more to read and then write the inode and finally one to write to the data block.

Caching and Buffering
*Caching is used to eliminate disk IO for future accesses of popular files. Popular files are loaded into virtual memory so that future accesses do not have to go through the process of reading from disk.
*Write buffering is used by the filesystem as a way of minimizing the number of disk accesses. The system builds a buffer of write's and schedules them minimizing IO.

Talk:Operating Systems 2017F Lecture 14

2017-10-28T21:39:06Z

Ssarazin: /* Review Notes */

Operating Systems 2017F Lecture 14

2017-10-28T21:38:46Z

Ssarazin: Created page with "==Review Notes== Definitions: - OS: A resource Manager that abstracts the hardware so that applications can get easy access to it. The OS facilitates resource sharing amon..."

==Review Notes==

Definitions:

- OS: A resource Manager that abstracts the hardware so that applications can get easy access to it. The OS facilitates resource sharing among concurrent programs.
- Kernel: The first program to launch when the computer boots. The kernel has absolute power over the hardware; it manages and controls access to these resources.
- Process: A running program. It has one or more threads and address space.
- Thread: A function that runs in the same memory space as other functions.
- Memory: A contiguous array of bytes. Modern systems virtualize memory, mapping it to physical memory so it can be shared among processes.
- Interrupts: A mechanism used by the kernel that interrupts the CPU when other tasks are higher priority.
- Signals: Inter-process communications. A mechanism used by the kernel and processes to communicate with one another.
- System Call: Specialized functions used by programs to request more resources from the kernel.
- File: A hierarchical name with a value mapping (Key/Value pair)
- Shell: Another program that is used to send system calls to the kernel or launch other programs.
- File system: A set of files grouped together with a common hierarchical root.
- Fragmentation: Variable sized memory allocation.
- Internal Fragmentation: Space wasted in an allocation.
- Concurrency: More than one thing happening at a time.
- Atomic Variables: Variables that guarantee a strict ordering of operations.
- Segmentation: Managing memory without paging.
- Segments: A variable sized block of memory that can be placed in different parts of the address space.
- Segfault: When a virtual address does not have a corresponding physical address.
- mmap: The virtual address range corresponds to the contents of a file or RAM.
- Concurrency: More than one thing is happening at the same time.

Creating a process:
* fork() creates a process and execve() loads the binary to the new process.
** returns a process id
*** pid > 0, parent process
*** pid < 0, error
*** pid = 0, child process

* When a parent does not deal with its child processes they become zombies
* If a parent dies before its children they become orphans that are re-parented by the OS

* wait()
** suspends the current process until one, some, or all of its children have terminated
** wait() can be called anytime to check on the status of children

IO redirection:
* >, direct to stdout
* <, direct to stdin
* |, connect the output of one process to the input of another process
* when a process starts it is given file descriptors to stdout, stdin, and stderr

Inodes:
* Contain information about the blocks corresponding to the contents of a file on disk, its owner, and the file access permissions.
* A filename points to an inode
* The inode count represents all of the files and folders on disk
* Hard links create another name to the same inode number
** Only point to files within the same file system
* Symbolic links are files with their own inode
** Can point to files in other partitions or file systems

Virtual Memory:
* Every process receives a memory map with virtual addresses that correspond to physical addresses.

Memory Allocation Challenges:
* Need to have a contiguous address range
* Data Structures can not be moved once allocated
* Memory is constantly allocated and deallocated

Solution to these challenges:
* Use fixed size chunks that can be stored anywhere
** Called a page in memory or a block on disk
** Sizes are always a power of 2 because addresses are indexed in bits
** The benefit: it removes the limitations on array sizes.
** Use a table to map virtual memory to physical memory -- called a page table
** In a 32 bit system: the top 20 bits are the page locations and the lower 12 bits are the page offset

Concurrency:
* Challenge: keeping the shared state consistent; want deterministic behavior from concurrent computations
* Solution: Need mechanisms that bypass the memory hierarchy.
** Use filesystem locking primitives -> atomic variables -> semaphores
** Use signals to synchronize the shared state

Segmentation:
* The physical address can be found by adding the segment register to the offset.

Talk:Operating Systems 2017F Lecture 14

2017-10-28T21:35:42Z

Operating Systems 2017F Lecture 3

2017-10-05T00:47:32Z

Ssarazin: /* Adding the execve() system call to the mix */

==Video==

Video from the lecture given on September 14, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec03-14Sep2017.mp4 is now available].

==Code==

Code and files from the lecture (captured as they were at the end) are available [http://homeostasis.scs.carleton.ca/~soma/os-2017f/code/lec03/ here].

==Notes==
====Commands ====
*-wall: show all warning
*less: allow screen navigation, hit q to quit

==== What is fork? ====
* Fork creates a new program but contains the same code (call the clone system call) and only argv[] and envp[] passed on to the new born program

- More on creating a process using fork().
<syntaxhighlight lang="c">
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>

int main(int argc, char* argv[], char* envp[]) {
int child;
int status;
pid_t retval;

child = fork();

printf("Hello! I am %d, my parent is %d.\n",
getpid(), getppid());

if (child == 0) {
/* We're the child */
sleep(3);
printf("Child all done!\n");
} else {
/* We're the parent */
retval = wait(&status);
printf("Parent all done! Child %d returned %d.\n",
(int) retval, WEXITSTATUS(status));
}

return 42;
}
</syntaxhighlight>
- sleep(int)
- causes the current process to do nothing for the specified number of seconds.

- pid_t = fork(void)
- fork creates a clone of the parent process
- pid is 0 when the child process is running and >0 when the parent process is running. Both processes will return from the system call to fork().

- pid_t = wait(int * status)
- this system call is equivalent to pid_t = waitpid(-1, &status, 0)
- the wait system call suspends the current process until one of its children has terminated.
- pid_t specifies which child to wait for. Various options are available that define how to wait.
- int* status holds the status of the wait system call. There are various macros that define what the status can be.

- '''What happens when the newline character is removed from the printf statement?'''
- printf holds the output in a buffer until it receives a newline character. The result is the output does not print when expected.
- The kernel is doing things as it is told, so the c library is at fault.
- We could get the expected output if we bypass the c library using a system call: "write"
- running an "strace" on the executable will show us the order in which the processes were run to help debug the problem.

- '''What happens in the conditional?'''
- When the parent process calls fork(), a child process is created and begins executing at the fork() statement. The OS returns a value from fork() which indicates which process
is currently running.
- If fork() returns zero then the child sleeps for 3 seconds before printing. The sleep system call forces the child process to do nothing for 3 seconds.
- if fork() returns anything else, it is assumed to be the parent process. In this case the wait() system call is run by the parent which causes the parent process to wait for its
child to finish; then the printf executes and prints the child process id and its return value from main. The macro WEXITSTATUS retrieves 42.

==== What Happen to the child process after the parent process died? ====
*Every child process must have a currently running parent. When the parent process die, kernel collect the dead body(return value) and re-parent the process
* In the demo showed in class, “system d user” became its new parent
**Use command "Pstree" to show process hierarchy
* Every process must have a valid process parent. Use "pstree" to see the tree structure
* When a parent dies before a child "init" will adopt every orphan on the system

==== What is zombie process? ====
*When a process die it become 'zombie' (s = sleep, z = zombie)
*We can use wait to be a 'good' parent

==== How to load a new binary manually====
*Use the execve command.
**We can fork child process to handle dangerous task (“Suicide Squad”), or handle all incoming request

==== Manipulating environment variables====
*argc tells the number of argument in command line, while argv[] contains the actual command obtained in the command line

=== Adding the execve() system call to the mix ===
<syntaxhighlight lang="c">
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>

int main(int argc, char* argv[], char* envp[]) {
int child;
int status;
pid_t retval;
int i;

printf("You called me with %d arguments.\n", argc);

for (i=0; i<argc; i++) {
printf("Arg %d: %s\n", i, argv[i]);
}

child = fork();

printf("Hello! I am %d, my parent is %d.\n",
getpid(), getppid());

if (child == 0) {
/* We're the child */

execve("/bin/ls", argv, envp);
sleep(3);
printf("Child all done!\n");
} else {
/* We're the parent */
retval = wait(&status);
printf("Parent all done! Child %d returned %d.\n",
(int) retval, WEXITSTATUS(status));
}

return 42;
}
</syntaxhighlight>
- int execve(const char *filename, char *const argv[], char *const envp[])
- this system call executes the program pointed to by filename
- The child process becomes independent from its parent -- a new program
- argv[] is an array of strings passed to the new program. The values are loaded into the array by execve not the kernel.
- envp[] is an array of strings of the form key=value which are passed as environment to the new program

- What happens when the if statement executes
- execve creates a new program -- an independent process and passes arguments to the "ls" program.
- execve stops the new process after it has completed.

- Some interesting points about main(int argc, char* argv[], char* envp[])
- Notice that the size of the arrays are not defined. We have to use argc to get this information. Not good programming practice but is
accepted because it is grandfathered from the early days of the c language.
- arg[0] is always the name of the program.

Operating Systems 2017F Lecture 3

2017-10-05T00:46:48Z

Ssarazin: /* What is fork? */

==Video==

Video from the lecture given on September 14, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec03-14Sep2017.mp4 is now available].

==Code==

Code and files from the lecture (captured as they were at the end) are available [http://homeostasis.scs.carleton.ca/~soma/os-2017f/code/lec03/ here].

==Notes==
====Commands ====
*-wall: show all warning
*less: allow screen navigation, hit q to quit

==== What is fork? ====
* Fork creates a new program but contains the same code (call the clone system call) and only argv[] and envp[] passed on to the new born program

- More on creating a process using fork().
<syntaxhighlight lang="c">
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>

int main(int argc, char* argv[], char* envp[]) {
int child;
int status;
pid_t retval;

child = fork();

printf("Hello! I am %d, my parent is %d.\n",
getpid(), getppid());

if (child == 0) {
/* We're the child */
sleep(3);
printf("Child all done!\n");
} else {
/* We're the parent */
retval = wait(&status);
printf("Parent all done! Child %d returned %d.\n",
(int) retval, WEXITSTATUS(status));
}

return 42;
}
</syntaxhighlight>
- sleep(int)
- causes the current process to do nothing for the specified number of seconds.

- pid_t = fork(void)
- fork creates a clone of the parent process
- pid is 0 when the child process is running and >0 when the parent process is running. Both processes will return from the system call to fork().

- pid_t = wait(int * status)
- this system call is equivalent to pid_t = waitpid(-1, &status, 0)
- the wait system call suspends the current process until one of its children has terminated.
- pid_t specifies which child to wait for. Various options are available that define how to wait.
- int* status holds the status of the wait system call. There are various macros that define what the status can be.

- '''What happens when the newline character is removed from the printf statement?'''
- printf holds the output in a buffer until it receives a newline character. The result is the output does not print when expected.
- The kernel is doing things as it is told, so the c library is at fault.
- We could get the expected output if we bypass the c library using a system call: "write"
- running an "strace" on the executable will show us the order in which the processes were run to help debug the problem.

- '''What happens in the conditional?'''
- When the parent process calls fork(), a child process is created and begins executing at the fork() statement. The OS returns a value from fork() which indicates which process
is currently running.
- If fork() returns zero then the child sleeps for 3 seconds before printing. The sleep system call forces the child process to do nothing for 3 seconds.
- if fork() returns anything else, it is assumed to be the parent process. In this case the wait() system call is run by the parent which causes the parent process to wait for its
child to finish; then the printf executes and prints the child process id and its return value from main. The macro WEXITSTATUS retrieves 42.

==== What Happen to the child process after the parent process died? ====
*Every child process must have a currently running parent. When the parent process die, kernel collect the dead body(return value) and re-parent the process
* In the demo showed in class, “system d user” became its new parent
**Use command "Pstree" to show process hierarchy
* Every process must have a valid process parent. Use "pstree" to see the tree structure
* When a parent dies before a child "init" will adopt every orphan on the system

==== What is zombie process? ====
*When a process die it become 'zombie' (s = sleep, z = zombie)
*We can use wait to be a 'good' parent

==== How to load a new binary manually====
*Use the execve command.
**We can fork child process to handle dangerous task (“Suicide Squad”), or handle all incoming request

==== Manipulating environment variables====
*argc tells the number of argument in command line, while argv[] contains the actual command obtained in the command line

=== Adding the execve() system call to the mix ===

#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>

int main(int argc, char* argv[], char* envp[]) {
int child;
int status;
pid_t retval;
int i;

printf("You called me with %d arguments.\n", argc);

for (i=0; i<argc; i++) {
printf("Arg %d: %s\n", i, argv[i]);
}

child = fork();

printf("Hello! I am %d, my parent is %d.\n",
getpid(), getppid());

if (child == 0) {
/* We're the child */

execve("/bin/ls", argv, envp);
sleep(3);
printf("Child all done!\n");
} else {
/* We're the parent */
retval = wait(&status);
printf("Parent all done! Child %d returned %d.\n",
(int) retval, WEXITSTATUS(status));
}

return 42;
}

- int execve(const char *filename, char *const argv[], char *const envp[])
- this system call executes the program pointed to by filename
- The child process becomes independent from its parent -- a new program
- argv[] is an array of strings passed to the new program. The values are loaded into the array by execve not the kernel.
- envp[] is an array of strings of the form key=value which are passed as environment to the new program

- What happens when the if statement executes
- execve creates a new program -- an independent process and passes arguments to the "ls" program.
- execve stops the new process after it has completed.

- Some interesting points about main(int argc, char* argv[], char* envp[])
- Notice that the size of the arrays are not defined. We have to use argc to get this information. Not good programming practice but is
accepted because it is grandfathered from the early days of the c language.
- arg[0] is always the name of the program.

Operating Systems 2017F Lecture 5

2017-10-05T00:44:18Z

Ssarazin: /* Notes */

==Video==

Video from the lecture given on September 21, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec05-21Sep2017.mp4 is now available].

==Code==

Code and files from the lecture (captured as they were at the end) are available [http://homeostasis.scs.carleton.ca/~soma/os-2017f/code/lec05/ here].

==Notes==

====IO Redirection====
*The > operator output to a location
**Eg. Ls > foo.txt

*The | operator connect the output of the 1st program to the input of the 2nd program
**Eg. Shuf foo.txt | shuf > shuf2.txt (pipe operater connect output of shuf foo.txt to input of shuf then output to shuf2.txt)

*File usually comes with standard input (0), standard output(1),standard error(error message, port 2). This defined when program started
**Ls foo > test.txt 2 >error.txt: redirect error message (port 2) of foo to error.txt
*Ls foo>& all.txt: redirect everything to output file
**Bc -l: This command bring up the calculater
*The convention is 3 file descriptors are available once the system boots. They can be accessed by a shell and used to send data in both directions.
**file descriptor 0 is stdin
**file descriptor 1 is stdout
**file descriptor 2 is stderr
*redirections is accomplished using the > (redirect to) or < (redirect from)
**example: ls > foo.txt creates a file called foo.txt and puts the result of ls in it
**example: ls foo > test.txt 2> errors.txt will generate errors because ls won't be able to find foo. In this case nothing will be put into test.txt instead the resulting error will be sent to errors.txt. 2> is the convention for selecting file descriptor 2.
*using >& will merge stdout and stderr
*The pipe | operator
**This operator allows us to join output from one process sending the data to another process.
**example: seq 1 100 num.txt | shuf > shuf2.txt
***The result is the sequence of numbers from 1 to 100 are put in num.txt then sent to shuf to be "shuffled" and finally redirected to shuf2.txt

====Manipulating environment variables====

<syntaxhighlight lang="c">
int main(int argc, char *argv[], char *envp[]){
While (envp[i] != NULL) {
If (strcmp(“User=“, envp[i], 5){
printf(“%s\n”, envp[i]);
char *username = envp[i] +5; //pointer arithmetic
}
I++;
}
printf (“%s\n”, username);
return 0;
}
</syntaxhighlight>

*Getopt: a better way to find environmental variable
*'''How many are there?'''
**arg c tells us how many are in argv[] not envp[]
**we can look at all of the environment variables using env in the shell -- There are a lot!
**Accessing argv[] and envp[] from a c programming can be accomplished using the unistd.h library
***getopt for manipulating argv[]
***getenv for manipulating envp[]

====Signals====
*Kill: send a signal to destroy a program
**Kill -stop 5617(This stop a progam) and Kill -cont 5617(resume a program)
**Sigkill, sigterm (ctrl+c), sigstop (ctrl+z), sigpipe (when | died), sighup (when close terminal window)
**Sigchild(when a child process terminated),
*Signals allow us to communicate with other processes through the kernel. The signal is a way to alert other processes that some event has happened giving the process some time to react.
**for example the kill command sends a signal that can suspend or terminate a running process
***kill -STOP pid or kill - CONT pid
**use man 7 signal to see the list of available signals
***SIGHUP: tells the process that the modem has just hung-up
***SIGCHLD: is a signal that gets sent when a child process terminates
***SIGUSR1: is a user defined signal
*sigaction() is a programming tool that allows us to interact with signals
**It is a structure that contains function pointers -- we have to define these functions in our code

*Implementing a shell
**What is the best strategy for reading and understanding code?
***Start with main or where execution starts. Most often we won't have a main, just a large body of code.
***temporarily ignore variable definitions until they are required
***Notice the different kinds of control structures and conditionals and start reviewing the code here. Examine the code line by line.
*Example of a shell program. What happens in the code?

<syntaxhighlight lang="c">

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <sys/types.h>
#define BUFFER_SIZE 1<<16
#define ARR_SIZE 1<<16

void parse_args(char *buffer, char** args,
size_t args_size, size_t *nargs)
{
char *buf_args[args_size]; /* You need C99 */
char **cp;
char *wbuf;
size_t i, j;

wbuf=buffer;
buf_args[0]=buffer;
args[0] =buffer;

for(cp=buf_args; (*cp=strsep(&wbuf, " \n\t")) != NULL ;){
if ((*cp != '\0') && (++cp >= &buf_args[args_size]))
break;
}

for (j=i=0; buf_args[i]!=NULL; i++){
if(strlen(buf_args[i])>0)
args[j++]=buf_args[i];
}

*nargs=j;
args[j]=NULL;
}

int main(int argc, char *argv[], char *envp[]){
char buffer[BUFFER_SIZE];
char *args[ARR_SIZE];

int *ret_status;
size_t nargs;
pid_t pid;

while(1){
printf("$ ");
fgets(buffer, BUFFER_SIZE, stdin);
parse_args(buffer, args, ARR_SIZE, &nargs);

if (nargs==0) continue;
if (!strcmp(args[0], "exit" )) exit(0);
pid = fork();
if (pid){
printf("Waiting for child (%d)\n", pid);
pid = wait(ret_status);
printf("Child (%d) finished\n", pid);
} else {
if( execvp(args[0], args)) {
puts(strerror(errno));
exit(127);
}
}
}
return 0;
}
</syntaxhighlight>

*Start in main at the infinite while loop
**printf prints the shell prompt
**parse-args takes the command line arguments it receives from fgets and places these values in the args[] array. it also returns the total number of arguments received via nargs. At this point understanding all of the details as to how this function works is not necessary. To get the full picture we would have to examine the code's fine details which takes time.
*After the aguments have been parsed, check to see if there are arguments to work with and is the argument exit. If no arguments are entered then jump to the start of the loop; if the argument is 'exit' then quit the program.
**The conditional is where all of the work gets done. Any argument other than 'none' or 'exit' cause a fork(). A child process is cloned and the parent is made to wait until the child process finishes. The execvp runs the program requested using the entered argument. An example would be 'ls'; in this case the child process would run the ls program.
***If the argument is not understood then an error results and is printed stdout and the child process terminates. Note that the error string is printed from errno.
*1 << 16 is interesting. it is a left shift operator that computes 65536. The most significant bit in a 16 bit number is set to 1. It's the same as saying 2^16. Defining this statement in a preprocessor definition means that it runs prior to compilation possibly saving some time.

Operating Systems 2017F Lecture 5

2017-10-05T00:28:08Z

Ssarazin: /* Signals */

Operating Systems 2017F Lecture 5

2017-10-05T00:26:19Z

Ssarazin: /* Manipulating environment variables */

Operating Systems 2017F Lecture 5

2017-10-05T00:23:54Z

Ssarazin: /* IO Redirection */

Operating Systems 2017F Lecture 4

2017-10-05T00:19:46Z

Ssarazin:

==Video==

Video from the lecture given on September 19, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec04-19Sep2017.mp4 is now available].

==Notes==
- The textbook
- Is just for reference as supporting material. The class lectures will not follow the text.

- Some concepts from the tutorial
- Running an ltrace on a statically compiled program prints an error. The error just explains that the file was not dynamically compiled "couldn't find .dynsym"
- What is obj code?
- machine code. It is important to have a good mental picture of what is happening. Notice that addresses are unique for each process.
- The point is that the system is not clean, it is complicated and not well understood.

- How to build a Linux machine using virtualbox and an iso image (concepts)
- iso: international organization for standardization for optical disk media
- it's a cd or dvd
- The VM uses a dynamically allocated hard drive. It is a file that has a filesystem and it grows and shrinks as needed.
- Hard drive partition: Creates memory segments on the disk that is used to store different types of data.
- Memory swapping: The processor uses hard disk space when it runs out of RAM.

- Things about linux
- '''Why is the kernel split?'''
- So the various hardware modules can be added to the system when the hardware is installed
- We can see the modules using ls mod or cd /sys/proc
- Environment Variables
- TERM or USER for example. They are capitalized by convention. The significance is that these variables belong to a process (like a shell).
- Child processes inherit the environment variables from the parent; envp[] stores the values of these variables.
- Things that can be done in a shell
- We can run programming commands directly in the shell
- things like loops and conditionals
- $ defines a variable
- echo is the shell's version of printf
- ssh
- used to access remote machines using a secure shell
- The ssh uses a file that stores authorized keys. If a user has a key for the remote terminal then they do not require a password.
- certificates
- used by websites to maintain security
- A secure website encrypts the data so it can't be viewed by hackers.

Operating Systems 2017F Lecture 3

2017-10-05T00:11:59Z

Ssarazin: /* Notes */

==Video==

Video from the lecture given on September 14, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec03-14Sep2017.mp4 is now available].

==Code==

Code and files from the lecture (captured as they were at the end) are available [http://homeostasis.scs.carleton.ca/~soma/os-2017f/code/lec03/ here].

==Notes==
====Commands ====
*-wall: show all warning
*less: allow screen navigation, hit q to quit

==== What is fork? ====
* Fork creates a new program but contains the same code (call the clone system call) and only argv[] and envp[] passed on to the new born program

- More on creating a process using fork().

#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>

int main(int argc, char* argv[], char* envp[]) {
int child;
int status;
pid_t retval;

child = fork();

printf("Hello! I am %d, my parent is %d.\n",
getpid(), getppid());

if (child == 0) {
/* We're the child */
sleep(3);
printf("Child all done!\n");
} else {
/* We're the parent */
retval = wait(&status);
printf("Parent all done! Child %d returned %d.\n",
(int) retval, WEXITSTATUS(status));
}

return 42;
}

- sleep(int)
- causes the current process to do nothing for the specified number of seconds.

- pid_t = fork(void)
- fork creates a clone of the parent process
- pid is 0 when the child process is running and >0 when the parent process is running. Both processes will return from the system call to fork().

- pid_t = wait(int * status)
- this system call is equivalent to pid_t = waitpid(-1, &status, 0)
- the wait system call suspends the current process until one of its children has terminated.
- pid_t specifies which child to wait for. Various options are available that define how to wait.
- int* status holds the status of the wait system call. There are various macros that define what the status can be.

- '''What happens when the newline character is removed from the printf statement?'''
- printf holds the output in a buffer until it receives a newline character. The result is the output does not print when expected.
- The kernel is doing things as it is told, so the c library is at fault.
- We could get the expected output if we bypass the c library using a system call: "write"
- running an "strace" on the executable will show us the order in which the processes were run to help debug the problem.

- '''What happens in the conditional?'''
- When the parent process calls fork(), a child process is created and begins executing at the fork() statement. The OS returns a value from fork() which indicates which process
is currently running.
- If fork() returns zero then the child sleeps for 3 seconds before printing. The sleep system call forces the child process to do nothing for 3 seconds.
- if fork() returns anything else, it is assumed to be the parent process. In this case the wait() system call is run by the parent which causes the parent process to wait for its
child to finish; then the printf executes and prints the child process id and its return value from main. The macro WEXITSTATUS retrieves 42.

==== What Happen to the child process after the parent process died? ====
*Every child process must have a currently running parent. When the parent process die, kernel collect the dead body(return value) and re-parent the process
* In the demo showed in class, “system d user” became its new parent
**Use command "Pstree" to show process hierarchy
* Every process must have a valid process parent. Use "pstree" to see the tree structure
* When a parent dies before a child "init" will adopt every orphan on the system

==== What is zombie process? ====
*When a process die it become 'zombie' (s = sleep, z = zombie)
*We can use wait to be a 'good' parent

==== How to load a new binary manually====
*Use the execve command.
**We can fork child process to handle dangerous task (“Suicide Squad”), or handle all incoming request

==== Manipulating environment variables====
*argc tells the number of argument in command line, while argv[] contains the actual command obtained in the command line

=== Adding the execve() system call to the mix ===

#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>

int main(int argc, char* argv[], char* envp[]) {
int child;
int status;
pid_t retval;
int i;

printf("You called me with %d arguments.\n", argc);

for (i=0; i<argc; i++) {
printf("Arg %d: %s\n", i, argv[i]);
}

child = fork();

printf("Hello! I am %d, my parent is %d.\n",
getpid(), getppid());

if (child == 0) {
/* We're the child */

execve("/bin/ls", argv, envp);
sleep(3);
printf("Child all done!\n");
} else {
/* We're the parent */
retval = wait(&status);
printf("Parent all done! Child %d returned %d.\n",
(int) retval, WEXITSTATUS(status));
}

return 42;
}

- int execve(const char *filename, char *const argv[], char *const envp[])
- this system call executes the program pointed to by filename
- The child process becomes independent from its parent -- a new program
- argv[] is an array of strings passed to the new program. The values are loaded into the array by execve not the kernel.
- envp[] is an array of strings of the form key=value which are passed as environment to the new program

- What happens when the if statement executes
- execve creates a new program -- an independent process and passes arguments to the "ls" program.
- execve stops the new process after it has completed.

- Some interesting points about main(int argc, char* argv[], char* envp[])
- Notice that the size of the arrays are not defined. We have to use argc to get this information. Not good programming practice but is
accepted because it is grandfathered from the early days of the c language.
- arg[0] is always the name of the program.

Operating Systems 2017F Lecture 3

2017-10-05T00:04:38Z

Ssarazin: /* What Happen to the child process after the parent process died? */

Operating Systems 2017F Lecture 3

2017-10-05T00:03:10Z

Ssarazin: /* What is fork? */

Operating Systems 2017F Lecture 3

2017-10-05T00:01:09Z

Ssarazin: /* What is fork? */

Operating Systems 2017F Lecture 2

2017-10-04T01:54:08Z

Ssarazin:

==Video==

Video from the lecture given on September 12, 2017 [http://homeostasis.scs.carleton.ca/~soma/os-2017f/lectures/comp3000-2017f-lec02-12Sep2017.mp4 is now available].

==Code==

Code and files from the lecture (captured as they were at the end) are available [http://homeostasis.scs.carleton.ca/~soma/os-2017f/code/lec02/ here].

==Notes==
'''What is a file?'''
- A hierarchical name with a value mapping, accessed using the API "open, read, close, seek" (roughly). Open and close are used for efficiency; read and write do all of the work.

'''Why is Unix and Linux interchangeable?'''
- Linux is a variant of the Unix, it contains the Unix kernel with custom built layers on top.

'''What is the architecture of the Unix kernel?'''
- A system of hierarchical files that represent all of the different hardware devices
for example: writing to stdout or reading from stdin. These are the computer monitor and keyborad file descriptors.

'''How do we add new devices to the system?'''
- We tell the operating system to mount the new device. The operating system responds with a file descriptor for the new device. Accessing the device means writing to its file.
- There are many files that represent a device. We can trace these files by calling the command "lspci" which gets the PCI bus hardware device info from the kernel. Using the
command "strace lspci" we can trace all of the system calls performed by lspci. Now we know exactly which files make up each device on the PCI bus.
- file descriptors like /dev or /sys are conduits to the kernel. The kernel then performs operations on the hardware.

The key point(s): The kernel abstracts the hardware by assigning file descriptors for every device. A program can gain information about the device using system calls to the kernel.

'''Virtualizing Memory'''
- Hard disks file descriptors appear as:
- /dev/sda -> unencrypted drive
- /dev/mapper… -> encrypted drive
- We can see the filesystem using the "df" command or view the size of files using "df."
- Why are some of the files showing a size of 0 bytes?
- When memory is virtualized, the kernel builds the file system as it goes.

'''Processes'''
- '''What is a Process?'''
- A running program
- Just like the devices, every process gets a directory in the files system "/proc"
- To view running process use the command "ps"
- To stop a running process use the command "Kill <process id>"
- '''What happens when multiple processes are created?'''
- The operating system shares the resources between the processes using time sharing -- also called virtualizing the resources.
- '''What is a shell in Linux?'''
- Just another program that can access the operating system and run other programs
- A shell can run multiple programs (processes), only control returns to the shell after the process has finished. To give the shell priority use the "&" to send the current process to
the background. This idea demonstrates how the CPU is virtualized because it is being shared by the shell and the running process. The concept was demonstrated using xclock.
- Use the "jobs" command to see which processes are running
- Use the "ps aux" command to view the full command line of all running processes.
- Use the "getpid()" function call in a c program to find the process id for the current process.
- Use the "getppid()" function call in a c program to find the parents process id for the current process.
- '''How do I make a process?'''
- by copying the current process
- fork(): a system call the clones the currently running process. The new process (child) begins executing from the statement that contained the call to fork().
- fork() returns zero it means the child process is running.
- fork() returns > 0 it means the parent process is running.
- fork() returns < 0 it means the fork() failed.

Using fork() in a c program

/* hello-world.c */

#include <stdio.h>
#include <unistd.h>

int main() {
fork();
fork();

printf("Hello world! I'm %d. My parent is %d\n", getpid(), getppid());
return 42;
}

'''What does the program do?'''
- The first call to fork() clones a child process running the hello-world program. The child begins operating as if it just called fork(). Only the value returned from fork()
is different for the child than the parent. The child process returns a zero.
- Now there are two processes running concurrently, executing the same code. A parent process and a child process.
- The parent process calls fork() again creating another child process, and the first child process also creates a child process. Now there are 4 processes running concurrently.
- All 4 processes will print their process id and their parent's process id.
- finally a return value of 42 is delivered to the parent of each process as it exits.
- Important: The order of execution for each process is not deterministic. We do not know which process will run next because the operating system decides.