Soma-notes - User contributions [en]

WebFund 2015W: Assignment 10

2015-03-31T14:58:19Z

Afry: There was a small edit to make the question more understandable.

'''This assignment is not yet finalized.'''

10 points + 1 bonus

Fix [http://homeostasis.scs.carleton.ca/~soma/webfund-2015w/code/tls-notes.zip tls-notes]

# [1] Make the server connect using TLS.
# [1] Make the "Register" button work.
# [1] Fix the password entry field so it is the same width as the username entry field.
# [1] Make the Edit Note screen show the old note title.
# [1] Fix the "New Note" button.
# [1] Fix the "Delete" button.
# [1] Fix "Change Username" so it doesn't generate a 500 error.
# [1] Fix "Change Username" so that it puts a "'s" to the end of the username in the page title immediately after successfully changing the username (i.e., without requiring the page to be reloaded).
# [2] Fix "Change Username" so it updates the users collection as well as the notes collection.
# [BONUS 1] Regenerate the keys so they have the same basic metadata except that the contact email address is yours. (Don't try to make the creation and expiry dates or the key hash the same. Focus on the data that is normally entered when creating a certificate.)

WebFund 2015W: Assignment 7

2015-03-16T14:09:32Z

Afry: changed name of directory

In this assignment you will be modifying [http://homeostasis.scs.carleton.ca/~soma/webfund-2015w/code/ajax-notes.zip the AJAX notes demo] from [[WebFund 2015W: Tutorial 7|Tutorial 7]]. There are 10 points in 3 tasks, 7 listed below and 3 for code style. This assignment is due by 10 AM on <del>Monday, March 16, 2015</del> '''Wednesday, March 18, 2015'''.

Please submit your answers as a zip file called "<username>-comp2406-assign7.zip", where username is your MyCarletonOne username. This zip file should uncompress to a directory called "<username>-comp2406-assign7" and inside this directory should be two things: a directory "ajax-notes" that contains the application and a text file "comments.txt".

"comments.txt" should:
* list any references you used to complete the assignment (documentation web sites, for example),
* list your collaborators, and
* optionally, should discuss any issues or concerns you had when completing this assignment.
Remember that while you are allowed to collaborate with others, plagiarism is not allowed. In other words you '''should not''' be copying any code or data directly from anywhere, and any assistance or inspiration should be credited. Any significant code similarity (beyond the code already given to you) will be considered plagiarism and will be reported to the Dean.

==Tasks==
# [2] Implement a [delete] button on the edit note view (with an id of "delete"). When pressed it should put up a [https://developer.mozilla.org/en-US/docs/Web/API/window/confirm confirm modal dialog box] that says 'Delete note "The note title"'? (Replace "The note title" with the actual title of the note to delete.) If the user says okay then it should do a post to "/deleteNote" where the form body contains an "id" value that has the _id of the note to be deleted. This post should return "note deleted" upon success or "ERROR: note not deleted" upon failure. When the POST returns the page should be refreshed with the current list of notes.
# [2] Make it so the contents of notes escape embedded HTML tags. However, allow links to be embedded in notes using the syntax of "[<link> <label>]" where link is a URL for an <a> tag and the label (the rest of the text in the square brackets) is the label for the URL. If there is no label then the link itself should be the <a> tag's label.
# [3] Add a Change Username button, besides the refresh button, that allows you to change the username for a user.
#* This button should have an <tt>id=changeusername</tt> assigned. It should cause the notesArea to be replaced with an interface for changing the username.
#* The text field for the changed username should have a <tt>id="username"</tt>. Below this should be two buttons, "Change Username" (with id of "doChangeUsername") and "Cancel" (with id of "cancelUsernameChange").
#* The Cancel button should cause the notes list to be redrawn. The Change Username button should do a POST to /changeusername to actually do the username change (with a body value of newUsername for the new username). When this post returns, if it was successful (returning a value "username changed") it should change the username on the page. It should NOT change the username if the name change failed (returning a value "ERROR: username not changed"). When the post returns the notes list should be redrawn.
#* The new username can be any username that is not currently being used by any stored notes. Note that you'll need to change the username in the session and update the owner field in stored notes as appropriate.

WebFund 2015W: Tutorial 1

2015-01-13T17:59:22Z

Afry: /* ASIDE: if you are unsure of your account information */

For this class you will be using a [http://www.lubuntu.net/ Lubuntu] virtual machine appliance. We will be using [https://www.virtualbox.org/ VirtualBox] as our preferred virtualization platform; however, VMware Workstation/Fusion and other virtualization platforms should be able to run the appliance as well. In this first tutorial you will be becoming familiar with [http://nodejs.org/ node.js]-based development environment provided by this appliance.

To get credit for this lab, show a TA or the instructor that you have gotten the class VM running, made simple changes to your first web app, and that you have started lessons on CodeAcademy (or convince them you don't need to).

If you finish early (which you are likely to do), try exploring node and the Lubuntu environment. You will be using them a lot this semester!

==Running the VM==

In the SCS labs you should be able to run the VM by starting Virtualbox (listed in the Applications menu) and selecting the COMP 2406 virtual machine. After the VM has fully booted up you should be logged in automatically as the user "student". If the screen locks or you need administrative access (via sudo) you'll need the password for the student account, however. This password is "tneduts!" (students backwards followed by an !). There is also an admin account in case your student account gets corrupted for any reason. The password for it is "nimda!".

We highly recommend running your VM in full-screen mode. (Don't maximize the window; instead select full screen from the view menu.) Do all of your work inside of the VM; it should be fast enough and you won't have any issues with sharing files or with host firewalls.

If you want to run the appliance on your own system (running essentially any desktop operating system you want), just [http://homeostasis.scs.carleton.ca/~soma/VMs/COMP%202406%20Winter%202015.ova download the
virtual appliance file] and import. The SHA1 hash of this file is:

47849f3c5a4b11e1c701bd95ba4bb8f88062d8ba COMP 2406 Winter 2015.ova

On Windows you can compute this hash for your downloaded file using the command [http://support.microsoft.com/kb/889768 <tt>FCIV -sha1 COMP 2406 Winter 2015.ova</tt>]. If the hash is different from above, your download has been corrupted.

If your virtualization application is not VirtualBox, you'll need to:
* Have the VM platform ignore any errors in the structure of the appliance when importing;
* Uninstall the VirtualBox guest additions by typing starting a terminal application and running
sudo /opt/VBoxGuestAdditions-4.3.10/uninstall.sh
* Install your platform's own Linux guest additions, if available.

Note as we will explain, you will have the ability to easily save the work you do from any VM to your SCS account and restore it to any other copy of the class VM. Thus feel free to play around with VMs; if you break anything, you can always revert. Remember though that in the labs you '''must''' save and restore your work, as all of your changes to the VM will be lost when you logout!

While you may update the software in the VM, those updates will be lost when you next login to the lab machines; thus, you probably only want to update a VM installed on your own system.

==Hello, World!==

To create your first node application, start [http://www.geany.org/ geany], [http://brackets.io/ brackets], [http://www.vim.org/ vim], or [http://www.gnu.org/software/emacs/ emacs] code editors by clicking on their quick launch icons at the bottom left of the screen (beside the LXDE start menu button).

(If you are a fan of vi but want to try emacs, you should type Alt-X [http://www.gnu.org/software/emacs/manual/html_mono/viper.html viper-mode]. You're welcome.)

In your editor of choice, create a file <tt>hello.js</tt> in your Documents folder with the following contents:

console.log("Hello World!");

You can now run this file by opening an LXTerminal (under Accessories) and typing:

cd Documents
node hello.js

And you should see <tt>Hello, World!</tt> output to your terminal.

You can also run node interactively by simply running <tt>node</tt> with no arguments. You'll then get a prompt where you can enter any code that you like and see what it does. To exit this environment, type Control-D.

Note that when run interactively, we say that node is running a [http://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop read-eval-print loop] (REPL). It reads input, evaluates it, and then prints the results. This structure is very old in computer science, going back to the first LISP interpreters from the early 1960's.

==Your First Web App==

Web applications, even simple ones, are a bit more complex than our "Hello, world!" example. Fortunately in node we have the [http://expressjs.com/ express] web application framework to make getting up and running quite easy.

Follow the directions for the [http://expressjs.com/starter/generator.html express application generator] in a terminal window. In short form, you should run the following commands to make "myapp":

sudo npm install express-generator -g
express myapp
cd myapp
npm install
DEBUG=myapp ./bin/www

To see what your app is doing, start up a web browser in your VM and visit the following URL:

http://localhost:3000

You should see a message from your first web application!

If you have any problems, particularly with network connections, you can get and run the basic app by instead doing the following:

wget http://homeostasis.scs.carleton.ca/~soma/webfund-2015w/code/myapp.zip
unzip myapp
cd myapp
DEBUG=myapp ./bin/www

==Observing/Debugging the Web App==

When you attempt to observe or debug a web application, you have to keep an eye on what happens on the server, what happens on the client (web browser), and what happens on the network in between.

You can use the browser's built-in developer tools to look at the network activity (as seen by the browser of course) and what happens with the actual rendering of the web page. In either Firefox or Chromium (the open source version of Chrome) you can get access to the developer tools by typing the key sequence Shift-Control-I or by selecting the developer tools from the menu. Pay particular attention to these tabs:
* The Network tab tells you about what has been sent and received.
* The Inspector or Elements tab lets you examine the DOM (document object model), i.e., the parsed version of the web page.
* The Console tab gives you a JavaScript prompt in the context of the web page.
For example, to find out the user agent string of the current browser type "navigator.userAgent" in the Console tab.

To find out what is happening on the server side, for now you'll have to either add console.log() or console.error() statements to print things out. Later we will learn how to use the node debugger.

==Simple Changes==

Now that you have an app up and running, make the following simple changes:
* Change the default port to 2000 (by editing bin/www)
* Change the title to "My First Web App" (<tt>routes/index.js</tt>)
* Prevent the default stylesheet <tt>style.css</tt> from being loaded (<tt>views/layout.jade</tt>)
* Add a paragraph to the initial page saying "This page is pretty boring."

Note that the Jade files (ending in .jade) are used to generate HTML files. Compare the source of the page as seen by the web browser to the original .jade files to see how they connect. Also, you can look at the [http://jade-lang.com/reference/ Jade documentation].

We don't expect you to understand everything that you are doing in this tutorial; these exercises are more designed to get your feet wet tos you can start asking the right kinds of questions.

==Saving your work==

You can save your work to your SCS account by running (where it says <SCS username> it should just be your username no <>brackets)

save2406 <SCS username>

This will rsync /home/student to the COMP2406 directory in your SCS account by connecting to access.scs.carleton.ca.

When you wish to restore your student account, run

restore2406 <SCS username>

Note that both of these commands are destructive - they will wipe out all the files in the COMP2406 folder on SCS or /home/student in your VM. If you want to see what the differences are between the two versions, run

compare2406 <SCS username>

=== ASIDE: If you are unsure of your account information (Please read this) ===

The SCS account would be your access.scs.carleton.ca account (which is your unix/linux account) if you do not have one or do not remember your password, please go to the main scs page [http://www.scs.carleton.ca here], click on Tech Support (top right corner), then click on accounts, Linux accounts and click the link [http://www.scs.carleton.ca/webacct/ http://www.scs.carleton.ca/webacct/], read the policy, accept, and proceed to retrieve your account information / setup your account. You will then use this password for all future save and restore operations!!)

==CodeAcademy==

Now that you've got your virtual machine running, it is time to start learning about web technologies. If you haven't already, you should either go through or make sure you know the material in all of the following CodeAcademy modules:
* [http://www.codecademy.com/tracks/web Web Fundamentals]
* [http://www.codecademy.com/tracks/javascript/ Javascript]
* [http://www.codecademy.com/tracks/jquery jQuery]
* [http://www.codecademy.com/tracks/projects Web Projects] (just the basic projects)

Feel free to skip around; these should be very simple for you, at least at the beginning. Try to do the last parts of each lesson to see if need to bother going through it. You'll be expected to be familiar with most of this material starting in next week's tutorial (and Assignment 2).

WebFund 2015W: Tutorial 1

2015-01-13T17:58:19Z

Afry: /* Saving your work */

For this class you will be using a [http://www.lubuntu.net/ Lubuntu] virtual machine appliance. We will be using [https://www.virtualbox.org/ VirtualBox] as our preferred virtualization platform; however, VMware Workstation/Fusion and other virtualization platforms should be able to run the appliance as well. In this first tutorial you will be becoming familiar with [http://nodejs.org/ node.js]-based development environment provided by this appliance.

To get credit for this lab, show a TA or the instructor that you have gotten the class VM running, made simple changes to your first web app, and that you have started lessons on CodeAcademy (or convince them you don't need to).

If you finish early (which you are likely to do), try exploring node and the Lubuntu environment. You will be using them a lot this semester!

==Running the VM==

In the SCS labs you should be able to run the VM by starting Virtualbox (listed in the Applications menu) and selecting the COMP 2406 virtual machine. After the VM has fully booted up you should be logged in automatically as the user "student". If the screen locks or you need administrative access (via sudo) you'll need the password for the student account, however. This password is "tneduts!" (students backwards followed by an !). There is also an admin account in case your student account gets corrupted for any reason. The password for it is "nimda!".

We highly recommend running your VM in full-screen mode. (Don't maximize the window; instead select full screen from the view menu.) Do all of your work inside of the VM; it should be fast enough and you won't have any issues with sharing files or with host firewalls.

If you want to run the appliance on your own system (running essentially any desktop operating system you want), just [http://homeostasis.scs.carleton.ca/~soma/VMs/COMP%202406%20Winter%202015.ova download the
virtual appliance file] and import. The SHA1 hash of this file is:

47849f3c5a4b11e1c701bd95ba4bb8f88062d8ba COMP 2406 Winter 2015.ova

On Windows you can compute this hash for your downloaded file using the command [http://support.microsoft.com/kb/889768 <tt>FCIV -sha1 COMP 2406 Winter 2015.ova</tt>]. If the hash is different from above, your download has been corrupted.

If your virtualization application is not VirtualBox, you'll need to:
* Have the VM platform ignore any errors in the structure of the appliance when importing;
* Uninstall the VirtualBox guest additions by typing starting a terminal application and running
sudo /opt/VBoxGuestAdditions-4.3.10/uninstall.sh
* Install your platform's own Linux guest additions, if available.

Note as we will explain, you will have the ability to easily save the work you do from any VM to your SCS account and restore it to any other copy of the class VM. Thus feel free to play around with VMs; if you break anything, you can always revert. Remember though that in the labs you '''must''' save and restore your work, as all of your changes to the VM will be lost when you logout!

While you may update the software in the VM, those updates will be lost when you next login to the lab machines; thus, you probably only want to update a VM installed on your own system.

==Hello, World!==

To create your first node application, start [http://www.geany.org/ geany], [http://brackets.io/ brackets], [http://www.vim.org/ vim], or [http://www.gnu.org/software/emacs/ emacs] code editors by clicking on their quick launch icons at the bottom left of the screen (beside the LXDE start menu button).

(If you are a fan of vi but want to try emacs, you should type Alt-X [http://www.gnu.org/software/emacs/manual/html_mono/viper.html viper-mode]. You're welcome.)

In your editor of choice, create a file <tt>hello.js</tt> in your Documents folder with the following contents:

console.log("Hello World!");

You can now run this file by opening an LXTerminal (under Accessories) and typing:

cd Documents
node hello.js

And you should see <tt>Hello, World!</tt> output to your terminal.

You can also run node interactively by simply running <tt>node</tt> with no arguments. You'll then get a prompt where you can enter any code that you like and see what it does. To exit this environment, type Control-D.

Note that when run interactively, we say that node is running a [http://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop read-eval-print loop] (REPL). It reads input, evaluates it, and then prints the results. This structure is very old in computer science, going back to the first LISP interpreters from the early 1960's.

==Your First Web App==

Web applications, even simple ones, are a bit more complex than our "Hello, world!" example. Fortunately in node we have the [http://expressjs.com/ express] web application framework to make getting up and running quite easy.

Follow the directions for the [http://expressjs.com/starter/generator.html express application generator] in a terminal window. In short form, you should run the following commands to make "myapp":

sudo npm install express-generator -g
express myapp
cd myapp
npm install
DEBUG=myapp ./bin/www

To see what your app is doing, start up a web browser in your VM and visit the following URL:

http://localhost:3000

You should see a message from your first web application!

If you have any problems, particularly with network connections, you can get and run the basic app by instead doing the following:

wget http://homeostasis.scs.carleton.ca/~soma/webfund-2015w/code/myapp.zip
unzip myapp
cd myapp
DEBUG=myapp ./bin/www

==Observing/Debugging the Web App==

When you attempt to observe or debug a web application, you have to keep an eye on what happens on the server, what happens on the client (web browser), and what happens on the network in between.

You can use the browser's built-in developer tools to look at the network activity (as seen by the browser of course) and what happens with the actual rendering of the web page. In either Firefox or Chromium (the open source version of Chrome) you can get access to the developer tools by typing the key sequence Shift-Control-I or by selecting the developer tools from the menu. Pay particular attention to these tabs:
* The Network tab tells you about what has been sent and received.
* The Inspector or Elements tab lets you examine the DOM (document object model), i.e., the parsed version of the web page.
* The Console tab gives you a JavaScript prompt in the context of the web page.
For example, to find out the user agent string of the current browser type "navigator.userAgent" in the Console tab.

To find out what is happening on the server side, for now you'll have to either add console.log() or console.error() statements to print things out. Later we will learn how to use the node debugger.

==Simple Changes==

Now that you have an app up and running, make the following simple changes:
* Change the default port to 2000 (by editing bin/www)
* Change the title to "My First Web App" (<tt>routes/index.js</tt>)
* Prevent the default stylesheet <tt>style.css</tt> from being loaded (<tt>views/layout.jade</tt>)
* Add a paragraph to the initial page saying "This page is pretty boring."

Note that the Jade files (ending in .jade) are used to generate HTML files. Compare the source of the page as seen by the web browser to the original .jade files to see how they connect. Also, you can look at the [http://jade-lang.com/reference/ Jade documentation].

We don't expect you to understand everything that you are doing in this tutorial; these exercises are more designed to get your feet wet tos you can start asking the right kinds of questions.

==Saving your work==

You can save your work to your SCS account by running (where it says <SCS username> it should just be your username no <>brackets)

save2406 <SCS username>

This will rsync /home/student to the COMP2406 directory in your SCS account by connecting to access.scs.carleton.ca.

When you wish to restore your student account, run

restore2406 <SCS username>

Note that both of these commands are destructive - they will wipe out all the files in the COMP2406 folder on SCS or /home/student in your VM. If you want to see what the differences are between the two versions, run

compare2406 <SCS username>

=== ASIDE: if you are unsure of your account information ===

The SCS account would be your access.scs.carleton.ca account (which is your unix/linux account) if you do not have one or do not remember your password, please go to the main scs page [http://www.scs.carleton.ca here], click on Tech Support (top right corner), then click on accounts, Linux accounts and click the link [http://www.scs.carleton.ca/webacct/ http://www.scs.carleton.ca/webacct/], read the policy, accept, and proceed to retrieve your account information / setup your account. You will then use this password for all future save and restore operations!!)

==CodeAcademy==

Now that you've got your virtual machine running, it is time to start learning about web technologies. If you haven't already, you should either go through or make sure you know the material in all of the following CodeAcademy modules:
* [http://www.codecademy.com/tracks/web Web Fundamentals]
* [http://www.codecademy.com/tracks/javascript/ Javascript]
* [http://www.codecademy.com/tracks/jquery jQuery]
* [http://www.codecademy.com/tracks/projects Web Projects] (just the basic projects)

Feel free to skip around; these should be very simple for you, at least at the beginning. Try to do the last parts of each lesson to see if need to bother going through it. You'll be expected to be familiar with most of this material starting in next week's tutorial (and Assignment 2).

Operating Systems 2014F Lecture 14

2014-11-05T14:51:14Z

Afry:

How does an operating system know it's accessed memory it doesn't have access to? A lot of you said segments.

Filesystems - normally operating system mechanisms are

talking about access to hardware systems, persistent storage.
- abstraction for persistent storage. Storage that maintains it's state when it loses power.
A couple of challenges with persistent storage. What's weird about storing things in persistent storage? It's slow. durability and persistent.

Going to make errors - we should be able to recover from them. Maybe not fix everything, but preserve most of the data. This is a huge burden on filesystems. Filesystem development tends to be very slow in practice. Whatever code you have doing this, has to do this right. Older filesystems tend to be more trustworthy, when hardware changes, bugs you didn't know where in the filesystem may come up.

what do we have today? Indexed filesystems.

There is typically a minimum storage allocation given to every file. That's the minimum size of a file, it's taking up 4 k / 8 k. this is not strictly true for all file systems. There was a filesystem that allowed arbitrarily sized files. (ReiserFS)

Unifying a keyvalue store for smaller and larger filesystems wasn't considered a priority.

Make filesystems do, on modern filesystem, rather than trying to optimize the storage of small files. It's not so much filesize that is the issue.

Floppy disks vs. hard disks

What is fast and what is slow?

fast - reading what is under the drive head at any given time. As long as you keep the head there, you can read the entire concentric circle really fast. That's the fastest operation you are going to get?

What's slow? Moving the head from one part of the disk to another - that's a slow operation.

Intuitively, why is it that slow? you have to move it with extreme precision..

moving data hard - seek time - the time it takes to move the drive head from one area to another on the disk.

coordinate system goes by Cylinder Head Sector - the geometry of the disk. If there is data I want to access in parallel, I can optimize this by putting the data on different platters.

How many heads there are, which of these, and which cylinder, and which sector, which is a count around.

The classic IBM pc BIOS expects cylinders, heads and sectors. It's a lie, your systems get rid of that completely. now they use LBA - linear block addressing. What does that mean? you give the count to the block. When I talk about hard disk blocks, how does that compare to ram. A block is the smallest addressable unit of storage. What is the smallest addressable unit in ram - a byte.

Older blocks could be 512 bytes. typical block sizes are 4k / 8k, when you do a transfer you read bytes in chunks. By blocks. On windows, common keyvalue store - the registry - a hierarchical keyvalue store for small keyvalues. it's like a filesystem, but every file is really small. It's stored in a file.

hard disks aren't perfect - they have bad blocks. They can't actually store 1s and 0s. in order to encode 4 bits, you have to encode 7 bits. Weird issues in the physics of storing the data. Teh signals they are trying to read off the hard disk is kind of messy.

Error correcting Codes

hard disk - proprietary information to the hardware manufacturers.

keys (filenames) -> mapped to values

the keys are really in directories - directory data structures that are storing the names of the files.

Random access for every block. Horrible on a hard drive, works well in RAM. Don't want to associate a list of blocks with every directory entry. Bad strategy. I have to do something better. What if I did something kind of like segment, a range of blocks? Modern systems normally do a list of extents (instead of doign a list of blocks) extents - list of segments instead of a list of blocks. The larger the file, the more blocks it's stored in. File is divided into one or more extents. This is exactly what I do not want to do in RAM. One simple way to do this? Have free space - have extra space on your harddisk. On a lot of filesystems - they will say they want to reserve a certain amount of space.

It wants that extra space, so that it can make sure it gets that long string of blocks. Files will all become fragmented, what happens to filesystem performance? It goes down. It's trying to resist fragmentation.

Operating Systems 2014F Lecture 14

2014-11-05T14:50:05Z

Afry:

Operating Systems 2014F Lecture 14

2014-11-05T14:46:24Z

Afry:

Operating Systems 2014F Lecture 14

2014-11-05T14:33:51Z

Afry:

Operating Systems 2014F Lecture 14

2014-11-05T14:19:07Z

Afry:

Operating Systems 2014F Lecture 14

2014-11-05T13:52:27Z

Afry: Created page with "How does an operating system know it's accessed memory it doesn't have access to? A lot of you said segments. Filesystems - normally operating system mechanisms are talkin..."

Operating Systems 2014F Lecture 11

2014-10-10T13:57:50Z

Afry:

Dining Philosophers problem

When can you have deadlock?

4 conditions must apply

- mutual exclusion

- hold and wait - you can grab a lock and wait for the next one, you can spin / go to sleep or something. You dont' just do things like try the lock if you are successful, and then continue with the computation.

- no pre-emption (pre-emption is taking the resource by force.) you can only have deadlock when people are polite.

- circular wait that's why the dining philosopher's problem has a circular table - have to have something that a) is waiting on one another - that's what gets it into the problem.

you break any of these, you can't have deadlock.

When people talk about deadlock, they talk about strategies for avoiding it (for removing the problem) in terms of these strategies:

1 prevention - construct your system so that deadlock can never happen. (Make it impossible) Design your system such that one of these or more go away.

let's say one thread has three locks to continue - whenever one goes to sleep, I'll take their chopstick and give it back to them ebfore they wake up and they'll never know the difference.

2 avoidance - prevention means you are making it impossible for this to happen.

All four conditions are there in principle, but you can watch the unfolding of the computation, and you can notice when you are getting into a situation that can lead to deadlock, I can avoid it. Allocating resources such that you know that it's never going to happen. It's not necessarily prediction, where you lay a schedule for how everything operates. For example, let's say we are talking about car accidents - complete prevention - don't get into the car. Avoidance - you see something coming, you steer around it, or you have strategies like, stay within the lanes, don't go off the road.

3 - detect and recover - you had an accident - sorry , call the police, call the bodyshop - fix it up.

in practice we mostly do detect and recover. you don't do all of them perfectly. Where watchdog timers come in? it's something that watches the system and then detects, an easy way of doing this ... let's say you have to call in to, a guard is going around, and when they are checking the perimeter, they have to check in periodically and say, ok that's fine, if someone was to attack the base, what would they do? they take out the guard. Then the signal wouldn't come in. Then you take steps to deal with it.

A watchdog timer, is a separate processor, that is periodically sending it messages - normal interrupts to the system, if the OS is working properly and keeping sending messages back to the interrupt. but if you don't respond to the watchdog timer's request, it goes uh-oh and restarts the system. spontaneous reboot is performed to ensure the system keeps running. The assumption being that when you reboot, you come back to a working state.

Two non-deadlock concurrency bugs:
- atomicity violations - you were supposed to lock it and you didn't, you were supposed to grab a lock and you didn't.
- order violation - you attempt to use something that hasn't been initialized, use before initialize

TOCTTOU

Time Of Check To Time Of Use

race conditions - a particular class of them, in talking about memory accesses to a variable, we check the value of the variable, and we try to make a change to it

temporary files - you have a program running, in the middle of running, it's potentially useful to generate temporary files, (dump data in the middle of running) where do they often go? in a shared directory (/tmp), your own files

when you run programs that are somehow priviledged (setuid / setgid programs) - when you normally runa program on a unix like system, it runs as you, ls - ls is running as a program. which means they can access any files that you own, they have the privileges that you have. When you need to run programs that need more access. Classic situations include: lpr or passwd

passwd program allows you to change your password - a secure hash of your password is stored in a file called: /etc/passwd /etc/shadow

These files, do you want any one else to be able to change these files? No sometimes regular users want to change their password, so I have this file that I need to keep protected, sometimes I have to allow access to it. This is not an OO system. These are files. So how do I make sure that only certain code can modify that file. You would have some programs that when they ran, they didnt' run with the privileges of the person who ran it, but with the privileges of the person who owns it. This password program we want to run it with extra privileges, we want to run it as root - to change those files. How do I denote this - there is a bit in the protections, there is also the setuid and setgid bit.If these bits are set, then the program runs with the user or group of how the file is own. The password program when you run it, it has the setuid bit set, and it's owned by root. When you run it, it's run by root. Since it is owned by root, it is run as root. you hope passwd doesn't have any bugs, otherwise it could corrupt your passwd file.

Why am I talking about this now? Because these sort of programs, setuid programs, are particularly vulnerable to these TOCTTOU vulnerabilities. It will want to access a file, and if you are not careful, it will access the file with the wrong privileges.

With what privileges does /bin/passwd - it runs w/ root privileges - but it has this command line option that lets it modify any file. So if you aren't careful you could use this to modify arbitrary files on the system. Well we should place some restrictions, what do you place on it ? you have it check the owner and group of the file, and the owner and group of the person who invoked the program. It will only let me modify files that are owned by that user. Standard check, but how does it know a file is owned by that user, it has to check (a system call to ask for the inode). It has to query the filesystem and ask what's that file, who is it owned by? What if I change the file after it's done the check but before it's modified the file.

Symbolic link - treat it as a password file, you replace it quickly and have it point to the real password file. If I win the race, in between the time it checks and the time it modifies the file, I can get in there, and do my damage. Particular problem at the filesystem level. All the stuff we talked about with locks, can't apply to files. It's harder to do atomic operations on files.

If you are programming a system and you are messing with temp files, make sure you use their mechanisms for doing secure temporary file allocation. One of the ways to do this, is it creates a very hard name to guess for your temporary files, it's hard for them to mess with it. Is it perfect no? but it protects you against these types of attacks. A lot of the built in programs to create temporary files claim they are secure, but they are not.

You know that a privileged program is going to mess with the file, it's doing a check for something, it's going to do it, and then it's going to modify it. You setup your race so that you wait for it to do it's check, once that check is done, make sure it passes, and then swap it around. How can I win that? I can potentially do a fork bomb, slow the system down. To win the race.

Switch gears, start thinking about TLBs. TLBs is the probably the most annoying term in operating systems - Translation Lookaside Buffer. When you are going from virtual to physical addresses. We do this on not a per address basis, but a per page basis. It's not actually a table, it's a tree with a couple of levels. But the important thing to realize, is that it's a big complex data structure that you can't access on every data access. It's too slow, why is it too slow? In order to access a big data structure, you have to access multiple locations in memory. The thing you are trying to do quickly, you are having to do many operations that are much slower in order to do the fast operation. The TLB is a cache of virtual to physical mappings. The smallest cache on the chip. question is why is it small? It's really weird. What's the operation I want to do on this cache. I want to give it a virtual address,a nd I want it to give me the corresponding physical address. how can you do a really fast table lookup?

1 have the table sorted, you go through and see where in the table it is and you find it, is a binary search of the table faster
2 what's a data structure that is a key value mapping - hash map - constant time - a hash table isn't fast enough for this
so instead of that, the fastest way to do a key value resolution - a content addressable memory. how our brains work - when you want to find something in memory, what's your way of accessing ram? It's by the address - RAM address accessible - ok I want whataver the value is at location 2000, you get that value. do you find anything else in memory, no you find out that particular thing. A TLB instead has all these entries. (each is a page table entry)

[[File:TLB.png]]

File:TLB.png

2014-10-10T13:57:19Z

Afry:

Operating Systems 2014F Lecture 11

2014-10-10T13:57:04Z

Afry:

Operating Systems 2014F Lecture 11

2014-10-10T13:45:16Z

Afry:

Operating Systems 2014F Lecture 11

2014-10-10T13:43:24Z

Afry:

Operating Systems 2014F Lecture 11

2014-10-10T13:40:18Z

Afry:

Operating Systems 2014F Lecture 11

2014-10-10T13:37:27Z

Afry:

Operating Systems 2014F Lecture 11

2014-10-10T13:35:54Z

Afry:

Operating Systems 2014F Lecture 11

2014-10-10T13:32:28Z

Afry:

Operating Systems 2014F Lecture 11

2014-10-10T13:15:46Z

Afry:

Operating Systems 2014F Lecture 11

2014-10-10T13:11:55Z

Afry:

Operating Systems 2014F Lecture 11

2014-10-10T13:11:37Z

Afry:

Operating Systems 2014F Lecture 11

2014-10-10T13:07:49Z

Afry: Created page with "Dining Philosophers problem When can you have deadlock? 4 conditions must apply - mutual exclusion - hold and wait - you can grab a lock and wait for the next one, you ca..."

Operating Systems 2014F Lecture 10

2014-10-08T13:56:44Z

Afry:

Assignment 5 visit:

[[File:Pagetableqass5.png]]

By using base 16 vs. base 10 - yes the alphabet is larger. when you talk about hex vs. decimal, you are changing how we read it. There's something more correct about using hex vs. base 10. A Hexadecimal digit represents 4 bits. How many bits does base 10 represent? (somewhere between 8 and 16 - a fractional bit) it's messy. Decimal is not the native base of a computer. Base 16 maps perfectly to base 2. That's why you also see octal - 3 bits vs. 4 bits. (but a reduced range), when you use hexadecimal it represents 4 bits, and is a bit cleaner.

When you say offset is a power of two, the page size is also a power of two. the storage used to encode the page size determines the range of the offset. It is a specific byte in a page. What is the range of bytes that can go in a page. Classic x86

what is the range of offsets of a 4 k page? The offsets are 0 - 4095. Another thing that you will need for the assignment, it's a question that refers to Peterson's algorithm. It doesn't work on modern machines. The big idea is the memory hierarchy. Key assumption: when one processor writes to memory the other processors will see that write, that value. Is that true in modern systems? No.

Concurrency Mechanisms:

locks
condition variables
Semaphores

you build all of these using the same mechanisms. Let's you check a value and change it in one atomic operation. The basic functionality of the hardware. The semantics are different in these higher level constructs. When do you use what? What are the purpose of these things? there's huge amounts of overlap. what you are talking about with these things, is that you have some sort of storage that has atomic operations so you can check it's value adn change it, all at one time (so you don't have race conditions when you manipulate the value) then you have a queue of threads.

The difference between these, is related to how you deal with the interaction between the storage and the queue of threads.
IF you have a lock, grab lock, and release the lock, does releasing the lock always succeed? Yes, if you have the lock, and you release it, it always succeeds. grab lock can block. when we talk about blocking, it's going to sit there and wait for the condition to happen, nothing is going to happen to that thread until that condition is met. You often have a try lock, try lock means check to see if the lock is grabbable or not. it will grab the lock if it's available, but if it's not available it will return, it's non-blocking, because maybe you don't want to sit there and block. Most of the time you will try to grab the lock and if the lock fails, it will wait until grabbing the lock succeeds.

what if 5 processes try to grab the lock all at once, what do the rest do? they are all blocked, there are two ways for them to wait. One is to just go into an infinite loop, it just keeps trying trying trying until it succeeds. It is sometimes a good idea, this is called a spin lock. If the CPU with running the spin lock, doing nothing else, it is useless. Not very good for making use of efficient resources. You expect the time for you to wait is very small. no one is going to be using hte lock structure for very long. it's better to sit there and wait. It's like if you go to a gov't office, and you want something to happen, you will probably have to wait in line. When you are spin locking, you aresitting in the queue doing nothing. What you can also do instead is you can have a wait queue: in here we say, they queue up, they wait, you are freeing up the resources to do something else. the particular processor running the thread, it encounters things behind it, it puts that thread to sleep. what does sleep mean? Sleep means, the thread goes to sleep that implies something about the scheduling of the thread. IT is kicked off the cpu it's running on. I am not going to consume any resources right now, when you wake up, in biological terms, it's still consuming resources. It's still consuming space resources (still consuming ram, still living in cache but it is not being actively run by a core. When you say a thread yields, I don't need this CPU anymore. when you call sleep you are making a system call that says, I don't want to do anything for at least 10 seconds. You are taken out of the list of running processes, you are put on the list of block processes, and you wait for the timer to go off. Then your process is brought back in and is allowed to run. It depends on the time of the CPU. In general you should sleep at least for that amount of time.

[[File:locks.png]]

When we are talking about locks we are talking about multiple threads. Processes do not have shared storage. It's logically separate copies. For threads they share this address space, and use this to coordinate their operations. The lock is not available, it queues up waiting for the lock. You have these threads lined up waiting for the service

Wait = sleep

if it is actually actively waiting running on the cpu that is it spinning, you don't queue up with spinning, because it's every core is just sitting there spinning. For 5 threads to be blocked on a spin lock, all 5 threads are running on 5 cores. You don't use spin locks on things you expect to wait long for. Sometimes you want to use a spin lock. Because it takes effort to put it to sleep. It's a relatively expensive thing to put a thread to sleep, and then have it wake up. When you put a thread to sleep it potentially can involve a context switch, it depends on how threads are implemented. It involves a system call, because then you call the sleep(). If you are going to be waiting less than the time it takes to execute 10 - 50 instructions, you want to use a spin lock.

with a lock all the threads queue up, then when the one that was active gets done. It's a fifo queue. They take their turns.

[[File:Conditionvariables.png]]

A condition variable has the same basic idea. we have a condition that we want to check and we have a wait queue, but the difference is, is in how we treat the queue. I'm waiting for something to happen, a key to be pressed / waiting for something to come in for the network, waiting for something to happen, the other threads too have gotten so far in the computation, what you are doing is check condition (blocking), check condition (non-blocking) and set condition. - seems kind of the same as a lock. What's really happening here is the normal expected thing when you check for the condition, is that you normally do the blocking one. i'm going to sleep until the condition has happened. The set condition says that it happened. The difference lies in how you treat the queue. with a lock you are waiting for exclusive access to something (an entire data structure, a tree or something) Only one thread can access the resource at a time, enforcing mutual exclusion, we want serial semantics. With a condition variable we are not trying to enforce serial semantics. Tell everyone! When the condition becomes true, everyone in the queue wakes up and gets service. Mechanism basic pieces underlying is the same but the semantics are different. Think of a club, are the doors open? Everyone flows in. Instead of sitting at the passport office, 1 person goes up, another person goes up. In a concurrent appplication you will use these different constructs. It's generalyl about protecting access to a datastructure. where as a condition variable is about waiting for some state of computation / state of the environment has come to pass. In the locks its saying can I get exclusive access to the resource? Normally in code that uses locks, there will be some mutex. (mutual exclusion) binary semaphore

[[File:semaphore.png]]

semaphores : up / down a general counting semaphore the value can be any integer. You can use semaphores to implement both condition variables and locks. The idea of a semaphore is: down blocks only if it is less than 0

when you say up, you increment it back up. A lock in this case, is if you set the value to 1. Instead of saying 1 can be present, really a semaphore is like a bathroom, 4 people can use the bathroom at a time. When someone goes in, someone's come out, now someone can go back in, if the room is empty, there is a certain number of resources, you can go in, but if it fills up, everyone is waiting. In practice if you are looking at how people build things. People use mutexs all the time condition variables have clear semantics, semaphores are more of a building block, I want to only have so many threads running at the same time. Its' much harder to reason about. You have to be very careful with your downs and ups. They are confusing but you should know about what they are.

What's in the midterm, the basic coverage of the midterm is the assignments and the tutorials, with a focus on the assignments. The key thing with pthreads, it's supposed to be portable across unix like systems. The exact system calls used for this can vary. when you do pthreads, it's doing the low level assembly. that's present as well. it's a mix of things, it can be implemented in user space partly and partly in the kernel. The mapping between the lowl evel mechanisms and the api is non-trivial. When you look at the code, it's surprisingly complex. Through to assignment 5 is going to be on the midterm. If you look at assignment 5, it has bits and pieces of the lectures last week and this week.

producer / consumer - you should think of a pipe. On the command line you could do these things like: sort foo | uniq

it will sort the lines of the file, and the thing uniq will remove any duplicates of the lines. If I had another command to add in linebreaks for every word. pipe operator means :take stdout from one program, and provide it as stdin for the following program.

We are talking about processes, the one on the left side of the pipe is the producer. The one on the right is the consumer. The pipe operation does not work on a file, it works on processes. ls you always think of as terminating, but you could do something that produces never ending values.

tail is runnign continuously, is this thing constantly cycling just waiting? is there input? no, there's not, is it every character that comes out of tail, does each character get processed by grep, is that how these are interacting? that would be dumb, we want these to be coordinated, but we don't want it to have to be, that every byte in the kernel it has to control the relative rates of how fast these run.

cat /dev/random | sort | uniq

tail -f /var/log/syslog | grep anil

Coordination problem, the producer could run faster, or the consumer runs faster. If the producer runs too fast for the consumer, you may want to to them to sleep. If the consumer runs too fast, you want it to run for a while and put it to sleep until you can let the producer to produce something.

Tail should not keep running, unless grep can keep up. this is the producer consumer pattern. in order for it to work you have to use condition variables and locks.
The key idea here is that you have some storage between these. You have instead a buffer, the buffer can be arbitrarily sized, it can grow and shrink, you let the producer produce for a while and the consumer consume for a while. A buffer is a circular list, when you get to the end you start at the beginning.

Guarantee exclusive access, you have to have a mutex to ensure they don't step on each other. (for the entire data structure) - buffer make sure both aren't messing with this at the same time.

Operating Systems 2014F Lecture 10

2014-10-08T13:53:36Z

Afry:

Operating Systems 2014F Lecture 10

2014-10-08T13:51:59Z

Afry:

Assignment 5 visit:

[[File:Pagetableqass5.png]]

By using base 16 vs. base 10 - yes the alphabet is larger. when you talk about hex vs. decimal, you are changing how we read it. There's something more correct about using hex vs. base 10. A Hexadecimal digit represents 4 bits. How many bits does base 10 represent? (somewhere between 8 and 16 - a fractional bit) it's messy. Decimal is not the native base of a computer. Base 16 maps perfectly to base 2. That's why you also see octal - 3 bits vs. 4 bits. (but a reduced range), when you use hexadecimal it represents 4 bits, and is a bit cleaner.

When you say offset is a power of two, the page size is also a power of two. the storage used to encode the page size determines the range of the offset. It is a specific byte in a page. What is the range of bytes that can go in a page. Classic x86

what is the range of offsets of a 4 k page? The offsets are 0 - 4095. Another thing that you will need for the assignment, it's a question that refers to Peterson's algorithm. It doesn't work on modern machines. The big idea is the memory hierarchy. Key assumption: when one processor writes to memory the other processors will see that write, that value. Is that true in modern systems? No.

Concurrency Mechanisms:

locks
condition variables
Semaphores

you build all of these using the same mechanisms. Let's you check a value and change it in one atomic operation. The basic functionality of the hardware. The semantics are different in these higher level constructs. When do you use what? What are the purpose of these things? there's huge amounts of overlap. what you are talking about with these things, is that you have some sort of storage that has atomic operations so you can check it's value adn change it, all at one time (so you don't have race conditions when you manipulate the value) then you have a queue of threads.

The difference between these, is related to how you deal with the interaction between the storage and the queue of threads.
IF you have a lock, grab lock, and release the lock, does releasing the lock always succeed? Yes, if you have the lock, and you release it, it always succeeds. grab lock can block. when we talk about blocking, it's going to sit there and wait for the condition to happen, nothing is going to happen to that thread until that condition is met. You often have a try lock, try lock means check to see if the lock is grabbable or not. it will grab the lock if it's available, but if it's not available it will return, it's non-blocking, because maybe you don't want to sit there and block. Most of the time you will try to grab the lock and if the lock fails, it will wait until grabbing the lock succeeds.

what if 5 processes try to grab the lock all at once, what do the rest do? they are all blocked, there are two ways for them to wait. One is to just go into an infinite loop, it just keeps trying trying trying until it succeeds. It is sometimes a good idea, this is called a spin lock. If the CPU with running the spin lock, doing nothing else, it is useless. Not very good for making use of efficient resources. You expect the time for you to wait is very small. no one is going to be using hte lock structure for very long. it's better to sit there and wait. It's like if you go to a gov't office, and you want something to happen, you will probably have to wait in line. When you are spin locking, you aresitting in the queue doing nothing. What you can also do instead is you can have a wait queue: in here we say, they queue up, they wait, you are freeing up the resources to do something else. the particular processor running the thread, it encounters things behind it, it puts that thread to sleep. what does sleep mean? Sleep means, the thread goes to sleep that implies something about the scheduling of the thread. IT is kicked off the cpu it's running on. I am not going to consume any resources right now, when you wake up, in biological terms, it's still consuming resources. It's still consuming space resources (still consuming ram, still living in cache but it is not being actively run by a core. When you say a thread yields, I don't need this CPU anymore. when you call sleep you are making a system call that says, I don't want to do anything for at least 10 seconds. You are taken out of the list of running processes, you are put on the list of block processes, and you wait for the timer to go off. Then your process is brought back in and is allowed to run. It depends on the time of the CPU. In general you should sleep at least for that amount of time.

[[File:locks.png]]

When we are talking about locks we are talking about multiple threads. Processes do not have shared storage. It's logically separate copies. For threads they share this address space, and use this to coordinate their operations. The lock is not available, it queues up waiting for the lock. You have these threads lined up waiting for the service

Wait = sleep

if it is actually actively waiting running on the cpu that is it spinning, you don't queue up with spinning, because it's every core is just sitting there spinning. For 5 threads to be blocked on a spin lock, all 5 threads are running on 5 cores. You don't use spin locks on things you expect to wait long for. Sometimes you want to use a spin lock. Because it takes effort to put it to sleep. It's a relatively expensive thing to put a thread to sleep, and then have it wake up. When you put a thread to sleep it potentially can involve a context switch, it depends on how threads are implemented. It involves a system call, because then you call the sleep(). If you are going to be waiting less than the time it takes to execute 10 - 50 instructions, you want to use a spin lock.

with a lock all the threads queue up, then when the one that was active gets done. It's a fifo queue. They take their turns.

[[File:Conditionvariables.png]]

A condition variable has the same basic idea. we have a condition that we want to check and we have a wait queue, but the difference is, is in how we treat the queue. I'm waiting for something to happen, a key to be pressed / waiting for something to come in for the network, waiting for something to happen, the other threads too have gotten so far in the computation, what you are doing is check condition (blocking), check condition (non-blocking) and set condition. - seems kind of the same as a lock. What's really happening here is the normal expected thing when you check for the condition, is that you normally do the blocking one. i'm going to sleep until the condition has happened. The set condition says that it happened. The difference lies in how you treat the queue. with a lock you are waiting for exclusive access to something (an entire data structure, a tree or something) Only one thread can access the resource at a time, enforcing mutual exclusion, we want serial semantics. With a condition variable we are not trying to enforce serial semantics. Tell everyone! When the condition becomes true, everyone in the queue wakes up and gets service. Mechanism basic pieces underlying is the same but the semantics are different. Think of a club, are the doors open? Everyone flows in. Instead of sitting at the passport office, 1 person goes up, another person goes up. In a concurrent appplication you will use these different constructs. It's generalyl about protecting access to a datastructure. where as a condition variable is about waiting for some state of computation / state of the environment has come to pass. In the locks its saying can I get exclusive access to the resource? Normally in code that uses locks, there will be some mutex. (mutual exclusion) binary semaphore

[[File:semaphore.png]]

semaphores : up / down a general counting semaphore the value can be any integer. You can use semaphores to implement both condition variables and locks. The idea of a semaphore is: down blocks only if it is less than 0

when you say up, you increment it back up. A lock in this case, is if you set the value to 1. Instead of saying 1 can be present, really a semaphore is like a bathroom, 4 people can use the bathroom at a time. When someone goes in, someone's come out, now someone can go back in, if the room is empty, there is a certain number of resources, you can go in, but if it fills up, everyone is waiting. In practice if you are looking at how people build things. People use mutexs all the time condition variables have clear semantics, semaphores are more of a building block, I want to only have so many threads running at the same time. Its' much harder to reason about. You have to be very careful with your downs and ups. They are confusing but you should know about what they are.

What's in the midterm, the basic coverage of the midterm is the assignments and the tutorials, with a focus on the assignments. The key thing with pthreads, it's supposed to be portable across unix like systems. The exact system calls used for this can vary. when you do pthreads, it's doing the low level assembly. that's present as well. it's a mix of things, it can be implemented in user space partly and partly in the kernel. The mapping between the lowl evel mechanisms and the api is non-trivial. When you look at the code, it's surprisingly complex. Through to assignment 5 is going to be on the midterm. If you look at assignment 5, it has bits and pieces of the lectures last week and this week.

producer / consumer - you should think of a pipe. On the command line you could do these things like: sort foo | uniq

it will sort the lines of the file, and the thing uniq will remove any duplicates of the lines. If I had another command to add in linebreaks for every word. pipe operator means :take stdout from one program, and provide it as stdin for the following program.

We are talking about processes, the one on the left side of the pipe is the producer. The one on the right is the consumer. The pipe operation does not work on a file, it works on processes. ls you always think of as terminating, but you could do something that produces never ending values.

tail is runnign continuously, is this thing constantly cycling just waiting? is there input? no, there's not, is it every character that comes out of tail, does each character get processed by grep, is that how these are interacting? that would be dumb, we want these to be coordinated, but we don't want it to have to be, that every byte in the kernel it has to control the relative rates of how fast these run.

cat /dev/random | sort | uniq

tail -f /var/log/syslog | grep anil

coordination problem, the producer could run faster, or the consumer runs faster. If the producer runs too fast for the consumer, you may want to to them to sleep. If the consumer runs too fast, you want it to run for a while and put it to sleep until you can let the producer to produce something.

Tail should not keep running, unless grep can keep up. this is the producer consumer pattern. in order for it to work you have to use condition variables and locks.

Operating Systems 2014F Lecture 10

2014-10-08T13:44:34Z

Afry:

Operating Systems 2014F Lecture 10

2014-10-08T13:43:09Z

Afry:

File:Pagetableqass5.png

2014-10-08T13:39:19Z

Afry:

Operating Systems 2014F Lecture 10

2014-10-08T13:39:06Z

Afry:

Assignment 5 visit:

By using base 16 vs. base 10 - yes the alphabet is larger. when you talk about hex vs. decimal, you are changing how we read it. There's something more correct about using hex vs. base 10. A Hexadecimal digit represents 4 bits. How many bits does base 10 represent? (somewhere between 8 and 16 - a fractional bit) it's messy. Decimal is not the native base of a computer. Base 16 maps perfectly to base 2. That's why you also see octal - 3 bits vs. 4 bits. (but a reduced range), when you use hexadecimal it represents 4 bits, and is a bit cleaner.

When you say offset is a power of two, the page size is also a power of two. the storage used to encode the page size determines the range of the offset. It is a specific byte in a page. What is the range of bytes that can go in a page. Classic x86

what is the range of offsets of a 4 k page? The offsets are 0 - 4095. Another thing that you will need for the assignment, it's a question that refers to Peterson's algorithm. It doesn't work on modern machines. The big idea is the memory hierarchy. Key assumption: when one processor writes to memory the other processors will see that write, that value. Is that true in modern systems? No.

Concurrency Mechanisms:

locks
condition variables
Semaphores

you build all of these using the same mechanisms. Let's you check a value and change it in one atomic operation. The basic functionality of the hardware. The semantics are different in these higher level constructs. When do you use what? What are the purpose of these things? there's huge amounts of overlap. what you are talking about with these things, is that you have some sort of storage that has atomic operations so you can check it's value adn change it, all at one time (so you don't have race conditions when you manipulate the value) then you have a queue of threads.

The difference between these, is related to how you deal with the interaction between the storage and the queue of threads.
IF you have a lock, grab lock, and release the lock, does releasing the lock always succeed? Yes, if you have the lock, and you release it, it always succeeds. grab lock can block. when we talk about blocking, it's going to sit there and wait for the condition to happen, nothing is going to happen to that thread until that condition is met. You often have a try lock, try lock means check to see if the lock is grabbable or not. it will grab the lock if it's available, but if it's not available it will return, it's non-blocking, because maybe you don't want to sit there and block. Most of the time you will try to grab the lock and if the lock fails, it will wait until grabbing the lock succeeds.

what if 5 processes try to grab the lock all at once, what do the rest do? they are all blocked, there are two ways for them to wait. One is to just go into an infinite loop, it just keeps trying trying trying until it succeeds. It is sometimes a good idea, this is called a spin lock. If the CPU with running the spin lock, doing nothing else, it is useless. Not very good for making use of efficient resources. You expect the time for you to wait is very small. no one is going to be using hte lock structure for very long. it's better to sit there and wait. It's like if you go to a gov't office, and you want something to happen, you will probably have to wait in line. When you are spin locking, you aresitting in the queue doing nothing. What you can also do instead is you can have a wait queue: in here we say, they queue up, they wait, you are freeing up the resources to do something else. the particular processor running the thread, it encounters things behind it, it puts that thread to sleep. what does sleep mean? Sleep means, the thread goes to sleep that implies something about the scheduling of the thread. IT is kicked off the cpu it's running on. I am not going to consume any resources right now, when you wake up, in biological terms, it's still consuming resources. It's still consuming space resources (still consuming ram, still living in cache but it is not being actively run by a core. When you say a thread yields, I don't need this CPU anymore. when you call sleep you are making a system call that says, I don't want to do anything for at least 10 seconds. You are taken out of the list of running processes, you are put on the list of block processes, and you wait for the timer to go off. Then your process is brought back in and is allowed to run. It depends on the time of the CPU. In general you should sleep at least for that amount of time.

[[File:locks.png]]

When we are talking about locks we are talking about multiple threads. Processes do not have shared storage. It's logically separate copies. For threads they share this address space, and use this to coordinate their operations. The lock is not available, it queues up waiting for the lock. You have these threads lined up waiting for the service

Wait = sleep

if it is actually actively waiting running on the cpu that is it spinning, you don't queue up with spinning, because it's every core is just sitting there spinning. For 5 threads to be blocked on a spin lock, all 5 threads are running on 5 cores. You don't use spin locks on things you expect to wait long for. Sometimes you want to use a spin lock. Because it takes effort to put it to sleep. It's a relatively expensive thing to put a thread to sleep, and then have it wake up. When you put a thread to sleep it potentially can involve a context switch, it depends on how threads are implemented. It involves a system call, because then you call the sleep(). If you are going to be waiting less than the time it takes to execute 10 - 50 instructions, you want to use a spin lock.

with a lock all the threads queue up, then when the one that was active gets done. It's a fifo queue. They take their turns.

[[File:Conditionvariables.png]]

A condition variable has the same basic idea. we have a condition that we want to check and we have a wait queue, but the difference is, is in how we treat the queue. I'm waiting for something to happen, a key to be pressed / waiting for something to come in for the network, waiting for something to happen, the other threads too have gotten so far in the computation, what you are doing is check condition (blocking), check condition (non-blocking) and set condition. - seems kind of the same as a lock. What's really happening here is the normal expected thing when you check for the condition, is that you normally do the blocking one. i'm going to sleep until the condition has happened. The set condition says that it happened. The difference lies in how you treat the queue. with a lock you are waiting for exclusive access to something (an entire data structure, a tree or something) Only one thread can access the resource at a time, enforcing mutual exclusion, we want serial semantics. With a condition variable we are not trying to enforce serial semantics. Tell everyone! When the condition becomes true, everyone in the queue wakes up and gets service. Mechanism basic pieces underlying is the same but the semantics are different. Think of a club, are the doors open? Everyone flows in. Instead of sitting at the passport office, 1 person goes up, another person goes up. In a concurrent appplication you will use these different constructs. It's generalyl about protecting access to a datastructure. where as a condition variable is about waiting for some state of computation / state of the environment has come to pass. In the locks its saying can I get exclusive access to the resource? Normally in code that uses locks, there will be some mutex. (mutual exclusion) binary semaphore

[[File:semaphore.png]]

semaphores : up / down a general counting semaphore the value can be any integer. You can use semaphores to implement both condition variables and locks. The idea of a semaphore is: down blocks only if it is less than 0

when you say up, you increment it back up. A lock in this case, is if you set the value to 1. Instead of saying 1 can be present, really a semaphore is like a bathroom, 4 people can use the bathroom at a time. When someone goes in, someone's come out, now someone can go back in, if the room is empty, there is a certain number of resources, you can go in, but if it fills up, everyone is waiting. In practice if you are looking at how people build things. People use mutexs all the time condition variables have clear semantics, semaphores are more of a building block, I want to only have so many threads running at the same time. Its' much harder to reason about. You have to be very careful with your downs and ups. They are confusing but you should know about what they are.

What's in the midterm, the basic coverage of the midterm is the assignments and the tutorials, with a focus on the assignments. The key thing with pthreads, it's supposed to be portable across unix like systems. The exact system calls used for this can vary. when you do pthreads, it's doing the low level assembly. that's present as well. it's a mix of things, it can be implemented in user space partly and partly in the kernel. The mapping between the lowl evel mechanisms and the api is non-trivial. When you look at the code, it's surprisingly complex. Through to assignment 5 is going to be on the midterm. If you look at assignment 5, it has bits and pieces of the lectures last week and this week.

Operating Systems 2014F Lecture 10

2014-10-08T13:32:41Z

Afry:

By using base 16 vs. base 10 - yes the alphabet is larger. when you talk about hex vs. decimal, you are changing how we read it. There's something more correct about using hex vs. base 10. A Hexadecimal digit represents 4 bits. How many bits does base 10 represent? (somewhere between 8 and 16 - a fractional bit) it's messy. Decimal is not the native base of a computer. Base 16 maps perfectly to base 2. That's why you also see octal - 3 bits vs. 4 bits. (but a reduced range), when you use hexadecimal it represents 4 bits, and is a bit cleaner.

When you say offset is a power of two, the page size is also a power of two. the storage used to encode the page size determines the range of the offset. It is a specific byte in a page. What is the range of bytes that can go in a page. Classic x86

what is the range of offsets of a 4 k page? The offsets are 0 - 4095. Another thing that you will need for the assignment, it's a question that refers to Peterson's algorithm. It doesn't work on modern machines. The big idea is the memory hierarchy. Key assumption: when one processor writes to memory the other processors will see that write, that value. Is that true in modern systems? No.

Concurrency Mechanisms:

locks
condition variables
Semaphores

you build all of these using the same mechanisms. Let's you check a value and change it in one atomic operation. The basic functionality of the hardware. The semantics are different in these higher level constructs. When do you use what? What are the purpose of these things? there's huge amounts of overlap. what you are talking about with these things, is that you have some sort of storage that has atomic operations so you can check it's value adn change it, all at one time (so you don't have race conditions when you manipulate the value) then you have a queue of threads.

The difference between these, is related to how you deal with the interaction between the storage and the queue of threads.
IF you have a lock, grab lock, and release the lock, does releasing the lock always succeed? Yes, if you have the lock, and you release it, it always succeeds. grab lock can block. when we talk about blocking, it's going to sit there and wait for the condition to happen, nothing is going to happen to that thread until that condition is met. You often have a try lock, try lock means check to see if the lock is grabbable or not. it will grab the lock if it's available, but if it's not available it will return, it's non-blocking, because maybe you don't want to sit there and block. Most of the time you will try to grab the lock and if the lock fails, it will wait until grabbing the lock succeeds.

what if 5 processes try to grab the lock all at once, what do the rest do? they are all blocked, there are two ways for them to wait. One is to just go into an infinite loop, it just keeps trying trying trying until it succeeds. It is sometimes a good idea, this is called a spin lock. If the CPU with running the spin lock, doing nothing else, it is useless. Not very good for making use of efficient resources. You expect the time for you to wait is very small. no one is going to be using hte lock structure for very long. it's better to sit there and wait. It's like if you go to a gov't office, and you want something to happen, you will probably have to wait in line. When you are spin locking, you aresitting in the queue doing nothing. What you can also do instead is you can have a wait queue: in here we say, they queue up, they wait, you are freeing up the resources to do something else. the particular processor running the thread, it encounters things behind it, it puts that thread to sleep. what does sleep mean? Sleep means, the thread goes to sleep that implies something about the scheduling of the thread. IT is kicked off the cpu it's running on. I am not going to consume any resources right now, when you wake up, in biological terms, it's still consuming resources. It's still consuming space resources (still consuming ram, still living in cache but it is not being actively run by a core. When you say a thread yields, I don't need this CPU anymore. when you call sleep you are making a system call that says, I don't want to do anything for at least 10 seconds. You are taken out of the list of running processes, you are put on the list of block processes, and you wait for the timer to go off. Then your process is brought back in and is allowed to run. It depends on the time of the CPU. In general you should sleep at least for that amount of time.

[[File:locks.png]]

When we are talking about locks we are talking about multiple threads. Processes do not have shared storage. It's logically separate copies. For threads they share this address space, and use this to coordinate their operations. The lock is not available, it queues up waiting for the lock. You have these threads lined up waiting for the service

Wait = sleep

if it is actually actively waiting running on the cpu that is it spinning, you don't queue up with spinning, because it's every core is just sitting there spinning. For 5 threads to be blocked on a spin lock, all 5 threads are running on 5 cores. You don't use spin locks on things you expect to wait long for. Sometimes you want to use a spin lock. Because it takes effort to put it to sleep. It's a relatively expensive thing to put a thread to sleep, and then have it wake up. When you put a thread to sleep it potentially can involve a context switch, it depends on how threads are implemented. It involves a system call, because then you call the sleep(). If you are going to be waiting less than the time it takes to execute 10 - 50 instructions, you want to use a spin lock.

with a lock all the threads queue up, then when the one that was active gets done. It's a fifo queue. They take their turns.

[[File:Conditionvariables.png]]

A condition variable has the same basic idea. we have a condition that we want to check and we have a wait queue, but the difference is, is in how we treat the queue. I'm waiting for something to happen, a key to be pressed / waiting for something to come in for the network, waiting for something to happen, the other threads too have gotten so far in the computation, what you are doing is check condition (blocking), check condition (non-blocking) and set condition. - seems kind of the same as a lock. What's really happening here is the normal expected thing when you check for the condition, is that you normally do the blocking one. i'm going to sleep until the condition has happened. The set condition says that it happened. The difference lies in how you treat the queue. with a lock you are waiting for exclusive access to something (an entire data structure, a tree or something) Only one thread can access the resource at a time, enforcing mutual exclusion, we want serial semantics. With a condition variable we are not trying to enforce serial semantics. Tell everyone! When the condition becomes true, everyone in the queue wakes up and gets service. Mechanism basic pieces underlying is the same but the semantics are different. Think of a club, are the doors open? Everyone flows in. Instead of sitting at the passport office, 1 person goes up, another person goes up. In a concurrent appplication you will use these different constructs. It's generalyl about protecting access to a datastructure. where as a condition variable is about waiting for some state of computation / state of the environment has come to pass. In the locks its saying can I get exclusive access to the resource? Normally in code that uses locks, there will be some mutex. (mutual exclusion) binary semaphore

[[File:semaphore.png]]

semaphores : up / down a general counting semaphore the value can be any integer. You can use semaphores to implement both condition variables and locks. The idea of a semaphore is: down blocks only if it is less than 0

when you say up, you increment it back up. A lock in this case, is if you set the value to 1. Instead of saying 1 can be present, really a semaphore is like a bathroom, 4 people can use the bathroom at a time. When someone goes in, someone's come out, now someone can go back in, if the room is empty, there is a certain number of resources, you can go in, but if it fills up, everyone is waiting. In practice if you are looking at how people build things. People use mutexs all the time condition variables have clear semantics, semaphores are more of a building block, I want to only have so many threads running at the same time. Its' much harder to reason about. You have to be very careful with your downs and ups. They are confusing but you should know about what they are.

What's in the midterm, the basic coverage of the midterm is the assignments and the tutorials, with a focus on the assignments. The key thing with pthreads, it's supposed to be portable across unix like systems. The exact system calls used for this can vary. when you do pthreads, it's doing the low level assembly. that's present as well. it's a mix of things, it can be implemented in user space partly and partly in the kernel. The mapping between the lowl evel mechanisms and the api is non-trivial. When you look at the code, it's surprisingly complex. Through to assignment 5 is going to be on the midterm. If you look at assignment 5, it has bits and pieces of the lectures last week and this week.

Operating Systems 2014F Lecture 10

2014-10-08T13:31:07Z

Afry:

By using base 16 vs. base 10 - yes the alphabet is larger. when you talk about hex vs. decimal, you are changing how we read it. There's something more correct about using hex vs. base 10. A Hexadecimal digit represents 4 bits. How many bits does base 10 represent? (somewhere between 8 and 16 - a fractional bit) it's messy. Decimal is not the native base of a computer. Base 16 maps perfectly to base 2. That's why you also see octal - 3 bits vs. 4 bits. (but a reduced range), when you use hexadecimal it represents 4 bits, and is a bit cleaner.

When you say offset is a power of two, the page size is also a power of two. the storage used to encode the page size determines the range of the offset. It is a specific byte in a page. What is the range of bytes that can go in a page. Classic x86

what is the range of offsets of a 4 k page? The offsets are 0 - 4095. Another thing that you will need for the assignment, it's a question that refers to Peterson's algorithm. It doesn't work on modern machines. The big idea is the memory hierarchy. Key assumption: when one processor writes to memory the other processors will see that write, that value. Is that true in modern systems? No.

Concurrency Mechanisms:

locks
condition variables
Semaphores

you build all of these using the same mechanisms. Let's you check a value and change it in one atomic operation. The basic functionality of the hardware. The semantics are different in these higher level constructs. When do you use what? What are the purpose of these things? there's huge amounts of overlap. what you are talking about with these things, is that you have some sort of storage that has atomic operations so you can check it's value adn change it, all at one time (so you don't have race conditions when you manipulate the value) then you have a queue of threads.

The difference between these, is related to how you deal with the interaction between the storage and the queue of threads.
IF you have a lock, grab lock, and release the lock, does releasing the lock always succeed? Yes, if you have the lock, and you release it, it always succeeds. grab lock can block. when we talk about blocking, it's going to sit there and wait for the condition to happen, nothing is going to happen to that thread until that condition is met. You often have a try lock, try lock means check to see if the lock is grabbable or not. it will grab the lock if it's available, but if it's not available it will return, it's non-blocking, because maybe you don't want to sit there and block. Most of the time you will try to grab the lock and if the lock fails, it will wait until grabbing the lock succeeds.

what if 5 processes try to grab the lock all at once, what do the rest do? they are all blocked, there are two ways for them to wait. One is to just go into an infinite loop, it just keeps trying trying trying until it succeeds. It is sometimes a good idea, this is called a spin lock. If the CPU with running the spin lock, doing nothing else, it is useless. Not very good for making use of efficient resources. You expect the time for you to wait is very small. no one is going to be using hte lock structure for very long. it's better to sit there and wait. It's like if you go to a gov't office, and you want something to happen, you will probably have to wait in line. When you are spin locking, you aresitting in the queue doing nothing. What you can also do instead is you can have a wait queue: in here we say, they queue up, they wait, you are freeing up the resources to do something else. the particular processor running the thread, it encounters things behind it, it puts that thread to sleep. what does sleep mean? Sleep means, the thread goes to sleep that implies something about the scheduling of the thread. IT is kicked off the cpu it's running on. I am not going to consume any resources right now, when you wake up, in biological terms, it's still consuming resources. It's still consuming space resources (still consuming ram, still living in cache but it is not being actively run by a core. When you say a thread yields, I don't need this CPU anymore. when you call sleep you are making a system call that says, I don't want to do anything for at least 10 seconds. You are taken out of the list of running processes, you are put on the list of block processes, and you wait for the timer to go off. Then your process is brought back in and is allowed to run. It depends on the time of the CPU. In general you should sleep at least for that amount of time.

When we are talking about locks we are talking about multiple threads. Processes do not have shared storage. It's logically separate copies. For threads they share this address space, and use this to coordinate their operations. The lock is not available, it queues up waiting for the lock. You have these threads lined up waiting for the service

Wait = sleep

if it is actually actively waiting running on the cpu that is it spinning, you don't queue up with spinning, because it's every core is just sitting there spinning. For 5 threads to be blocked on a spin lock, all 5 threads are running on 5 cores. You don't use spin locks on things you expect to wait long for. Sometimes you want to use a spin lock. Because it takes effort to put it to sleep. It's a relatively expensive thing to put a thread to sleep, and then have it wake up. When you put a thread to sleep it potentially can involve a context switch, it depends on how threads are implemented. It involves a system call, because then you call the sleep(). If you are going to be waiting less than the time it takes to execute 10 - 50 instructions, you want to use a spin lock.

with a lock all the threads queue up, then when the one that was active gets done. It's a fifo queue. They take their turns.

[[File:Conditionvariables.png]]

A condition variable has the same basic idea. we have a condition that we want to check and we have a wait queue, but the difference is, is in how we treat the queue. I'm waiting for something to happen, a key to be pressed / waiting for something to come in for the network, waiting for something to happen, the other threads too have gotten so far in the computation, what you are doing is check condition (blocking), check condition (non-blocking) and set condition. - seems kind of the same as a lock. What's really happening here is the normal expected thing when you check for the condition, is that you normally do the blocking one. i'm going to sleep until the condition has happened. The set condition says that it happened. The difference lies in how you treat the queue. with a lock you are waiting for exclusive access to something (an entire data structure, a tree or something) Only one thread can access the resource at a time, enforcing mutual exclusion, we want serial semantics. With a condition variable we are not trying to enforce serial semantics. Tell everyone! When the condition becomes true, everyone in the queue wakes up and gets service. Mechanism basic pieces underlying is the same but the semantics are different. Think of a club, are the doors open? Everyone flows in. Instead of sitting at the passport office, 1 person goes up, another person goes up. In a concurrent appplication you will use these different constructs. It's generalyl about protecting access to a datastructure. where as a condition variable is about waiting for some state of computation / state of the environment has come to pass. In the locks its saying can I get exclusive access to the resource? Normally in code that uses locks, there will be some mutex. (mutual exclusion) binary semaphore

[[File:semaphore.png]]

semaphores : up / down a general counting semaphore the value can be any integer. You can use semaphores to implement both condition variables and locks. The idea of a semaphore is: down blocks only if it is less than 0

when you say up, you increment it back up. A lock in this case, is if you set the value to 1. Instead of saying 1 can be present, really a semaphore is like a bathroom, 4 people can use the bathroom at a time. When someone goes in, someone's come out, now someone can go back in, if the room is empty, there is a certain number of resources, you can go in, but if it fills up, everyone is waiting. In practice if you are looking at how people build things. People use mutexs all the time condition variables have clear semantics, semaphores are more of a building block, I want to only have so many threads running at the same time. Its' much harder to reason about. You have to be very careful with your downs and ups. They are confusing but you should know about what they are.

What's in the midterm, the basic coverage of the midterm is the assignments and the tutorials, with a focus on the assignments. The key thing with pthreads, it's supposed to be portable across unix like systems. The exact system calls used for this can vary. when you do pthreads, it's doing the low level assembly. that's present as well. it's a mix of things, it can be implemented in user space partly and partly in the kernel. The mapping between the lowl evel mechanisms and the api is non-trivial. When you look at the code, it's surprisingly complex. Through to assignment 5 is going to be on the midterm. If you look at assignment 5, it has bits and pieces of the lectures last week and this week.

Operating Systems 2014F Lecture 10

2014-10-08T13:27:53Z

Afry:

By using base 16 vs. base 10 - yes the alphabet is larger. when you talk about hex vs. decimal, you are changing how we read it. There's something more correct about using hex vs. base 10. A Hexadecimal digit represents 4 bits. How many bits does base 10 represent? (somewhere between 8 and 16 - a fractional bit) it's messy. Decimal is not the native base of a computer. Base 16 maps perfectly to base 2. That's why you also see octal - 3 bits vs. 4 bits. (but a reduced range), when you use hexadecimal it represents 4 bits, and is a bit cleaner.

When you say offset is a power of two, the page size is also a power of two. the storage used to encode the page size determines the range of the offset. It is a specific byte in a page. What is the range of bytes that can go in a page. Classic x86

what is the range of offsets of a 4 k page? The offsets are 0 - 4095. Another thing that you will need for the assignment, it's a question that refers to Peterson's algorithm. It doesn't work on modern machines. The big idea is the memory hierarchy. Key assumption: when one processor writes to memory the other processors will see that write, that value. Is that true in modern systems? No.

Concurrency Mechanisms:

locks
condition variables
Semaphores

you build all of these using the same mechanisms. Let's you check a value and change it in one atomic operation. The basic functionality of the hardware. The semantics are different in these higher level constructs. When do you use what? What are the purpose of these things? there's huge amounts of overlap. what you are talking about with these things, is that you have some sort of storage that has atomic operations so you can check it's value adn change it, all at one time (so you don't have race conditions when you manipulate the value) then you have a queue of threads.

The difference between these, is related to how you deal with the interaction between the storage and the queue of threads.
IF you have a lock, grab lock, and release the lock, does releasing the lock always succeed? Yes, if you have the lock, and you release it, it always succeeds. grab lock can block. when we talk about blocking, it's going to sit there and wait for the condition to happen, nothing is going to happen to that thread until that condition is met. You often have a try lock, try lock means check to see if the lock is grabbable or not. it will grab the lock if it's available, but if it's not available it will return, it's non-blocking, because maybe you don't want to sit there and block. Most of the time you will try to grab the lock and if the lock fails, it will wait until grabbing the lock succeeds.

what if 5 processes try to grab the lock all at once, what do the rest do? they are all blocked, there are two ways for them to wait. One is to just go into an infinite loop, it just keeps trying trying trying until it succeeds. It is sometimes a good idea, this is called a spin lock. If the CPU with running the spin lock, doing nothing else, it is useless. Not very good for making use of efficient resources. You expect the time for you to wait is very small. no one is going to be using hte lock structure for very long. it's better to sit there and wait. It's like if you go to a gov't office, and you want something to happen, you will probably have to wait in line. When you are spin locking, you aresitting in the queue doing nothing. What you can also do instead is you can have a wait queue: in here we say, they queue up, they wait, you are freeing up the resources to do something else. the particular processor running the thread, it encounters things behind it, it puts that thread to sleep. what does sleep mean? Sleep means, the thread goes to sleep that implies something about the scheduling of the thread. IT is kicked off the cpu it's running on. I am not going to consume any resources right now, when you wake up, in biological terms, it's still consuming resources. It's still consuming space resources (still consuming ram, still living in cache but it is not being actively run by a core. When you say a thread yields, I don't need this CPU anymore. when you call sleep you are making a system call that says, I don't want to do anything for at least 10 seconds. You are taken out of the list of running processes, you are put on the list of block processes, and you wait for the timer to go off. Then your process is brought back in and is allowed to run. It depends on the time of the CPU. In general you should sleep at least for that amount of time.

When we are talking about locks we are talking about multiple threads. Processes do not have shared storage. It's logically separate copies. For threads they share this address space, and use this to coordinate their operations. The lock is not available, it queues up waiting for the lock. You have these threads lined up waiting for the service

Wait = sleep

if it is actually actively waiting running on the cpu that is it spinning, you don't queue up with spinning, because it's every core is just sitting there spinning. For 5 threads to be blocked on a spin lock, all 5 threads are running on 5 cores. You don't use spin locks on things you expect to wait long for. Sometimes you want to use a spin lock. Because it takes effort to put it to sleep. It's a relatively expensive thing to put a thread to sleep, and then have it wake up. When you put a thread to sleep it potentially can involve a context switch, it depends on how threads are implemented. It involves a system call, because then you call the sleep(). If you are going to be waiting less than the time it takes to execute 10 - 50 instructions, you want to use a spin lock.

with a lock all the threads queue up, then when the one that was active gets done. It's a fifo queue. They take their turns.

[[File:Conditionvariables.png]]

A condition variable has the same basic idea. we have a condition that we want to check and we have a wait queue, but the difference is, is in how we treat the queue. I'm waiting for something to happen, a key to be pressed / waiting for something to come in for the network, waiting for something to happen, the other threads too have gotten so far in the computation, what you are doing is check condition (blocking), check condition (non-blocking) and set condition. - seems kind of the same as a lock. What's really happening here is the normal expected thing when you check for the condition, is that you normally do the blocking one. i'm going to sleep until the condition has happened. The set condition says that it happened. The difference lies in how you treat the queue. with a lock you are waiting for exclusive access to something (an entire data structure, a tree or something) Only one thread can access the resource at a time, enforcing mutual exclusion, we want serial semantics. With a condition variable we are not trying to enforce serial semantics. Tell everyone! When the condition becomes true, everyone in the queue wakes up and gets service. Mechanism basic pieces underlying is the same but the semantics are different. Think of a club, are the doors open? Everyone flows in. Instead of sitting at the passport office, 1 person goes up, another person goes up. In a concurrent appplication you will use these different constructs. It's generalyl about protecting access to a datastructure. where as a condition variable is about waiting for some state of computation / state of the environment has come to pass. In the locks its saying can I get exclusive access to the resource? Normally in code that uses locks, there will be some mutex. (mutual exclusion) binary semaphore

[[File:semaphore.png]]

semaphores : up / down a general counting semaphore the value can be any integer. You can use semaphores to implement both condition variables and locks. The idea of a semaphore is: down blocks only if it is less than 0

when you say up, you increment it back up. A lock in this case, is if you set the value to 1. Instead of saying 1 can be present, really a semaphore is like a bathroom, 4 people can use the bathroom at a time. When someone goes in, someone's come out, now someone can go back in, if the room is empty, there is a certain number of resources, you can go in, but if it fills up, everyone is waiting. In practice if you are looking at how people build things. People use mutexs all the time condition variables have clear semantics, semaphores are more of a building block, I want to only have so many threads running at the same time. Its' much harder to reason about. You have to be very careful with your downs and ups. They are confusing but you should know about what they are.

Operating Systems 2014F Lecture 10

2014-10-08T13:27:16Z

Afry:

By using base 16 vs. base 10 - yes the alphabet is larger. when you talk about hex vs. decimal, you are changing how we read it. There's something more correct about using hex vs. base 10. A Hexadecimal digit represents 4 bits. How many bits does base 10 represent? (somewhere between 8 and 16 - a fractional bit) it's messy. Decimal is not the native base of a computer. Base 16 maps perfectly to base 2. That's why you also see octal - 3 bits vs. 4 bits. (but a reduced range), when you use hexadecimal it represents 4 bits, and is a bit cleaner.

When you say offset is a power of two, the page size is also a power of two. the storage used to encode the page size determines the range of the offset. It is a specific byte in a page. What is the range of bytes that can go in a page. Classic x86

what is the range of offsets of a 4 k page? The offsets are 0 - 4095. Another thing that you will need for the assignment, it's a question that refers to Peterson's algorithm. It doesn't work on modern machines. The big idea is the memory hierarchy. Key assumption: when one processor writes to memory the other processors will see that write, that value. Is that true in modern systems? No.

Concurrency Mechanisms:

locks
condition variables
Semaphores

you build all of these using the same mechanisms. Let's you check a value and change it in one atomic operation. The basic functionality of the hardware. The semantics are different in these higher level constructs. When do you use what? What are the purpose of these things? there's huge amounts of overlap. what you are talking about with these things, is that you have some sort of storage that has atomic operations so you can check it's value adn change it, all at one time (so you don't have race conditions when you manipulate the value) then you have a queue of threads.

The difference between these, is related to how you deal with the interaction between the storage and the queue of threads.
IF you have a lock, grab lock, and release the lock, does releasing the lock always succeed? Yes, if you have the lock, and you release it, it always succeeds. grab lock can block. when we talk about blocking, it's going to sit there and wait for the condition to happen, nothing is going to happen to that thread until that condition is met. You often have a try lock, try lock means check to see if the lock is grabbable or not. it will grab the lock if it's available, but if it's not available it will return, it's non-blocking, because maybe you don't want to sit there and block. Most of the time you will try to grab the lock and if the lock fails, it will wait until grabbing the lock succeeds.

what if 5 processes try to grab the lock all at once, what do the rest do? they are all blocked, there are two ways for them to wait. One is to just go into an infinite loop, it just keeps trying trying trying until it succeeds. It is sometimes a good idea, this is called a spin lock. If the CPU with running the spin lock, doing nothing else, it is useless. Not very good for making use of efficient resources. You expect the time for you to wait is very small. no one is going to be using hte lock structure for very long. it's better to sit there and wait. It's like if you go to a gov't office, and you want something to happen, you will probably have to wait in line. When you are spin locking, you aresitting in the queue doing nothing. What you can also do instead is you can have a wait queue: in here we say, they queue up, they wait, you are freeing up the resources to do something else. the particular processor running the thread, it encounters things behind it, it puts that thread to sleep. what does sleep mean? Sleep means, the thread goes to sleep that implies something about the scheduling of the thread. IT is kicked off the cpu it's running on. I am not going to consume any resources right now, when you wake up, in biological terms, it's still consuming resources. It's still consuming space resources (still consuming ram, still living in cache but it is not being actively run by a core. When you say a thread yields, I don't need this CPU anymore. when you call sleep you are making a system call that says, I don't want to do anything for at least 10 seconds. You are taken out of the list of running processes, you are put on the list of block processes, and you wait for the timer to go off. Then your process is brought back in and is allowed to run. It depends on the time of the CPU. In general you should sleep at least for that amount of time.

When we are talking about locks we are talking about multiple threads. Processes do not have shared storage. It's logically separate copies. For threads they share this address space, and use this to coordinate their operations. The lock is not available, it queues up waiting for the lock. You have these threads lined up waiting for the service

Wait = sleep

if it is actually actively waiting running on the cpu that is it spinning, you don't queue up with spinning, because it's every core is just sitting there spinning. For 5 threads to be blocked on a spin lock, all 5 threads are running on 5 cores. You don't use spin locks on things you expect to wait long for. Sometimes you want to use a spin lock. Because it takes effort to put it to sleep. It's a relatively expensive thing to put a thread to sleep, and then have it wake up. When you put a thread to sleep it potentially can involve a context switch, it depends on how threads are implemented. It involves a system call, because then you call the sleep(). If you are going to be waiting less than the time it takes to execute 10 - 50 instructions, you want to use a spin lock.

with a lock all the threads queue up, then when the one that was active gets done. It's a fifo queue. They take their turns.
[[File:Conditionvariables.png]]
A condition variable has the same basic idea. we have a condition that we want to check and we have a wait queue, but the difference is, is in how we treat the queue. I'm waiting for something to happen, a key to be pressed / waiting for something to come in for the network, waiting for something to happen, the other threads too have gotten so far in the computation, what you are doing is check condition (blocking), check condition (non-blocking) and set condition. - seems kind of the same as a lock. What's really happening here is the normal expected thing when you check for the condition, is that you normally do the blocking one. i'm going to sleep until the condition has happened. The set condition says that it happened. The difference lies in how you treat the queue. with a lock you are waiting for exclusive access to something (an entire data structure, a tree or something) Only one thread can access the resource at a time, enforcing mutual exclusion, we want serial semantics. With a condition variable we are not trying to enforce serial semantics. Tell everyone! When the condition becomes true, everyone in the queue wakes up and gets service. Mechanism basic pieces underlying is the same but the semantics are different. Think of a club, are the doors open? Everyone flows in. Instead of sitting at the passport office, 1 person goes up, another person goes up. In a concurrent appplication you will use these different constructs. It's generalyl about protecting access to a datastructure. where as a condition variable is about waiting for some state of computation / state of the environment has come to pass. In the locks its saying can I get exclusive access to the resource? Normally in code that uses locks, there will be some mutex. (mutual exclusion) binary semaphore

semaphores : up / down a general counting semaphore the value can be any integer. You can use semaphores to implement both condition variables and locks. The idea of a semaphore is: down blocks only if it is less than 0

when you say up, you increment it back up. A lock in this case, is if you set the value to 1. Instead of saying 1 can be present, really a semaphore is like a bathroom, 4 people can use the bathroom at a time. When someone goes in, someone's come out, now someone can go back in, if the room is empty, there is a certain number of resources, you can go in, but if it fills up, everyone is waiting. In practice if you are looking at how people build things. People use mutexs all the time condition variables have clear semantics, semaphores are more of a building block, I want to only have so many threads running at the same time. Its' much harder to reason about. You have to be very careful with your downs and ups. They are confusing but you should know about what they are.

File:Conditionvariables.png

2014-10-08T13:25:53Z

Afry:

File:Semaphore.png

2014-10-08T13:25:43Z

Afry:

File:Locks.png

2014-10-08T13:25:29Z

Afry:

Operating Systems 2014F Lecture 10

2014-10-08T13:21:30Z

Afry:

By using base 16 vs. base 10 - yes the alphabet is larger. when you talk about hex vs. decimal, you are changing how we read it. There's something more correct about using hex vs. base 10. A Hexadecimal digit represents 4 bits. How many bits does base 10 represent? (somewhere between 8 and 16 - a fractional bit) it's messy. Decimal is not the native base of a computer. Base 16 maps perfectly to base 2. That's why you also see octal - 3 bits vs. 4 bits. (but a reduced range), when you use hexadecimal it represents 4 bits, and is a bit cleaner.

When you say offset is a power of two, the page size is also a power of two. the storage used to encode the page size determines the range of the offset. It is a specific byte in a page. What is the range of bytes that can go in a page. Classic x86

what is the range of offsets of a 4 k page? The offsets are 0 - 4095. Another thing that you will need for the assignment, it's a question that refers to Peterson's algorithm. It doesn't work on modern machines. The big idea is the memory hierarchy. Key assumption: when one processor writes to memory the other processors will see that write, that value. Is that true in modern systems? No.

Concurrency Mechanisms:

locks
condition variables
Semaphores

you build all of these using the same mechanisms. Let's you check a value and change it in one atomic operation. The basic functionality of the hardware. The semantics are different in these higher level constructs. When do you use what? What are the purpose of these things? there's huge amounts of overlap. what you are talking about with these things, is that you have some sort of storage that has atomic operations so you can check it's value adn change it, all at one time (so you don't have race conditions when you manipulate the value) then you have a queue of threads.

The difference between these, is related to how you deal with the interaction between the storage and the queue of threads.
IF you have a lock, grab lock, and release the lock, does releasing the lock always succeed? Yes, if you have the lock, and you release it, it always succeeds. grab lock can block. when we talk about blocking, it's going to sit there and wait for the condition to happen, nothing is going to happen to that thread until that condition is met. You often have a try lock, try lock means check to see if the lock is grabbable or not. it will grab the lock if it's available, but if it's not available it will return, it's non-blocking, because maybe you don't want to sit there and block. Most of the time you will try to grab the lock and if the lock fails, it will wait until grabbing the lock succeeds.

what if 5 processes try to grab the lock all at once, what do the rest do? they are all blocked, there are two ways for them to wait. One is to just go into an infinite loop, it just keeps trying trying trying until it succeeds. It is sometimes a good idea, this is called a spin lock. If the CPU with running the spin lock, doing nothing else, it is useless. Not very good for making use of efficient resources. You expect the time for you to wait is very small. no one is going to be using hte lock structure for very long. it's better to sit there and wait. It's like if you go to a gov't office, and you want something to happen, you will probably have to wait in line. When you are spin locking, you aresitting in the queue doing nothing. What you can also do instead is you can have a wait queue: in here we say, they queue up, they wait, you are freeing up the resources to do something else. the particular processor running the thread, it encounters things behind it, it puts that thread to sleep. what does sleep mean? Sleep means, the thread goes to sleep that implies something about the scheduling of the thread. IT is kicked off the cpu it's running on. I am not going to consume any resources right now, when you wake up, in biological terms, it's still consuming resources. It's still consuming space resources (still consuming ram, still living in cache but it is not being actively run by a core. When you say a thread yields, I don't need this CPU anymore. when you call sleep you are making a system call that says, I don't want to do anything for at least 10 seconds. You are taken out of the list of running processes, you are put on the list of block processes, and you wait for the timer to go off. Then your process is brought back in and is allowed to run. It depends on the time of the CPU. In general you should sleep at least for that amount of time.

When we are talking about locks we are talking about multiple threads. Processes do not have shared storage. It's logically separate copies. For threads they share this address space, and use this to coordinate their operations. The lock is not available, it queues up waiting for the lock. You have these threads lined up waiting for the service

Wait = sleep

if it is actually actively waiting running on the cpu that is it spinning, you don't queue up with spinning, because it's every core is just sitting there spinning. For 5 threads to be blocked on a spin lock, all 5 threads are running on 5 cores. You don't use spin locks on things you expect to wait long for. Sometimes you want to use a spin lock. Because it takes effort to put it to sleep. It's a relatively expensive thing to put a thread to sleep, and then have it wake up. When you put a thread to sleep it potentially can involve a context switch, it depends on how threads are implemented. It involves a system call, because then you call the sleep(). If you are going to be waiting less than the time it takes to execute 10 - 50 instructions, you want to use a spin lock.

with a lock all the threads queue up, then when the one that was active gets done. It's a fifo queue. They take their turns.

A condition variable has the same basic idea. we have a condition that we want to check and we have a wait queue, but the difference is, is in how we treat the queue. I'm waiting for something to happen, a key to be pressed / waiting for something to come in for the network, waiting for something to happen, the other threads too have gotten so far in the computation, what you are doing is check condition (blocking), check condition (non-blocking) and set condition. - seems kind of the same as a lock. What's really happening here is the normal expected thing when you check for the condition, is that you normally do the blocking one. i'm going to sleep until the condition has happened. The set condition says that it happened. The difference lies in how you treat the queue. with a lock you are waiting for exclusive access to something (an entire data structure, a tree or something) Only one thread can access the resource at a time, enforcing mutual exclusion, we want serial semantics. With a condition variable we are not trying to enforce serial semantics. Tell everyone! When the condition becomes true, everyone in the queue wakes up and gets service. Mechanism basic pieces underlying is the same but the semantics are different. Think of a club, are the doors open? Everyone flows in. Instead of sitting at the passport office, 1 person goes up, another person goes up. In a concurrent appplication you will use these different constructs. It's generalyl about protecting access to a datastructure. where as a condition variable is about waiting for some state of computation / state of the environment has come to pass. In the locks its saying can I get exclusive access to the resource? Normally in code that uses locks, there will be some mutex. (mutual exclusion) binary semaphore

semaphores : up / down a general counting semaphore the value can be any integer. You can use semaphores to implement both condition variables and locks. The idea of a semaphore is: down blocks only if it is less than 0

when you say up, you increment it back up. A lock in this case, is if you set the value to 1. Instead of saying 1 can be present, really a semaphore is like a bathroom, 4 people can use the bathroom at a time. When someone goes in, someone's come out, now someone can go back in, if the room is empty, there is a certain number of resources, you can go in, but if it fills up, everyone is waiting. In practice if you are looking at how people build things. People use mutexs all the time condition variables have clear semantics, semaphores are more of a building block, I want to only have so many threads running at the same time. Its' much harder to reason about. You have to be very careful with your downs and ups. They are confusing but you should know about what they are.

Operating Systems 2014F Lecture 10

2014-10-08T13:11:21Z

Afry:

By using base 16 vs. base 10 - yes the alphabet is larger. when you talk about hex vs. decimal, you are changing how we read it. There's something more correct about using hex vs. base 10. A Hexadecimal digit represents 4 bits. How many bits does base 10 represent? (somewhere between 8 and 16 - a fractional bit) it's messy. Decimal is not the native base of a computer. Base 16 maps perfectly to base 2. That's why you also see octal - 3 bits vs. 4 bits. (but a reduced range), when you use hexadecimal it represents 4 bits, and is a bit cleaner.

When you say offset is a power of two, the page size is also a power of two. the storage used to encode the page size determines the range of the offset. It is a specific byte in a page. What is the range of bytes that can go in a page. Classic x86

what is the range of offsets of a 4 k page? The offsets are 0 - 4095. Another thing that you will need for the assignment, it's a question that refers to Peterson's algorithm. It doesn't work on modern machines. The big idea is the memory hierarchy. Key assumption: when one processor writes to memory the other processors will see that write, that value. Is that true in modern systems? No.

Concurrency Mechanisms:

locks
condition variables
Semaphores

you build all of these using the same mechanisms. Let's you check a value and change it in one atomic operation. The basic functionality of the hardware. The semantics are different in these higher level constructs. When do you use what? What are the purpose of these things? there's huge amounts of overlap. what you are talking about with these things, is that you have some sort of storage that has atomic operations so you can check it's value adn change it, all at one time (so you don't have race conditions when you manipulate the value) then you have a queue of threads.

The difference between these, is related to how you deal with the interaction between the storage and the queue of threads.
IF you have a lock, grab lock, and release the lock, does releasing the lock always succeed? Yes, if you have the lock, and you release it, it always succeeds. grab lock can block. when we talk about blocking, it's going to sit there and wait for the condition to happen, nothing is going to happen to that thread until that condition is met. You often have a try lock, try lock means check to see if the lock is grabbable or not. it will grab the lock if it's available, but if it's not available it will return, it's non-blocking, because maybe you don't want to sit there and block. Most of the time you will try to grab the lock and if the lock fails, it will wait until grabbing the lock succeeds.

what if 5 processes try to grab the lock all at once, what do the rest do? they are all blocked, there are two ways for them to wait. One is to just go into an infinite loop, it just keeps trying trying trying until it succeeds. It is sometimes a good idea, this is called a spin lock. If the CPU with running the spin lock, doing nothing else, it is useless. Not very good for making use of efficient resources. You expect the time for you to wait is very small. no one is going to be using hte lock structure for very long. it's better to sit there and wait. It's like if you go to a gov't office, and you want something to happen, you will probably have to wait in line. When you are spin locking, you aresitting in the queue doing nothing. What you can also do instead is you can have a wait queue: in here we say, they queue up, they wait, you are freeing up the resources to do something else. the particular processor running the thread, it encounters things behind it, it puts that thread to sleep. what does sleep mean? Sleep means, the thread goes to sleep that implies something about the scheduling of the thread. IT is kicked off the cpu it's running on. I am not going to consume any resources right now, when you wake up, in biological terms, it's still consuming resources. It's still consuming space resources (still consuming ram, still living in cache but it is not being actively run by a core. When you say a thread yields, I don't need this CPU anymore. when you call sleep you are making a system call that says, I don't want to do anything for at least 10 seconds. You are taken out of the list of running processes, you are put on the list of block processes, and you wait for the timer to go off. Then your process is brought back in and is allowed to run. It depends on the time of the CPU. In general you should sleep at least for that amount of time.

When we are talking about locks we are talking about multiple threads. Processes do not have shared storage. It's logically separate copies. For threads they share this address space, and use this to coordinate their operations. The lock is not available, it queues up waiting for the lock. You have these threads lined up waiting for the service

Wait = sleep

if it is actually actively waiting running on the cpu that is it spinning, you don't queue up with spinning, because it's every core is just sitting there spinning. For 5 threads to be blocked on a spin lock, all 5 threads are running on 5 cores. You don't use spin locks on things you expect to wait long for. Sometimes you want to use a spin lock. Because it takes effort to put it to sleep. It's a relatively expensive thing to put a thread to sleep, and then have it wake up. When you put a thread to sleep it potentially can involve a context switch, it depends on how threads are implemented. It involves a system call, because then you call the sleep(). If you are going to be waiting less than the time it takes to execute 10 - 50 instructions, you want to use a spin lock.

with a lock all the threads queue up, then when the one that was active gets done. It's a fifo queue. They take their turns.

A condition variable has the same basic idea. we have a condition that we want to check and we have a wait queue, but the difference is, is in how we treat the queue. I'm waiting for something to happen, a key to be pressed / waiting for something to come in for the network, waiting for something to happen, the other threads too have gotten so far in the computation, what you are doing is check condition (blocking), check condition (non-blocking) and set condition. - seems kind of the same as a lock. What's really happening here is the normal expected thing when you check for the condition, is that you normally do the blocking one. i'm going to sleep until the condition has happened. The set condition says that it happened. The difference lies in how you treat the queue. with a lock you are waiting for exclusive access to something (an entire data structure, a tree or something) Only one thread can access the resource at a time, enforcing mutual exclusion, we want serial semantics. With a condition variable we are not trying to enforce serial semantics. Tell everyone! When the condition becomes true, everyone in the queue wakes up and gets service.

Operating Systems 2014F Lecture 10

2014-10-08T13:06:02Z

Afry:

Operating Systems 2014F Lecture 10

2014-10-08T13:02:33Z

Afry:

Operating Systems 2014F Lecture 10

2014-10-08T12:55:57Z

Afry: Created page with "By using base 16 vs. base 10 - yes the alphabet is larger. when you talk about hex vs. decimal, you are changing how we read it. There's something more correct about using hex..."

Operating Systems 2014F Lecture 2

2014-10-06T17:55:47Z

Afry:

Audio from the lecture given on September 10, 2014 [http://homeostasis.scs.carleton.ca/~soma/os-2014f/lectures/comp3000-2014f-lec02-10Sep2014.mp3 is now available].

{{{
machine state
program counter
process states
paging / swapping
process
running program
virtualizing
time/space sharing
mechanisms
policy
}}}

Chapter 4 in the book:

processes - key abstraction in a modern operating system

Sitting all day kills you - seriously reduces your life expectancy. Working out, doesn't necessarily make up for sitting all day. If you walk around for 5 minutes every hour. Anyone use typing break programs? Occupational hazard of the career path you have chosen is that you sit in front of the computer typing. Anil started typing dvorak early in order to avoid repetitive strain injuries.

[http://www.lcdf.org/xwrits/ xwrits] link to save your wrists on a *nix machine gives you hand gestures to say it's time to get up. you can get it to insult you with hand gestures. It does hand gesture depending on culture. In order to tell you to take a break.

When typing it is important to take breaks.

You need to distinguish between programs and processes. A program is inprecise in the context of O/S.

Program is precise in context of operating systems - Your web browser is that a program?
Web browser is a lot of little programs, but makes up one big program. It is not a precise thing. What is precise is an executable. An executable is a file on disk that can be exec'd. (Disks are no longer disks they are all kinds of things such as filestate, etc) This is the unix version of the statement. There is a system call - called execve - takes as one of it's parameters a file and that file is then loaded into a process obliterating whatever else was in the process.

Code can take many forms in a computer systems, this is just one form of data.

For example, you ahve a text file that has a javascript / perl program / something, that is a program, it is also a text document, the operating system kernel does not really recognize it as an executable. You cannot give it as an argument to the execve system call. It has to run it indirectly, it has to find another exec executable to run that code. You have executables and you have processes.

A process - is an executable that has been executed - loaded into memory and started running. A process you should think of as an abstraction of a computer that can only run one program at a time. (Older personal computers, early 1960's or something, there is no abstraction of a process. There is no notion of running more than one program at a time. Logically speaking: when you wanted to run a program, all of memory would be loaded with that program, when you wanted to quit the program, you cut the power (turn the computer off).) They run one program at a time, you load it off the disk, and it has complete control of the machine. A process is the abstraction you get when you say, we don't want every program to have complete control of the computer because I do not want to have to reboot the computer to switch programs. I want to run different programs concurrently, for multiple reasons. Want to chain multiple programs in order to produce a result. (A Unix pipeline) The process - giving each running program (each executable) it's own virtual computer to run.

Virtualizing / virtualization (term is rather overloaded) What am I talking about when I say virtual? Something that isn't real. It's not a real thing. When people talk about virtual reality, they are talking about something that can be experienced. What we are saying in a computer science context: When we say virtual, we are really talking about an abstraction - What we actually have, the real thing is not good enough, it doesn't have qualities that you want, so you want to transform it into something more useful (in some way). When we talk about a virtual machine, we are talking about a machine (computer) that does not exist, in the sense that it is not embodied in actual hardware.

(from the theoretical side of computer science): All programming languages or programming system to a first approximation are equivalent, a system is known as Turing complete it can run anything. Turning one Turing complete system into another Turing complete system is the process of virtualization. The ones you've often heard of are: Language Based Virtual machine - an example: java virtual machine. Really you could talk about any time you run a higher level language (perl, javascript, python, etc) That code does not run directly on the processor. It runs inside of another program which has some kind of virtual machine. Strictly speaking, a lot of languages can be interpreted, which means that you have a program that goes through line by line and figures out what that line is supposed to do and what the next instruction is. The point is that no modern language operates that way. What they all go through is some sort of translation phase, converts it to some binary code, and then it runs the byte code. That runtime is what's called a virtual machine. But virtual machines are everywhere when we are talking about trying to run programs. Operating systems can be thought of as implementing a virtual machine and that virtual machine it implements is the process. Key difference between a virtual machine that makes processes and the vm that is typically in these language based virtual machines. The difference between these is getting smaller. Any idea what this difference is?

Java based Virtual Machine - executes byte codes.
hardware can't interpret byte code

What is the nature of the binary format that is being run in an operating system process? What format is that code? - machine code - it's the code that is understood by the processor. Machine code here, byte code here, what's the difference? The hardware can't interpret this, This language the processor needs another program to translate. Why can't the processor understand java byte code? It could, there are chips that run java byte code natively. What's worse, the machine code that your processor understand? it actually doesn't. Modern processors such as x86 or x86-64 these are the most common things for a pc, arm byte code, arm machine language, that sort of thing.

This language is too annoying to use internally inside the microprocessor, it's not efficient, it was not designed to run very fast. It actually has a front end that takes that code and translates it to another byte code. There have been processor startups, where instead of having it done directly on the chip, they put something like a java virtual machine on the processor. Why am I saying this? The virtual and the real in computer science are often hard to tell apart. Virtual to one group could actually be real to another group. When you are coding in java / c, that is the language, that is real to you. That is the abstraction you are working i. but there are actually other levels below you. That generally is not the real level. when you are dealing with millions of transistors, there is a lot of abstraction. The process is the virtual machine that you run processes in. You take a file and loads it to disk. there is a little problem with this concept: Program on disk, is it a one to one mapping between programs on disk, and programs in memory? Not at all! Most programs on disk are not running at any given time. A given program on disk can be running in many different processes, you can have multiple instances of the same program running at the same time. Logically in an API for an operating system, you have to distinguish between the creation of a process, and the loading of an executable into that process. because you want to be able to facilitate the many to many mapping. What does that API look like? I'll give you another funny thing, if you are running one program, can you make that program do multiple things at the same time? Yes, now there is this whole notion of threading, but a thread is not a process.

==A thread is not a process.==

Process = thread(s) + address space

The cPU is virtual, because that would be really annoying to only have 4 things running at a time. Talking about things that are running at the same time. how many things are running @ the same time? Hyperthreading. What does it mean to run things at the same time, is it actually running? or is it logically running. we want the abstraction so that every process has it's own computer. That's what we are talking about when we talk about threads. Process has memory in which it can run, inside that process, how many processors do I have running? Classic only has 1. When you get to multithreaded processors, there is more than one logical solution. If you think about this? That's a mess! Having more than one program counter to track inside of one address space. this causes lots of problems! What happens when they step on each other. They do crazy things like chance teh loop index, from outside the loop. how do you reason about your code when things like this happen? When you put more than one logical cpu inside an address space that can happen. For a long time Operating systems, only supported processes with a single thread of execution. They supported lots of processes. Just made sure each one only had 1 thread.

Having more than one cpu running around inside the address space. What happens if they try running the same code twice? Change the loop index in the loop from outside of the loop. The right way to think of this is don't do that. When you put more than one cpu inside the address space. For a long time operating system supported one cpu inside the address space. They supported lots of processes but only one cpu. That's kind of limiting, but why do you want your running programs to be sharing memory, the main reason you want them to share memory is to communicate. Shared memory has only one advantage - it can be very fast. How do you make sure you don't overwrite each other's messages? In modern computation, distributed systems, big systems - you do almost everything you can - when you share memory you put some sort of API on top of it to control access. The only problem is - other than having potential communication overhead, you have higher overhead in general, you now have an address space to keep track of, it's own version of memory for each running program. That's so much overhead, it was a good while before computers started to allow multithreading, because it takes a lot of transistors to do. Used to be you'd have a Completely separate processor (a MMU) to take care of every running program in it's own address space, it is now integrated into cpus.

A running program is a process, address space, plus one or more threads. That's the virtual machine in which you are running in. It's got machine code.

Did you see any assembly in 2401? Not really.

=Sharing=

Explain terminology. Time space sharing, when we talk about virtualizing resources, talking about virtualizing cpu, virtualizing ram, what we are actually talking about is sharing. Like on a playground, we need to place nicely together.
An operating system is a set of mechanisms and policies for allowing for time and space sharing of the (computer resources) processor. In Time sharing: (taking turns) The processor is a limited resource. one program gets it for a while, another gets it for another while, then another gets it for a while. Space sharing means that you have all this RAM, split it up, you have this disk, split it up, one program gets part of it, and another program gets part of it. That's what we mean by space sharing.

=Virtual memory and physical memory=

[[File:Virtualmemory.png]]

There's this distinction between virtual memory and physical memory. Physical memory, is the RAM your computer actually has. You buy those little chips, and plug them in (voila, it goes faster)! (Sims & DIMMS, you expand the RAM) you see them at startup, that's physical memory, it's a real thing, you get gigs of it now. Virtual memory is the memory each running program thinks it has. Memory is shared, we want to share it between multiple processes. A wierd consequence of this is how do we refer to memory, with addresses, with 4 gigs of ram, you get 4 million memory locations. 2^32 locations. That's a number of addresses, one way you could say to share a program running in memory. the first program you run, it will get address ranges 200 - 400, the second 500 - 800. when you load any program binary, you have to then change it to use the addresses it is supposed to use. What address range is it supposed to be? Where is it going to get? What if your program wants more RAM? It's a bit of a pain. It's annoying. What we actually have is physical addresses and virtual addresses: When you load a process, all of the memory references are referencing virtual addresses. Unless you have special hardware to accellerate it, you have to do some sort of table lookup. This is happening on every memory access. There's a lot of hardware, there's a lot of operating system mechanism to make this run pretty damn fast. That's what we mean, every process can have an address 2000, but the address 2000 is the virtual address 2000, each is mapped to a different physical address. (it's OVER 9000)

It can be as big as you want it to be. When you talk about 64 bit processors vs. 32 bit processors. It's not the difference in how much physical memory you have, it's the virtual memory. It is much bigger. Do we have any computer with 2^64 bytes of RAM? No, it is a really big number. Not in our lifetime. Everyone has their own private address space. It's just like namespaces, when you write program, and run another program. Do you expect the x in one program to be the same in another? No. the scoping is between processes. These address spaces are the same thing, but they are mapped differently. The address space is fixed in size, because it's limited by the processor word size. How much of it is allocated to the program? Might have a program that's taking up 50 Mb, but I want some more memory, asking for more address space? Asking for part of your virtual address space to be turned into storage, ask the operating system, please can I have some more ram? (Can I have some more sir?)

The operating system notices when you access parts of physical memory that you are not allowed to. It raises an exception, handles it, and you get a segmentation fault. Accessing memory you shouldn't be accessing. The operating system is very firm about it, if you don't handle it in your process, you die.
In modern operating systems, some programs are not equal. Just because you are root, does not mean you are in control! It si not the root user, it is the kernel? Can the kernel have a segmentation fault? A program running with full privileges can still have a segmentation fault. The kernel can also have segmentation fault. When this happens the machine will crash hard.

[http://slacksite.com/slackware/oops.html For Oops vs. Panics]

On the linux system, an oops is a kernel message system being logged. when the kernel fully panics, it just stops. Kernel panic in linux is rare, blue screen of death in windows is a kernel panic. It is possible the system recovers, deep enough in the kernel if you make mistakes, you are just done. you aren't just managing your own memory, you are managing on behalf of other processes. If a kernel is corrupt, it will corrupt everything on disk. Which would you rather have the kernel stop, or continue to do uncontrollable things to other parts of the system?

Environment variables for X:

Did you see the command for listing all the environment variables? env - when you ran env on it, when you look through that list of environment variables. There's an environment variables called display. Looking at systems going, what the heck is that? Looking really far, another is just to look at teh environment variables, which programs require this. Look at the environment variables, which one could it be? Google for Display environment variable, and you see all kinds of stuff. one of the skills you need to have, you need to be able to figure this out! The longer story is, the way you see things in a unix system, is that there is a program controlling the display. There is a part of the kernel that controls their access to the display. Wayland, what does ubuntu use? Next generation replacements for the xwindow system. before the xwindow system, there was the wwindow system.to allow you to separate the notion of a program to interact with the output, it was designed to be network separate. It's actually for display. It's the computer you are running on. That's the server, because all the programs you are running, on it. the program running the process sending all the graphical commands could be somewhere else completely. Xwindow system was based on the xnetwork protocol. processes can be arbitrarily separate from their displays. how does it know where to display things? Provides network transparency.

New view - direct graphical output from another computer on the network - bad because of latency

How to get around lag - run more of the code on the client instead of the server. Have the xclients have some code - transfer code to the xserver, to run on the server. Invisible website, downloads the page and it runs in your browser. Same thing different technology stack.

Mechanisms vs. Policy -

mechanisms - things to do things - the knobs that let us manipulate program state - should be maximally flexible so that they can implement whatever policies you want to do.

policy are what you should do

X Server <= mechanism

window manager, toolkit <= policy

Windows - two calls - create process () <--- many different parameters

unix - fork() and execve(file, cmdline, env)

Operating Systems 2014F Lecture 2

2014-10-06T17:53:26Z

Afry:

Audio from the lecture given on September 10, 2014 [http://homeostasis.scs.carleton.ca/~soma/os-2014f/lectures/comp3000-2014f-lec02-10Sep2014.mp3 is now available].

{{{
machine state
program counter
process states
paging / swapping
process
running program
virtualizing
time/space sharing
mechanisms
policy
}}}

Chapter 4 in the book:

processes - key abstraction in a modern operating system

Sitting all day kills you - seriously reduces your life expectancy. Working out, doesn't necessarily make up for sitting all day. If you walk around for 5 minutes every hour. Anyone use typing break programs? Occupational hazard of the career path you have chosen is that you sit in front of the computer typing. Anil started typing dvorak early in order to avoid repetitive strain injuries.

[http://www.lcdf.org/xwrits/ xwrits] link to save your wrists on a *nix machine gives you hand gestures to say it's time to get up. you can get it to insult you with hand gestures. It does hand gesture depending on culture. In order to tell you to take a break.

When typing it is important to take breaks.

You need to distinguish between programs and processes. A program is inprecise in the context of O/S.

Program is precise in context of operating systems - Your web browser is that a program?
Web browser is a lot of little programs, but makes up one big program. It is not a precise thing. What is precise is an executable. An executable is a file on disk that can be exec'd. (Disks are no longer disks they are all kinds of things such as filestate, etc) This is the unix version of the statement. There is a system call - called execve - takes as one of it's parameters a file and that file is then loaded into a process obliterating whatever else was in the process.

Code can take many forms in a computer systems, this is just one form of data.

For example, you ahve a text file that has a javascript / perl program / something, that is a program, it is also a text document, the operating system kernel does not really recognize it as an executable. You cannot give it as an argument to the execve system call. It has to run it indirectly, it has to find another exec executable to run that code. You have executables and you have processes.

A process - is an executable that has been executed - loaded into memory and started running. A process you should think of as an abstraction of a computer that can only run one program at a time. (Older personal computers, early 1960's or something, there is no abstraction of a process. There is no notion of running more than one program at a time. Logically speaking: when you wanted to run a program, all of memory would be loaded with that program, when you wanted to quit the program, you cut the power (turn the computer off).) They run one program at a time, you load it off the disk, and it has complete control of the machine. A process is the abstraction you get when you say, we don't want every program to have complete control of the computer because I do not want to have to reboot the computer to switch programs. I want to run different programs concurrently, for multiple reasons. Want to chain multiple programs in order to produce a result. (A Unix pipeline) The process - giving each running program (each executable) it's own virtual computer to run.

Virtualizing / virtualization (term is rather overloaded) What am I talking about when I say virtual? Something that isn't real. It's not a real thing. When people talk about virtual reality, they are talking about something that can be experienced. What we are saying in a computer science context: When we say virtual, we are really talking about an abstraction - What we actually have, the real thing is not good enough, it doesn't have qualities that you want, so you want to transform it into something more useful (in some way). When we talk about a virtual machine, we are talking about a machine (computer) that does not exist, in the sense that it is not embodied in actual hardware.

(from the theoretical side of computer science): All programming languages or programming system to a first approximation are equivalent, a system is known as Turing complete it can run anything. Turning one Turing complete system into another Turing complete system is the process of virtualization. The ones you've often heard of are: Language Based Virtual machine - an example: java virtual machine. Really you could talk about any time you run a higher level language (perl, javascript, python, etc) That code does not run directly on the processor. It runs inside of another program which has some kind of virtual machine. Strictly speaking, a lot of languages can be interpreted, which means that you have a program that goes through line by line and figures out what that line is supposed to do and what the next instruction is. The point is that no modern language operates that way. What they all go through is some sort of translation phase, converts it to some binary code, and then it runs the byte code. That runtime is what's called a virtual machine. But virtual machines are everywhere when we are talking about trying to run programs. Operating systems can be thought of as implementing a virtual machine and that virtual machine it implements is the process. Key difference between a virtual machine that makes processes and the vm that is typically in these language based virtual machines. The difference between these is getting smaller. Any idea what this difference is?

Java based Virtual Machine - executes byte codes.
hardware can't interpret byte code

What is the nature of the binary format that is being run in an operating system process? What format is that code? - machine code - it's the code that is understood by the processor. Machine code here, byte code here, what's the difference? The hardware can't interpret this, This language the processor needs another program to translate. Why can't the processor understand java byte code? It could, there are chips that run java byte code natively. What's worse, the machine code that your processor understand? it actually doesn't. Modern processors such as x86 or x86-64 these are the most common things for a pc, arm byte code, arm machine language, that sort of thing.

This language is too annoying to use internally inside the microprocessor, it's not efficient, it was not designed to run very fast. It actually has a front end that takes that code and translates it to another byte code. There have been processor startups, where instead of having it done directly on the chip, they put something like a java virtual machine on the processor. Why am I saying this? The virtual and the real in computer science are often hard to tell apart. Virtual to one group could actually be real to another group. When you are coding in java / c, that is the language, that is real to you. That is the abstraction you are working i. but there are actually other levels below you. That generally is not the real level. when you are dealing with millions of transistors, there is a lot of abstraction. The process is the virtual machine that you run processes in. You take a file and loads it to disk. there is a little problem with this concept: Program on disk, is it a one to one mapping between programs on disk, and programs in memory? Not at all! Most programs on disk are not running at any given time. A given program on disk can be running in many different processes, you can have multiple instances of the same program running at the same time. Logically in an API for an operating system, you have to distinguish between the creation of a process, and the loading of an executable into that process. because you want to be able to facilitate the many to many mapping. What does that API look like? I'll give you another funny thing, if you are running one program, can you make that program do multiple things at the same time? Yes, now there is this whole notion of threading, but a thread is not a process.

==A thread is not a process.==

Process = thread(s) + address space

The cPU is virtual, because that would be really annoying to only have 4 things running at a time. Talking about things that are running at the same time. how many things are running @ the same time? Hyperthreading. What does it mean to run things at the same time, is it actually running? or is it logically running. we want the abstraction so that every process has it's own computer. That's what we are talking about when we talk about threads. Process has memory in which it can run, inside that process, how many processors do I have running? Classic only has 1. When you get to multithreaded processors, there is more than one logical solution. If you think about this? That's a mess! Having more than one program counter to track inside of one address space. this causes lots of problems! What happens when they step on each other. They do crazy things like chance teh loop index, from outside the loop. how do you reason about your code when things like this happen? When you put more than one logical cpu inside an address space that can happen. For a long time Operating systems, only supported processes with a single thread of execution. They supported lots of processes. Just made sure each one only had 1 thread.

Having more than one cpu running around inside the address space. What happens if they try running the same code twice? Change the loop index in the loop from outside of the loop. The right way to think of this is don't do that. When you put more than one cpu inside the address space. For a long time operating system supported one cpu inside the address space. They supported lots of processes but only one cpu. That's kind of limiting, but why do you want your running programs to be sharing memory, the main reason you want them to share memory is to communicate. Shared memory has only one advantage - it can be very fast. How do you make sure you don't overwrite each other's messages? In modern computation, distributed systems, big systems - you do almost everything you can - when you share memory you put some sort of API on top of it to control access. The only problem is - other than having potential communication overhead, you have higher overhead in general, you now have an address space to keep track of, it's own version of memory for each running program. That's so much overhead, it was a good while before computers started to allow multithreading, because it takes a lot of transistors to do. Used to be you'd have a Completely separate processor (a MMU) to take care of every running program in it's own address space, it is now integrated into cpus.

A running program is a process, address space, plus one or more threads. That's the virtual machine in which you are running in. It's got machine code.

Did you see any assembly in 2401? Not really.

=Sharing=

Explain terminology. Time space sharing, when we talk about virtualizing resources, talking about virtualizing cpu, virtualizing ram, what we are actually talking about is sharing. Like on a playground, we need to place nicely together.
An operating system is a set of mechanisms and policies for allowing for time and space sharing of the (computer resources) processor. In Time sharing: (taking turns) The processor is a limited resource. one program gets it for a while, another gets it for another while, then another gets it for a while. Space sharing means that you have all this RAM, split it up, you have this disk, split it up, one program gets part of it, and another program gets part of it. That's what we mean by space sharing.

=Virtual memory and physical memory=

[[File:Virtualmemory.png]]

There's this distinction between virtual memory and physical memory. Physical memory, is the RAM your computer actually has. You buy those little chips, and plug them in (voila, it goes faster)! (Sims & DIMMS, you expand the RAM) you see them at startup, that's physical memory, it's a real thing, you get gigs of it now. Virtual memory is the memory each running program thinks it has. Memory is shared, we want to share it between multiple processes. A wierd consequence of this is how do we refer to memory, with addresses, with 4 gigs of ram, you get 4 million memory locations. 2^32 locations. That's a number of addresses, one way you could say to share a program running in memory. the first program you run, it will get address ranges 200 - 400, the second 500 - 800. when you load any program binary, you have to then change it to use the addresses it is supposed to use. What address range is it supposed to be? Where is it going to get? What if your program wants more RAM? It's a bit of a pain. It's annoying. What we actually have is physical addresses and virtual addresses: When you load a process, all of the memory references are referencing virtual addresses. Unless you have special hardware to accellerate it, you have to do some sort of table lookup. This is happening on every memory access. There's a lot of hardware, there's a lot of operating system mechanism to make this run pretty damn fast. That's what we mean, every process can have an address 2000, but the address 2000 is the virtual address 2000, each is mapped to a different physical address. (it's OVER 9000)

It can be as big as you want it to be. When you talk about 64 bit processors vs. 32 bit processors. It's not the difference in how much physical memory you have, it's the virtual memory. It is much bigger. Do we have any computer with 2^64 bytes of RAM? No, it is a really big number. Not in our lifetime. Everyone has their own private address space. It's just like namespaces, when you write program, and run another program. Do you expect the x in one program to be the same in another? No. the scoping is between processes. These address spaces are the same thing, but they are mapped differently. The address space is fixed in size, because it's limited by the processor word size. How much of it is allocated to the program? Might have a program that's taking up 50 Mb, but I want some more memory, asking for more address space? Asking for part of your virtual address space to be turned into storage, ask the operating system, please can I have some more ram? (Can I have some more sir?)

The operating system notices when you access parts of physical memory that you are not allowed to. It raises an exception, handles it, and you get a segmentation fault. Accessing memory you shouldn't be accessing. The operating system is very firm about it, if you don't handle it in your process, you die.
In modern operating systems, some programs are not equal. Just because you are root, does not mean you are in control! It si not the root user, it is the kernel? Can the kernel have a segmentation fault? A program running with full privileges can still have a segmentation fault. The kernel can also have segmentation fault. When this happens the machine will crash hard.

[http://slacksite.com/slackware/oops.html For Oops vs. Panics]

On the linux system, an oops is a kernel message system being logged. when the kernel fully panics, it just stops. Kernel panic in linux is rare, blue screen of death in windows is a kernel panic. It is possible the system recovers, deep enough in the kernel if you make mistakes, you are just done. you aren't just managing your own memory, you are managing on behalf of other processes. If a kernel is corrupt, it will corrupt everything on disk. Which would you rather have the kernel stop, or continue to do uncontrollable things to other parts of the system?

Environment variables for X:

Did you see the command for listing all the environment variables? env - when you ran env on it, when you look through that list of environment variables. There's an environment variables called display. Looking at systems going, what the heck is that? Looking really far, another is just to look at teh environment variables, which programs require this. Look at the environment variables, which one could it be? Google for Display environment variable, and you see all kinds of stuff. one of the skills you need to have, you need to be able to figure this out! The longer story is, the way you see things in a unix system, is that there is a program controlling the display. There is a part of the kernel that controls their access to the display. Wayland, what does ubuntu use? Next generation replacements for the xwindow system. before the xwindow system, there was the wwindow system.to allow you to separate the notion of a program to interact with the output, it was designed to be network separate. It's actually for display. It's the computer you are running on. That's the server, because all the programs you are running, on it. the program running the process sending all the graphical commands could be somewhere else completely. Xwindow system was based on the xnetwork protocol. processes can be arbitrarily separate from their displays. how does it know where to display things?

new view - direct graphical output from another computer on the network - bad because of latency - new

How to get around lag - run more of the code on the client instead of the server. Have the xclients have some code - transfer code to the xserver, to run on the server. Invisible website, downloads the page and it runs in your browser. Same thing different technology stack.

Mechanisms vs. Policy -

mechanisms - things to do things - the knobs that let us manipulate program state - should be maximally flexible so that they can implement whatever policies you want to do.

policy are what you should do

X Server <= mechanism

window manager, toolkit <= policy

Windows - two calls - create process () <--- many different parameters

unix - fork() and execve(file, cmdline, env)

Operating Systems 2014F Lecture 2

2014-10-06T17:46:12Z

Afry:

Audio from the lecture given on September 10, 2014 [http://homeostasis.scs.carleton.ca/~soma/os-2014f/lectures/comp3000-2014f-lec02-10Sep2014.mp3 is now available].

{{{
machine state
program counter
process states
paging / swapping
process
running program
virtualizing
time/space sharing
mechanisms
policy
}}}

Chapter 4 in the book:

processes - key abstraction in a modern operating system

Sitting all day kills you - seriously reduces your life expectancy. Working out, doesn't necessarily make up for sitting all day. If you walk around for 5 minutes every hour. Anyone use typing break programs? Occupational hazard of the career path you have chosen is that you sit in front of the computer typing. Anil started typing dvorak early in order to avoid repetitive strain injuries.

[http://www.lcdf.org/xwrits/ xwrits] link to save your wrists on a *nix machine gives you hand gestures to say it's time to get up. you can get it to insult you with hand gestures. It does hand gesture depending on culture. In order to tell you to take a break.

When typing it is important to take breaks.

You need to distinguish between programs and processes. A program is inprecise in the context of O/S.

Program is precise in context of operating systems - Your web browser is that a program?
Web browser is a lot of little programs, but makes up one big program. It is not a precise thing. What is precise is an executable. An executable is a file on disk that can be exec'd. (Disks are no longer disks they are all kinds of things such as filestate, etc) This is the unix version of the statement. There is a system call - called execve - takes as one of it's parameters a file and that file is then loaded into a process obliterating whatever else was in the process.

Code can take many forms in a computer systems, this is just one form of data.

For example, you ahve a text file that has a javascript / perl program / something, that is a program, it is also a text document, the operating system kernel does not really recognize it as an executable. You cannot give it as an argument to the execve system call. It has to run it indirectly, it has to find another exec executable to run that code. You have executables and you have processes.

A process - is an executable that has been executed - loaded into memory and started running. A process you should think of as an abstraction of a computer that can only run one program at a time. (Older personal computers, early 1960's or something, there is no abstraction of a process. There is no notion of running more than one program at a time. Logically speaking: when you wanted to run a program, all of memory would be loaded with that program, when you wanted to quit the program, you cut the power (turn the computer off).) They run one program at a time, you load it off the disk, and it has complete control of the machine. A process is the abstraction you get when you say, we don't want every program to have complete control of the computer because I do not want to have to reboot the computer to switch programs. I want to run different programs concurrently, for multiple reasons. Want to chain multiple programs in order to produce a result. (A Unix pipeline) The process - giving each running program (each executable) it's own virtual computer to run.

Virtualizing / virtualization (term is rather overloaded) What am I talking about when I say virtual? Something that isn't real. It's not a real thing. When people talk about virtual reality, they are talking about something that can be experienced. What we are saying in a computer science context: When we say virtual, we are really talking about an abstraction - What we actually have, the real thing is not good enough, it doesn't have qualities that you want, so you want to transform it into something more useful (in some way). When we talk about a virtual machine, we are talking about a machine (computer) that does not exist, in the sense that it is not embodied in actual hardware.

(from the theoretical side of computer science): All programming languages or programming system to a first approximation are equivalent, a system is known as Turing complete it can run anything. Turning one Turing complete system into another Turing complete system is the process of virtualization. The ones you've often heard of are: Language Based Virtual machine - an example: java virtual machine. Really you could talk about any time you run a higher level language (perl, javascript, python, etc) That code does not run directly on the processor. It runs inside of another program which has some kind of virtual machine. Strictly speaking, a lot of languages can be interpreted, which means that you have a program that goes through line by line and figures out what that line is supposed to do and what the next instruction is. The point is that no modern language operates that way. What they all go through is some sort of translation phase, converts it to some binary code, and then it runs the byte code. That runtime is what's called a virtual machine. But virtual machines are everywhere when we are talking about trying to run programs. Operating systems can be thought of as implementing a virtual machine and that virtual machine it implements is the process. Key difference between a virtual machine that makes processes and the vm that is typically in these language based virtual machines. The difference between these is getting smaller. Any idea what this difference is?

Java based Virtual Machine - executes byte codes.
hardware can't interpret byte code

What is the nature of the binary format that is being run in an operating system process? What format is that code? - machine code - it's the code that is understood by the processor. Machine code here, byte code here, what's the difference? The hardware can't interpret this, This language the processor needs another program to translate. Why can't the processor understand java byte code? It could, there are chips that run java byte code natively. What's worse, the machine code that your processor understand? it actually doesn't. Modern processors such as x86 or x86-64 these are the most common things for a pc, arm byte code, arm machine language, that sort of thing.

This language is too annoying to use internally inside the microprocessor, it's not efficient, it was not designed to run very fast. It actually has a front end that takes that code and translates it to another byte code. There have been processor startups, where instead of having it done directly on the chip, they put something like a java virtual machine on the processor. Why am I saying this? The virtual and the real in computer science are often hard to tell apart. Virtual to one group could actually be real to another group. When you are coding in java / c, that is the language, that is real to you. That is the abstraction you are working i. but there are actually other levels below you. That generally is not the real level. when you are dealing with millions of transistors, there is a lot of abstraction. The process is the virtual machine that you run processes in. You take a file and loads it to disk. there is a little problem with this concept: Program on disk, is it a one to one mapping between programs on disk, and programs in memory? Not at all! Most programs on disk are not running at any given time. A given program on disk can be running in many different processes, you can have multiple instances of the same program running at the same time. Logically in an API for an operating system, you have to distinguish between the creation of a process, and the loading of an executable into that process. because you want to be able to facilitate the many to many mapping. What does that API look like? I'll give you another funny thing, if you are running one program, can you make that program do multiple things at the same time? Yes, now there is this whole notion of threading, but a thread is not a process.

==A thread is not a process.==

Process = thread(s) + address space

The cPU is virtual, because that would be really annoying to only have 4 things running at a time. Talking about things that are running at the same time. how many things are running @ the same time? Hyperthreading. What does it mean to run things at the same time, is it actually running? or is it logically running. we want the abstraction so that every process has it's own computer. That's what we are talking about when we talk about threads. Process has memory in which it can run, inside that process, how many processors do I have running? Classic only has 1. When you get to multithreaded processors, there is more than one logical solution. If you think about this? That's a mess! Having more than one program counter to track inside of one address space. this causes lots of problems! What happens when they step on each other. They do crazy things like chance teh loop index, from outside the loop. how do you reason about your code when things like this happen? When you put more than one logical cpu inside an address space that can happen. For a long time Operating systems, only supported processes with a single thread of execution. They supported lots of processes. Just made sure each one only had 1 thread.

Having more than one cpu running around inside the address space. What happens if they try running the same code twice? Change the loop index in the loop from outside of the loop. The right way to think of this is don't do that. When you put more than one cpu inside the address space. For a long time operating system supported one cpu inside the address space. They supported lots of processes but only one cpu. That's kind of limiting, but why do you want your running programs to be sharing memory, the main reason you want them to share memory is to communicate. Shared memory has only one advantage - it can be very fast. How do you make sure you don't overwrite each other's messages? In modern computation, distributed systems, big systems - you do almost everything you can - when you share memory you put some sort of API on top of it to control access. The only problem is - other than having potential communication overhead, you have higher overhead in general, you now have an address space to keep track of, it's own version of memory for each running program. That's so much overhead, it was a good while before computers started to allow multithreading, because it takes a lot of transistors to do. Used to be you'd have a Completely separate processor (a MMU) to take care of every running program in it's own address space, it is now integrated into cpus.

A running program is a process, address space, plus one or more threads. That's the virtual machine in which you are running in. It's got machine code.

Did you see any assembly in 2401? Not really.

=Sharing=

Explain terminology. Time space sharing, when we talk about virtualizing resources, talking about virtualizing cpu, virtualizing ram, what we are actually talking about is sharing. Like on a playground, we need to place nicely together.
An operating system is a set of mechanisms and policies for allowing for time and space sharing of the (computer resources) processor. In Time sharing: (taking turns) The processor is a limited resource. one program gets it for a while, another gets it for another while, then another gets it for a while. Space sharing means that you have all this RAM, split it up, you have this disk, split it up, one program gets part of it, and another program gets part of it. That's what we mean by space sharing.

=Virtual memory and physical memory=

[[File:Virtualmemory.png]]

There's this distinction between virtual memory and physical memory. Physical memory, is the RAM your computer actually has. You buy those little chips, and plug them in (voila, it goes faster)! (Sims & DIMMS, you expand the RAM) you see them at startup, that's physical memory, it's a real thing, you get gigs of it now. Virtual memory is the memory each running program thinks it has. Memory is shared, we want to share it between multiple processes. A wierd consequence of this is how do we refer to memory, with addresses, with 4 gigs of ram, you get 4 million memory locations. 2^32 locations. That's a number of addresses, one way you could say to share a program running in memory. the first program you run, it will get address ranges 200 - 400, the second 500 - 800. when you load any program binary, you have to then change it to use the addresses it is supposed to use. What address range is it supposed to be? Where is it going to get? What if your program wants more RAM? It's a bit of a pain. It's annoying. What we actually have is physical addresses and virtual addresses: When you load a process, all of the memory references are referencing virtual addresses. Unless you have special hardware to accellerate it, you have to do some sort of table lookup. This is happening on every memory access. There's a lot of hardware, there's a lot of operating system mechanism to make this run pretty damn fast. That's what we mean, every process can have an address 2000, but the address 2000 is the virtual address 2000, each is mapped to a different physical address. (it's OVER 9000)

It can be as big as you want it to be. When you talk about 64 bit processors vs. 32 bit processors. It's not the difference in how much physical memory you have, it's the virtual memory. It is much bigger. Do we have any computer with 2^64 bytes of RAM? No, it is a really big number. Not in our lifetime. Everyone has their own private address space. It's just like namespaces, when you write program, and run another program. Do you expect the x in one program to be the same in another? No. the scoping is between processes. These address spaces are the same thing, but they are mapped differently. The address space is fixed in size, because it's limited by the processor word size. How much of it is allocated to the program? Might have a program that's taking up 50 Mb, but I want some more memory, asking for more address space? Asking for part of your virtual address space to be turned into storage, ask the operating system, please can I have some more ram? (Can I have some more sir?)

The operating system notices when you access parts of physical memory that you are not allowed to. It raises an exception, handles it, and you get a segmentation fault. Accessing memory you shouldn't be accessing. The operating system is very firm about it, if you don't handle it in your process, you die.
In modern operating systems, some programs are not equal. Just because you are root, does not mean you are in control! It si not the root user, it is the kernel? Can the kernel have a segmentation fault? A program running with full privileges can still have a segmentation fault. The kernel can also have segmentation fault. When this happens the machine will crash hard.

[http://slacksite.com/slackware/oops.html For Oops vs. Panics]

On the linux system, an oops is a kernel message system being logged. when the kernel fully panics, it just stops. Kernel panic in linux is rare, blue screen of death in windows is a kernel panic. It is possible the system recovers, deep enough in the kernel if you make mistakes, you are just done. you aren't just managing your own memory, you are managing on behalf of other processes. If a kernel is corrupt, it will corrupt everything on disk. Which would you rather have the kernel stop, or continue to do uncontrollable things to other parts of the system?

Environment variables for X:

new view - direct graphical output from another computer on the network - bad because of latency - new

How to get around lag - run more of the code on the client instead of the server. Have the xclients have some code - transfer code to the xserver, to run on the server. Invisible website, downloads the page and it runs in your browser. Same thing different technology stack.

Mechanisms vs. Policy -

mechanisms - things to do things - the knobs that let us manipulate program state - should be maximally flexible so that they can implement whatever policies you want to do.

policy are what you should do

X Server <= mechanism

window manager, toolkit <= policy

Windows - two calls - create process () <--- many different parameters

unix - fork() and execve(file, cmdline, env)

Operating Systems 2014F Lecture 2

2014-10-06T17:43:50Z

Afry:

Audio from the lecture given on September 10, 2014 [http://homeostasis.scs.carleton.ca/~soma/os-2014f/lectures/comp3000-2014f-lec02-10Sep2014.mp3 is now available].

{{{
machine state
program counter
process states
paging / swapping
process
running program
virtualizing
time/space sharing
mechanisms
policy
}}}

Chapter 4 in the book:

processes - key abstraction in a modern operating system

Sitting all day kills you - seriously reduces your life expectancy. Working out, doesn't necessarily make up for sitting all day. If you walk around for 5 minutes every hour. Anyone use typing break programs? Occupational hazard of the career path you have chosen is that you sit in front of the computer typing. Anil started typing dvorak early in order to avoid repetitive strain injuries.

[http://www.lcdf.org/xwrits/ xwrits] link to save your wrists on a *nix machine gives you hand gestures to say it's time to get up. you can get it to insult you with hand gestures. It does hand gesture depending on culture. In order to tell you to take a break.

When typing it is important to take breaks.

You need to distinguish between programs and processes. A program is inprecise in the context of O/S.

Program is precise in context of operating systems - Your web browser is that a program?
Web browser is a lot of little programs, but makes up one big program. It is not a precise thing. What is precise is an executable. An executable is a file on disk that can be exec'd. (Disks are no longer disks they are all kinds of things such as filestate, etc) This is the unix version of the statement. There is a system call - called execve - takes as one of it's parameters a file and that file is then loaded into a process obliterating whatever else was in the process.

Code can take many forms in a computer systems, this is just one form of data.

For example, you ahve a text file that has a javascript / perl program / something, that is a program, it is also a text document, the operating system kernel does not really recognize it as an executable. You cannot give it as an argument to the execve system call. It has to run it indirectly, it has to find another exec executable to run that code. You have executables and you have processes.

A process - is an executable that has been executed - loaded into memory and started running. A process you should think of as an abstraction of a computer that can only run one program at a time. (Older personal computers, early 1960's or something, there is no abstraction of a process. There is no notion of running more than one program at a time. Logically speaking: when you wanted to run a program, all of memory would be loaded with that program, when you wanted to quit the program, you cut the power (turn the computer off).) They run one program at a time, you load it off the disk, and it has complete control of the machine. A process is the abstraction you get when you say, we don't want every program to have complete control of the computer because I do not want to have to reboot the computer to switch programs. I want to run different programs concurrently, for multiple reasons. Want to chain multiple programs in order to produce a result. (A Unix pipeline) The process - giving each running program (each executable) it's own virtual computer to run.

Virtualizing / virtualization (term is rather overloaded) What am I talking about when I say virtual? Something that isn't real. It's not a real thing. When people talk about virtual reality, they are talking about something that can be experienced. What we are saying in a computer science context: When we say virtual, we are really talking about an abstraction - What we actually have, the real thing is not good enough, it doesn't have qualities that you want, so you want to transform it into something more useful (in some way). When we talk about a virtual machine, we are talking about a machine (computer) that does not exist, in the sense that it is not embodied in actual hardware.

(from the theoretical side of computer science): All programming languages or programming system to a first approximation are equivalent, a system is known as Turing complete it can run anything. Turning one Turing complete system into another Turing complete system is the process of virtualization. The ones you've often heard of are: Language Based Virtual machine - an example: java virtual machine. Really you could talk about any time you run a higher level language (perl, javascript, python, etc) That code does not run directly on the processor. It runs inside of another program which has some kind of virtual machine. Strictly speaking, a lot of languages can be interpreted, which means that you have a program that goes through line by line and figures out what that line is supposed to do and what the next instruction is. The point is that no modern language operates that way. What they all go through is some sort of translation phase, converts it to some binary code, and then it runs the byte code. That runtime is what's called a virtual machine. But virtual machines are everywhere when we are talking about trying to run programs. Operating systems can be thought of as implementing a virtual machine and that virtual machine it implements is the process. Key difference between a virtual machine that makes processes and the vm that is typically in these language based virtual machines. The difference between these is getting smaller. Any idea what this difference is?

Java based Virtual Machine - executes byte codes.
hardware can't interpret byte code

What is the nature of the binary format that is being run in an operating system process? What format is that code? - machine code - it's the code that is understood by the processor. Machine code here, byte code here, what's the difference? The hardware can't interpret this, This language the processor needs another program to translate. Why can't the processor understand java byte code? It could, there are chips that run java byte code natively. What's worse, the machine code that your processor understand? it actually doesn't. Modern processors such as x86 or x86-64 these are the most common things for a pc, arm byte code, arm machine language, that sort of thing.

This language is too annoying to use internally inside the microprocessor, it's not efficient, it was not designed to run very fast. It actually has a front end that takes that code and translates it to another byte code. There have been processor startups, where instead of having it done directly on the chip, they put something like a java virtual machine on the processor. Why am I saying this? The virtual and the real in computer science are often hard to tell apart. Virtual to one group could actually be real to another group. When you are coding in java / c, that is the language, that is real to you. That is the abstraction you are working i. but there are actually other levels below you. That generally is not the real level. when you are dealing with millions of transistors, there is a lot of abstraction. The process is the virtual machine that you run processes in. You take a file and loads it to disk. there is a little problem with this concept: Program on disk, is it a one to one mapping between programs on disk, and programs in memory? Not at all! Most programs on disk are not running at any given time. A given program on disk can be running in many different processes, you can have multiple instances of the same program running at the same time. Logically in an API for an operating system, you have to distinguish between the creation of a process, and the loading of an executable into that process. because you want to be able to facilitate the many to many mapping. What does that API look like? I'll give you another funny thing, if you are running one program, can you make that program do multiple things at the same time? Yes, now there is this whole notion of threading, but a thread is not a process.

==A thread is not a process.==

Process = thread(s) + address space

The cPU is virtual, because that would be really annoying to only have 4 things running at a time. Talking about things that are running at the same time. how many things are running @ the same time? Hyperthreading. What does it mean to run things at the same time, is it actually running? or is it logically running. we want the abstraction so that every process has it's own computer. That's what we are talking about when we talk about threads. Process has memory in which it can run, inside that process, how many processors do I have running? Classic only has 1. When you get to multithreaded processors, there is more than one logical solution. If you think about this? That's a mess! Having more than one program counter to track inside of one address space. this causes lots of problems! What happens when they step on each other. They do crazy things like chance teh loop index, from outside the loop. how do you reason about your code when things like this happen? When you put more than one logical cpu inside an address space that can happen. For a long time Operating systems, only supported processes with a single thread of execution. They supported lots of processes. Just made sure each one only had 1 thread.

Having more than one cpu running around inside the address space. What happens if they try running the same code twice? Change the loop index in the loop from outside of the loop. The right way to think of this is don't do that. When you put more than one cpu inside the address space. For a long time operating system supported one cpu inside the address space. They supported lots of processes but only one cpu. That's kind of limiting, but why do you want your running programs to be sharing memory, the main reason you want them to share memory is to communicate. Shared memory has only one advantage - it can be very fast. How do you make sure you don't overwrite each other's messages? In modern computation, distributed systems, big systems - you do almost everything you can - when you share memory you put some sort of API on top of it to control access. The only problem is - other than having potential communication overhead, you have higher overhead in general, you now have an address space to keep track of, it's own version of memory for each running program. That's so much overhead, it was a good while before computers started to allow multithreading, because it takes a lot of transistors to do. Used to be you'd have a Completely separate processor (a MMU) to take care of every running program in it's own address space, it is now integrated into cpus.

A running program is a process, address space, plus one or more threads. That's the virtual machine in which you are running in. It's got machine code.

Did you see any assembly in 2401? Not really.

=Sharing=

Explain terminology. Time space sharing, when we talk about virtualizing resources, talking about virtualizing cpu, virtualizing ram, what we are actually talking about is sharing. Like on a playground, we need to place nicely together.
An operating system is a set of mechanisms and policies for allowing for time and space sharing of the (computer resources) processor. In Time sharing: (taking turns) The processor is a limited resource. one program gets it for a while, another gets it for another while, then another gets it for a while. Space sharing means that you have all this RAM, split it up, you have this disk, split it up, one program gets part of it, and another program gets part of it. That's what we mean by space sharing.

=Virtual memory and physical memory=

[[File:Virtualmemory.png]]

There's this distinction between virtual memory and physical memory. Physical memory, is the RAM your computer actually has. You buy those little chips, and plug them in (voila, it goes faster)! (Sims & DIMMS, you expand the RAM) you see them at startup, that's physical memory, it's a real thing, you get gigs of it now. Virtual memory is the memory each running program thinks it has. Memory is shared, we want to share it between multiple processes. A wierd consequence of this is how do we refer to memory, with addresses, with 4 gigs of ram, you get 4 million memory locations. 2^32 locations. That's a number of addresses, one way you could say to share a program running in memory. the first program you run, it will get address ranges 200 - 400, the second 500 - 800. when you load any program binary, you have to then change it to use the addresses it is supposed to use. What address range is it supposed to be? Where is it going to get? What if your program wants more RAM? It's a bit of a pain. It's annoying. What we actually have is physical addresses and virtual addresses: When you load a process, all of the memory references are referencing virtual addresses. Unless you have special hardware to accellerate it, you have to do some sort of table lookup. This is happening on every memory access. There's a lot of hardware, there's a lot of operating system mechanism to make this run pretty damn fast. That's what we mean, every process can have an address 2000, but the address 2000 is the virtual address 2000, each is mapped to a different physical address. (it's OVER 9000)

It can be as big as you want it to be. When you talk about 64 bit processors vs. 32 bit processors. It's not the difference in how much physical memory you have, it's the virtual memory. It is much bigger. Do we have any computer with 2^64 bytes of RAM? No, it is a really big number. Not in our lifetime. Everyone has their own private address space. It's just like namespaces, when you write program, and run another program. Do you expect the x in one program to be the same in another? No. the scoping is between processes. These address spaces are the same thing, but they are mapped differently. The address space is fixed in size, because it's limited by the processor word size. How much of it is allocated to the program? Might have a program that's taking up 50 Mb, but I want some more memory, asking for more address space? Asking for part of your virtual address space to be turned into storage, ask the operating system, please can I have some more ram? (Can I have some more sir?)

The operating system notices when you access parts of physical memory that you are not allowed to. It raises an exception, handles it, and you get a segmentation fault. Accessing memory you shouldn't be accessing. The operating system is very firm about it, if you don't handle it in your process, you die.
In modern operating systems, some programs are not equal. Just because you are root, does not mean you are in control! It si not the root user, it is the kernel? Can the kernel have a segmentation fault? A program running with full privileges can still have a segmentation fault. The kernel can also have segmentation fault. When this happens the machine will crash hard.

[http://slacksite.com/slackware/oops.html For Oops vs. Panics]

Environment variables for X:

new view - direct graphical output from another computer on the network - bad because of latency - new

How to get around lag - run more of the code on the client instead of the server. Have the xclients have some code - transfer code to the xserver, to run on the server. Invisible website, downloads the page and it runs in your browser. Same thing different technology stack.

Mechanisms vs. Policy -

mechanisms - things to do things - the knobs that let us manipulate program state - should be maximally flexible so that they can implement whatever policies you want to do.

policy are what you should do

X Server <= mechanism

window manager, toolkit <= policy

Windows - two calls - create process () <--- many different parameters

unix - fork() and execve(file, cmdline, env)

Operating Systems 2014F Lecture 2

2014-10-06T17:30:43Z

Afry:

Audio from the lecture given on September 10, 2014 [http://homeostasis.scs.carleton.ca/~soma/os-2014f/lectures/comp3000-2014f-lec02-10Sep2014.mp3 is now available].

{{{
machine state
program counter
process states
paging / swapping
process
running program
virtualizing
time/space sharing
mechanisms
policy
}}}

Chapter 4 in the book:

processes - key abstraction in a modern operating system

Sitting all day kills you - seriously reduces your life expectancy. Working out, doesn't necessarily make up for sitting all day. If you walk around for 5 minutes every hour. Anyone use typing break programs? Occupational hazard of the career path you have chosen is that you sit in front of the computer typing. Anil started typing dvorak early in order to avoid repetitive strain injuries.

[http://www.lcdf.org/xwrits/ xwrits] link to save your wrists on a *nix machine gives you hand gestures to say it's time to get up. you can get it to insult you with hand gestures. It does hand gesture depending on culture. In order to tell you to take a break.

When typing it is important to take breaks.

You need to distinguish between programs and processes. A program is inprecise in the context of O/S.

Program is precise in context of operating systems - Your web browser is that a program?
Web browser is a lot of little programs, but makes up one big program. It is not a precise thing. What is precise is an executable. An executable is a file on disk that can be exec'd. (Disks are no longer disks they are all kinds of things such as filestate, etc) This is the unix version of the statement. There is a system call - called execve - takes as one of it's parameters a file and that file is then loaded into a process obliterating whatever else was in the process.

Code can take many forms in a computer systems, this is just one form of data.

For example, you ahve a text file that has a javascript / perl program / something, that is a program, it is also a text document, the operating system kernel does not really recognize it as an executable. You cannot give it as an argument to the execve system call. It has to run it indirectly, it has to find another exec executable to run that code. You have executables and you have processes.

A process - is an executable that has been executed - loaded into memory and started running. A process you should think of as an abstraction of a computer that can only run one program at a time. (Older personal computers, early 1960's or something, there is no abstraction of a process. There is no notion of running more than one program at a time. Logically speaking: when you wanted to run a program, all of memory would be loaded with that program, when you wanted to quit the program, you cut the power (turn the computer off).) They run one program at a time, you load it off the disk, and it has complete control of the machine. A process is the abstraction you get when you say, we don't want every program to have complete control of the computer because I do not want to have to reboot the computer to switch programs. I want to run different programs concurrently, for multiple reasons. Want to chain multiple programs in order to produce a result. (A Unix pipeline) The process - giving each running program (each executable) it's own virtual computer to run.

Virtualizing / virtualization (term is rather overloaded) What am I talking about when I say virtual? Something that isn't real. It's not a real thing. When people talk about virtual reality, they are talking about something that can be experienced. What we are saying in a computer science context: When we say virtual, we are really talking about an abstraction - What we actually have, the real thing is not good enough, it doesn't have qualities that you want, so you want to transform it into something more useful (in some way). When we talk about a virtual machine, we are talking about a machine (computer) that does not exist, in the sense that it is not embodied in actual hardware.

(from the theoretical side of computer science): All programming languages or programming system to a first approximation are equivalent, a system is known as Turing complete it can run anything. Turning one Turing complete system into another Turing complete system is the process of virtualization. The ones you've often heard of are: Language Based Virtual machine - an example: java virtual machine. Really you could talk about any time you run a higher level language (perl, javascript, python, etc) That code does not run directly on the processor. It runs inside of another program which has some kind of virtual machine. Strictly speaking, a lot of languages can be interpreted, which means that you have a program that goes through line by line and figures out what that line is supposed to do and what the next instruction is. The point is that no modern language operates that way. What they all go through is some sort of translation phase, converts it to some binary code, and then it runs the byte code. That runtime is what's called a virtual machine. But virtual machines are everywhere when we are talking about trying to run programs. Operating systems can be thought of as implementing a virtual machine and that virtual machine it implements is the process. Key difference between a virtual machine that makes processes and the vm that is typically in these language based virtual machines. The difference between these is getting smaller. Any idea what this difference is?

Java based Virtual Machine - executes byte codes.
hardware can't interpret byte code

What is the nature of the binary format that is being run in an operating system process? What format is that code? - machine code - it's the code that is understood by the processor. Machine code here, byte code here, what's the difference? The hardware can't interpret this, This language the processor needs another program to translate. Why can't the processor understand java byte code? It could, there are chips that run java byte code natively. What's worse, the machine code that your processor understand? it actually doesn't. Modern processors such as x86 or x86-64 these are the most common things for a pc, arm byte code, arm machine language, that sort of thing.

This language is too annoying to use internally inside the microprocessor, it's not efficient, it was not designed to run very fast. It actually has a front end that takes that code and translates it to another byte code. There have been processor startups, where instead of having it done directly on the chip, they put something like a java virtual machine on the processor. Why am I saying this? The virtual and the real in computer science are often hard to tell apart. Virtual to one group could actually be real to another group. When you are coding in java / c, that is the language, that is real to you. That is the abstraction you are working i. but there are actually other levels below you. That generally is not the real level. when you are dealing with millions of transistors, there is a lot of abstraction. The process is the virtual machine that you run processes in. You take a file and loads it to disk. there is a little problem with this concept: Program on disk, is it a one to one mapping between programs on disk, and programs in memory? Not at all! Most programs on disk are not running at any given time. A given program on disk can be running in many different processes, you can have multiple instances of the same program running at the same time. Logically in an API for an operating system, you have to distinguish between the creation of a process, and the loading of an executable into that process. because you want to be able to facilitate the many to many mapping. What does that API look like? I'll give you another funny thing, if you are running one program, can you make that program do multiple things at the same time? Yes, now there is this whole notion of threading, but a thread is not a process.

==A thread is not a process.==

Process = thread(s) + address space

The cPU is virtual, because that would be really annoying to only have 4 things running at a time. Talking about things that are running at the same time. how many things are running @ the same time? Hyperthreading. What does it mean to run things at the same time, is it actually running? or is it logically running. we want the abstraction so that every process has it's own computer. That's what we are talking about when we talk about threads. Process has memory in which it can run, inside that process, how many processors do I have running? Classic only has 1. When you get to multithreaded processors, there is more than one logical solution. If you think about this? That's a mess! Having more than one program counter to track inside of one address space. this causes lots of problems! What happens when they step on each other. They do crazy things like chance teh loop index, from outside the loop. how do you reason about your code when things like this happen? When you put more than one logical cpu inside an address space that can happen. For a long time Operating systems, only supported processes with a single thread of execution. They supported lots of processes. Just made sure each one only had 1 thread.

Having more than one cpu running around inside the address space. What happens if they try running the same code twice? Change the loop index in the loop from outside of the loop. The right way to think of this is don't do that. When you put more than one cpu inside the address space. For a long time operating system supported one cpu inside the address space. They supported lots of processes but only one cpu. That's kind of limiting, but why do you want your running programs to be sharing memory, the main reason you want them to share memory is to communicate. Shared memory has only one advantage - it can be very fast. How do you make sure you don't overwrite each other's messages? In modern computation, distributed systems, big systems - you do almost everything you can - when you share memory you put some sort of API on top of it to control access. The only problem is - other than having potential communication overhead, you have higher overhead in general, you now have an address space to keep track of, it's own version of memory for each running program. That's so much overhead, it was a good while before computers started to allow multithreading, because it takes a lot of transistors to do. Used to be you'd have a Completely separate processor (a MMU) to take care of every running program in it's own address space, it is now integrated into cpus.

A running program is a process, address space, plus one or more threads. That's the virtual machine in which you are running in. It's got machine code.

Did you see any assembly in 2401? Not really.

=Sharing=

Explain terminology. Time space sharing, when we talk about virtualizing resources, talking about virtualizing cpu, virtualizing ram, what we are actually talking about is sharing. Like on a playground, we need to place nicely together.
An operating system is a set of mechanisms and policies for allowing for time and space sharing of the (computer resources) processor. In Time sharing: (taking turns) The processor is a limited resource. one program gets it for a while, another gets it for another while, then another gets it for a while. Space sharing means that you have all this RAM, split it up, you have this disk, split it up, one program gets part of it, and another program gets part of it. That's what we mean by space sharing.

=Virtual memory and physical memory=

[[File:Virtualmemory.png]]

There's this distinction between virtual memory and physical memory. Physical memory, is the RAM your computer actually has. You buy those little chips, and plug them in (voila, it goes faster)!
A program running with full privileges can still have a segmentation fault. The kernel can also have segmentation fault. When this happens the machine will crash hard.

[http://slacksite.com/slackware/oops.html For Oops vs. Panics]

Environment variables for X:

new view - direct graphical output from another computer on the network - bad because of latency - new

How to get around lag - run more of the code on the client instead of the server. Have the xclients have some code - transfer code to the xserver, to run on the server. Invisible website, downloads the page and it runs in your browser. Same thing different technology stack.

Mechanisms vs. Policy -

mechanisms - things to do things - the knobs that let us manipulate program state - should be maximally flexible so that they can implement whatever policies you want to do.

policy are what you should do

X Server <= mechanism

window manager, toolkit <= policy

Windows - two calls - create process () <--- many different parameters

unix - fork() and execve(file, cmdline, env)

Operating Systems 2014F Lecture 2

2014-10-06T17:27:34Z

Afry:

Audio from the lecture given on September 10, 2014 [http://homeostasis.scs.carleton.ca/~soma/os-2014f/lectures/comp3000-2014f-lec02-10Sep2014.mp3 is now available].

{{{
machine state
program counter
process states
paging / swapping
process
running program
virtualizing
time/space sharing
mechanisms
policy
}}}

Chapter 4 in the book:

processes - key abstraction in a modern operating system

Sitting all day kills you - seriously reduces your life expectancy. Working out, doesn't necessarily make up for sitting all day. If you walk around for 5 minutes every hour. Anyone use typing break programs? Occupational hazard of the career path you have chosen is that you sit in front of the computer typing. Anil started typing dvorak early in order to avoid repetitive strain injuries.

[http://www.lcdf.org/xwrits/ xwrits] link to save your wrists on a *nix machine gives you hand gestures to say it's time to get up. you can get it to insult you with hand gestures. It does hand gesture depending on culture. In order to tell you to take a break.

When typing it is important to take breaks.

You need to distinguish between programs and processes. A program is inprecise in the context of O/S.

Program is precise in context of operating systems - Your web browser is that a program?
Web browser is a lot of little programs, but makes up one big program. It is not a precise thing. What is precise is an executable. An executable is a file on disk that can be exec'd. (Disks are no longer disks they are all kinds of things such as filestate, etc) This is the unix version of the statement. There is a system call - called execve - takes as one of it's parameters a file and that file is then loaded into a process obliterating whatever else was in the process.

Code can take many forms in a computer systems, this is just one form of data.

For example, you ahve a text file that has a javascript / perl program / something, that is a program, it is also a text document, the operating system kernel does not really recognize it as an executable. You cannot give it as an argument to the execve system call. It has to run it indirectly, it has to find another exec executable to run that code. You have executables and you have processes.

A process - is an executable that has been executed - loaded into memory and started running. A process you should think of as an abstraction of a computer that can only run one program at a time. (Older personal computers, early 1960's or something, there is no abstraction of a process. There is no notion of running more than one program at a time. Logically speaking: when you wanted to run a program, all of memory would be loaded with that program, when you wanted to quit the program, you cut the power (turn the computer off).) They run one program at a time, you load it off the disk, and it has complete control of the machine. A process is the abstraction you get when you say, we don't want every program to have complete control of the computer because I do not want to have to reboot the computer to switch programs. I want to run different programs concurrently, for multiple reasons. Want to chain multiple programs in order to produce a result. (A Unix pipeline) The process - giving each running program (each executable) it's own virtual computer to run.

Virtualizing / virtualization (term is rather overloaded) What am I talking about when I say virtual? Something that isn't real. It's not a real thing. When people talk about virtual reality, they are talking about something that can be experienced. What we are saying in a computer science context: When we say virtual, we are really talking about an abstraction - What we actually have, the real thing is not good enough, it doesn't have qualities that you want, so you want to transform it into something more useful (in some way). When we talk about a virtual machine, we are talking about a machine (computer) that does not exist, in the sense that it is not embodied in actual hardware.

(from the theoretical side of computer science): All programming languages or programming system to a first approximation are equivalent, a system is known as Turing complete it can run anything. Turning one Turing complete system into another Turing complete system is the process of virtualization. The ones you've often heard of are: Language Based Virtual machine - an example: java virtual machine. Really you could talk about any time you run a higher level language (perl, javascript, python, etc) That code does not run directly on the processor. It runs inside of another program which has some kind of virtual machine. Strictly speaking, a lot of languages can be interpreted, which means that you have a program that goes through line by line and figures out what that line is supposed to do and what the next instruction is. The point is that no modern language operates that way. What they all go through is some sort of translation phase, converts it to some binary code, and then it runs the byte code. That runtime is what's called a virtual machine. But virtual machines are everywhere when we are talking about trying to run programs. Operating systems can be thought of as implementing a virtual machine and that virtual machine it implements is the process. Key difference between a virtual machine that makes processes and the vm that is typically in these language based virtual machines. The difference between these is getting smaller. Any idea what this difference is?

Java based Virtual Machine - executes byte codes.
hardware can't interpret byte code

What is the nature of the binary format that is being run in an operating system process? What format is that code? - machine code - it's the code that is understood by the processor. Machine code here, byte code here, what's the difference? The hardware can't interpret this, This language the processor needs another program to translate. Why can't the processor understand java byte code? It could, there are chips that run java byte code natively. What's worse, the machine code that your processor understand? it actually doesn't. Modern processors such as x86 or x86-64 these are the most common things for a pc, arm byte code, arm machine language, that sort of thing.

This language is too annoying to use internally inside the microprocessor, it's not efficient, it was not designed to run very fast. It actually has a front end that takes that code and translates it to another byte code. There have been processor startups, where instead of having it done directly on the chip, they put something like a java virtual machine on the processor. Why am I saying this? The virtual and the real in computer science are often hard to tell apart. Virtual to one group could actually be real to another group. When you are coding in java / c, that is the language, that is real to you. That is the abstraction you are working i. but there are actually other levels below you. That generally is not the real level. when you are dealing with millions of transistors, there is a lot of abstraction. The process is the virtual machine that you run processes in. You take a file and loads it to disk. there is a little problem with this concept: Program on disk, is it a one to one mapping between programs on disk, and programs in memory? Not at all! Most programs on disk are not running at any given time. A given program on disk can be running in many different processes, you can have multiple instances of the same program running at the same time. Logically in an API for an operating system, you have to distinguish between the creation of a process, and the loading of an executable into that process. because you want to be able to facilitate the many to many mapping. What does that API look like? I'll give you another funny thing, if you are running one program, can you make that program do multiple things at the same time? Yes, now there is this whole notion of threading, but a thread is not a process.

==A thread is not a process.==

Process = thread(s) + address space

The cPU is virtual, because that would be really annoying to only have 4 things running at a time. Talking about things that are running at the same time. how many things are running @ the same time? Hyperthreading. What does it mean to run things at the same time, is it actually running? or is it logically running. we want the abstraction so that every process has it's own computer. That's what we are talking about when we talk about threads. Process has memory in which it can run, inside that process, how many processors do I have running? Classic only has 1. When you get to multithreaded processors, there is more than one logical solution. If you think about this? That's a mess! Having more than one program counter to track inside of one address space. this causes lots of problems! What happens when they step on each other. They do crazy things like chance teh loop index, from outside the loop. how do you reason about your code when things like this happen? When you put more than one logical cpu inside an address space that can happen. For a long time Operating systems, only supported processes with a single thread of execution. They supported lots of processes. Just made sure each one only had 1 thread.

Having more than one cpu running around inside the address space. What happens if they try running the same code twice? Change the loop index in the loop from outside of the loop. The right way to think of this is don't do that. When you put more than one cpu inside the address space. For a long time operating system supported one cpu inside the address space. They supported lots of processes but only one cpu. That's kind of limiting, but why do you want your running programs to be sharing memory, the main reason you want them to share memory is to communicate. Shared memory has only one advantage - it can be very fast. How do you make sure you don't overwrite each other's messages? In modern computation, distributed systems, big systems - you do almost everything you can - when you share memory you put some sort of API on top of it to control access. The only problem is - other than having potential communication overhead, you have higher overhead in general, you now have an address space to keep track of, it's own version of memory for each running program. That's so much overhead, it was a good while before computers started to allow multithreading, because it takes a lot of transistors to do. Used to be you'd have a Completely separate processor (a MMU) to take care of every running program in it's own address space, it is now integrated into cpus.

A running program is a process, address space, plus one or more threads. That's the virtual machine in which you are running in. It's got machine code.

Did you see any assembly in 2401? Not really.

Explain terminology. Time space sharing, when we talk about virtualizing resources, talking about virtualizing cpu, virtualizing ram, what we are actually talking about is sharing. Like on a playground, we need to place nicely together.
An operating system is a set of mechanisms and policies for allowing for time and space sharing of the (computer resources) processor. In Time sharing: (taking turns) The processor is a limited resource. one program gets it for a while, another gets it for another while, then another gets it for a while. Space sharing means that you have all this RAM, split it up, you have this disk, split it up, one program gets part of it, and another program gets part of it. That's what we mean by space sharing.

Virtual memory and physical memory

[[File:Virtualmemory.png]]

A program running with full privileges can still have a segmentation fault. The kernel can also have segmentation fault. When this happens the machine will crash hard.

[http://slacksite.com/slackware/oops.html For Oops vs. Panics]

Environment variables for X:

new view - direct graphical output from another computer on the network - bad because of latency - new

How to get around lag - run more of the code on the client instead of the server. Have the xclients have some code - transfer code to the xserver, to run on the server. Invisible website, downloads the page and it runs in your browser. Same thing different technology stack.

Mechanisms vs. Policy -

mechanisms - things to do things - the knobs that let us manipulate program state - should be maximally flexible so that they can implement whatever policies you want to do.

policy are what you should do

X Server <= mechanism

window manager, toolkit <= policy

Windows - two calls - create process () <--- many different parameters

unix - fork() and execve(file, cmdline, env)

Operating Systems 2014F Lecture 2

2014-10-06T17:09:42Z

Afry:

Operating Systems 2014F Lecture 2

2014-10-06T16:58:44Z

Afry: