Soma-notes - User contributions [en]

SystemsSec 2016W Lecture 13

2016-03-08T01:20:30Z

Hassansaid: /* Classic Buffer Overflow Attack */

== Topics & Readings ==
* Reading: Aleph One (aka Elias Levy), [http://www.phrack.com/issues/49/14.html#article Smashing The Stack For Fun And Profit] (Phrack 49, 1996)
* Buffer Overflow Attack
* Memory Corruption Attack

== Class Notes ==

==== Unsafe Language ====
Defined as languages, such as C, that are susceptible to buffer overflow attacks. Safe languages are not susceptible because they have really strong compilers.

Secure systems are not susceptible to overflow attacks because the reference monitor is small enough to verify and remove any bugs. The reference monitor is also resilient due to the fact that it is outside the domain of process execution.

==== Memory Corruption Attack ====
[[File:systemMemory.png]] 
'''Figure 1:''' System structure with certain portions of memory re severed for those types 

In a buffer overflow attack you can essentially over write some other portion of memory. For example an attacker may put too much information into the memory portion reserved for string such that it overflows into the memory space reserved for array. Safe languages such as Java are not susceptible to memory corruption attacks unless there is some bug in the compiler.

A common way to know that your code is vulnerable to a buffer overflow/memory corruption attack is if you get a segmentation fault.

==== Classic Buffer Overflow Attack ====
[[File:ClassicBufferOverflow.png]] 
'''Figure 2:''' 
'''Code Pointer:''' Not typically writing to pointer. You are writing to region of memory where the pointer is. 
'''Bad Code:''' The bad code attacker injects is "shell code" 

The bad code comes from user input (network, keyboard, disk), in the program itself (just run in different way), environment variable. You can use function pointers, etc, but most code pointers used are return addresses. Return addresses are stored on stack (built in modern processes). You have stack for local variables (when you enter function you allocate space for that function, and when you return from that function you clean up local variables). There is no requirement that local variable stack and return address stack be the same but in practice they are. Local variables stick around longer, and it can be something that attackers can modify. Local variables can be modified using C functions such as gets().

Since attackers are trying to change the code pointer to their bad code, they must figure out where their code is in memory. To do this the bad code is created in such a way that there is a "landing strip" at the beginning of the code. This "landing strip" is usually a bunch of NO OPS. This gives an attacker a bit of extra cushion to ensure that their code could be executed.

==== Buffer Overflow Prevention ====
''' Fuzzing ''' 
To find Buffer Overflow/Memory Corruption Attack vulnerability you try to find segmentation fault using fuzzing. Fuzzing: you take program and throw well formed legitimate data at it first. As the fuzzing progresses you tweak it (essentially starts well formed and then gets more random). Fuzzing can be done without source code.

''' Randomization of Stack ''' 
Buffer Overflow defenses make it hard to find landing strip. The best method is to make it hard for attacker to store bad code in memory (example randomize stack).

SystemsSec 2016W Lecture 13

2016-03-08T01:13:27Z

Hassansaid: /* Memory Corruption Attack */

== Topics & Readings ==
* Reading: Aleph One (aka Elias Levy), [http://www.phrack.com/issues/49/14.html#article Smashing The Stack For Fun And Profit] (Phrack 49, 1996)
* Buffer Overflow Attack
* Memory Corruption Attack

== Class Notes ==

==== Unsafe Language ====
Defined as languages, such as C, that are susceptible to buffer overflow attacks. Safe languages are not susceptible because they have really strong compilers.

Secure systems are not susceptible to overflow attacks because the reference monitor is small enough to verify and remove any bugs. The reference monitor is also resilient due to the fact that it is outside the domain of process execution.

==== Memory Corruption Attack ====
[[File:systemMemory.png]] 
'''Figure 1:''' System structure with certain portions of memory re severed for those types 

In a buffer overflow attack you can essentially over write some other portion of memory. For example an attacker may put too much information into the memory portion reserved for string such that it overflows into the memory space reserved for array. Safe languages such as Java are not susceptible to memory corruption attacks unless there is some bug in the compiler.

A common way to know that your code is vulnerable to a buffer overflow/memory corruption attack is if you get a segmentation fault.

==== Classic Buffer Overflow Attack ====
[[File:ClassicBufferOverflow.png]] 
'''Figure 2:''' 
'''Code Pointer:''' Not typically writing to pointer. You are writing to region you where pointer is.
'''Bad Code:''' The bad code attacker injects is "shell code" 

The bad code comes from user input (network, keyboard, disk), in the program itself (just run in different way), environment variable. You can use function pointers, etc, but most code pointers used are return addresses. Return addresses are stored on stack (built in modern processes). You have stack for local variables (when you enter function you allocate space for that function, and when you return from that function you clean up local variables). There is no requirement that local variable stack and return address stack be the same but in practice they are. Local variables stick around longer, and it can be something that attackers can modify. Local variables can be modified using C functions such as gets().

Since attackers are trying to change the code pointer to their bad code, they must figure out where their code is in memory. To do this the bad code is created in such a way that there is a "landing strip" at the beginning of the code. This "landing strip" is usually a bunch of NO OPS. This gives an attacker a bit of extra cushion to ensure that their code could be executed.

==== Buffer Overflow Prevention ====
''' Fuzzing ''' 
To find Buffer Overflow/Memory Corruption Attack vulnerability you try to find segmentation fault using fuzzing. Fuzzing: you take program and throw well formed legitimate data at it first. As the fuzzing progresses you tweak it (essentially starts well formed and then gets more random). Fuzzing can be done without source code.

''' Randomization of Stack ''' 
Buffer Overflow defenses make it hard to find landing strip. The best method is to make it hard for attacker to store bad code in memory (example randomize stack).

Computer Systems Security (Winter 2016)

2016-03-07T08:22:50Z

Hassansaid: /* Lectures and Exams */

==Course Outline==

[[Computer Systems Security: Winter 2016 Course Outline|Here]] is the course outline.

==Hacking Opportunities==

The [[SystemsSec 2016W Hacking Opportunities|Hacking Opportunities]] page lists potential hacking opportunities that you can attempt for your hacking journal. If you attempt but do not successfully accomplish one of them, be sure to document what you tried. As you learn more, you may come back to them and try again.

==Resources==

===Readings===

* For the first part of the course we will be reading selections from Trent Jaeger's [http://www.morganclaypool.com/doi/abs/10.2200/S00126ED1V01Y200808SPT001 Operating Systems Security] textbook. You can download the PDF [http://www.morganclaypool.com.proxy.library.carleton.ca/doi/abs/10.2200/S00126ED1V01Y200808SPT001 through Carleton's library]. In the reading assignments this text will be referred to as "Jaeger".
* An excellent but dated text on browser security is Michal Zalewski's [https://code.google.com/p/browsersec/wiki/Main Browser Security Handbook].

===Other Courses===

* Dan Boneh ran an excellent course at Stanford in Spring 2015 on [https://crypto.stanford.edu/cs155/ Computer and Network Security]. This course has many interesting readings that we will not be covering. Also, the assignments are very good sources for hacking opportunities.
* The assignments from the Winter 2015 run of COMP 4108 [https://ccsl.carleton.ca/~dmccarney/COMP4108/ are available]. They are a reasonable start for several hacking opportunities.

==Lectures and Exams==

<table style="width: 100%;" border="1" cellpadding="4" cellspacing="0">
<tr valign="top">
<th>
Date
</th>
<th>
Topic
</th>
<th>
Readings
</th>
</tr>
<tr>
<td>
Jan. 7

</td>
<td>
[[SystemsSec 2016W Lecture 1|Introduction]]

</td>
<td>Jaeger, Chapter 1 (Introduction)</td></tr>
<tr>
<td>
Jan. 12

</td>
<td>
[[SystemsSec 2016W Lecture 2|Access Control, Security Hacking 101]]

</td>
<td>Jaeger, Chapter 2 (Access Control Fundamentals)</td></tr>
<tr>
<td>
Jan. 14

</td>
<td>
[[SystemsSec 2016W Lecture 3|Multics, UNIX, and Windows]]

</td>
<td>Jaeger, Chapter 3 (Multics) and Chapter 4 (UNIX & Windows) </td></tr>
<tr>
<td>
Jan. 19

</td>
<td>
[[SystemsSec 2016W Lecture 4|Secure OSs, theory and practice]]

</td>
<td>Jaeger, Chapter 6 (Security Kernels) and Chapter 7 (Securing Commercial Operating Systems)</td></tr>
<tr>
<td>
Jan. 21

</td>
<td>
[[SystemsSec 2016W Lecture 5|LSM, SELinux, & Capabilities]]

</td>
<td>Jaeger, Chapter 9 (LSM & SELinux) and Chapter 10 (Secure Capability Systems)</td></tr>
<tr>
<td>
Jan. 26

</td>
<td>
[[SystemsSec 2016W Lecture 6|Secure Virtual Machines, Systems Assurance]]

</td>
<td>Jaeger, Chapter 11 (Secure Virtual Machine Systems) and Chapter 12 (System Assurance)</td></tr>
<tr>
<td>
Jan. 28

</td>
<td>
[[SystemsSec 2016W Lecture 7|Lecture 7]]

</td>
<td></td></tr>
<tr>
<td>
Feb. 2

</td>
<td>
[[SystemsSec 2016W Lecture 8|Lecture 8]]

</td>
<td></td></tr>
<tr>
<td>
Feb. 4

</td>
<td>
[[SystemsSec 2016W Lecture 9|Defensive Security Technologies / Hacking Opportunities]]

</td>
<td></td></tr>
<tr>
<td>
Feb. 9

</td>
<td>
[[SystemsSec 2016W Lecture 10|Security Research, Hashes, and Secure Protocols]]

</td>
<td></td></tr>
<tr>
<td>
Feb. 11

</td>
<td>
[[SystemsSec 2016W Lecture 11|Modeling a potential attack/ Midterm FAQ]]

</td>
<td></td></tr>
<tr>
<td>
Feb. 23

</td>
<td>
[[SystemsSec 2016W Lecture 12|Midterm Review]]

</td>
<td></td></tr>
<tr>
<td>
Feb. 25

</td>
<td>
Midterm (in class)

</td>
<td></td></tr>
<tr>
<td>
Mar. 1

</td>
<td>
[[SystemsSec 2016W Lecture 13|Buffer Overflow/Memory Corruption Attacks]]

</td>
<td>Aleph One (aka Elias Levy), [http://www.phrack.com/issues/49/14.html#article Smashing The Stack For Fun And Profit] (Phrack 49, 1996)</td></tr>
<tr>
<td>
Mar. 3

</td>
<td>
[[SystemsSec 2016W Lecture 14|Lecture 14]]

</td>
<td>
Wikipedia, [https://en.wikipedia.org/wiki/Buffer_overflow_protection Buffer Overflow Protection] 
Crispin Cowan et al., [https://www.usenix.org/legacy/publications/library/proceedings/sec98/cowan.html StackGuard: Automatic Adaptive Detection and Prevention of Buffer-Overflow Attacks] (USENIX Security, 1998)
</td>
</tr>
<tr>
<td>
Mar. 8

</td>
<td>
[[SystemsSec 2016W Lecture 15|Lecture 15]]

</td>
<td>Hovav Shacham et al., [http://dx.doi.org/10.1145/1030083.1030124 On the effectiveness of address-space randomization] (ACM CCS, 2004) [http://dl.acm.org.proxy.library.carleton.ca/ft_gateway.cfm?id=1030124&ftid=285463&dwn=1&CFID=588127386&CFTOKEN=74533951 (proxy)] 
Hovav Shachem [http://dx.doi.org/10.1145/1315245.1315313 The geometry of innocent flesh on the bone: return-into-libc without function calls (on the x86)] (ACM CCS 2007) [http://dl.acm.org.proxy.library.carleton.ca/ft_gateway.cfm?id=1315313&ftid=476749&dwn=1&CFID=588127386&CFTOKEN=74533951 (proxy)]</td></tr>
<tr>
<td>
Mar. 10

</td>
<td>
[[SystemsSec 2016W Lecture 16|Lecture 16]]

</td>
<td>Bellovin and Cheswick, [http://dx.doi.org/10.1109/35.312843 Network Firewalls] (IEEE Communications Magazine, 1994) [http://ieeexplore.ieee.org.proxy.library.carleton.ca/stamp/stamp.jsp?tp=&arnumber=312843 (proxy)]</td></tr>
<tr>
<td>
Mar. 15

</td>
<td>
[[SystemsSec 2016W Lecture 17|Lecture 17]]

</td>
<td></td></tr>
<tr>
<td>
Mar. 17

</td>
<td>
[[SystemsSec 2016W Lecture 18|Lecture 18]]

</td>
<td></td></tr>
<tr>
<td>
Mar. 22

</td>
<td>
[[SystemsSec 2016W Lecture 19|Lecture 19]]

</td>
<td></td></tr>
<tr>
<td>
Mar. 24

</td>
<td>
[[SystemsSec 2016W Lecture 20|Lecture 20]]

</td>
<td></td></tr>
<tr>
<td>
Mar. 29

</td>
<td>
[[SystemsSec 2016W Lecture 21|Lecture 21]]

</td>
<td></td></tr>
<tr>
<td>
Mar. 31

</td>
<td>
[[SystemsSec 2016W Lecture 22|Lecture 22]]

</td>
<td></td></tr>
<tr>
<td>
Apr. 5

</td>
<td>
[[SystemsSec 2016W Lecture 23|Lecture 23]]

</td>
<td></td></tr>
<tr>
<td>
April 7

</td>
<td>
[[SystemsSec 2016W Lecture 24|Lecture 24]]

</td>
<td></td></tr>
<tr>
<td>
April 19, 9 AM

</td>
<td>
Final Exam

</td>
<td></td></tr>
</table>

==Assignments==
<table style="width: 100%;" border="1" cellpadding="4" cellspacing="0">
<tr valign="top">
<th>
Due Date
</th>
<th>
Assignments
</th>
</tr>
<tr>
<td>
Jan. 30

</td>
<td>
[[SystemsSec 2016W: Assignment 1|Assignment 1]]

</td>
</tr>
<tr>
<td>
Feb. 22

</td>
<td>
[[SystemsSec 2016W: Assignment 2|Assignment 2]]

</td>
</tr>
<tr>
<td>
Mar. 19

</td>
<td>
[[SystemsSec 2016W: Assignment 3|Assignment 3]]

</td>
</tr>
<tr>
<td>
April 4

</td>
<td>
[[SystemsSec 2016W: Assignment 4|Assignment 4]]

</td>
</tr>
</table>

==Lecture Notes Guidelines==

Part of your participation mark is doing notes for at least one of the lectures. Here are the guidelines for those notes.

The class TA Borke (BorkeObadaObieh at cmail.carleton.ca) will be handling course notes. Please contact her to schedule your class to take notes.

Borke or Anil will set you up with an account on this wiki. You'll enter your initial draft notes here and then work with Borke to make sure they are of sufficient quality. This may require a few rounds of revisions; however, if you follow the guidelines below it shouldn't be too bad.

You should plan on organizing your notes as follows:
* Organize them in at least the following sections: Topics & Readings and Notes.
* The Topics & Readings section lists the main topics covered in the class, e.g. "buffer overflows". Please use an unordered bulleted list (using *'s in wiki markup). In this section also list readings relevant to the lecture that were mentioned in class.
* Put your notes in the Notes section.

Use (nested) lists if appropriate for the notes; however, please have some text that isn't bulleted. Please try to make the notes even if you did not attend lecture; however, you don't need to cover every small bit of information that was covered. In particular the notes do not need to include digressions into topics only tangentially related to the course. Complete sentences are welcome but not required.

==Security Reading Analysis Guidelines==

A security reading analysis is a detailed analysis of a security research paper. In it you analyze the key arguments of the paper and give your informed opinion.

Most security papers can be classified as attack or defense papers. You should analyze them differently.

For attack papers:
* What systems are vulnerable to the attack?
* What is the nature of the vulnerability?
* What is the the exploit? In particular, what is its technical core?
* How reproducible is the exploit?
* Are there likely to be many similar exploits, in the targeted system or other systems?
* How difficult will it be mitigate/fix the vulnerability in targeted systems?

For defense papers:
* What is the security problem the paper addresses? In what kind of threat model(s) does the problem exist?
* How significant is the problem? Specifically, to what degree do existing solutions not work sufficiently well?
* What is the defense? How does it work?
* To what degree will the defense potentially solve the targeted security problem? In particular, how difficult will it be for attackers to adapt to this defense?
* What are the challenges facing deployment of the defense? Are they likely to be overcome?

For both kinds of papers, you should give your reaction by addressing questions like the following:
* Did you like the paper?
* What it easy to understand, or was it hard to read?
* Did you learn much from the paper?
* How surprised were you by the result?

Your analysis should not cover the above questions separately (this would tend to make for a very wordy analysis); instead, use these questions as a guide in writing a short essay (1-2 pages) on the paper in question.

SystemsSec 2016W Lecture 13

2016-03-03T00:31:54Z

Hassansaid:

== Topics & Readings ==
* Reading: Aleph One (aka Elias Levy), [http://www.phrack.com/issues/49/14.html#article Smashing The Stack For Fun And Profit] (Phrack 49, 1996)
* Buffer Overflow Attack
* Memory Corruption Attack

== Class Notes ==

==== Unsafe Language ====
Defined as languages, such as C, that are susceptible to buffer overflow attacks. Safe languages are not susceptible because they have really strong compilers.

Secure systems are not susceptible to overflow attacks because the reference monitor is small enough to verify and remove any bugs. The reference monitor is also resilient due to the fact that it is outside the domain of process execution.

==== Memory Corruption Attack ====
[[File:systemMemory.png]] 
'''Figure 1:''' System structure with certain portions of memory re severed for those types 

In a buffer overflow attack you can essentially over write some other portion of memory. For example an attacker may put to much information into the memory portion reserved for string such that it overflows into the memory space reserved for array. Safe languages such as Java are not susceptible to memory corruption attacks unless there is some bug in the compiler.

A common way to know that your code is vulnerable to a buffer overflow/memory corruption attack is if you get a segmentation fault.

==== Classic Buffer Overflow Attack ====
[[File:ClassicBufferOverflow.png]] 
'''Figure 2:''' 
'''Code Pointer:''' Not typically writing to pointer. You are writing to region you where pointer is.
'''Bad Code:''' The bad code attacker injects is "shell code" 

The bad code comes from user input (network, keyboard, disk), in the program itself (just run in different way), environment variable. You can use function pointers, etc, but most code pointers used are return addresses. Return addresses are stored on stack (built in modern processes). You have stack for local variables (when you enter function you allocate space for that function, and when you return from that function you clean up local variables). There is no requirement that local variable stack and return address stack be the same but in practice they are. Local variables stick around longer, and it can be something that attackers can modify. Local variables can be modified using C functions such as gets().

Since attackers are trying to change the code pointer to their bad code, they must figure out where their code is in memory. To do this the bad code is created in such a way that there is a "landing strip" at the beginning of the code. This "landing strip" is usually a bunch of NO OPS. This gives an attacker a bit of extra cushion to ensure that their code could be executed.

==== Buffer Overflow Prevention ====
''' Fuzzing ''' 
To find Buffer Overflow/Memory Corruption Attack vulnerability you try to find segmentation fault using fuzzing. Fuzzing: you take program and throw well formed legitimate data at it first. As the fuzzing progresses you tweak it (essentially starts well formed and then gets more random). Fuzzing can be done without source code.

''' Randomization of Stack ''' 
Buffer Overflow defenses make it hard to find landing strip. The best method is to make it hard for attacker to store bad code in memory (example randomize stack).

File:ClassicBufferOverflow.png

2016-03-03T00:00:08Z

Hassansaid:

File:SystemMemory.png

2016-03-03T00:00:00Z

Hassansaid:

SystemsSec 2016W Lecture 13

2016-03-02T23:59:23Z

Hassansaid:

SystemsSec 2016W Lecture 13

2016-03-02T23:48:02Z

Hassansaid:

Class Notes
Unsafe Language
Defined as languages, such as C, that are susceptible to buffer overflow attacks. Safe languages are not susceptible because they have really strong compilers.

Secure systems are not susceptible to overflow attacks because the reference monitor is small enough to verify and remove any bugs. The reference monitor is also resilient due to the fact that it is outside the domain of process execution.

Memory Corruption Attack
Figure 1: System structure with certain portions of memory re severed for those types

In a buffer overflow attack you can essentially over write some other portion of memory. For example an attacker may put to much information into the memory portion reserved for string such that it overflows into the memory space reserved for array. Safe languages such as Java are not susceptible to memory corruption attacks unless there is some bug in the compiler.

A common way to know that your code is vulnerable to a buffer overflow/memory corruption attack is if you get a segmentation fault.

Classic Buffer Overflow Attack

SystemsSec 2016W Lecture 13

2016-03-02T23:27:42Z

Hassansaid: Created page with "Class Notes Unsafe Language Defined as languages, such as C, that are susceptible to buffer overflow attacks. Safe languages are not susceptible because they have really stron..."

User:Hassansaid

2015-03-16T19:16:32Z

Hassansaid: Created page with "Hello World!"

Hello World!

DistOS 2015W Session 10

2015-03-16T19:14:06Z

Hassansaid: /* Kademlia */

Feel free to tweak the questions!

==Kademlia==
Members: Kirill, Deep, Jason, Hassan

* '''Why are DHTs relevant to distributed OSs?'''
** Using many system, having repetition
** DHT to distributed content over multiple nodes
** Decentralized therefore peer-to-peer

* '''How is content divided?'''
** File hashes
** Node ID to locate the value
** 160 bit key space, binary tree for partition and searching down the tree

* '''How is the network traversed?'''
** Highest prefix and increase the number of digit it used to get close to the target nodes

* '''What trust assumptions does the system make?'''
** DHT by itself is insecure
** The academic and practitioner communities have realized that all current DHT designs suffer from a security weakness, known as the Sybil attack
** K-buckets
*** Binary tree with each node having k-buckets as leaf
*** Any given k-node is very unlikely to fail within an hour of each other
*** New nodes are only inserted when there is room in the bucket or the oldest node doesn't respond
** Uses UDP therefore packets are often lost

* '''Performance constraints?'''
** Binary tree traversing, therefore traversing is maximum O(log n)

* '''What non-DHT internet infrastructure would you replace with a DHT? How suitable is Kademlia for this purpose?'''
** DNS
** Any kind of meta-data service

==Comet==
Members: Mohamed Ahmed, Apoorv Sangal, Ambalica Sharma

* Why are DHTs relevant to distributed OSs?
DHT is an infrastructure than enables many clients to share information, and scale to handle node arrival, departure and failure. DHT's serve many of the design goals of disbtributed operating systems. The paper states that "DHTs are increasingly used to support a variety of distributed applications, such as file-sharing, distributed resource tracking, end-system multicast, publish-subscribe systems, distributed search engines"

* How is content divided?

One of the three main components of the comet system is a routing substrate which
implements the value/node mapping. This allows a client to find the node htat stores
a specific data item. Since Comet uses a DHT implementation, routing occurs by applying
a hash function to the key to compute node ID's that store the associated value.

* How is the network traversed?

* What trust assumptions does the system make?

A client node running COmet should be protected from the execution of handlers
e.g. an executing handler cannot corrupt the node or use unlimited resources.
Handlers should not be able to mount messaging attacks on other nodes.

Users downloading Comet must trust it and have guarantees about its behavior. For this reason, Comet enforces four important restrictions:
1. Limited knowledge: an ASO is not aware of other objects
or resources stored on the same node and has no
direct way to learn about them.
2. Limited access: an object handler can manipulate only its own value and cannot modify the values of other objects on its storage node.
3. Limited communication: an active storage object cannot
send arbitrary messages over the network.
4. Limited resource consumption: an ASO’s resource usage is strictly bounded, e.g., the system limits the amount of computation and memory it can consume.

* Performance constraints?

* What non-DHT internet infrastructure would you replace with a DHT? How suitable is Kademlia for this purpose?

==Tapestry==
Members:

* Why are DHTs relevant to distributed OSs?

Because they provide a way to distribute information over large networks (distributed key/value store).

* How is content divided?

Uses consistent hashing (SHA-1), upon node creation (join) creates optimal routing table.

* How is the network traversed?

You look at your neighbours, you see which neighbour is closest to your destination, and recurse.

* What trust assumptions does the system make?

It assumes the system is entirely trustworthy from adversaries. While network failures may happen and nodes may go down, no node will explicitly try to mess with the network.

* Performance constraints?

O(log n) access times to any given node. Best effort publishing/unpublishing via decentralized object location routing.

* What non-DHT internet infrastructure would you replace with a DHT? How suitable is Tapestry for this purpose?

DistOS 2015W Session 10

2015-03-16T19:11:37Z

Hassansaid: /* Kademlia */

Feel free to tweak the questions!

==Kademlia==
Members: Kirill, Deep, Jason, Hassan

* Why are DHTs relevant to distributed OSs?
** Using many system, having repetition
** DHT to distributed content over multiple nodes
** Decentralized therefore peer-to-peer

* How is content divided?
** File hashes
** Node ID to locate the value
** 160 bit key space, binary tree for partition and searching down the tree

* How is the network traversed?
** Highest prefix and increase the number of digit it used to get close to the target nodes

* What trust assumptions does the system make?
** DHT by itself is insecure
** The academic and practitioner communities have realized that all current DHT designs suffer from a security weakness, known as the Sybil attack
** K-buckets: binary tree with each node having k-buckets as leaf. Any given k-node is very unlikely to fail within an hour of each other. New nodes are only inserted when there is room in the bucket or the oldest node doesn't respond
** Uses UDP therefore packets are often lost

* Performance constraints?
** Binary tree traversing, therefore traversing is maximum O(log n)

* What non-DHT internet infrastructure would you replace with a DHT? How suitable is Kademlia for this purpose?

==Comet==
Members: Mohamed Ahmed, Apoorv Sangal, Ambalica Sharma

* Why are DHTs relevant to distributed OSs?
DHT is an infrastructure than enables many clients to share information, and scale to handle node arrival, departure and failure. DHT's serve many of the design goals of disbtributed operating systems. The paper states that "DHTs are increasingly used to support a variety of distributed applications, such as file-sharing, distributed resource tracking, end-system multicast, publish-subscribe systems, distributed search engines"

* How is content divided?

One of the three main components of the comet system is a routing substrate which
implements the value/node mapping. This allows a client to find the node htat stores
a specific data item. Since Comet uses a DHT implementation, routing occurs by applying
a hash function to the key to compute node ID's that store the associated value.

* How is the network traversed?

* What trust assumptions does the system make?

A client node running COmet should be protected from the execution of handlers
e.g. an executing handler cannot corrupt the node or use unlimited resources.
Handlers should not be able to mount messaging attacks on other nodes

* Performance constraints?

* What non-DHT internet infrastructure would you replace with a DHT? How suitable is Kademlia for this purpose?

==Tapestry==
Members:

* Why are DHTs relevant to distributed OSs?

Because they provide a way to distribute information over large networks (distributed key/value store).

* How is content divided?

Uses consistent hashing (SHA-1), upon node creation (join) creates optimal routing table.

* How is the network traversed?

You look at your neighbours, you see which neighbour is closest to your destination, and recurse.

* What trust assumptions does the system make?

It assumes the system is entirely trustworthy from adversaries. While network failures may happen and nodes may go down, no node will explicitly try to mess with the network.

* Performance constraints?

O(log n) access times to any given node. Best effort publishing/unpublishing via decentralized object location routing.

* What non-DHT internet infrastructure would you replace with a DHT? How suitable is Tapestry for this purpose?

DistOS 2015W Session 10

2015-03-16T18:59:52Z

Hassansaid: /* Kademlia */

Feel free to tweak the questions!

==Kademlia==
Members: Kirill, Deep, Jason, Hassan

* Why are DHTs relevant to distributed OSs?

* How is content divided?

* How is the network traversed?

* What trust assumptions does the system make?

* Performance constraints?

* What non-DHT internet infrastructure would you replace with a DHT? How suitable is Kademlia for this purpose?

==Comet==
Members: Mohamed Ahmed, Apoorv Sangal, Ambalica Sharma

* Why are DHTs relevant to distributed OSs?

* How is content divided?

* How is the network traversed?

* What trust assumptions does the system make?

* Performance constraints?

* What non-DHT internet infrastructure would you replace with a DHT? How suitable is Kademlia for this purpose?

==Tapestry==
Members:

* Why are DHTs relevant to distributed OSs?

Because they provide a way to distribute information over large networks (distributed key/value store).

* How is content divided?

* How is the network traversed?

* What trust assumptions does the system make?

* Performance constraints?

* What non-DHT internet infrastructure would you replace with a DHT? How suitable is Tapestry for this purpose?

DistOS 2015W Session 6

2015-02-09T21:45:30Z

Hassansaid: /* Group 2 */

==Midterm==

The midterm from last year [http://homeostasis.scs.carleton.ca/~soma/distos/2015w/comp4000-2014w-midterm.pdf is now available].

==Group 1==

'''Team:''' Kirill, Jamie, Alexis, Veena, Khaled, Hassan

{| class="wikitable"
|-
!
! FARSITE
! OceanStore
|-
! Fault Tolerance
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
|-
! Cryptography
| Trusted Certificates
| A strong cryptographic algorithm on read-only operations
|-
! Implementation
| Did not mention what programming they used, but it was based on Windows. They did not implement the file system
| Implemented in JAVA
|-
! Scalability
| Scalable to a University or large corporations, maximum 105
| Worldwide scalability, maximum 1010
|-
! File Usage
| Was designed for general purpose files
| Was designed for small file sizes
|-
! Scope
| All clients sharing the available resources
| Transient centralized service
|-
! Object Model
| Didn't use the object model
| Used the object model
|}

==Group 2==
Team Members: Apoorv, Ambalica, Ashley, Eric, Mert, Shivjot

==Group 3==
DANY, MOE, DEEP, SAMEER, TROY

'''FARSITE'''

'''Security'''
• Cascading certificates system through directory hierarchy
• Keys
• Three types of certificates
• CFS required to authorized certificate
• Because directory groups only modify their shared state via a Byzantine-fault-tolerant protocol, we trust the group not to make
an incorrect update to directory metadata. This metadata includes an access control list (ACL) of public keys of all users
who are authorized writers to that directory and to files in it
• Both file content and user-sensitive metadata (meaning file and directory names) are encrypted for privacy.

'''System Architecture'''
• Client Monitor, directory group, file host
• When space runs out in directory group, delegate’s ownership to sub tree to other delegate group.

'''OCEANSTORE'''

'''Security'''
• GUID and ACLs used for write, encryption used for reads.
• To prevent unauthorized reads, it encrypts
all data in the system that is not completely public and distributes the encryption key to those users with read permission

DistOS 2015W Session 6

2015-02-09T21:11:17Z

Hassansaid: /* Group 1 */

DistOS 2015W Session 6

2015-02-09T21:10:32Z

Hassansaid: /* Group 3 */

DistOS 2015W Session 6

2015-02-09T21:06:53Z

Hassansaid: /* Group 3 */

==Midterm==

The midterm from last year [http://homeostasis.scs.carleton.ca/~soma/distos/2015w/comp4000-2014w-midterm.pdf is now available].

==Group 1==

==Group 2==
Team Members: Apoorv, Ambalica, Ashley, Eric, Mert, Shivjot

==Group 3==

'''Team:''' Kirill, Jamie, Alexis, Veena, Khaled, Hassan

{| class="wikitable"
|-
!
! FARSITE
! OceanStore
|-
! Fault Tolerance
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
|-
! Cryptography
| Trusted Certificates
| A strong cryptographic algorithm on read-only operations
|-
! Implementation
| Did not mention what programming they used, but it was based on Windows. They did not implement the file system
| Implemented in JAVA
|-
! Scalability
| Scalable to a University or large corporations, maximum 105
| Worldwide scalability, maximum 1010
|-
! File Usage
| Was designed for general purpose files
| Was designed for small file sizes
|-
! Scope
| All clients sharing the available resources
| Transient centralized service
|-
! Object Model
| Didn't use the object model
| Used the object model
|}

DistOS 2015W Session 6

2015-02-09T21:04:33Z

Hassansaid: /* Group 3 */

==Midterm==

The midterm from last year [http://homeostasis.scs.carleton.ca/~soma/distos/2015w/comp4000-2014w-midterm.pdf is now available].

==Group 1==

==Group 2==
Team Members: Apoorv, Ambalica, Ashley, Eric, Mert, Shivjot

==Group 3==

{| class="wikitable"
|-
!
! FARSITE
! OceanStore
|-
! Fault Tolerance
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
|-
! Cryptography
| Trusted Certificates
| A strong cryptographic algorithm on read-only operations
|-
! Implementation
| Did not mention what programming they used, but it was based on Windows. They did not implement the file system
| Implemented in JAVA
|-
! Scalability
| Scalable to a University or large corporations, maximum 10{{sup|5}}
| Worldwide scalability, maximum 10{{sup|10}}
|-
! File Usage
| Was designed for general purpose files
| Was designed for small file sizes
|-
! Scope
| All clients sharing the available resources
| Transient centralized service
|-
! Object Model
| Didn't use the object model
| Used the object model
|}

DistOS 2015W Session 6

2015-02-09T21:03:34Z

Hassansaid: /* Group 3 */

==Midterm==

The midterm from last year [http://homeostasis.scs.carleton.ca/~soma/distos/2015w/comp4000-2014w-midterm.pdf is now available].

==Group 1==

==Group 2==
Team Members: Apoorv, Ambalica, Ashley, Eric, Mert, Shivjot

==Group 3==

{| class="wikitable"
|-
!
! FARSITE
! OceanStore
|-
! Fault Tolerance
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
|-
! Cryptography
| Trusted Certificates
| A strong cryptographic algorithm on read-only operations
|-
! Implementation
| Did not mention what programming they used, but it was based on Windows. They did not implement the file system
| Implemented in JAVA
|-
! Scalability
| Scalable to a University or large corporations, maximum 10^5
| Worldwide scalability, maximum 10{{su|10}}
|-
! File Usage
| Was designed for general purpose files
| Was designed for small file sizes
|-
! Scope
| All clients sharing the available resources
| Transient centralized service
|-
! Object Model
| Didn't use the object model
| Used the object model
|}

DistOS 2015W Session 4

2015-02-01T22:34:50Z

Hassansaid:

= Andrew File System =
AFS (Andrew File System) was set up as a direct response to NFS. Essentially universities found issues when they tried to scale NFS in a way that would allow them to share files amongst their staff effectively. AFS was more scalable than NFS because read-write operations happened locally before they were committed to the server (data store). As such the largest file size that AFS could deal with was the size of its local disk cache.

AFS open-close operations were more time-consuming, while for NFS its read-write operations were more time consuming.

= Amoeba Operating System =

=== Capablities: ===
* Pointer to the object
* Capability assigning right to perform to some operation to the object ticket
* Communicate wide area network
* a kind of ticket or key that allows the holder of the capa- bility to perform some (not neces- sarily all)
* Each user process owns some collection of capabilities, which together define the set of objects it may access and the types of operations that my ne performed on each
* After the server has performed the operation, it sends back a reply message that unblocks the client
* Sending messages, blocking and accepting forms the remote procedure call that can be encapsulate using to make entire remote operation look like local procedure
* Second field: used by the sever to identify which of its objects is being addressed server port and object number identify object which operation to performed
* Generates 48-bit random number
* The third field is the right field which contains a bit map telling which operation the holder of the capability may performed

=== Thread Management: ===
* Same process have multiple thread and each process has its own registered counter and stack
* Behave like process
* It can synchronized using mutex semaphore
* File: Multiple thread,
* Blocked when there's multiple threads
* Buttlet thread the mutex
* The careful reader may have noticed that user process can pull 813kbytes/sec

= The V Distributed System =

* First tent in V design: High Performance communication is the most critical facility for distributed systems.
* Second; The protocols, not the software, define the system.
* Third; a relatively small operating system kernel can implement the basic protocols and services providing a simple network-transparent process, address space & communication model.

=== Ideas that significantly affected the design ===
* Shared Memory.
* Dealing with group of entities same as they deal w/individual entities.
* Efficient file caching mechanism using the virtual memory caching mechanism.

=== Design Decisions ===
* Designed for a cluster of workstations with high speed network access ( only really supports LAN ).
* Abstract the physical architecture of the participating workstations, by defining common protocols providing well-defined interfaces.

V was run on LAN, and its developers developed a really fast IPC protocol which allowed for it to be a fasted distributed operating system in a small geographic area. Aside from the IPC protocols, V also implemented RPC calls in the background.

V uses the strong consistency model. This model can cause issues with concurrency because in V files are a memory space. Thus two different users accessing the same file and in fact accessing the same memory location. This could result in issues unless there is an effective implementation to deal with multiple versions, etc.

DistOS 2015W Session 3

2015-02-01T22:09:39Z

Hassansaid:

= '''Reading Response Discussion''' =

== Multics ==

'''Team:''' Sameer, Shivjot, Ambalica, Veena

It came into being in the 1960s and it completely vanished in 2000s. It was started by Bell, General Electric and MIT but Bell backed out of the project in 1969.
Multics is a time sharing OS which provides Multitasking and Multiprogramming. It is not a Distributed OS but it a Centralized system which was written in the assembly language.

It provides following features:
# Utility Computing
# Access Control Lists
# Single level storage
# Dynamic linking
#* Sharded libraries or files can be loaded and linked to Random Access Memory at run time.
# Hot swapping
# Multiprocessing System
# Ring oriented Security
#* It provides number of levels of authorization within the computer System.

== Unix ==

Unix in its original conception is a small, minimal API system designed by two guys from Bell Labs. It was essentially a OS that would be easy to grasp for an programmer but not much beyond that. The UNIX OS ran on one computer, and terminals ran from that one computer. Thus it is not a distributed operating system as it is centralized and implements time sharing.

== Sun NFS ==

The Sun NFS OS implemented networking using RPC connections. These connections are not secure. Sun wanted to encrypt these RPC connections but encryption would result in government regulations that Sun wanted to avoid in order to sell NFS over seas (See below).

Sun wanted to encrypt their NFS system but decided not to. In the US encryption is regulated like munitions. Thus when Sun wanted to sell the NFS
system abroad they would have to contend with government regulations. To avoid these regulations, Sun decided to sell the unencrypted NFS system abroad.

== Locus ==

# Not scalable
#* Every computer stores a copy of every file
#* Also used CAS to manage files
# Not efficient with abstractions
#* Trying to distribute files and processes
#Allowed for process migration

== Sprite ==

''' Team: ''' Jamie, Hassan, Khaled

Sprite had the following Design Features:
# Network Transparency
# Process Migration, file transfer between computers
#* User could initiate a process migration to an idle machine, and if the machine was no longer idle; due to it being used be another user, the system would take care of the process migration to another machine
# Handling Cache Consistency
#* Sequential file sharing ==> By using a version number for each file
#* Concurrent write sharing ==> Disable cache to clients, enable write-blocking and other methods
# Implemented a caching system that sped up performance
# Implemented a log structured file system
#* Log structured file systems are optimized for writes; data is written to cache instead of memory, over reads. This is a way of minimizing the performance issues of doing writes
#* Example: SSD (Solid-state disks)

The ''' main features ''' to take away from the Sprite system is that it implemented a log structured file system, and implemented caching to increase performanace.

DistOS 2015W Session 3

2015-02-01T21:37:37Z

Hassansaid:

= '''Reading Response Discussion''' =

----
'''Multics'''

----

Team: Sameer, Shivjot, Ambalica, Veena

It came into being in the 1960s and it completely vanished in 2000s. It was started by Bell, General Electric and MIT but Bell backed out of the project in 1969.
Multics is a time sharing OS which provides Multitasking and Multiprogramming.

It provides following features:
1. Utility Computing
2. Access Control Lists
3. Single level storage
4. Dynamic linking
*Sharded libraries or files can be loaded and linked to Random Access Memory at run time.
5. Hot swapping
6. Multiprocessing System
7. Ring oriented Security
* It provides number of levels of authorization within the computer System.
It is not a Distributed OS but it a Centralized system which was written in the assembly language.

'''Sprite'''

Panel Group: Jamie, Hassan, Khaled

Sprite had the following Design Features:
1. Network Transparency.
2. Process Migration.
3. Handling Cache Consistency
a. Sequential file sharing===> By using a version number for each file.
b. Concurrent write sharing==> Disable cache to clients.
4. The main design theme is to make aggressive use of RAM for caching files.

DistOS 2015W Session 2

2015-01-13T02:01:03Z

Hassansaid: /* Discussion: Easy on one computer, hard on multiple computers */

= Reading Response Discussion =

== Mother of All Demos: ==

Team: Jason, Kirill, Sravan, Agustin, Hassan Ambalica, Apoorv, Khaled

* 1968, Doug Inglebard
* One computer with multiple terminals controlling
* Video conferencing
* Computer mouse
* Word processing, rudimentary copy and paste
* Dynamic file linking
* Revision control/version control/source control
* Collaborative real-time editor
** User privilege control in that user can provide read-only access, read-write privilege to file
* Cord keyboard
** Marco keyboard that allows messages quickly, while using mouse at same time
* Not really a distributed operating system, but great start because multiple users at different terminals could share same resources

== Early Internet ==

== Alto ==

Team: Tory, Veena, Sameer, Mert, Deep, Nameet, Moe

* 1973
* High speed (3 Mbps @ 1km difference)
* Connects up to 256 machines
* Protocol similar to cross between UDP/TCP, before TCP invented
* Allowed sharing printers
* Also allowed for distributing files across computers (redundancy/reliability)
* Sort of early cloud
* Allowed for remote debugging, store error logs
* Allowed machines to use processing power of others
* much of the time it would be doing nothing

= Professor lead discussion 1 =

A true distributed operating system does not actually exist, its more of a dream.

You'll find in the history of trying to achieve this dream of a distributed operating system there has always been a road block as a result of a technical issue. People would come up with a solution to this technical issue but then they would find that some other technical issue would crop up. For example 'The Mother of All Demos' tried to build a distributed operating system but they had no networking. They had to point a television camera at the computer monitor to be able to demo their concept. The early internet came along and dealt with the issue of no networking. However the early internet itself had other technical issues.

A common buzz word during the early days of development was '''Time Sharing'''. Time sharing is about the sharing of resources between users. Today the concept of time sharing is similar to sharing a processor between multiple processes.

= Discussion: Easy on one computer, hard on multiple computers =

{| class="wikitable" width="100%"
|+ '''Instant Messaging'''
|-
! Why is it easy?
! Why is it hard?
|-
| Can't be out sync
| Synchronization issues
* Server-client modal
** If two or more servers they get requests same time, they have to figure out how to synchronize the data
* Peer-to-Peer
** Each peer has to figure out how to sync each message
|-
| Only one data store/source
|-
|}

{| class="wikitable" width="100%"
|+ '''Mortal Kombat/Twitch Gaming'''
|-
! Why is it easy?
! Why is it hard?
|-
|}

{| class="wikitable" width="100%"
|+ '''Photo Album'''
|-
! Why is it easy?
! Why is it hard?
|-
|}

=== Commonalities ===
* Synchronization
* Bandwidth
* Reliability/Fault Tolerance
* Interoperability
* Discovery
* Routing
** In modern system this issue has been mostly abstracted away
** Classic example would be Wireless Ad-hoc networking

The above list is not hard on a single system because all the cores have equal access to all the resources. Also as a result of excellent engineering modern single systems can very effective deal with any errors that may result.

DistOS 2015W Session 2

2015-01-13T01:54:40Z

Hassansaid: /* Reading Response Discussion */

DistOS 2015W Session 2

2015-01-13T01:30:38Z

Hassansaid: /* Reading Response Discussion */

DistOS 2015W Session 2

2015-01-13T01:17:52Z

Hassansaid: Created page with "== Reading Response Discussion =="

== Reading Response Discussion ==

DistOS 2015W Session 1

2015-01-13T01:15:06Z

Hassansaid: /* Distributed OS is kind of OS */

Notes for the first session that happened on Jan. 5th, 2015.

= Course Outline =

Undergrad Grading Scheme
* 15% Class Participation
* 15% Reading Responses
* 10% lecture Notes/Wiki Contributions
* 25% Midterm
* 35% Final Exam

Grads Grading Scheme
* 15% Class Participation
* 15% Reading Responses
* 10% Lectures Notes/Wiki contributions
* 10% Project Proposal
* 15% Project Presentation
* 35% Final Project

=== Project ===
* A literature review of distributing operations
* Research proposal on a problem related to distributed systems

= Discussion =

== Q: What do you think of when you hear 'Distributed System'? ==

=== Group 1 ===
* Sharing Resources
* Spreading Work Loads
* Scheduling
* Process migration
* Different nodes have different purpose
* Google
* Parallel running processes
* Nodes
* Resource Allocation across multiple nodes
* Scheduling multiple nodes
* Resources availability among nodes
* Problem: across multiple machine

=== Group 2 ===
* Request comes from (usually) one computer and processing is usually handled by more than one computer
* Tasks are divided into small parts which can be processed individually before coming back together
* Usually deals with large scale data sets
* Globalization
* Fault tolerance
* Usually emphasize distributed agents to not work on certain schedule
* Examples: Map Reduce, cloud, cloud software (Google Drive)

=== Group 3 ===
* Separated, networked machines
* Coordinated by similar or identical software
* Error recovery/redundancy
* No controlled storage
* Coordinated communication facilitating operation on coordinated task
* Leader/hierarchy for task delegation

=== Group 4 ===
* Distributed, multiple systems
* OS: The root level system that operates a computer system
* Dist. OS more complexity

=== Prof Discussion Notes ===
Key words: network, parallel, fault tolerant, redundancy, complexity

== Distributed OS is kind of OS ==

'''But what is an OS in the first place?'''
* Connection software and hardware
* Resource allocation
* Abstraction layers
* Makes it easy to run higher level programs on different computers
* Sharing: resources are split up and each process is isolated, this improves security and programming (no need to worry about sharing)
* Virtual memory, process scheduling

'''What is a distributed system?'''
* Feels like it is only a single node, but runs on many
* Do the things users take for granted on regular computers
* Doesn't actually work as a single (general use) computer - with 1000 computers one would expect it to be like a single computer with 1000 times the resources, but it's not
* In a way, opposite of an operating system: instead of splitting up a single computer into many, it takes many and tries to merge into one
* Resources can be divided as you want
* Centralized control solves some issues, but has issues of its own
* Since it cannot truly act as a single computer, we fake it as much as possible - and it works in some scenarios
* The more specialized the task, the better it will scale
* Interact it, process update, callback,
* Gmail so responsible - it perfected , downloads in your browser
* Cache - they predict what the processor going to do

[[File:Dos.png|frame|center|Distributed Operating System represented as a Venn diagram. OS denotes an operating system, and DS denotes a distributed system. The black center represents a distributed operating system; a system with properties of an operating system and a distributed system.]]

== Operating System Examples ==
'''Mobile devices - Phones'''
* iOS
* Android

'''Embedded OS'''
* Linux
* QNX
* xBSD firewalls

'''Desktop'''
* Windows
* OSX
* Chrome OS

'''Server'''
* Windows
* Linux
* BSD

'''Main Frames'''
* OS/400

Is the cloud an OS? Important question.

'''Cloud'''
* MPI
* AWS
* Google App Engine

No, they are at best proto-OSs because the abstractions they provide are very leaky. They only provide limited APIs.

== Pick a system and show how it is and is not an OS ==

=== Group 1 ===
{| class="wikitable" width="100%"
|+ '''BOINC'''
|-
! Similarities to a traditional OS
! Differences from a traditional OS
|-
| Very parallel
| Can handle only very specific (trivially parallelizable) types of problems
|-
| Availability
| Abstraction layer is poor
|-
| Scheduling
|-
| Networked
|-
| Nodes may come and leave as they like (it is active when the computer is idle)
|-
| Same problem at the same time
|-
| Redundant/Fault tolerant
|-
| Allocation can be handle by system
|-
| Large problem managed by one system
|-
| Anyone can submit projects
|}

=== Group 2 ===
{| class="wikitable" width="100%"
|+ '''Facebook'''
|-
! Similarities to a traditional OS
! Differences from a traditional OS
|-
| OK abstraction (for human communication/control)
| Doesn't really take a single resources and split it up into smaller ones
|-
| The programming API is stable
| The human API (user interface) is not stable
|-
| Able to separate resources such as wall posts, photos, etc
| Limited control
|}

=== Group 3 ===
{| class="wikitable" width="100%"
|+ '''Google Docs (Drive)'''
|-
! Similarities to a traditional OS
! Differences from a traditional OS
|-
| Has a file system
| Somehow still not a true OS
|-
| Provides hardware abstraction (users don't care how the requests are carried out)
|-
| Has an API
|}

=== Group 4 ===
{| class="wikitable" width="100%"
|+ '''LinkedIn'''
|-
! Similarities to a traditional OS
! Differences from a traditional OS
|-
| Supports multiple location/users
| Specific functionality (not abstract)
|-
| Multiple servers around the world
|-
| Resource/Security management
|-
| Networked
|-
| Cross-platform (Desktop/Mobile)
|}

File:Dos.png

2015-01-13T01:10:26Z

Hassansaid: Distributed Operating System represented as a Venn diagram. OS denotes an operating system, and DS denotes a distributed system. The black center represents a distributed operating system; a system with properties of an operating system and a distribut...

Distributed Operating System represented as a Venn diagram. OS denotes an operating system, and DS denotes a distributed system. The black center represents a distributed operating system; a system with properties of an operating system and a distributed system.

DistOS 2015W Session 1

2015-01-13T00:45:42Z

Hassansaid: /* Group 2 */

Notes for the first session that happened on Jan. 5th, 2015.

= Course Outline =

Undergrad Grading Scheme
* 15% Class Participation
* 15% Reading Responses
* 10% lecture Notes/Wiki Contributions
* 25% Midterm
* 35% Final Exam

Grads Grading Scheme
* 15% Class Participation
* 15% Reading Responses
* 10% Lectures Notes/Wiki contributions
* 10% Project Proposal
* 15% Project Presentation
* 35% Final Project

=== Project ===
* A literature review of distributing operations
* Research proposal on a problem related to distributed systems

= Discussion =

== Q: What do you think of when you hear 'Distributed System'? ==

=== Group 1 ===
* Sharing Resources
* Spreading Work Loads
* Scheduling
* Process migration
* Different nodes have different purpose
* Google
* Parallel running processes
* Nodes
* Resource Allocation across multiple nodes
* Scheduling multiple nodes
* Resources availability among nodes
* Problem: across multiple machine

=== Group 2 ===
* Request comes from (usually) one computer and processing is usually handled by more than one computer
* Tasks are divided into small parts which can be processed individually before coming back together
* Usually deals with large scale data sets
* Globalization
* Fault tolerance
* Usually emphasize distributed agents to not work on certain schedule
* Examples: Map Reduce, cloud, cloud software (Google Drive)

=== Group 3 ===
* Separated, networked machines
* Coordinated by similar or identical software
* Error recovery/redundancy
* No controlled storage
* Coordinated communication facilitating operation on coordinated task
* Leader/hierarchy for task delegation

=== Group 4 ===
* Distributed, multiple systems
* OS: The root level system that operates a computer system
* Dist. OS more complexity

=== Prof Discussion Notes ===
Key words: network, parallel, fault tolerant, redundancy, complexity

== Distributed OS is kind of OS ==

'''But what is an OS in the first place?'''
* Connection software and hardware
* Resource allocation
* Abstraction layers
* Makes it easy to run higher level programs on different computers
* Sharing: resources are split up and each process is isolated, this improves security and programming (no need to worry about sharing)
* Virtual memory, process scheduling

'''What is a distributed system?'''
* Feels like it is only a single node, but runs on many
* Do the things users take for granted on regular computers
* Doesn't actually work as a single (general use) computer - with 1000 computers one would expect it to be like a single computer with 1000 times the resources, but it's not
* In a way, opposite of an operating system: instead of splitting up a single computer into many, it takes many and tries to merge into one
* Resources can be divided as you want
* Centralized control solves some issues, but has issues of its own
* Since it cannot truly act as a single computer, we fake it as much as possible - and it works in some scenarios
* The more specialized the task, the better it will scale
* Interact it, process update, callback,
* Gmail so responsible - it perfected , downloads in your browser
* Cache - they predict what the processor going to do

== Operating System Examples ==
'''Mobile devices - Phones'''
* iOS
* Android

'''Embedded OS'''
* Linux
* QNX
* xBSD firewalls

'''Desktop'''
* Windows
* OSX
* Chrome OS

'''Server'''
* Windows
* Linux
* BSD

'''Main Frames'''
* OS/400

Is the cloud an OS? Important question.

'''Cloud'''
* MPI
* AWS
* Google App Engine

No, they are at best proto-OSs because the abstractions they provide are very leaky. They only provide limited APIs.

== Pick a system and show how it is and is not an OS ==

=== Group 1 ===
{| class="wikitable" width="100%"
|+ '''BOINC'''
|-
! Similarities to a traditional OS
! Differences from a traditional OS
|-
| Very parallel
| Can handle only very specific (trivially parallelizable) types of problems
|-
| Availability
| Abstraction layer is poor
|-
| Scheduling
|-
| Networked
|-
| Nodes may come and leave as they like (it is active when the computer is idle)
|-
| Same problem at the same time
|-
| Redundant/Fault tolerant
|-
| Allocation can be handle by system
|-
| Large problem managed by one system
|-
| Anyone can submit projects
|}

=== Group 2 ===
{| class="wikitable" width="100%"
|+ '''Facebook'''
|-
! Similarities to a traditional OS
! Differences from a traditional OS
|-
| OK abstraction (for human communication/control)
| Doesn't really take a single resources and split it up into smaller ones
|-
| The programming API is stable
| The human API (user interface) is not stable
|-
| Able to separate resources such as wall posts, photos, etc
| Limited control
|}

=== Group 3 ===
{| class="wikitable" width="100%"
|+ '''Google Docs (Drive)'''
|-
! Similarities to a traditional OS
! Differences from a traditional OS
|-
| Has a file system
| Somehow still not a true OS
|-
| Provides hardware abstraction (users don't care how the requests are carried out)
|-
| Has an API
|}

=== Group 4 ===
{| class="wikitable" width="100%"
|+ '''LinkedIn'''
|-
! Similarities to a traditional OS
! Differences from a traditional OS
|-
| Supports multiple location/users
| Specific functionality (not abstract)
|-
| Multiple servers around the world
|-
| Resource/Security management
|-
| Networked
|-
| Cross-platform (Desktop/Mobile)
|}

DistOS 2015W Session 1

2015-01-13T00:45:28Z

Hassansaid: /* Group 3 */

Notes for the first session that happened on Jan. 5th, 2015.

= Course Outline =

Undergrad Grading Scheme
* 15% Class Participation
* 15% Reading Responses
* 10% lecture Notes/Wiki Contributions
* 25% Midterm
* 35% Final Exam

Grads Grading Scheme
* 15% Class Participation
* 15% Reading Responses
* 10% Lectures Notes/Wiki contributions
* 10% Project Proposal
* 15% Project Presentation
* 35% Final Project

=== Project ===
* A literature review of distributing operations
* Research proposal on a problem related to distributed systems

= Discussion =

== Q: What do you think of when you hear 'Distributed System'? ==

=== Group 1 ===
* Sharing Resources
* Spreading Work Loads
* Scheduling
* Process migration
* Different nodes have different purpose
* Google
* Parallel running processes
* Nodes
* Resource Allocation across multiple nodes
* Scheduling multiple nodes
* Resources availability among nodes
* Problem: across multiple machine

=== Group 2 ===
* Request comes from (usually) one computer and processing is usually handled by more than one computer
* Tasks are divided into small parts which can be processed individually before coming back together
* Usually deals with large scale data sets
* Globalization
* Fault tolerance
* Usually emphasize distributed agents to not work on certain schedule

=== Group 3 ===
* Separated, networked machines
* Coordinated by similar or identical software
* Error recovery/redundancy
* No controlled storage
* Coordinated communication facilitating operation on coordinated task
* Leader/hierarchy for task delegation

=== Group 4 ===
* Distributed, multiple systems
* OS: The root level system that operates a computer system
* Dist. OS more complexity

=== Prof Discussion Notes ===
Key words: network, parallel, fault tolerant, redundancy, complexity

== Distributed OS is kind of OS ==

'''But what is an OS in the first place?'''
* Connection software and hardware
* Resource allocation
* Abstraction layers
* Makes it easy to run higher level programs on different computers
* Sharing: resources are split up and each process is isolated, this improves security and programming (no need to worry about sharing)
* Virtual memory, process scheduling

'''What is a distributed system?'''
* Feels like it is only a single node, but runs on many
* Do the things users take for granted on regular computers
* Doesn't actually work as a single (general use) computer - with 1000 computers one would expect it to be like a single computer with 1000 times the resources, but it's not
* In a way, opposite of an operating system: instead of splitting up a single computer into many, it takes many and tries to merge into one
* Resources can be divided as you want
* Centralized control solves some issues, but has issues of its own
* Since it cannot truly act as a single computer, we fake it as much as possible - and it works in some scenarios
* The more specialized the task, the better it will scale
* Interact it, process update, callback,
* Gmail so responsible - it perfected , downloads in your browser
* Cache - they predict what the processor going to do

== Operating System Examples ==
'''Mobile devices - Phones'''
* iOS
* Android

'''Embedded OS'''
* Linux
* QNX
* xBSD firewalls

'''Desktop'''
* Windows
* OSX
* Chrome OS

'''Server'''
* Windows
* Linux
* BSD

'''Main Frames'''
* OS/400

Is the cloud an OS? Important question.

'''Cloud'''
* MPI
* AWS
* Google App Engine

No, they are at best proto-OSs because the abstractions they provide are very leaky. They only provide limited APIs.

== Pick a system and show how it is and is not an OS ==

=== Group 1 ===
{| class="wikitable" width="100%"
|+ '''BOINC'''
|-
! Similarities to a traditional OS
! Differences from a traditional OS
|-
| Very parallel
| Can handle only very specific (trivially parallelizable) types of problems
|-
| Availability
| Abstraction layer is poor
|-
| Scheduling
|-
| Networked
|-
| Nodes may come and leave as they like (it is active when the computer is idle)
|-
| Same problem at the same time
|-
| Redundant/Fault tolerant
|-
| Allocation can be handle by system
|-
| Large problem managed by one system
|-
| Anyone can submit projects
|}

=== Group 2 ===
{| class="wikitable" width="100%"
|+ '''Facebook'''
|-
! Similarities to a traditional OS
! Differences from a traditional OS
|-
| OK abstraction (for human communication/control)
| Doesn't really take a single resources and split it up into smaller ones
|-
| The programming API is stable
| The human API (user interface) is not stable
|-
| Able to separate resources such as wall posts, photos, etc
| Limited control
|}

=== Group 3 ===
{| class="wikitable" width="100%"
|+ '''Google Docs (Drive)'''
|-
! Similarities to a traditional OS
! Differences from a traditional OS
|-
| Has a file system
| Somehow still not a true OS
|-
| Provides hardware abstraction (users don't care how the requests are carried out)
|-
| Has an API
|}

=== Group 4 ===
{| class="wikitable" width="100%"
|+ '''LinkedIn'''
|-
! Similarities to a traditional OS
! Differences from a traditional OS
|-
| Supports multiple location/users
| Specific functionality (not abstract)
|-
| Multiple servers around the world
|-
| Resource/Security management
|-
| Networked
|-
| Cross-platform (Desktop/Mobile)
|}