Soma-notes - User contributions [en]

Class Review, Future Directions

2008-04-02T19:34:58Z

Emmellst: /* Day Two */

==Exam Prep==
===Topics Covered===
- DSM

- Distributed File Systems
*GFS Major feature? FAULT TOLERANCE!

- RPC

- Process Migration
*Planetlab?
**Central admin control - this is very restrictive model
***Likely why only researchers are using this kind of system

- Fault tolerance
*Some dealt with this extensively, others not much at all

- Security
*Again, some systems used security as a tenant of design, while others simply said it was to be developed in the future and did not do anything for it*

===Possible Study Questions===
1. Example paper, then quiz on the paper?

2. What were the systems that implemented DSM, and what were the problems they faced?
*Cons
**Must remember that all the facets of all the systems
**Mostly regurgitation - not really synthesizing new opinion or information

3. What kind of problems could you use DSM to help solve? What environments is DSM most suitable for?

4. Scenario based question - Suggest that you are a designer for a software project* You are charged with designing a system that implements some kind of DSM solution* Which system would you implement and what would the possible advantages/disadvantages be?
*A circumstance where you would need enormous amounts of memory
*Mostly reading - caching would be the win
*DNS as a big DSM program - one gigantic table
**Good idea? Need something like signature based submission to control and secure the system
**Access control and security would be big problem

5. Scenario based question - Suggest a problem and request various solutions, requesting pros/cons of each - (RPC based, DSM based, etc)

6. Which was a more successful distributed operating system?
*How do you define successful?
**Was this solely for research? Or real implementation?
**In terms of championing ideas? Or deployed implementations?

7. Evaluate past work - 'make you turn your head sideways' - evaluate from different perspective

8. Which system best captured "UNIX" in a distributed operating system
*Best captured the 'flavour' of unix, and which one least captured it?
*Out of the following... which one is most 'unix-like' (plan 9, locus, mach, etc)... which is least?
*Out of the following... which distributed file-system is most 'unix-like'(GFS, locus, etc)...

9. Build something to solve X - or - Build X using Y

10. Opinion question
*What was your favourite system that we covered?
**What were the key characteristics of this system that you liked?
**What criteria are you using to evaluate it?
**Why is your liking it not justified? IE, how did this system fail in some way?
**Talk about at least two other systems that don't meet this same criteria
***Criteria being the reasons that you prefer THIS operating system
*What was your least favourite?
*Take what you have chosen as your favourite, and then explain why it is the worst! (Will not do this, but great for debating)

11. What were the key problems addressed by most of these systems?
*Which of these problems are most important to solve in todays computing environment
**What is todays computing environment? Should we only be optimizing for clusters given that we are not building for systems that cross administrative boundaries? What technology would make these clusters better?

===Answers===
What do we know about building these systems? What can we do well?
*Message passing
*RPC
*Local files
*Distributed files? depending on scenario, depending on what your file IS
**A normal POSIX file in a distributed environment? No, not really
***Which semantics do you let slip?
**Append-only files? Sure
*Single domain authentication
*Distributed read-only anything (files, memory)
*Concurrent writing? No, that's the hard part
**When you try to update the same piece of data from multiple locations, possibly at the same time
**We know that the less communication, then better
***Reduces latency problem and minimizes multiple writes
*Backwards compatibility
**Completely duplicating the specification of non-distributed systems is HARD (synchronicity = SLOW)
**Slip the standards enough, minimize changes required and most problems can be alleviated enough to make the system usable
**Metadata often a big problem
***Needs higher visibility
***Typically has higher contention than other data
**Some have abandoned backwards compatibility
***Some systems have done this
*General purpose solutions are generally bad, in distinct contrast to the local case
**Specific solutions to solve specific problems
*Security - not easily implemented in distributed OSes
**Crypto "ain't enough"
**Typically added on after the fact
***Without security as design tenant, often design choices are made along the way that make securing the system very difficult
**Very hard to test accurately, how do you plan to secure "any hole"
**Often developed before security was a real concern
**Often adding security makes the system very slow
**''We have yet to develop a good model for multiple administrative domains of control''

===Study tips===
- Go through each paper and ask ''"Why do I care?"''

==Day Two==
===More questions===
- Why have distOSs failed?

- What '''should''' a distOS do?
* Networking (TCP/IP)
* Administrative domains
** Who administers the whole package? (compared to the internet where there IS no single administration system)
* Share resources
** If you participate in the internet, you are sharing resources (or at least, USING resources)
** When you run javascript on a page, you are using your machine to run someone else's code
** This philosophy is flawed!
*** Should someone not want to play the game nicely, they can consume more than their share
** All these locks, permissions, etc are there to help make sure users only consume their share

- What is the state of the internet today?
* Anarchy
** There IS NO HIGHER STRUCTURE (no global cohersion, no 'police force' or 'rule of law')
* Are distOSs trying to implement 'communism' on the internet?
* A rule of law is different than having good laws
** A good set of rules implies that users live by a certain model
*** A good society is a society that isn't based on following the letter of the law, but of helping others

* Traditional OSs try to implement a set of rules that DO NOT ALLOW others to do 'bad things'
* We all have the capacity to cause harm to others, but our '''culture''' helps create a safe environment, that results in social enforcement. More often than not it isn't the law that discourages bad behaviour, but the judgments of others.
* Related to DistOSs?
** Each machine has the power to cause problems
** Supposition: It is impossible to make fixed rules that limit a single machine's power without fundamentally ruining the internet.

* Who defines appropriate behaviour?
** The computers themselves, in some kind of distributed framework.
** Perhaps a series of connected frameworks
** Though this solution would need to be adaptive, and something that allows the computers to decide what is appropriate

* How to enforce that behaviour?
** Attribution - make cause/effect connections - who did what behaviour
*** This is not trivial, and in general not totally possible
** Punishment
*** Some kind of prison, some kind of process that results in privilege reduction, or resource reduction
*** How would that work? You need a LOT of evidence before punishment can be enforced
**** Therefore there would need to be a lot of damage before punishment can be enforced
**** Shame, austricism
* A gossip mechanism
** Spread the knowledge - spread 'interesting information' to other computers
** gossip is always suspect, but usually contains at least mildly correct information
** Low level mechanism - what to do with the gossip? How to evaluate it?

If we don't do something, some way to 'teach the mob some manners' - there will be a HUGE problem with a massive number of resources outside of the core infrastructure, to the point that these resources would be easily able to dominate these core systems.

''Perhaps a model similar to an oligarchy''
* Machines locked down
* Specific rules that require certain behaviour
* A set of small organizations that decides these rules, and in theory punishes deviation from that behaviour
* Would this work?
* This is largely what we have now, and are moving more towards

Rather than get humans to behave properly, get the computers to behave properly.

Though in systems that are managed top-down there tends to be vary many deficiencies.

Computers could develop 'opinions' of other computers, so then computers in similar environments might share their opinions, or tend to align themselves to certain other groups. And when things happen different behaviours can result across that whole group, which can then influence another group, etc
* Though outside of your community your status might not be well defined, you might not get the same level of privileges.
* Even to the point of stereotyping -> If I don't know you: Where are you coming from? What OS are you running? Use my default level of trust for BLA
* The idea of rehabilitation? Computers don't have moral behaviour, or psychoses, given that a computer could always be re-imaged, or used by a different user... so it would need to work differently.

Class Review, Future Directions

2008-04-02T19:27:24Z

Emmellst:

==Exam Prep==
===Topics Covered===
- DSM

- Distributed File Systems
*GFS Major feature? FAULT TOLERANCE!

- RPC

- Process Migration
*Planetlab?
**Central admin control - this is very restrictive model
***Likely why only researchers are using this kind of system

- Fault tolerance
*Some dealt with this extensively, others not much at all

- Security
*Again, some systems used security as a tenant of design, while others simply said it was to be developed in the future and did not do anything for it*

===Possible Study Questions===
1. Example paper, then quiz on the paper?

2. What were the systems that implemented DSM, and what were the problems they faced?
*Cons
**Must remember that all the facets of all the systems
**Mostly regurgitation - not really synthesizing new opinion or information

3. What kind of problems could you use DSM to help solve? What environments is DSM most suitable for?

4. Scenario based question - Suggest that you are a designer for a software project* You are charged with designing a system that implements some kind of DSM solution* Which system would you implement and what would the possible advantages/disadvantages be?
*A circumstance where you would need enormous amounts of memory
*Mostly reading - caching would be the win
*DNS as a big DSM program - one gigantic table
**Good idea? Need something like signature based submission to control and secure the system
**Access control and security would be big problem

5. Scenario based question - Suggest a problem and request various solutions, requesting pros/cons of each - (RPC based, DSM based, etc)

6. Which was a more successful distributed operating system?
*How do you define successful?
**Was this solely for research? Or real implementation?
**In terms of championing ideas? Or deployed implementations?

7. Evaluate past work - 'make you turn your head sideways' - evaluate from different perspective

8. Which system best captured "UNIX" in a distributed operating system
*Best captured the 'flavour' of unix, and which one least captured it?
*Out of the following... which one is most 'unix-like' (plan 9, locus, mach, etc)... which is least?
*Out of the following... which distributed file-system is most 'unix-like'(GFS, locus, etc)...

9. Build something to solve X - or - Build X using Y

10. Opinion question
*What was your favourite system that we covered?
**What were the key characteristics of this system that you liked?
**What criteria are you using to evaluate it?
**Why is your liking it not justified? IE, how did this system fail in some way?
**Talk about at least two other systems that don't meet this same criteria
***Criteria being the reasons that you prefer THIS operating system
*What was your least favourite?
*Take what you have chosen as your favourite, and then explain why it is the worst! (Will not do this, but great for debating)

11. What were the key problems addressed by most of these systems?
*Which of these problems are most important to solve in todays computing environment
**What is todays computing environment? Should we only be optimizing for clusters given that we are not building for systems that cross administrative boundaries? What technology would make these clusters better?

===Answers===
What do we know about building these systems? What can we do well?
*Message passing
*RPC
*Local files
*Distributed files? depending on scenario, depending on what your file IS
**A normal POSIX file in a distributed environment? No, not really
***Which semantics do you let slip?
**Append-only files? Sure
*Single domain authentication
*Distributed read-only anything (files, memory)
*Concurrent writing? No, that's the hard part
**When you try to update the same piece of data from multiple locations, possibly at the same time
**We know that the less communication, then better
***Reduces latency problem and minimizes multiple writes
*Backwards compatibility
**Completely duplicating the specification of non-distributed systems is HARD (synchronicity = SLOW)
**Slip the standards enough, minimize changes required and most problems can be alleviated enough to make the system usable
**Metadata often a big problem
***Needs higher visibility
***Typically has higher contention than other data
**Some have abandoned backwards compatibility
***Some systems have done this
*General purpose solutions are generally bad, in distinct contrast to the local case
**Specific solutions to solve specific problems
*Security - not easily implemented in distributed OSes
**Crypto "ain't enough"
**Typically added on after the fact
***Without security as design tenant, often design choices are made along the way that make securing the system very difficult
**Very hard to test accurately, how do you plan to secure "any hole"
**Often developed before security was a real concern
**Often adding security makes the system very slow
**''We have yet to develop a good model for multiple administrative domains of control''

===Study tips===
- Go through each paper and ask ''"Why do I care?"''

==Day Two==
===More questions===
- Why have distOSs failed?

- What '''should''' a distOS do?
* Networking (TCP/IP)
* Administrative domains
** Who administers the whole package? (compared to the internet where there IS no single administration system)
* Share resources
** If you participate in the internet, you are sharing resources (or at least, USING resources)
** When you run javascript on a page, you are using your machine to run someone else's code
** This philosophy is flawed!
*** Should someone not want to play the game nicely, they can consume more than their share
** All these locks, permissions, etc are there to help make sure users only consume their share

- What is the state of the internet today?
* Anarchy
** There IS NO HIGHER STRUCTURE (no global cohersion, no 'police force' or 'rule of law')
* Are distOSs trying to implement 'communism' on the internet?
* A rule of law is different than having good laws
** A good set of rules implies that users live by a certain model
*** A good society is a society that isn't based on following the letter of the law, but of helping others

* Traditional OSs try to implement a set of rules that DO NOT ALLOW others to do 'bad things'
* We all have the capacity to cause harm to others, but our '''culture''' helps create a safe environment, that results in social enforcement. More often than not it isn't the law that discourages bad behaviour, but the judgments of others.
* Related to DistOSs?
** Each machine has the power to cause problems
** Supposition: It is impossible to make fixed rules that limit a single machine's power without fundamentally ruining the internet.

* Who defines appropriate behaviour?
** The computers themselves, in some kind of distributed framework.
** Perhaps a series of connected frameworks
** Though this solution would need to be adaptive, and something that allows the computers to decide what is appropriate

* How to enforce that behaviour?
** Attribution - make cause/effect connections - who did what behaviour
*** This is not trivial, and in general not totally possible
** Punishment
*** Some kind of prison, some kind of process that results in privilege reduction, or resource reduction
*** How would that work? You need a LOT of evidence before punishment can be enforced
**** Therefore there would need to be a lot of damage before punishment can be enforced
**** Shame, austricism
* A gossip mechanism
** Spread the knowledge - spread 'interesting information' to other computers
** gossip is always suspect, but usually contains at least mildly correct information
** Low level mechanism - what to do with the gossip? How to evaluate it?

If we don't do something, some way to 'teach the mob some manners' - there will be a HUGE problem with a massive number of resources outside of the core infrastructure, to the point that these resources would be easily able to dominate these core systems.

====Perhaps a model similar to an oligarchy====
* Machines locked down
* Specific rules that require certain behaviour
* A set of small organizations that decides these rules, and in theory punishes deviation from that behaviour
* Would this work?
* This is largely what we have now, and are moving more towards

Rather than get humans to behave properly, get the computers to behave properly.

Though in systems that are managed top-down there tends to be vary many deficiencies.

Computers could develop 'opinions' of other computers, so then computers in similar environments might share their opinions, or tend to align themselves to certain other groups. And when things happen different behaviours can result across that whole group, which can then influence another group, etc
* Though outside of your community your status might not be well defined, you might not get the same level of privileges.

Class Review, Future Directions

2008-03-31T19:54:27Z

Emmellst:

Class Review, Future Directions

2008-03-31T19:51:46Z

Emmellst:

==Exam Prep==
===Topics Covered===
- DSM

- Distributed File Systems
.GFS Major feature? FAULT TOLERANCE!

- RPC

- Process Migration
.Planetlab?
..Central admin control - this is very restrictive model
...Likely why only researchers are using this kind of system

- Fault tolerance
.Some dealt with this extensively, others not much at all

- Security
.Again, some systems used security as a tenant of design, while others simply said it was to be developed in the future and did not do anything for it.

===Possible Study Questions===
1. Example paper, then quiz on the paper?

2. What were the systems that implemented DSM, and what were the problems they faced?
.Cons
..Must remember that all the facets of all the systems
..Mostly regurgitation - not really synthesizing new opinion or information

3. What kind of problems could you use DSM to help solve? What environments is DSM most suitable for?

4. Scenario based question - Suggest that you are a designer for a software project. You are charged with designing a system that implements some kind of DSM solution. Which system would you implement and what would the possible advantages/disadvantages be?
.A circumstance where you would need enormous amounts of memory
.Mostly reading - caching would be the win
.DNS as a big DSM program - one gigantic table
..Good idea? Need something like signature based submission to control and secure the system
..Access control and security would be big problem

5. Scenario based question - Suggest a problem and request various solutions, requesting pros/cons of each - (RPC based, DSM based, etc)

6. Which was a more successful distributed operating system?
.How do you define successful?
..Was this solely for research? Or real implementation?
..In terms of championing ideas? Or deployed implementations?

7. Evaluate past work - 'make you turn your head sideways' - evaluate from different perspective

8. Which system best captured "UNIX" in a distributed operating system
.Best captured the 'flavour' of unix, and which one least captured it?
.Out of the following... which one is most 'unix-like' (plan 9, locus, mach, etc)... which is least?
.Out of the following... which distributed file-system is most 'unix-like'(GFS, locus, etc)...

9. Build something to solve X - or - Build X using Y

10. Opinion question
.What was your favourite system that we covered?
..What were the key characteristics of this system that you liked?
..What criteria are you using to evaluate it?
..Why is your liking it not justified? IE, how did this system fail in some way?
..Talk about at least two other systems that don't meet this same criteria
...Criteria being the reasons that you prefer THIS operating system
.What was your least favourite?
.Take what you have chosen as your favourite, and then explain why it is the worst! (Will not do this, but great for debating)

11. What were the key problems addressed by most of these systems?
.Which of these problems are most important to solve in todays computing environment
..What is todays computing environment? Should we only be optimizing for clusters given that we are not building for systems that cross administrative boundaries? What technology would make these clusters better?

===Answers===
What do we know about building these systems? What can we do well?
.Message passing
.RPC
.Local files
.Distributed files? depending on scenario, depending on what your file IS
..A normal POSIX file in a distributed environment? No, not really
...Which semantics do you let slip?
..Append-only files? Sure
.Single domain authentication
.Distributed read-only anything (files, memory)
.Concurrent writing? No, that's the hard part
..When you try to update the same piece of data from multiple locations, possibly at the same time
..We know that the less communication, then better
...Reduces latency problem and minimizes multiple writes
.Backwards compatibility
..Completely duplicating the specification of non-distributed systems is HARD (synchronicity = SLOW)
..Slip the standards enough, minimize changes required and most problems can be alleviated enough to make the system usable
..Metadata often a big problem
...Needs higher visibility
...Typically has higher contention than other data
..Some have abandoned backwards compatibility
...Some systems have done this
.General purpose solutions are generally bad, in distinct contrast to the local case
..Specific solutions to solve specific problems
.Security - not easily implemented in distributed OSes
..Crypto "ain't enough"
..Typically added on after the fact
...Without security as design tenant, often design choices are made along the way that make securing the system very difficult
..Very hard to test accurately, how do you plan to secure "any hole"
..Often developed before security was a real concern
..Often adding security makes the system very slow
..''We have yet to develop a good model for multiple administrative domains of control''

===Study tips===
- Go through each paper and ask ''"Why do I care?"''

Distributed OS: Winter 2008

2008-03-31T18:41:00Z

Emmellst:

Welcome to the Distributed Operating Systems (COMP 4000/5102) wiki for Winter 2008!

==Course Outline==

The course outline for COMP 4000/5102 is available [http://www.scs.carleton.ca/~courses/course_outline.php?number=COMP%205102&term=Winter&year=2008 here]. A backup copy is available [http://homeostasis.scs.carleton.ca/~soma/distos/outline.html here].

==Reading Responses==

In your reading response, you can reflect on whatever came to mind
when reading these papers. The one key requirement is that you
demonstrate that you read the papers; however, you have to do so
without merely summarizing them! While you may discuss the in-class
questions in your response, I'm more interested in your own personal
perspective on the readings.

==Class Project==

==Course Notes==

Notes for the lectures & discussions are as follows:

{| border=1 cellpadding=5
|-

! Date

! Topic

|-

| January 7 & 9, 2008

| 0. [[Distributed OS Overview]]

|-

| January 14 & 16, 2008

| 1. [[Early Internet & RPC]]

|-

| January 21 & 23, 2008

| 2. [[Locus, V, Mach]]

|-

| January 28 & 30, 2008

| 3. [[Sprite, Amoeba, Clouds]]

|-

| February 4 & 6, 2008

| 4. [[DSM: IVY]]

|-

| February 11 & 13, 2008

| 5. [[DSM Review, NFS, AFS]]

|-

| February 18 & 20, 2008

| Winter Break

|-

| February 25 & 27, 2008

| 6. [[OceanStore & GPFS]]

|-

| March 3 & 5, 2008

| 7. [[Bell Labs]], project topic discussion Wed.

|-

| March 10 & 12, 2008

| 8. [[NASD, GoogleFS, Farsite]]

|-

| March 17 & 19, 2008

| 9. [[WebOS, PlanetLab, Starfish]]

|-

| March 24 & 26, 2008

| 10. [[MapReduce, Globus, BOINC]]

|-

| March 31 & April 2, 2008

| [[Class Review, Future Directions]]

|-

| April 16, 2008

| Final Exam (2-4:30 PM) & Final Projects Due

|}

MapReduce, Globus, BOINC

2008-03-26T19:32:44Z

Emmellst:

==Readings==

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/foster-grid.pdf Ian Foster and Carl Kesselman, "Computational Grids" (1998)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/foster-globus-intro.pdf Ian Foster, "Globus Toolkit Version 4: Software for Service-Oriented Systems" (2006)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/anderson-boinc.pdf David P. Anderson, "BOINC: A System for Public-Resource Computing and Storage" (2004)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/mapreduce-osdi04.pdf Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simpliﬁed Data Processing on Large Clusters" (2004)]

==Notes==
===BOINC===
*Premise? Local client on your machine downloads a 'workunit', churns the data, dumps the results and downloads a new 'workunit'
*Why are we caring?
**Entertainment?
**How is this an OS paradigm? What is it useful for?
***It isn't really an OS, just a method to have your mass computation done
***More of a distributed scheduler?
****Not even, central scheduler, but mass computation
***How many systems have we seen that have accomplished mass computation on millions of uncontrolled computers?
****ummm... none?
***As an OS?
****An OS is something that is created to run programs
****This is a special case allowing us to run specific programs (BUT IS IT AN OS?)
***Useful for "embarassingly parallel programs"
*Perfect for large scale simulation?
**But then you need LOTS of communication, and this system does not have interconnects
*The type of problems that we most care about tend not to be THAT parallel

*So what would a distributed OS be for?
**Shared communication!
***But we don't have much in the way that works well.
*An OS typically provides a lot of services, together in one package
**We have been seeing that there are no complete packages, just pieces and parts. Why?
***Computers are changing too fast? Same *NIX OS, same TCP/IP stack... so more of the same, why no true solution?
***Communication is unreliable? Yes, but that is also nothing new

*If people found that distributed file systems were successful, they would be in use all the time, but they aren't. Reason? PERFORMANCE

*Take away message?
*Can't handle communication - how do you abstract access to resources when driven through a network?
**As a result, we have many many specialized solutions for particular workloads.
*If you are willing to not have communication between nodes, you gain a HUGE amount of computation.

*The most reliable systems are the one that forget communication.
**The more you system tolerates bad stuff with a network, the better is scales.

*We dont have general cluster distributed OS.

===MapReduce===
*The communication happens when you reduce the problem.
**MapReduce works because there is mapping and there is reducing.
***There is no side effects (enabling things).
*Why is it a good fit to a thousands of machines?
**They first had all these pieces, and if one of them does not replay, then they just do it over :)
***You create the algorithm to fit this model, create this pieces, you have a combining function.
****You have to have some back end that keeps track of who got work done. But you don't care if any machine fail in the middle of the computation.
*Compare MapReduce to POSIX
**The difference is in efficiency. MapReduce is an extension to POSIX.
***Distributed OSs trying to run the programs that run on different APIs. The systems that work, they are relaxed.
****Here is the model, loose compatibility by gaining scalability.

MapReduce, Globus, BOINC

2008-03-26T19:14:22Z

Emmellst:

MapReduce, Globus, BOINC

2008-03-26T19:01:43Z

Emmellst: notes in class

==Readings==

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/foster-grid.pdf Ian Foster and Carl Kesselman, "Computational Grids" (1998)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/foster-globus-intro.pdf Ian Foster, "Globus Toolkit Version 4: Software for Service-Oriented Systems" (2006)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/anderson-boinc.pdf David P. Anderson, "BOINC: A System for Public-Resource Computing and Storage" (2004)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/mapreduce-osdi04.pdf Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simpliﬁed Data Processing on Large Clusters" (2004)]

==Notes==
===BOINC===
*Premise? Local client on your machine downloads a 'workunit', churns the data, dumps the results and downloads a new 'workunit'
*Why are we caring?
**Entertainment?
**How is this an OS paradigm? What is it useful for?
***It isn't really an OS, just a method to have your mass computation done
***More of a distributed scheduler?
****Not even, central scheduler, but mass computation
***How many systems have we seen that have accomplished mass computation on millions of uncontrolled computers?
****ummm... none?
***As an OS?
****An OS is something that is created to run programs
****This is a special case allowing us to run specific programs (BUT IS IT AN OS?)
***Useful for "embarassingly parallel programs"
*Perfect for large scale simulation?
**But then you need LOTS of communication, and this system does not have interconnects
*The type of problems that we most care about tend not to be THAT parallel

*So what would a distrbuted OS be for?
**Shared communication!
***But we don't have much in the way that works well.
*An OS typically provides a lot of services, together in one package
**We have been seeing that there are no complete packages, just pieces and parts. Why?
***Computers are changing too fast? Same *NIX OS, same tcp/ip stack... so more of the same, why no true solution?
***Communication is unreliable? Yes, but that is also nothing new

*If people found that distributed file systems were succesful, they would be in use all the time, but they aren't. Reason? PERFORMANCE

*Take away message?
*Can't handle communication - how do you abstract access to resources when driven through a network?
**As a result, we have many many specialised solutions for particular workloads.
*If you are willing to not have communication between nodes, you gain a HUGE amount of computation

NASD, GoogleFS, Farsite

2008-03-12T19:56:05Z

Emmellst:

==Readings==

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-10/gibson-nasd.pdf Garth A. Gibson et al., "A Cost-Effective, High-Bandwidth Storage Architecture" (1998)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-10/gfs-sosp2003.pdf Sanjay Ghemawat et al., "The Google File System" (2003)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-10/adya-farsite-intro.pdf Atul Adya et al.,"FARSITE: Federated, Available, and Reliable Storage for an Incompletely Trusted Environment" (2002)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-10/bolosky-farsite-retro.pdf William J. Bolosky et al., "The Farsite Project: A Retrospective" (2007)]

==Questions==

# What were the target environments for these filesystems? How did these environments shape their assumptions?
:*Farsite was geared towards distributing a company's resources
:**Just imagine debugging that sucker
:*GFS Is very much geared for their specific requirements

# What are the key ideas behind each filesystem?
:*Scalability
:*Separating control & metadata and the data itself
:**Separate everything, machines protocols, etc
:*Farsite?
:**No real notion of striping, their model was "small files distributed everywhere"
:*GFS?
:**Lots of BIG FILES!

# What are the strengths and weaknesses of each design?
:*Would you want to play with Farsite?
:**Very baroque, "like windows" :)
:*While GFS is not any more applicable to average users, it has a much simpler design
:*NASD?
:**Minus crypto, this is very close to what NAS is now
:**Good idea in principle, but added hardware requirement likely prevented this actual implementation

# What are the strengths and weaknesses of each implementation?
# Which system is best suited for today's Internet? How about tomorrow's?

==Questions for NASD==
# Is giving direct access between client and drive a good idea?
# Are there substantial advantages in storing variable-length objects over fixed-sized blocks?
# Is putting the filesystem on the drive a good idea? Should more control and awareness be given to hardware devices?
# What are the strengths and weaknesses of the capability-based cryptography which NASD makes use of?

==Questions for GoogleFS==
# How does the Google file system implement security?
:*Doesn't

# Is using a central server (point of access) a good design decision?
:*It certainly works
:*Makes administration easier
:*As long as redundant and fast, why bother with the hassle of synchronization?

# Is removing random writes a good idea?
:*They didn't actually remove it, but it is horribly inneficient
:*BigTable specifically reduces the instances of random write and implements a way to append the same information
:*Implementing this style would have killed their model

# Is the speedup attained by GFS's record-append method worth the sacrifice of Application overhead?
:*Needing to manage duplication yourself
:*Guaranteed access to specific offsets, which helps consistency, though wastes space

==Questions for Farsite==
# Byzantine fault tolerance?
:*Have several entities, some of which may be compromised in some way. They might either be corrupted, compromised, or simply down.
:**Assumptions for a Byzantine protocol? Failures are independent, so they are not colluding.
:*Good model for hardware failures
:*Bad model for software failures (infection, etc)
:*Not really the appropriate solution, software is your main likely culprit, not hardware problems.
:*Tried to implement a simpler version using checksums

# How similar and different compared to OceanStore?
:*Uses crypto (same)
:*Uses commodity hardware (different)
:*Byzantine Fault tolerance (same)
:*Namespaces are different
:*Simpler version than OceanStore
:*Only one administrative domain (someone HAS admin access)
:*Planned for complete distribution, though ended up implementing a central server
:*Made sure that every machine was identified through different keys
:*Oceanstore was originally designed for dedicated distributed network servers, Farsite was designed for local commodity machines

# What's up with the file lease mechanism?
:*Four kinds
:**Likely discovered an application class that broke and needed different semantics
:*Unable to give truly seamless access as if local
:*Content, name, access, mode, machine leases
:*Likely a windows semantics problem, not a file-system problem. But due to the desire that the file system should accomodate, rather than the OS, many 'hacks' were added

==Questions for Farsite retrospective==
# If using different programming methods... how does this file-system work given different programming models
# Details of Byzantine fault tolerance

*Mentioned that they started to use formal methods, really good for their design
*A bit of a reality check that was necessary for this paper. Ultimate realization was that the proposed system was a little grandiose.

===Notes===
*GFS is great because it works on 'crap hardware'
*Oceanstore is likely better for regular document storage
*NASD on top of GFS? More messages, likely too slow and could defeat the purpose of GFS
*How do you go to the REALLY LARGE SCALE and have things work?
**Great question, only application specific?
**Definitely a need for large scale resource sharing
**Currently no unified way to share resources across administrative domains, so resources are all silo'd

NASD, GoogleFS, Farsite

2008-03-12T19:31:02Z

Emmellst:

==Readings==

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-10/gibson-nasd.pdf Garth A. Gibson et al., "A Cost-Effective, High-Bandwidth Storage Architecture" (1998)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-10/gfs-sosp2003.pdf Sanjay Ghemawat et al., "The Google File System" (2003)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-10/adya-farsite-intro.pdf Atul Adya et al.,"FARSITE: Federated, Available, and Reliable Storage for an Incompletely Trusted Environment" (2002)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-10/bolosky-farsite-retro.pdf William J. Bolosky et al., "The Farsite Project: A Retrospective" (2007)]

==Questions==

# What were the target environments for these filesystems? How did these environments shape their assumptions?
# What are the key ideas behind each filesystem?
# What are the strengths and weaknesses of each design?
# What are the strengths and weaknesses of each implementation?
# Which system is best suited for today's Internet? How about tomorrow's?

==Questions for NASD==
# Is giving direct access between client and drive a good idea?
# Are there substantial advantages in storing variable-length objects over fixed-sized blocks?
# Is putting the filesystem on the drive a good idea? Should more control and awareness be given to hardware devices?
# What are the strengths and weaknesses of the capability-based cryptography which NASD makes use of?

==Questions for GoogleFS==
# How does the Google file system implement security?
:*Doesn't

# Is using a central server (point of access) a good design decision?
:*It certainly works
:*Makes administration easier
:*As long as redundant and fast, why bother with the hassle of synchronization?

# Is removing random writes a good idea?
:*They didn't actually remove it, but it is horribly inneficient
:*BigTable specifically reduces the instances of random write and implements a way to append the same information
:*Implementing this style would have killed their model

# Is the speedup attained by GFS's record-append method worth the sacrifice of Application overhead?
:*Needing to manage duplication yourself
:*Guaranteed access to specific offsets, which helps consistency, though wastes space

==Questions for Farsite==
# Byzantine fault tolerance?
:*Have several entities, some of which may be compromised in some way. They might either be corrupted, compromised, or simply down.
:**Assumptions for a Byzantine protocol? Failures are independent, so they are not colluding.
:*Good model for hardware failures
:*Bad model for software failures (infection, etc)
:*Not really the appropriate solution, software is your main likely culprit, not hardware problems.
:*Tried to implement a simpler version using checksums

# How similar and different compared to OceanStore?
:*Uses crypto
:*Uses commodity hardware
:*Byzantine Fault tolerance
:*Namespaces are different
:*Simpler version than OceanStore
:*Only one administrative domain (someone HAS admin access)
:*Planned for complete distribution, though ended up implementing a central server
:*Made sure that every machine was identified through different keys
:*Oceanstore was originally designed for dedicated distributed network servers, Farsite was designed for local commodity machines

# What's up with the file lease mechanism?
:*Four kinds
:**Likely discovered an application class that broke and needed different semantics
:*Unable to give truly seamless access as if local
:*Content, name, access, mode, machine leases
:*Likely a windows semantics problem, not a file-system problem. But due to the desire that the file system should accomodate, rather than the OS, many 'hacks' were added

==Questions for Farsite retrospective==
# If using different programming methods... how does this file-system work given different programming models
# Details of Byzantine fault tolerance

NASD, GoogleFS, Farsite

2008-03-12T19:26:55Z

Emmellst: Edit during class discussion

NASD, GoogleFS, Farsite

2008-03-12T19:18:32Z

Emmellst:

NASD, GoogleFS, Farsite

2008-03-12T19:08:56Z

Emmellst: /* Questions for GoogleFS */

NASD, GoogleFS, Farsite

2008-03-12T19:05:13Z

Emmellst: /* Questions for GoogleFS */

NASD, GoogleFS, Farsite

2008-03-10T19:53:36Z

Emmellst:

OceanStore & GPFS

2008-02-25T20:52:24Z

Emmellst: /* GPFS */

==Readings==

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/oceanstore-sigplan.pdf John Kubiatowicz et al., "OceanStore: An Architecture for Global-Scale Persistent Storage" (2000)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/fast2003-pond.pdf Sean Rhea et al., "Pond: the OceanStore Prototype" (2003)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/gpfs-fast02.pdf Frank Schmuck and Roger Haskin, "GPFS: A Shared-Disk File System for Large Computing Clusters" (2002)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/walker-xufs-worlds06.pdf Edward Walker, "A Distributed File System for a Wide-Area High Performance Computing Infrastructure" (2006)]

==Questions==
Is it worth it??

=Ocean Store=
Pros
*Only trust required is own box
*Data is highly durable due to file versioning
*Information divorced from location
**So long as you can reliably obtain information, it doesn't matter where it is located
*Applicable to many data storage situations, not for a specific case
*Routing is decentralized
*2/3 of network is up? All is available

Cons
*Very expensive to computer cryptography (slow generation of keys)
*Utility models don't make economic sense, people prefer not to pay for access to their data

=GPFS=
Distributed local OS designed for clusters
Max size of 4096TB

Pros
*Massively parallel - data is striped across many many disks
*Therefor read/write is very fast
*Option of redundancy
*Locking mechanism
**Two options
***1. Data shipping
****Distributed
****First client to request access to file receives token
****Other clients must request the current owner of the token
*****The current owner of the file grants portional access to their file (breaks token and gives portion access)
***2. Centralized locking
****Faster in a small disk circumstance
*Extreme reliability
**Able to literally remove a hotswap disk and insert a blank one in its place, only to have the blank disk completely regenerate the missing data
**Journalling to record token ownership - helps recovery when node in possession dies

Cons
*Everything must be trusted! Designed for clusters, not across LAN/WAN
*Not appropriate for distributed networks.

=XUFS=
*User-space implementation
*Designed to be simple
*Very generic

OceanStore & GPFS

2008-02-25T20:50:55Z

Emmellst: /* GPFS */

==Readings==

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/oceanstore-sigplan.pdf John Kubiatowicz et al., "OceanStore: An Architecture for Global-Scale Persistent Storage" (2000)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/fast2003-pond.pdf Sean Rhea et al., "Pond: the OceanStore Prototype" (2003)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/gpfs-fast02.pdf Frank Schmuck and Roger Haskin, "GPFS: A Shared-Disk File System for Large Computing Clusters" (2002)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/walker-xufs-worlds06.pdf Edward Walker, "A Distributed File System for a Wide-Area High Performance Computing Infrastructure" (2006)]

==Questions==
Is it worth it??

=Ocean Store=
Pros
*Only trust required is own box
*Data is highly durable due to file versioning
*Information divorced from location
**So long as you can reliably obtain information, it doesn't matter where it is located
*Applicable to many data storage situations, not for a specific case
*Routing is decentralized
*2/3 of network is up? All is available

Cons
*Very expensive to computer cryptography (slow generation of keys)
*Utility models don't make economic sense, people prefer not to pay for access to their data

=GPFS=
Distributed local OS designed for clusters
Max size of 4096TB

Pros
*Massively parallel - data is striped across many many disks
*Therefor read/write is very fast
*Option of redundancy
*Locking mechanism
**Two options
***Data shipping
****Distributed
****First client to request access to file receives token
****Other clients must request the current owner of the token
*****The current owner of the file grants portional access to their file (breaks token and gives portion access)
***Centralized locking
****Faster in a small disk circumstance
*Extreme reliability
**Able to literally remove a hotswap disk and insert a blank one in its place, only to have the blank disk completely regenerate the missing data
**Journalling to record token ownership - helps recovery when node in possession dies

Cons
*Everything must be trusted! Designed for clusters, not across LAN/WAN
*Not appropriate for distributed networks.

=XUFS=
*User-space implementation
*Designed to be simple
*Very generic

OceanStore & GPFS

2008-02-25T20:50:32Z

Emmellst:

==Readings==

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/oceanstore-sigplan.pdf John Kubiatowicz et al., "OceanStore: An Architecture for Global-Scale Persistent Storage" (2000)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/fast2003-pond.pdf Sean Rhea et al., "Pond: the OceanStore Prototype" (2003)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/gpfs-fast02.pdf Frank Schmuck and Roger Haskin, "GPFS: A Shared-Disk File System for Large Computing Clusters" (2002)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/walker-xufs-worlds06.pdf Edward Walker, "A Distributed File System for a Wide-Area High Performance Computing Infrastructure" (2006)]

==Questions==
Is it worth it??

=Ocean Store=
Pros
*Only trust required is own box
*Data is highly durable due to file versioning
*Information divorced from location
**So long as you can reliably obtain information, it doesn't matter where it is located
*Applicable to many data storage situations, not for a specific case
*Routing is decentralized
*2/3 of network is up? All is available

Cons
*Very expensive to computer cryptography (slow generation of keys)
*Utility models don't make economic sense, people prefer not to pay for access to their data

=GPFS=
Distributed local OS designed for clusters
Max size of 4096TB

Pros
**Massively parallel - data is striped across many many disks
**Therefor read/write is very fast
*Option of redundancy
*Locking mechanism
**Two options
***Data shipping
****Distributed
****First client to request access to file receives token
****Other clients must request the current owner of the token
*****The current owner of the file grants portional access to their file (breaks token and gives portion access)
***Centralized locking
****Faster in a small disk circumstance
*Extreme reliability
**Able to literally remove a hotswap disk and insert a blank one in its place, only to have the blank disk completely regenerate the missing data
**Journalling to record token ownership - helps recovery when node in possession dies

Cons
*Everything must be trusted! Designed for clusters, not across LAN/WAN
*Not appropriate for distributed networks.

=XUFS=
*User-space implementation
*Designed to be simple
*Very generic

OceanStore & GPFS

2008-02-25T20:46:26Z

Emmellst:

==Readings==

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/oceanstore-sigplan.pdf John Kubiatowicz et al., "OceanStore: An Architecture for Global-Scale Persistent Storage" (2000)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/fast2003-pond.pdf Sean Rhea et al., "Pond: the OceanStore Prototype" (2003)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/gpfs-fast02.pdf Frank Schmuck and Roger Haskin, "GPFS: A Shared-Disk File System for Large Computing Clusters" (2002)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/walker-xufs-worlds06.pdf Edward Walker, "A Distributed File System for a Wide-Area High Performance Computing Infrastructure" (2006)]

==Questions==
Is it worth it??

=Ocean Store=
Pros
-Only trust required is own box
-Data is highly durable due to file versioning
-Information divorced from location
--So long as you can reliably obtain information, it doesn't matter where it is located
-Applicable to many data storage situations, not for a specific case
-Routing is decentralized
-2/3 of network is up? All is available

Cons
-Very expensive to computer cryptography (slow generation of keys)
-Utility models don't make economic sense, people prefer not to pay for access to their data

=Pond=
Example of oceanstore

=GPFS=
Distributed local OS designed for clusters
Max size of 4096TB
Pros
-Massively parallel - data is striped across many many disks
--Therefor read/write is very fast
-Option of redundancy
-Locking mechanism
--Two options
---Data shipping
----Distributed
----First client to request access to file receives token
----Other clients must request the current owner of the token
-----The current owner of the file grants portional access to their file (breaks token and gives portion access)
---Centralized locking
----Faster in a small disk circumstance
-Extreme reliability
--Able to literally remove a hotswap disk and insert a blank one in its place, only to have the blank disk completely regenerate the missing data
--Journalling to record token ownership - helps recovery when node in possession dies

Cons
-Everything must be trusted! Designed for clusters, not across LAN/WAN
-Not appropriate for distributed networks.

=XUFS=
User-space implementation
Designed to be simple
Very generic

OceanStore & GPFS

2008-02-25T20:43:42Z

Emmellst: /* Questions */

==Readings==

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/oceanstore-sigplan.pdf John Kubiatowicz et al., "OceanStore: An Architecture for Global-Scale Persistent Storage" (2000)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/fast2003-pond.pdf Sean Rhea et al., "Pond: the OceanStore Prototype" (2003)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/gpfs-fast02.pdf Frank Schmuck and Roger Haskin, "GPFS: A Shared-Disk File System for Large Computing Clusters" (2002)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-02-25/walker-xufs-worlds06.pdf Edward Walker, "A Distributed File System for a Wide-Area High Performance Computing Infrastructure" (2006)]

==Questions==
Is it worth it??

=Ocean Store=
Pros
-Only trust required is own box
-Data is highly durable due to file versioning
-Information divorced from location
--So long as you can reliably obtain information, it doesn't matter where it is located
-Applicable to many data storage situations, not for a specific case
-Routing is decentralized
-2/3 of network is up? All is available

Cons
-Very expensive to computer cryptography (slow generation of keys)
-Utility models don't make economic sense, people prefer not to pay for access to their data

=Pond=
Example of oceanstore

=GPFS=
Distributed local OS designed for clusters
Max size of 4096TB
Pros
-Massively parallel - data is striped across many many disks
--Therefor read/write is very fast
-Option of redundancy
-Locking mechanism
--Two options
---Data shipping
----Distributed
----First client to request access to file receives token
----Other clients must request the current owner of the token
-----The current owner of the file grants portional access to their file (breaks token and gives portion access)
---Centralized locking
----Faster in a small disk circumstance
-Extreme reliability
--Able to literally remove a hotswap disk and insert a blank one in its place, only to have the blank disk completely regenerate the missing data
--Journalling to record token ownership - helps recovery when node in possession dies

Cons
-Everything must be trusted! Designed for clusters, not across LAN/WAN
-Not appropriate for distributed networks.

=XUFS=
Designed for OS with