Soma-notes - User contributions [en]

DistOS 2015W Session 9

2015-04-09T23:08:31Z

Ambalica: /* BOINC */

== BOINC ==

*Public Resource Computing Platform
*Gives scientists the ability to use large amounts of computation resources.
*The clients do not connect directly with each other but instead they talk to a central server located at Berkley
*The goals of Boinc are:
:*1) reduce the barriers of entry
:*2) Share resources among autonomous projects
:*3) Support diverse applications
:*4) Reward participants.
:*5) Provide screensaver graphics

*It can run as applications in common language with no modifications
*A BOINC application can be identified by a single master URL, which serves as the homepage as well as the directory of the servers.
*Servers perform set of function using:
**Scheduling servers: handles Remote Procedure Call from clients
** Data servers:helps to manage the uploads

== SETI@Home ==

*Uses public resource computing to analyze radio signals to find extraterrestrial intelligence
*Need good quality telescope to search for radio signals, and lots of computational power, which was unavailable locally
*It has not yet found extraterrestrial intelligence, but its has established credibility of public resource computing projects
*Originally custom, now uses BOINC as a backbone for the project
*Uses relational database to store information on a large scale, further it uses a multi-threaded server to distribute work to clients
*Quality of data in this architecture is untrustworthy, the main incentive to use it, however, is that it is a cheap and easy way of scaling the work exponentially.
*Provided social incentives to encourage users to join the system.
*This computation model still exists but not in the legitimate world.
*Formed a good concept of public resource computing and a distributed computing by providing a platform independent framework

== MapReduce ==

*A programming model presented by Google to do large scale parallel computations
*Uses the <code>Map()</code> and <code>Reduce()</code> functions from functional style programming languages
:*Map (Filtering)
::*Takes a function and applies it to a bunch of keys to produce values
* Hides parallelization, fault tolerance, locality optimization and load balancing
:*Reduce (Summary)
::*Accumulates results from the data set using a given function
* Very easy to use and understand, with many classic problems fitting this pattern
* Otherwise quite constrained in what exactly can be done
* Uses hashing to distribute similar keys to similar machines, but otherwise spread the load

== Naiad ==

*A programming model similar to <code>MapReduce</code> but with streaming capabilities so that data results are almost instantaneous
*A distributed system for executing data parallel cyclic dataflow programs offering high throughput and low latency
*Aims to provide a general purpose system which will fulfill the requirements and the will also support wide variety of high level programming models.
*Highly used for parallel execution of data
*Provides the functionality of checkpoint and restoring
*A complex framework that can be the backend for simpler models of computation like LINQ or MapReduce to be built on top of.
*Real Time Applications:
:*Batch iterative Machine Learning:
VW, an open source distributed machine learning performs iteration in 3 phases: each process updates local state; processes independently training on local data; and process jointly performed global average which is All Reduce.
:*Streaming Acyclic Computation
When compared to a system called [http://research.microsoft.com/apps/pubs/default.aspx?id=163832 Kineograph] ( also done by Microsoft ), which processes twitter handles and provides counts of the occurrence of hashtags as well as links between popular tags, was written using Naiad in 26 lines of code and ran close to 2X faster.
* Naiad paper won the best paper award in SOSP 2013, check-out this link in Microsoft Research website http://research.microsoft.com/en-us/projects/naiad/ . Down in this page you can see some videos that explains naiad including Derek's Murray presentation at SOSP 2013.

DistOS 2015W Session 9

2015-04-09T23:06:18Z

Ambalica: /* BOINC */

== BOINC ==

*Public Resource Computing Platform
*Gives scientists the ability to use large amounts of computation resources.
*The clients do not connect directly with each other but instead they talk to a central server located at Berkley
*The goals of Boinc are:
:*1) reduce the barriers of entry
:*2) Share resources among autonomous projects
:*3) Support diverse applications
:*4) Reward participants.
:*5) Provide screensaver graphics

*It can run as applications in common language with no modifications
*A BOINC application can be identified by a single master URL, <br/>which serves as the homepage as well as the directory of the servers.
*Servers perform set of function using:
**Scheduling servers: handles Remote Process Call from clients
** Data servers:helps to manage the uploads

== SETI@Home ==

*Uses public resource computing to analyze radio signals to find extraterrestrial intelligence
*Need good quality telescope to search for radio signals, and lots of computational power, which was unavailable locally
*It has not yet found extraterrestrial intelligence, but its has established credibility of public resource computing projects
*Originally custom, now uses BOINC as a backbone for the project
*Uses relational database to store information on a large scale, further it uses a multi-threaded server to distribute work to clients
*Quality of data in this architecture is untrustworthy, the main incentive to use it, however, is that it is a cheap and easy way of scaling the work exponentially.
*Provided social incentives to encourage users to join the system.
*This computation model still exists but not in the legitimate world.
*Formed a good concept of public resource computing and a distributed computing by providing a platform independent framework

== MapReduce ==

*A programming model presented by Google to do large scale parallel computations
*Uses the <code>Map()</code> and <code>Reduce()</code> functions from functional style programming languages
:*Map (Filtering)
::*Takes a function and applies it to a bunch of keys to produce values
* Hides parallelization, fault tolerance, locality optimization and load balancing
:*Reduce (Summary)
::*Accumulates results from the data set using a given function
* Very easy to use and understand, with many classic problems fitting this pattern
* Otherwise quite constrained in what exactly can be done
* Uses hashing to distribute similar keys to similar machines, but otherwise spread the load

== Naiad ==

*A programming model similar to <code>MapReduce</code> but with streaming capabilities so that data results are almost instantaneous
*A distributed system for executing data parallel cyclic dataflow programs offering high throughput and low latency
*Aims to provide a general purpose system which will fulfill the requirements and the will also support wide variety of high level programming models.
*Highly used for parallel execution of data
*Provides the functionality of checkpoint and restoring
*A complex framework that can be the backend for simpler models of computation like LINQ or MapReduce to be built on top of.
*Real Time Applications:
:*Batch iterative Machine Learning:
VW, an open source distributed machine learning performs iteration in 3 phases: each process updates local state; processes independently training on local data; and process jointly performed global average which is All Reduce.
:*Streaming Acyclic Computation
When compared to a system called [http://research.microsoft.com/apps/pubs/default.aspx?id=163832 Kineograph] ( also done by Microsoft ), which processes twitter handles and provides counts of the occurrence of hashtags as well as links between popular tags, was written using Naiad in 26 lines of code and ran close to 2X faster.
* Naiad paper won the best paper award in SOSP 2013, check-out this link in Microsoft Research website http://research.microsoft.com/en-us/projects/naiad/ . Down in this page you can see some videos that explains naiad including Derek's Murray presentation at SOSP 2013.

DistOS 2015W Session 9

2015-04-09T22:55:16Z

Ambalica: /* BOINC */

== BOINC ==

*Public Resource Computing Platform
*Gives scientists the ability to use large amounts of computation resources.
*The clients do not connect directly with each other but instead they talk to a central server located at Berkley
*The goals of Boinc are:
:*1) reduce the barriers of entry
:*2) Share resources among autonomous projects
:*3) Support diverse applications
:*4) Reward participants.
:*5) Provide screensaver graphics

*It can run as applications in common language with no modifications
A BOINC application can be identified by a single master URL, <br/>which serves as the homepage as well as the directory of the servers.

== SETI@Home ==

*Uses public resource computing to analyze radio signals to find extraterrestrial intelligence
*Need good quality telescope to search for radio signals, and lots of computational power, which was unavailable locally
*It has not yet found extraterrestrial intelligence, but its has established credibility of public resource computing projects
*Originally custom, now uses BOINC as a backbone for the project
*Uses relational database to store information on a large scale, further it uses a multi-threaded server to distribute work to clients
*Quality of data in this architecture is untrustworthy, the main incentive to use it, however, is that it is a cheap and easy way of scaling the work exponentially.
*Provided social incentives to encourage users to join the system.
*This computation model still exists but not in the legitimate world.
*Formed a good concept of public resource computing and a distributed computing by providing a platform independent framework

== MapReduce ==

*A programming model presented by Google to do large scale parallel computations
*Uses the <code>Map()</code> and <code>Reduce()</code> functions from functional style programming languages
:*Map (Filtering)
::*Takes a function and applies it to a bunch of keys to produce values
* Hides parallelization, fault tolerance, locality optimization and load balancing
:*Reduce (Summary)
::*Accumulates results from the data set using a given function
* Very easy to use and understand, with many classic problems fitting this pattern
* Otherwise quite constrained in what exactly can be done
* Uses hashing to distribute similar keys to similar machines, but otherwise spread the load

== Naiad ==

*A programming model similar to <code>MapReduce</code> but with streaming capabilities so that data results are almost instantaneous
*A distributed system for executing data parallel cyclic dataflow programs offering high throughput and low latency
*Aims to provide a general purpose system which will fulfill the requirements and the will also support wide variety of high level programming models.
*Highly used for parallel execution of data
*Provides the functionality of checkpoint and restoring
*A complex framework that can be the backend for simpler models of computation like LINQ or MapReduce to be built on top of.
*Real Time Applications:
:*Batch iterative Machine Learning:
VW, an open source distributed machine learning performs iteration in 3 phases: each process updates local state; processes independently training on local data; and process jointly performed global average which is All Reduce.
:*Streaming Acyclic Computation
When compared to a system called [http://research.microsoft.com/apps/pubs/default.aspx?id=163832 Kineograph] ( also done by Microsoft ), which processes twitter handles and provides counts of the occurrence of hashtags as well as links between popular tags, was written using Naiad in 26 lines of code and ran close to 2X faster.
* Naiad paper won the best paper award in SOSP 2013, check-out this link in Microsoft Research website http://research.microsoft.com/en-us/projects/naiad/ . Down in this page you can see some videos that explains naiad including Derek's Murray presentation at SOSP 2013.

DistOS 2015W Session 12

2015-04-09T22:20:54Z

Ambalica: /* Comet */

=Haystack=
* Facebook's Photo Application Storage System.
* Previous Fb photo storage based on NFS design. The reason why NFS didn't work is that it took 3 file-system accesses per logical photo read. Haystack only needs one access.
*Main goals of Haystack:
** High throughput with low latency. It uses one disk operation to provide these.
**Fault tolerance
**Cost effective
**Simple
*Facebook stored all images in haystack with a CDN in front to cache hot data. Haystack still needs to be fast since accessing non-cached data is still common.
*Haystack reduces the memory used for ''filesystem metadata''
*It has 2 types of metadata:
**''Application metadata''
**''File System metadata''
* The architecture consists of 3 components:
**Haystack Store
**Haystack Directory
**Haystack Cache
*Pitchfork and bulk sync were used to tolerate faults. theTfault tolerance works in a very profound way to make haystack feasible and reliable

=Comet=
*Introduced the concept of distributed shared memory (DSM). In a DSM, RAMs from multiple servers would appear as if they are all belonging to one server, allowing better scalability for caching.
*DSM provides advatage over RPC(Remote Procedure Call) including multi threading suuport, thread migration during execution.
*client and server model maintain consistency using DSM
*Comet model works by offloading the computation intensive process from the mobile to only one server.
*The offloading process works by passing the computation intensive process to the server and hold it on the mobile device. Once the process on the server completes, it returns the results and the handle back to the mobile device. In other words, the process does not get physically offloaded to the server but instead it runs on the server and stopped on the mobile device.

=F4=
* Warm Blob Storage System.
** Warm Blob is a store for large quantities of immutable data that isn't frequently accessed, but must still be available.
** Built to reduce the overhead of haystack for old data that doesn't need to be quite as available. Generally data that is a few months old is moved from Haystack to Warm Blob.
** F4 reduce the space usage of Haystack from a replication factor of 3.6 to 2.8 or 2.1 using Reed Solomon coding and XOR coding respectively but still provides consistency.
** Less robust to data center failures as a result.
*Reed Solomon coding basically use(10,4) which means 10 data and 4 parity blocks in a stripe, and can thus tolerate losing up to 4 blocks which means it can tolerate 4 rack failure and use 1.4 expansion factor.Two copies of this would be 2* 1.4= 2.8 effective replication factor.
*XOR coding use(2,1) across three data center and use 1.5 expansion factor which gives 1.5*1.4= 2.1 effective replication factor.
*The caching mechanism provides the reduction in load on storage system and it makes BLOB scaleable.

=Sapphire=
*Represents a building block towards building this global distributed systems. The main critique to it is that it didn’t present a specific use case upon which their design is built upon.
*Sapphire does not show their scalability boundaries. There is no such distributed system model that can be “one size fits all”, most probably it will break in some large scale distributed application.
*Reaching this global distributed system that address all the distributed OS use cases will be a cumulative work of many big bodies and building it block by block and then this system will evolve by putting all these different building blocks together. In other words, reaching a global distributed system will come from a “bottom up not top down approach” [Somayaji, 2015].
*The concept of separate application logic from deployment logic helps programmers in making a flexible system. The other important part that makes it as a scalable system was that it is object based and could be integrated with any object oriented language.

DistOS 2015W Session 12

2015-04-09T22:15:15Z

Ambalica: /* Comet */

=Haystack=
* Facebook's Photo Application Storage System.
* Previous Fb photo storage based on NFS design. The reason why NFS didn't work is that it took 3 file-system accesses per logical photo read. Haystack only needs one access.
*Main goals of Haystack:
** High throughput with low latency. It uses one disk operation to provide these.
**Fault tolerance
**Cost effective
**Simple
*Facebook stored all images in haystack with a CDN in front to cache hot data. Haystack still needs to be fast since accessing non-cached data is still common.
*Haystack reduces the memory used for ''filesystem metadata''
*It has 2 types of metadata:
**''Application metadata''
**''File System metadata''
* The architecture consists of 3 components:
**Haystack Store
**Haystack Directory
**Haystack Cache
*Pitchfork and bulk sync were used to tolerate faults. theTfault tolerance works in a very profound way to make haystack feasible and reliable

=Comet=
*Introduced the concept of distributed shared memory (DSM). In a DSM, RAMs from multiple servers would appear as if they are all belonging to one server, allowing better scalability for caching.
*DSM provides advatage over Rpc(Remote Procedure Call) including multi threading suuport, thread migration during execution.
*client and server model maintain consistency using DSM
*Comet model works by offloading the computation intensive process from the mobile to only one server.
*The offloading process works by passing the computation intensive process to the server and hold it on the mobile device. Once the process on the server completes, it returns the results and the handle back to the mobile device. In other words, the process does not get physically offloaded to the server but instead it runs on the server and stopped on the mobile device.

=F4=
* Warm Blob Storage System.
** Warm Blob is a store for large quantities of immutable data that isn't frequently accessed, but must still be available.
** Built to reduce the overhead of haystack for old data that doesn't need to be quite as available. Generally data that is a few months old is moved from Haystack to Warm Blob.
** F4 reduce the space usage of Haystack from a replication factor of 3.6 to 2.8 or 2.1 using Reed Solomon coding and XOR coding respectively but still provides consistency.
** Less robust to data center failures as a result.
*Reed Solomon coding basically use(10,4) which means 10 data and 4 parity blocks in a stripe, and can thus tolerate losing up to 4 blocks which means it can tolerate 4 rack failure and use 1.4 expansion factor.Two copies of this would be 2* 1.4= 2.8 effective replication factor.
*XOR coding use(2,1) across three data center and use 1.5 expansion factor which gives 1.5*1.4= 2.1 effective replication factor.
*The caching mechanism provides the reduction in load on storage system and it makes BLOB scaleable.

=Sapphire=
*Represents a building block towards building this global distributed systems. The main critique to it is that it didn’t present a specific use case upon which their design is built upon.
*Sapphire does not show their scalability boundaries. There is no such distributed system model that can be “one size fits all”, most probably it will break in some large scale distributed application.
*Reaching this global distributed system that address all the distributed OS use cases will be a cumulative work of many big bodies and building it block by block and then this system will evolve by putting all these different building blocks together. In other words, reaching a global distributed system will come from a “bottom up not top down approach” [Somayaji, 2015].
*The concept of separate application logic from deployment logic helps programmers in making a flexible system. The other important part that makes it as a scalable system was that it is object based and could be integrated with any object oriented language.

DistOS 2015W Session 12

2015-04-02T04:58:19Z

Ambalica: /* F4 */

=Haystack=
* Facebook's Photo Application Storage System.
* Previous Fb photo storage based on NFS design. The reason why NFS dint work is because it gave 3 reads for every photo. The issue here was that they needed 1 read per photo.
*Main goals of Haystack:
** High throughput with low latency
**Fault tolerance
**Cost effective
**SImple
*Facebook utilises CDN to serve popular images and further uses haystack to respond to photo requests in the long tail effectively.
*Haystack reduces the memory used for ''filesystem metadata''
*It has 2 types of metadata:
**''Application metadata''
**''File System metadata''
* The architecture consists of 3 components:
**Haystack Store
**Haystack Directory
**Haystack Cache

=Comet=
*Introduced the concept of distributed shared memory (DSM). In a DSM, RAMs from multiple servers would appear as if they are all belonging to one server, allowing better scalability for caching.
*Comet model works by offloading the computation intensive process from the mobile to only one server.
*The offloading process works by passing the computation intensive process to the server and hold it on the mobile device. Once the process on the server completes, it returns the results and the handle back to the mobile device. In other words, the process does not get physically offloaded to the server but instead it runs on the server and stopped on the mobile device.
=F4=
* Warm Blob Storage System.
** Warm Blob is a immutable data that gets cool very rapidly.
** F4 reduce the space usage by 3.6 to 2.8 or 2.1 replication factor using Reed Solomon coding and XOR coding respectively but still provides consistency.

=Sapphire=
*Represents a building block towards building this global distributed systems. The main critique to it is that it didn’t present a specific use case upon which their design is built upon.
*Sapphire does not show their scalability boundaries. There is no such distributed system model that can be “one size fits all”, most probably it will break in some large scale distributed application.
*Reaching this global distributed system that address all the distributed OS use cases will be a cumulative work of many big bodies and building it block by block and then this system will evolve by putting all these different building blocks together. In other words, reaching a global distributed system will come from a “bottom up not top down approach” [Somayaji, 2015].

DistOS 2015W Session 12

2015-04-02T04:51:41Z

Ambalica: /* F4 */

=Haystack=
* Facebook's Photo Application Storage System.
* Previous Fb photo storage based on NFS design. The reason why NFS dint work is because it gave 3 reads for every photo. The issue here was that they needed 1 read per photo.
*Main goals of Haystack:
** High throughput with low latency
**Fault tolerance
**Cost effective
**SImple
*Facebook utilises CDN to serve popular images and further uses haystack to respond to photo requests in the long tail effectively.
*Haystack reduces the memory used for ''filesystem metadata''
*It has 2 types of metadata:
**''Application metadata''
**''File System metadata''
* The architecture consists of 3 components:
**Haystack Store
**Haystack Directory
**Haystack Cache

=Comet=
*Introduced the concept of distributed shared memory (DSM). In a DSM, RAMs from multiple servers would appear as if they are all belonging to one server, allowing better scalability for caching.
*Comet model works by offloading the computation intensive process from the mobile to only one server.
*The offloading process works by passing the computation intensive process to the server and hold it on the mobile device. Once the process on the server completes, it returns the results and the handle back to the mobile device. In other words, the process does not get physically offloaded to the server but instead it runs on the server and stopped on the mobile device.
=F4=
* Warm Blob Storage System.
** Warm Blob is a immutable data that gets cool very rapidly.
** F4 reduce the space usage by 3.6 to 2.8 or 2.1 replication factor but still provides consistency.

=Sapphire=
*Represents a building block towards building this global distributed systems. The main critique to it is that it didn’t present a specific use case upon which their design is built upon.
*Sapphire does not show their scalability boundaries. There is no such distributed system model that can be “one size fits all”, most probably it will break in some large scale distributed application.
*Reaching this global distributed system that address all the distributed OS use cases will be a cumulative work of many big bodies and building it block by block and then this system will evolve by putting all these different building blocks together. In other words, reaching a global distributed system will come from a “bottom up not top down approach” [Somayaji, 2015].

DistOS 2015W Session 12

2015-04-02T04:38:33Z

Ambalica:

=Haystack=
* Facebook's Photo Application Storage System.
* Previous Fb photo storage based on NFS design. The reason why NFS dint work is because it gave 3 reads for every photo. The issue here was that they needed 1 read per photo.
*Main goals of Haystack:
** High throughput with low latency
**Fault tolerance
**Cost effective
**SImple
*Facebook utilises CDN to serve popular images and further uses haystack to respond to photo requests in the long tail effectively.
*Haystack reduces the memory used for ''filesystem metadata''
*It has 2 types of metadata:
**''Application metadata''
**''File System metadata''
* The architecture consists of 3 components:
**Haystack Store
**Haystack Directory
**Haystack Cache

=Comet=
*Introduced the concept of distributed shared memory (DSM). In a DSM, RAMs from multiple servers would appear as if they are all belonging to one server, allowing better scalability for caching.
*Comet model works by offloading the computation intensive process from the mobile to only one server.
*The offloading process works by passing the computation intensive process to the server and hold it on the mobile device. Once the process on the server completes, it returns the results and the handle back to the mobile device. In other words, the process does not get physically offloaded to the server but instead it runs on the server and stopped on the mobile device.
=F4=
* Warm Blob Storage System.
** Warm Blob is a immutable data that gets cool very rapidly.
** F4 reduce the space usage by 3.6 to 2.8 or 1.4 replication factor but still provides consistency.

=Sapphire=
*Represents a building block towards building this global distributed systems. The main critique to it is that it didn’t present a specific use case upon which their design is built upon.
*Sapphire does not show their scalability boundaries. There is no such distributed system model that can be “one size fits all”, most probably it will break in some large scale distributed application.
*Reaching this global distributed system that address all the distributed OS use cases will be a cumulative work of many big bodies and building it block by block and then this system will evolve by putting all these different building blocks together. In other words, reaching a global distributed system will come from a “bottom up not top down approach” [Somayaji, 2015].

DistOS 2015W Session 12

2015-04-02T04:37:26Z

Ambalica:

=Haystack=
* Facebook's Photo Application Storage System.
* Previous Fb photo storage based on NFS design. The reason why NFS dint work is because it gave 3 reads for every photo. The issue here was that they needed 1 read per photo.
*Main goals of Haystack:
** High throughput with low latency
**Fault tolerance
**Cost effective
**SImple
*Facebook utilises CDN to serve popular images and further uses haystack to respond to photo requests in the long tail effectively.
*Haystack reduces the memory used for ''filesystem metadata''
*It has 2 types of metadata:
**''Application metadata''
**''File System metadata''
* The architecture consists of 3 components:
**Haystack Store
**Haystack Directory
**Haystack Cache

=Comet=
*Introduced the concept of distributed shared memory (DSM). In a DSM, RAMs from multiple servers would appear as if they are all belonging to one server, allowing better scalability for caching.
*Comet model works by offloading the computation intensive process from the mobile to only one server.
*The offloading process works by passing the computation intensive process to the server and hold it on the mobile device. Once the process on the server completes, it returns the results and the handle back to the mobile device. In other words, the process does not get physically offloaded to the server but instead it runs on the server and stopped on the mobile device.
=F4=
* Warm Blob Storage System.
** Warm Blob is a immutable data that gets cool very rapidly.
** F4 reduce the space usage by 3.6 to 2.8 or 1.4 replication factor but still provides consistency.

=Sapphire=
*Represents a building block towards building this global distributed systems. The main critique to it is that it didn’t present a specific use case upon which their design is built upon.
*Sapphire does not show their scalability boundaries. There is no such distributed system model that can be “one size fits all”, most probably it will break in some large scale distributed application.
*Reaching this global distributed system that address all the distributed OS use cases will be a cumulative work of many big bodies and building it block by block and then this system will evolve by putting all these different building blocks together. In other words, reaching a global distributed system will come from a “bottom up not top down approach” [Somayaji, 2015].

DistOS 2015W Session 12

2015-04-02T04:09:42Z

Ambalica: /* F4 */

=Haystack=
* Facebook's Photo Application Storage System.
* Previous Fb photo storage based on NFS design. The reason why NFS dint work is because it gave 3 reads for every photo. The issue here was that they needed 1 read per photo.
*Main goals of Haystack:
** High throughput with low latency
**Fault tolerance
**Cost effective
**SImple
*Facebook utilises CDN to serve popular images and further uses haystack to respond to photo requests in the long tail effectively.
*Haystack reduces the memory used for ''filesystem metadata''
*It has 2 types of metadata:
**''Application metadata''
**''File System metadata''
* The architecture consists of 3 components:
**Haystack Store
**Haystack Directory
**Haystack Cache

=Comet=
*Introduced the concept of distributed shared memory (DSM). In a DSM, RAMs from multiple servers would appear as if they are all belonging to one server, allowing better scalability for caching.
*Comet model works by offloading the computation intensive process from the mobile to only one server.
*The offloading process works by passing the computation intensive process to the server and hold it on the mobile device. Once the process on the server completes, it returns the results and the handle back to the mobile device. In other words, the process does not get physically offloaded to the server but instead it runs on the server and stopped on the mobile device.
=F4=
'''It is a warm blob storage system.
'''

=Sapphire=
*Represents a building block towards building this global distributed systems. The main critique to it is that it didn’t present a specific use case upon which their design is built upon.
*Sapphire does not show their scalability boundaries. There is no such distributed system model that can be “one size fits all”, most probably it will break in some large scale distributed application.
*Reaching this global distributed system that address all the distributed OS use cases will be a cumulative work of many big bodies and building it block by block and then this system will evolve by putting all these different building blocks together. In other words, reaching a global distributed system will come from a “bottom up not top down approach” [Somayaji, 2015].

DistOS 2015W Session 12

2015-04-02T04:08:44Z

Ambalica: /* F4 */

=Haystack=
* Facebook's Photo Application Storage System.
* Previous Fb photo storage based on NFS design. The reason why NFS dint work is because it gave 3 reads for every photo. The issue here was that they needed 1 read per photo.
*Main goals of Haystack:
** High throughput with low latency
**Fault tolerance
**Cost effective
**SImple
*Facebook utilises CDN to serve popular images and further uses haystack to respond to photo requests in the long tail effectively.
*Haystack reduces the memory used for ''filesystem metadata''
*It has 2 types of metadata:
**''Application metadata''
**''File System metadata''
* The architecture consists of 3 components:
**Haystack Store
**Haystack Directory
**Haystack Cache

=Comet=
*Introduced the concept of distributed shared memory (DSM). In a DSM, RAMs from multiple servers would appear as if they are all belonging to one server, allowing better scalability for caching.
*Comet model works by offloading the computation intensive process from the mobile to only one server.
*The offloading process works by passing the computation intensive process to the server and hold it on the mobile device. Once the process on the server completes, it returns the results and the handle back to the mobile device. In other words, the process does not get physically offloaded to the server but instead it runs on the server and stopped on the mobile device.
=F4=

== warm blob storage system ==

=Sapphire=
*Represents a building block towards building this global distributed systems. The main critique to it is that it didn’t present a specific use case upon which their design is built upon.
*Sapphire does not show their scalability boundaries. There is no such distributed system model that can be “one size fits all”, most probably it will break in some large scale distributed application.
*Reaching this global distributed system that address all the distributed OS use cases will be a cumulative work of many big bodies and building it block by block and then this system will evolve by putting all these different building blocks together. In other words, reaching a global distributed system will come from a “bottom up not top down approach” [Somayaji, 2015].

DistOS 2015W Session 7

2015-02-24T06:02:32Z

Ambalica: /* Chubby */

= Ceph =
* Key advantage is that it is a general purpose distributed file system.
* System is composed of three units:
*Client,
*Cluster of Object Storage device (OSDs): It is basically stores data and metadata and clients communicate directly with it to perform IO operations.
*MetaData Server (MDS): It is used to manage the file and directories.Client basically interacts with it to perform metadata operations like open, rename. It manages the capabilities of a client.
* system has three key features:
* decoupled data and metadata:
* Dynamic Distributed Metadata Management: It distribute the metadata among multiple metadata servers using dynamic subtree partitioning to increase the performance and avoid metadata access hot spots.
* Object based storage: Using cluster of OSDs to form a Reliable Autonomic Distributed Object-Store(RADOS) for ceph failure detection and recovery.

*CRUSH (Controlled, Replicated, Under Scalable, Hashing) is the hashing algorithm used to calculate the location of object instead of looking for them. The CRUSH paper on Ceph’s website can be downloaded from here http://ceph.com/papers/weil-crush-sc06.pdf.
* RADOS (Reliable Autonomic Distributed Object-Store) is the object store for Ceph.

= Chubby =
It is basically a coarse grained lock service that serves multiple clients with small number of servers (chubby cell).

== system consists ==
*Chubby Cell: mainly consists of 5 servers known as replicas. Consensus protocol is used to elect the master from replicas.
*Client: Find the master between the replicas. Consensus protocol is used to propagate the write request to the majority of servers. Read request is handled by master only.
communication between client and server is via RPCs.

* Is a consensus algorithm among a set of servers to agree on who is the master that is in charge of the metadata.
* Can be considered a distributed file system for small size files only “256 KB” with very low scalability “5 servers”.
* Is defined in the paper as “A lock service used within a loosely-coupled distributed system consisting of moderately large number of small machines connected by a high speed network”.

DistOS 2015W Session 5

2015-02-04T03:47:23Z

Ambalica: /* Google File System */

== '''Cloud Distributed Operating System'''==
It is a distributed OS running on a set of computers that are interconnected by a group of network. It basically unifies different computers into a single component.
The OS is based on 2 patterns:
1. Message Based OS
2. Object Based OS

The structure of this is based on '''Object Thread Model.'''
It has set of objects which are defined by the class. Objects respond to messages.
Sending message to object causes object to execute the method and then reply back.

It has Active Objects and Passive objects

1.'''Active Objects''' are the objects which have one or more processes associated with them and further they can communicate with the external environment.
2.'''Passive Objects''' are the object which have no processes in them.

The contents of the Cloud are long lived. They exist forever and can survive system crashes and shut downs.

Another important part of Cloud DOS are '''threads'''

The threads are the logical path of execution that traverse objects and executes code in them.

Note: The cloud thread is not bound to a single address space. Several threads can enter an object simultaneously and execute concurrently.

The nature of the Cloud object prohibits a thread from accessing any data outside the current address space in which it is executing.

Interaction between '''Objects''' and '''Threads'''
1)Inter object interfaces are procedural
2)Invocations work across machine boundaries
3)Objects in cloud unify concept of persistent storage and memory to create address space, thus making the programming simpler.
4)Control flow achieved by threads invoking objects.

'''Cloud Environment'''
1) Integrates set of homogeneous machines into one seamless environment
2) There are three logical categories of machines- Compute Server, User Workstation and Data server.

----

== '''Plan 9''' ==

Plan 9 is a general purpose, multiuser and mobile computing environment physically distributed across machines.
The Plan 9 began in late 1980s. The aims of this system are:
1) To built a system that should be centrally administered
2) Cost effective using cheap modern microcomputers.
The distribution itself is transparent to most programs.
This property is made possible by 2 properties:
1) A per process group name space
2) A uniform access to all the resources by representing them as a file.

It is quite similar to the Unix yet so different. The commands, libraries and system calls are similar to that of Unix and therefore a casual user cannot distinguish between these two. The problems in UNIX were too deep to fix but still the various ideas were brought along. The problems addressed badly by UNIX were improved. Old tools were dropped and others were polished and reused.

What actually distinguishes Plan 9 is its '''organization'''.

Plan 9 is divided along the lines of service function.
* CPU services and terminals use same kernel
* Users may choose to run programs locally or remotely on CPU servers
*Gives the user a choice to choose whether they want distributed or centralized.

The design of Plan 9 is based on 3 principles:
1) Resources are named and accessed like files in hierarchical file system.
2) Standard protocol 9P
3) Disjoint hierarchical provided by different services are joined together into single private hierarchal file name space.

Another concept in Plan 9 is the '''Virtual Name Space'''
In a '''Virtual Name Space''', a user boots a terminal or connects to a CPU server and then a new process group is created.
Processes in group can either add to or rearrange their name space using two system calls- [[Mount]] and [[Bind]]
* '''Mount''' is used to attach new file system to a point in name space.
*'''Bind''' is used to attach a kernel resident file system to name space and also arrange pieces of name space.

The plan 9 provides mechanism to customize one's view of the system with the help of the software rather than the hardware.
It is built for the traditional system but it can be extended to the other resources.

'''Parallel Programming'''
The parallel programming has two aspects:
* Kernel provides simple process model and carefully designed system calls for synchronization.
*Programming language supports concurrent programming.

'''Implementation of Name Spaces'''
User processes construct name specs using three system calls- mount, bind, unmount.
Mount- System call attaches a tree served by a file server to the current name specs
Bind-Duplicates pieces of existing name specs at another point
Unmount- Allows components to be removed.
----

== '''Google File System''' ==

It is scalable file system for large distributed data intensive applications. The design is driven by providing previous applications workloads and technical environments, both current and anticipated.

The architecture of the Google file system consists of a single master, multiple chunk-servers and multiple clients. These chunk-servers store the data or file in unit of named chunks. Each chunk is identified by globally unique 64 bit chunk handle assigned by master at the end of the time of chunk creation. For more reliability and availability chunks are replicated on more chunk servers. The master maintains all the file system meta data which include the name space, chunk location and also the access control information.

Master and Chunk-Server Communication:
a) To check whether there is any chunk-server is down
b) To check if any file is corrupted.
c) Whether to create or delete any chunk.

Operation of GFS:
a) Client communicate with master to get the matadata.
b) client get chunk location from matadata.
c) Communicate with the one of that chunk-server to retrieve the data to perform operations on it.

DistOS 2015W Session 3

2015-01-20T04:44:30Z

Ambalica:

'''Reading Response Discussion'''

----
'''Multics'''

----

Team: Sameer, Shivjot, Ambalica, Veena

It came into being in the 1960s and it completely vanished in 2000s. It was started by Bell, General Electric and MIT but Bell backed out of the project in 1969.
Multics is a time sharing OS which provides Multitasking and Multiprogramming.

It provides following features:
1. Utility Computing
2. Access Control Lists
3. Single level storage
4. Dynamic linking
*Sharded libraries or files can be loaded and linked to Random Access Memory at run time.
5. Hot swapping
6. Multiprocessing System
7. Ring oriented Security
* It provides number of levels of authorization within the computer System.
It is not a Distributed OS but it a Centralized system which was written in the assembly language.

DistOS 2015W Session 3

2015-01-20T04:07:29Z

Ambalica:

'''Reading Response Discussion'''

----
'''Multics'''

----

Team: Sameer, Shivjot, Ambalica, Veena

It came into being in the 1960s and it completely vanished in 2000s. It was started by Bell and MIT but Bell backed out of the project in 1969.
Multics is a time sharing OS which provides Multitasking and Multiprogramming.

It provides following features:
1. Utility Computing
2. Access Control Lists
3. Single level storage
4. Dynamic linking
*Sharded libraries or files can be loaded and linked to Random Access Memory at run time.
5. Hot swapping

It is not a Distributed OS but it a Centralized system which was written in the assembly language.