Soma-notes - User contributions [en]

DistOS 2015W Session 5

2015-04-12T18:41:54Z

Veenarose: /* Google File System */

= The Clouds Distributed Operating System =
It is a distributed OS running on a set of computers that are interconnected by a group of network. It basically unifies different computers into a single component.

The OS is based on 2 patterns:
* Message Based OS
* Object Based OS

== Object Thread Model ==

The structure of this is based on object thread model. It has set of objects which are defined by the class. Objects respond to messages. Sending message to object causes object to execute the method and then reply back.

The system has '''active objects''' and '''passive objects'''.

# Active objects are the objects which have one or more processes associated with them and further they can communicate with the external environment.
# Passive objects are those that currently do not have an active thread executing in them.

The content of the Clouds data is long lived. Since the memory is implemented as a single-level store, the data exists forever and can survive system crashes and shut downs.

== Threads ==

The threads are the logical path of execution that traverse objects and executes code in them. The Clouds thread is not bound to a single address space. Several threads can enter an object simultaneously and execute concurrently. The nature of the Clouds object prohibits a thread from accessing any data outside the current address space in which it is executing.

== Interaction Between Objects and Threads ==

# Inter object interfaces are procedural
# Invocations work across machine boundaries
# Objects in clouds unify concept of persistent storage and memory to create address space, thus making the programming simpler.
# Control flow achieved by threads invoking objects.

== Clouds Environment ==

# Integrates set of homogeneous machines into one seamless environment
# There are three logical categories of machines- Compute Server, User Workstation and Data server.

= Plan 9 =

Plan 9 is a general purpose, multi-user and mobile computing environment physically distributed across machines. Development of the system began in the late 1980s. The system was built at Bell Labs - the birth place of Unix. The original Unix OS had no support for networking, and there were many attempts over the years by others to create distributed systems with Unix compatibility. Plan 9, however, is distributed systems done following the original Unix philosophy.

The goals of this system were:
# To build a distributed system that can be centrally administered.
# To be cost effective using cheap, modern microcomputers.

The distribution itself is transparent to most programs. This is made possible by two properties:
# A per process group namespace.
# Uniform access to most resources by representing them as a file.

== Unix Compatibility ==

The commands, libraries and system calls are similar to that of Unix and therefore a casual user cannot distinguish between these two. The problems in UNIX were too deep to fix but still the various ideas were brought along. The problems addressed badly by UNIX were improved. Old tools were dropped and others were polished and reused.

== Similaritieis with the UNIX ==
<ul>
<li>shell</li>
<li>Various C compilers</li>
</ul>

== Unique Features ==

What actually distinguishes Plan 9 is its '''organization'''. Plan 9 is divided along the lines of service function.
* CPU services and terminals use same kernel.
* Users may choose to run programs locally or remotely on CPU servers.
* It lets the user choose whether they want a distributed or centralized system.

The design of Plan 9 is based on 3 principles:
# Resources are named and accessed like files in hierarchical file system.
# Standard protocol 9P.
# Disjoint hierarchical provided by different services are joined together into single private hierarchical file name space.

=== Virtual Namespaces ===

In a virtual namespace, a user boots a terminal or connects to a CPU server and then a new process group is created. Processes in group can either add to or rearrange their name space using two system calls - mount and bind.

* '''Mount''' is used to attach new file system to a point in name space.
* '''Bind''' is used to attach a kernel resident (existing, mounted) file system to name space and also arrange pieces of name space.
* There is also '''unbind''' which undoes the effects of the other two calls.

Namespaces in Plan 9 are on a per-process basis. While everything had a way reference resources with a unique name, using mount and bind every process could build a custom namespace as they saw fit.

Since most resources are in the form of files (and folders), the term ''namespace'' really only refers to the filesystem layout.

=== Parallel Programming ===
Parallel programming was supported in two ways:
* Kernel provides simple process model and carefully designed system calls for synchronization.
* Programming language supports concurrent programming.

== Legacy ==

Even though Plan 9 is no longer developed, the good ideas from the system still exist today. For example, the ''/proc'' virtual filesystem which displays current process information in the form of files exists in modern Linux kernels.

= Google File System =

It is scalable, distributed file system for large, data intensive applications. It is crafted to Google's unique needs as a search engine company.

Unlike most filesystems, GFS must be implemented by individual applications and is not part of the kernel. While this introduces some technical overhead, it gives the system more freedom to implement or not implement certain non-standard features.

Link to an explanation on how GFS works
[http://computer.howstuffworks.com/internet/basics/google-file-system1.htm]

== Architecture ==

The architecture of the Google file system consists of a single master, multiple chunk-servers and multiple clients. Chunk servers are used to store the data in uniformly sized chunks. Each chunk is identified by globally unique 64 bit handle assigned by master at the time of creation. The chunks are split into 64KB blocks, each with its own hashsum for data integrity checks. The chunks are replicated between servers, 3 by default. The master maintains all the file system metadata which includes the namespace and chunk location.

Each of the chunks is 64 MB large (contrast this to the typical filesystem sectors of 512 or 4096 bytes) as the system is meant to hold enormous amount of data - namely the internet. The large chunk size is also important for the scalability of the system - the larger the chunk size, the less metadata the master server has to store for any given amount of data. With the current size, the master server is able to store the entirety of the metadata in memory, increasing performance by a significant margin.

== Operation ==

Master and Chunk server communication consists of
# checking whether there any chunk-server is down
# checking if any file is corrupted
# deleting stale chunks

When a client wants to do some operations on the chunks
# it first asks the master server for the list of servers that store the parts of a file it wants to access
# it receives a list of chunk servers, with multiple servers for each chunk
# it finally communicates with the the chunk servers to perform the operation

The system is geared towards appends and sequential reads. This is why the master server responds with multiple server addresses for each chunk - the client can then request a small piece from each server, increasing the data throughput linearly with the number of servers. Writes, in general, are in the form of a special ''append'' system call. When appending, there is no chance that two clients will want to write to the same location at the same time. This helps avoid any potential synchronization issues. If there are multiple appends to the same file at the same time, the chunk servers are free to order them as they wish (chunks on each server are not guaranteed to be byte-for-byte identical). Changes may also be applied multiple times. These issues are left for the application using GFS to resolve themselves. While a problem in the general sense, this is good enough for Google's needs.

== Redundancy ==

GFS is built with failure in mind. The system expects that at any time, there is some server or disk that is malfunctioning. The system deals with the failures as follows.

=== Chunk Servers ===

By default, chunks are replicated to three servers. This exact number depends on the application in question doing the write. When a chunk server finds that some of its data is corrupt, it grabs the data from other servers to repair itself{{Citation needed}}.

=== Master Server ===

For efficiency, there is only a single live master server at a time. While not making the system completely distributed, it avoid many synchronization problems and suits Google's needs. At any point in time, there are multiple read-only master servers that copy metadata from the currently live master. Should it go down, they will serve any read operations from the clients, until one of the hot spares is promoted to being the new live master server.

DistOS 2015W Session 6

2015-04-12T18:37:04Z

Veenarose: /* OCEAN STORE */

==Midterm==

The midterm from last year [http://homeostasis.scs.carleton.ca/~soma/distos/2015w/comp4000-2014w-midterm.pdf is now available].

==FARSITE==

Farsite is a secure, scalable file system that logically functions as a centralized file server but is physically distributed among a set of untrusted computers. Farsite provides file availability and reliability through randomized replicated storage; it ensures the secrecy of file contents with cryptographic techniques; it maintains the integrity of file and directory data with a Byzantine-fault-tolerant protocol; it is designed to be scalable by using a distributed hint mechanism and delegation certificates for pathname translations; and it achieves good performance by locally caching file data, lazily propagating file updates, and varying the duration and granularity of content leases. We report on the design of Farsite and the lessons we have learned by implementing much of that design.

[http://research.microsoft.com/apps/pubs/default.aspx?id=67917]

==OCEAN STORE==
Link to a mini review on the OceanStore Paper...

[http://www.cse.lehigh.edu/~brian/course/advanced-networking/reviews/manura-oceanstore.html]

==Group 1==

'''Team:''' Kirill, Jamie, Alexis, Veena, Khaled, Hassan

{| class="wikitable"
|-
!
! FARSITE
! OceanStore
|-
! Fault Tolerance
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
|-
! Cryptography
| Trusted Certificates
| A strong cryptographic algorithm on read-only operations
|-
! Implementation
| Did not mention what programming they used, but it was based on Windows. They did not implement the file system
| Implemented in Java
|-
! Scalability
| Scalable to a University or large corporations, maximum 105
| Worldwide scalability, maximum 1010
|-
! File Usage
| Was designed for general purpose files
| Was designed for small file sizes
|-
! Scope
| All clients sharing the available resources
| Transient centralized service
|-
! Object Model
| Didn't use the object model
| Used the object model
|}

==Group 2==
'''Team''': Apoorv, Ambalica, Ashley, Eric, Mert, Shivjot

<table style="width: 100%;" border="1" cellpadding="4" cellspacing="0" class="wikitable">
<tr>
<td>'''Farsite'''</td>
<td>'''OceanStore'''</td>
</tr>
<tr>
<td>Implemented Content Leases</td>
<td>Update Model handled data consistency, no Leases</td>
</tr>
<tr>
<td>Single tier, peer to peer model</td>
<td>Two tier, server client model</td>
</tr>
<tr>
<td>Scope of ten to the five</td>
<td>Global Scope (ten to the ten)</td>
</tr>
<tr>
<td>Cryptographic public, private key security</td>
<td>Read and write privileges</td>
</tr>
<tr>
<td>Randomized data replication</td>
<td>Nomadic Data concept</td>
</tr>
</table>

==Group 3==
'''Team''': DANY, MOE, DEEP, SAMEER, TROY

----
'''
== FARSITE ==
'''

'''Security'''
• Cascading certificates system through directory hierarchy
• Keys
• Three types of certificates
• CFS required to authorized certificate
• Because directory groups only modify their shared state via a Byzantine-fault-tolerant protocol, we trust the group not to make
an incorrect update to directory metadata. This metadata includes an access control list (ACL) of public keys of all users
who are authorized writers to that directory and to files in it
• Both file content and user-sensitive metadata (meaning file and directory names) are encrypted for privacy.

'''System Architecture'''
• Client Monitor, directory group, file host
• When space runs out in directory group, delegate’s ownership to sub tree to other delegate group.

----
'''
== OCEANSTORE ==
'''

'''Security'''
• GUID and ACLs used for write, encryption used for reads.
• To prevent unauthorized reads, it encrypts
all data in the system that is not completely public and distributes the encryption key to those users with read permission

DistOS 2015W Session 6

2015-04-12T18:36:39Z

Veenarose: /* OCEAN STORE */

==Midterm==

The midterm from last year [http://homeostasis.scs.carleton.ca/~soma/distos/2015w/comp4000-2014w-midterm.pdf is now available].

==FARSITE==

Farsite is a secure, scalable file system that logically functions as a centralized file server but is physically distributed among a set of untrusted computers. Farsite provides file availability and reliability through randomized replicated storage; it ensures the secrecy of file contents with cryptographic techniques; it maintains the integrity of file and directory data with a Byzantine-fault-tolerant protocol; it is designed to be scalable by using a distributed hint mechanism and delegation certificates for pathname translations; and it achieves good performance by locally caching file data, lazily propagating file updates, and varying the duration and granularity of content leases. We report on the design of Farsite and the lessons we have learned by implementing much of that design.

[http://research.microsoft.com/apps/pubs/default.aspx?id=67917]

==OCEAN STORE==
Linc to a mini review on the OceanStore Paper...

[http://www.cse.lehigh.edu/~brian/course/advanced-networking/reviews/manura-oceanstore.html]

==Group 1==

'''Team:''' Kirill, Jamie, Alexis, Veena, Khaled, Hassan

{| class="wikitable"
|-
!
! FARSITE
! OceanStore
|-
! Fault Tolerance
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
|-
! Cryptography
| Trusted Certificates
| A strong cryptographic algorithm on read-only operations
|-
! Implementation
| Did not mention what programming they used, but it was based on Windows. They did not implement the file system
| Implemented in Java
|-
! Scalability
| Scalable to a University or large corporations, maximum 105
| Worldwide scalability, maximum 1010
|-
! File Usage
| Was designed for general purpose files
| Was designed for small file sizes
|-
! Scope
| All clients sharing the available resources
| Transient centralized service
|-
! Object Model
| Didn't use the object model
| Used the object model
|}

==Group 2==
'''Team''': Apoorv, Ambalica, Ashley, Eric, Mert, Shivjot

<table style="width: 100%;" border="1" cellpadding="4" cellspacing="0" class="wikitable">
<tr>
<td>'''Farsite'''</td>
<td>'''OceanStore'''</td>
</tr>
<tr>
<td>Implemented Content Leases</td>
<td>Update Model handled data consistency, no Leases</td>
</tr>
<tr>
<td>Single tier, peer to peer model</td>
<td>Two tier, server client model</td>
</tr>
<tr>
<td>Scope of ten to the five</td>
<td>Global Scope (ten to the ten)</td>
</tr>
<tr>
<td>Cryptographic public, private key security</td>
<td>Read and write privileges</td>
</tr>
<tr>
<td>Randomized data replication</td>
<td>Nomadic Data concept</td>
</tr>
</table>

==Group 3==
'''Team''': DANY, MOE, DEEP, SAMEER, TROY

----
'''
== FARSITE ==
'''

'''Security'''
• Cascading certificates system through directory hierarchy
• Keys
• Three types of certificates
• CFS required to authorized certificate
• Because directory groups only modify their shared state via a Byzantine-fault-tolerant protocol, we trust the group not to make
an incorrect update to directory metadata. This metadata includes an access control list (ACL) of public keys of all users
who are authorized writers to that directory and to files in it
• Both file content and user-sensitive metadata (meaning file and directory names) are encrypted for privacy.

'''System Architecture'''
• Client Monitor, directory group, file host
• When space runs out in directory group, delegate’s ownership to sub tree to other delegate group.

----
'''
== OCEANSTORE ==
'''

'''Security'''
• GUID and ACLs used for write, encryption used for reads.
• To prevent unauthorized reads, it encrypts
all data in the system that is not completely public and distributes the encryption key to those users with read permission

DistOS 2015W Session 6

2015-04-12T18:35:12Z

Veenarose: /* FARSITE */

==Midterm==

The midterm from last year [http://homeostasis.scs.carleton.ca/~soma/distos/2015w/comp4000-2014w-midterm.pdf is now available].

==FARSITE==

Farsite is a secure, scalable file system that logically functions as a centralized file server but is physically distributed among a set of untrusted computers. Farsite provides file availability and reliability through randomized replicated storage; it ensures the secrecy of file contents with cryptographic techniques; it maintains the integrity of file and directory data with a Byzantine-fault-tolerant protocol; it is designed to be scalable by using a distributed hint mechanism and delegation certificates for pathname translations; and it achieves good performance by locally caching file data, lazily propagating file updates, and varying the duration and granularity of content leases. We report on the design of Farsite and the lessons we have learned by implementing much of that design.

[http://research.microsoft.com/apps/pubs/default.aspx?id=67917]

==OCEAN STORE==
“OceanStore: An Architecture for Global-Scale Persistent Storage” proposes an architecture for a global, durable, and highly available persistent storage network composed of untrusted servers. Data can migrate and replicate to where it is most needed, thereby having similar benefits to caching. For security, all stored data in encrypted. A type of routing mechanism is needed on top of IP to locate OceanStore data, and this is achieved by two methods: an attenuated Bloom filter (i.e. a vector of Bloom filters per node, where each element locates objects at a given graph distance) and, if the previous method fails, a highly redundant version of the Plaxton scheme. Various error-correcting schemes are also used. The network supports a type of introspection, in which the network can automatically tune itself to network conditions. In addition to providing local access to global data, the system is a guard against natural disasters and denial-of-service (DOS) attacks.

[http://www.cse.lehigh.edu/~brian/course/advanced-networking/reviews/manura-oceanstore.html]

==Group 1==

'''Team:''' Kirill, Jamie, Alexis, Veena, Khaled, Hassan

{| class="wikitable"
|-
!
! FARSITE
! OceanStore
|-
! Fault Tolerance
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
|-
! Cryptography
| Trusted Certificates
| A strong cryptographic algorithm on read-only operations
|-
! Implementation
| Did not mention what programming they used, but it was based on Windows. They did not implement the file system
| Implemented in Java
|-
! Scalability
| Scalable to a University or large corporations, maximum 105
| Worldwide scalability, maximum 1010
|-
! File Usage
| Was designed for general purpose files
| Was designed for small file sizes
|-
! Scope
| All clients sharing the available resources
| Transient centralized service
|-
! Object Model
| Didn't use the object model
| Used the object model
|}

==Group 2==
'''Team''': Apoorv, Ambalica, Ashley, Eric, Mert, Shivjot

<table style="width: 100%;" border="1" cellpadding="4" cellspacing="0" class="wikitable">
<tr>
<td>'''Farsite'''</td>
<td>'''OceanStore'''</td>
</tr>
<tr>
<td>Implemented Content Leases</td>
<td>Update Model handled data consistency, no Leases</td>
</tr>
<tr>
<td>Single tier, peer to peer model</td>
<td>Two tier, server client model</td>
</tr>
<tr>
<td>Scope of ten to the five</td>
<td>Global Scope (ten to the ten)</td>
</tr>
<tr>
<td>Cryptographic public, private key security</td>
<td>Read and write privileges</td>
</tr>
<tr>
<td>Randomized data replication</td>
<td>Nomadic Data concept</td>
</tr>
</table>

==Group 3==
'''Team''': DANY, MOE, DEEP, SAMEER, TROY

----
'''
== FARSITE ==
'''

'''Security'''
• Cascading certificates system through directory hierarchy
• Keys
• Three types of certificates
• CFS required to authorized certificate
• Because directory groups only modify their shared state via a Byzantine-fault-tolerant protocol, we trust the group not to make
an incorrect update to directory metadata. This metadata includes an access control list (ACL) of public keys of all users
who are authorized writers to that directory and to files in it
• Both file content and user-sensitive metadata (meaning file and directory names) are encrypted for privacy.

'''System Architecture'''
• Client Monitor, directory group, file host
• When space runs out in directory group, delegate’s ownership to sub tree to other delegate group.

----
'''
== OCEANSTORE ==
'''

'''Security'''
• GUID and ACLs used for write, encryption used for reads.
• To prevent unauthorized reads, it encrypts
all data in the system that is not completely public and distributes the encryption key to those users with read permission

DistOS 2015W Session 6

2015-04-12T18:32:27Z

Veenarose: /* Midterm */

==Midterm==

The midterm from last year [http://homeostasis.scs.carleton.ca/~soma/distos/2015w/comp4000-2014w-midterm.pdf is now available].

==FARSITE==

Farsite is a secure, scalable file system that logically functions as a centralized file server but is physically distributed among a set of untrusted computers. Farsite provides file availability and reliability through randomized replicated storage; it ensures the secrecy of file contents with cryptographic techniques; it maintains the integrity of file and directory data with a Byzantine-fault-tolerant protocol; it is designed to be scalable by using a distributed hint mechanism and delegation certificates for pathname translations; and it achieves good performance by locally caching file data, lazily propagating file updates, and varying the duration and granularity of content leases. We report on the design of Farsite and the lessons we have learned by implementing much of that design.

[http://research.microsoft.com/apps/pubs/default.aspx?id=67917]

==Group 1==

'''Team:''' Kirill, Jamie, Alexis, Veena, Khaled, Hassan

{| class="wikitable"
|-
!
! FARSITE
! OceanStore
|-
! Fault Tolerance
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
|-
! Cryptography
| Trusted Certificates
| A strong cryptographic algorithm on read-only operations
|-
! Implementation
| Did not mention what programming they used, but it was based on Windows. They did not implement the file system
| Implemented in Java
|-
! Scalability
| Scalable to a University or large corporations, maximum 105
| Worldwide scalability, maximum 1010
|-
! File Usage
| Was designed for general purpose files
| Was designed for small file sizes
|-
! Scope
| All clients sharing the available resources
| Transient centralized service
|-
! Object Model
| Didn't use the object model
| Used the object model
|}

==Group 2==
'''Team''': Apoorv, Ambalica, Ashley, Eric, Mert, Shivjot

<table style="width: 100%;" border="1" cellpadding="4" cellspacing="0" class="wikitable">
<tr>
<td>'''Farsite'''</td>
<td>'''OceanStore'''</td>
</tr>
<tr>
<td>Implemented Content Leases</td>
<td>Update Model handled data consistency, no Leases</td>
</tr>
<tr>
<td>Single tier, peer to peer model</td>
<td>Two tier, server client model</td>
</tr>
<tr>
<td>Scope of ten to the five</td>
<td>Global Scope (ten to the ten)</td>
</tr>
<tr>
<td>Cryptographic public, private key security</td>
<td>Read and write privileges</td>
</tr>
<tr>
<td>Randomized data replication</td>
<td>Nomadic Data concept</td>
</tr>
</table>

==Group 3==
'''Team''': DANY, MOE, DEEP, SAMEER, TROY

----
'''
== FARSITE ==
'''

'''Security'''
• Cascading certificates system through directory hierarchy
• Keys
• Three types of certificates
• CFS required to authorized certificate
• Because directory groups only modify their shared state via a Byzantine-fault-tolerant protocol, we trust the group not to make
an incorrect update to directory metadata. This metadata includes an access control list (ACL) of public keys of all users
who are authorized writers to that directory and to files in it
• Both file content and user-sensitive metadata (meaning file and directory names) are encrypted for privacy.

'''System Architecture'''
• Client Monitor, directory group, file host
• When space runs out in directory group, delegate’s ownership to sub tree to other delegate group.

----
'''
== OCEANSTORE ==
'''

'''Security'''
• GUID and ACLs used for write, encryption used for reads.
• To prevent unauthorized reads, it encrypts
all data in the system that is not completely public and distributes the encryption key to those users with read permission

DistOS 2015W Session 7

2015-04-12T18:27:47Z

Veenarose: /* Chubby */

= Ceph =
Unlike GFS that was talked about previously, this is a general purpose distributed file system. It follows the same general model of distribution as GFS and Amoeba.

HOW IT WORKS
Ceph file system runs on top of the same object storage system that provides object storage and block device interfaces. The Ceph metadata server cluster provides a service that maps the directories and filenames of the file system to objects stored within RADOS clusters. The metadata server cluster can expand or contract, and it can rebalance the file system dynamically to distribute data evenly among cluster hosts. This ensures high performance and prevents heavy loads on specific hosts within the cluster.

BENEFITS
The Ceph file system provides numerous benefits:
It provides stronger data safety for mission-critical applications.
It provides virtually unlimited storage to file systems.
Applications that use file systems can use Ceph FS with POSIX semantics. No integration or customization required!
Ceph automatically balances the file system to deliver maximum performance.

== Advantages of using File systems ==
* It supports heterogeneous operating systems including all flavors of the unix operating system as well as Linux and windows
* Multiple client machines can access a single resource simultaneously.
enables sharing common application binaries and read only information instead of putting them on each single machine. This results in reduced overall disk storage cost and administration overhead.
*Gives access to uniform data to groups of users.
*Useful when many users exists on many systems with each user's home directory located on every single machine. Network file systems allows you to all users home directories on a single machine under /home

== Main Components ==
* Client

* Cluster of Object Storage Devices (OSD)
** It basically stores data and metadata and clients communicate directly with it to perform IO operations
** Data is stored in objects (variable size chunks)

* Meta-data Server (MDS)
** It is used to manage the file and directories. Clients interact with it to perform metadata operations like open, rename. It manages the capabilities of a client.
** Clients '''do not''' need to access MDSs to find where data is stored, improving scalability (more on that below)

== Key Features ==
* Decoupled data and metadata

* Dynamic Distributed Metadata Management

** It distributes the metadata among multiple metadata servers using dynamic sub-tree partitioning, meaning folders that get used more often get their meta-data replicated to more servers, spreading the load. This happens completely automatically

* Object based storage
** Uses cluster of OSDs to form a Reliable Autonomic Distributed Object-Store (RADOS) for Ceph failure detection and recovery

* CRUSH (Controlled, Replicated, Under Scalable, Hashing)
** The hashing algorithm used to calculate the location of object instead of looking for them
** This significantly reduces the load on the MDSs because each client has enough information to independently determine where things should be located.
** Responsible for automatically moving data when ODSs are added or removed (can be simplified as ''location = CRUSH(filename) % num_servers'')
** The CRUSH paper on Ceph’s website can be [http://ceph.com/papers/weil-crush-sc06.pdf viewed here]

* RADOS (Reliable Autonomic Distributed Object-Store) is the object store for Ceph

= Chubby =
It is a coarse grained lock service, made and used internally by Google, that serves multiple clients with small number of servers (chubby cell).

Chubby was designed to be a lock service, a distributed system that clients could connect to and share access to small files. The servers providing the system are partitioned into a variety of cells and access for a particular file is managed through one elected master node in one cell. This master makes all decisions and informs the rest of the cell nodes of that decision. If the master fails, the other nodes elect a new master. The problem of asynchronous consensus is solved through the use of timeouts as a failure detector. To avoid the scaling problem of a single bottleneck, the number of cells can be increased with the cost of making some cells smaller.

== System Components ==
* Chubby Cell
** Handles the actual locks
** Mainly consists of 5 servers known as replicas
** Consensus protocol ([https://en.wikipedia.org/wiki/Paxos_(computer_science) Paxos]) is used to elect the master from replicas

* Client
** Used by programs to request and use locks

== Main Features ==
* Implemented as a semi POSIX-compliant file-system with a 256KB file limit
** Permissions are for files only, not folders, thus breaking some compatibility
** Trivial to use by programs; just use the standard ''fopen()'' family of calls

* Uses consensus algorithm (paxos) among a set of servers to agree on who is the master that is in charge of the metadata

* Meant for locks that last hours or days, not seconds (thus, "coarse grained")

* A master server can handle tens of thousands of simultaneous connections
** This can be further improved by using caching servers as most of the traffic are keep-alive messages

* Used by GFS for electing master server

* Also used by Google as a nameserver

== Issues ==
* Due to the use of Paxos (the only algorithm for this problem), the chubby cell is limited to only 5 servers. While this limits fault-tolerance, in practice this is more than enough.
* Since it has a file-system client interface, many programmers tend to abuse the system and need education (even inside Google)
* Clients need to constantly ping Chubby to verify that they still exist. This is to ensure that a server disappearing while holding a lock does not indefinitely hold that lock.
* Clients must also consequently re-verify that they hold a lock that they think they hold because Chubby may have timed them out.

DistOS 2015W Session 7

2015-04-11T19:25:31Z

Veenarose: /* Ceph */

= Ceph =
Unlike GFS that was talked about previously, this is a general purpose distributed file system. It follows the same general model of distribution as GFS and Amoeba.

HOW IT WORKS
Ceph file system runs on top of the same object storage system that provides object storage and block device interfaces. The Ceph metadata server cluster provides a service that maps the directories and filenames of the file system to objects stored within RADOS clusters. The metadata server cluster can expand or contract, and it can rebalance the file system dynamically to distribute data evenly among cluster hosts. This ensures high performance and prevents heavy loads on specific hosts within the cluster.

BENEFITS
The Ceph file system provides numerous benefits:
It provides stronger data safety for mission-critical applications.
It provides virtually unlimited storage to file systems.
Applications that use file systems can use Ceph FS with POSIX semantics. No integration or customization required!
Ceph automatically balances the file system to deliver maximum performance.

== Advantages of using File systems ==
* It supports heterogeneous operating systems including all flavors of the unix operating system as well as Linux and windows
* Multiple client machines can access a single resource simultaneously.
enables sharing common application binaries and read only information instead of putting them on each single machine. This results in reduced overall disk storage cost and administration overhead.
*Gives access to uniform data to groups of users.
*Useful when many users exists on many systems with each user's home directory located on every single machine. Network file systems allows you to all users home directories on a single machine under /home

== Main Components ==
* Client

* Cluster of Object Storage Devices (OSD)
** It basically stores data and metadata and clients communicate directly with it to perform IO operations
** Data is stored in objects (variable size chunks)

* Meta-data Server (MDS)
** It is used to manage the file and directories. Clients interact with it to perform metadata operations like open, rename. It manages the capabilities of a client.
** Clients '''do not''' need to access MDSs to find where data is stored, improving scalability (more on that below)

== Key Features ==
* Decoupled data and metadata

* Dynamic Distributed Metadata Management

** It distributes the metadata among multiple metadata servers using dynamic sub-tree partitioning, meaning folders that get used more often get their meta-data replicated to more servers, spreading the load. This happens completely automatically

* Object based storage
** Uses cluster of OSDs to form a Reliable Autonomic Distributed Object-Store (RADOS) for Ceph failure detection and recovery

* CRUSH (Controlled, Replicated, Under Scalable, Hashing)
** The hashing algorithm used to calculate the location of object instead of looking for them
** This significantly reduces the load on the MDSs because each client has enough information to independently determine where things should be located.
** Responsible for automatically moving data when ODSs are added or removed (can be simplified as ''location = CRUSH(filename) % num_servers'')
** The CRUSH paper on Ceph’s website can be [http://ceph.com/papers/weil-crush-sc06.pdf viewed here]

* RADOS (Reliable Autonomic Distributed Object-Store) is the object store for Ceph

= Chubby =
It is a coarse grained lock service, made and used internally by Google, that serves multiple clients with small number of servers (chubby cell).

== System Components ==
* Chubby Cell
** Handles the actual locks
** Mainly consists of 5 servers known as replicas
** Consensus protocol ([https://en.wikipedia.org/wiki/Paxos_(computer_science) Paxos]) is used to elect the master from replicas

* Client
** Used by programs to request and use locks

== Main Features ==
* Implemented as a semi POSIX-compliant file-system with a 256KB file limit
** Permissions are for files only, not folders, thus breaking some compatibility
** Trivial to use by programs; just use the standard ''fopen()'' family of calls

* Uses consensus algorithm (paxos) among a set of servers to agree on who is the master that is in charge of the metadata

* Meant for locks that last hours or days, not seconds (thus, "coarse grained")

* A master server can handle tens of thousands of simultaneous connections
** This can be further improved by using caching servers as most of the traffic are keep-alive messages

* Used by GFS for electing master server

* Also used by Google as a nameserver

== Issues ==
* Due to the use of Paxos (the only algorithm for this problem), the chubby cell is limited to only 5 servers. While this limits fault-tolerance, in practice this is more than enough.
* Since it has a file-system client interface, many programmers tend to abuse the system and need education (even inside Google)
* Clients need to constantly ping Chubby to verify that they still exist. This is to ensure that a server disappearing while holding a lock does not indefinitely hold that lock.
* Clients must also consequently re-verify that they hold a lock that they think they hold because Chubby may have timed them out.

DistOS 2015W Session 7

2015-04-11T19:23:48Z

Veenarose: /* Ceph */

= Ceph =
Unlike GFS that was talked about previously, this is a general purpose distributed file system. It follows the same general model of distribution as GFS and Amoeba.

HOW IT WORKS
Ceph file system runs on top of the same object storage system that provides object storage and block device interfaces. The Ceph metadata server cluster provides a service that maps the directories and filenames of the file system to objects stored within RADOS clusters. The metadata server cluster can expand or contract, and it can rebalance the file system dynamically to distribute data evenly among cluster hosts. This ensures high performance and prevents heavy loads on specific hosts within the cluster.

BENEFITS
The Ceph file system provides numerous benefits:
It provides stronger data safety for mission-critical applications.
It provides virtually unlimited storage to file systems.
Applications that use file systems can use Ceph FS with POSIX semantics. No integration or customization required!
Ceph automatically balances the file system to deliver maximum performance.

== Main Components ==
* Client

* Cluster of Object Storage Devices (OSD)
** It basically stores data and metadata and clients communicate directly with it to perform IO operations
** Data is stored in objects (variable size chunks)

* Meta-data Server (MDS)
** It is used to manage the file and directories. Clients interact with it to perform metadata operations like open, rename. It manages the capabilities of a client.
** Clients '''do not''' need to access MDSs to find where data is stored, improving scalability (more on that below)

== Key Features ==
* Decoupled data and metadata

* Dynamic Distributed Metadata Management

** It distributes the metadata among multiple metadata servers using dynamic sub-tree partitioning, meaning folders that get used more often get their meta-data replicated to more servers, spreading the load. This happens completely automatically

* Object based storage
** Uses cluster of OSDs to form a Reliable Autonomic Distributed Object-Store (RADOS) for Ceph failure detection and recovery

* CRUSH (Controlled, Replicated, Under Scalable, Hashing)
** The hashing algorithm used to calculate the location of object instead of looking for them
** This significantly reduces the load on the MDSs because each client has enough information to independently determine where things should be located.
** Responsible for automatically moving data when ODSs are added or removed (can be simplified as ''location = CRUSH(filename) % num_servers'')
** The CRUSH paper on Ceph’s website can be [http://ceph.com/papers/weil-crush-sc06.pdf viewed here]

* RADOS (Reliable Autonomic Distributed Object-Store) is the object store for Ceph

= Chubby =
It is a coarse grained lock service, made and used internally by Google, that serves multiple clients with small number of servers (chubby cell).

== System Components ==
* Chubby Cell
** Handles the actual locks
** Mainly consists of 5 servers known as replicas
** Consensus protocol ([https://en.wikipedia.org/wiki/Paxos_(computer_science) Paxos]) is used to elect the master from replicas

* Client
** Used by programs to request and use locks

== Main Features ==
* Implemented as a semi POSIX-compliant file-system with a 256KB file limit
** Permissions are for files only, not folders, thus breaking some compatibility
** Trivial to use by programs; just use the standard ''fopen()'' family of calls

* Uses consensus algorithm (paxos) among a set of servers to agree on who is the master that is in charge of the metadata

* Meant for locks that last hours or days, not seconds (thus, "coarse grained")

* A master server can handle tens of thousands of simultaneous connections
** This can be further improved by using caching servers as most of the traffic are keep-alive messages

* Used by GFS for electing master server

* Also used by Google as a nameserver

== Issues ==
* Due to the use of Paxos (the only algorithm for this problem), the chubby cell is limited to only 5 servers. While this limits fault-tolerance, in practice this is more than enough.
* Since it has a file-system client interface, many programmers tend to abuse the system and need education (even inside Google)
* Clients need to constantly ping Chubby to verify that they still exist. This is to ensure that a server disappearing while holding a lock does not indefinitely hold that lock.
* Clients must also consequently re-verify that they hold a lock that they think they hold because Chubby may have timed them out.