DistOS 2015W Session 5 - Revision history

Jasons: /* Google File System */

2015-04-20T10:03:57Z

Google File System

← Older revision		Revision as of 10:03, 20 April 2015
Line 131:		Line 131:

	For efficiency, there is only a single live master server at a time. While not making the system completely distributed, it avoid many synchronization problems and suits Google's needs. At any point in time, there are multiple read-only master servers that copy metadata from the currently live master. Should it go down, they will serve any read operations from the clients, until one of the hot spares is promoted to being the new live master server.		For efficiency, there is only a single live master server at a time. While not making the system completely distributed, it avoid many synchronization problems and suits Google's needs. At any point in time, there are multiple read-only master servers that copy metadata from the currently live master. Should it go down, they will serve any read operations from the clients, until one of the hot spares is promoted to being the new live master server.

			=== Server:Stateless ===
			*the servers does not store states about clients
			*no caching at client either
			**since most program only cares about the output
			**if client wants up-to-date result, rerun the program
			*use heartbeat messages to monitor servers
			**good for system with assumption that changes (or failures) are often

Veenarose: /* Google File System */

2015-04-12T18:41:54Z

Google File System

← Older revision		Revision as of 18:41, 12 April 2015
Line 96:		Line 96:

	Unlike most filesystems, GFS must be implemented by individual applications and is not part of the kernel. While this introduces some technical overhead, it gives the system more freedom to implement or not implement certain non-standard features.		Unlike most filesystems, GFS must be implemented by individual applications and is not part of the kernel. While this introduces some technical overhead, it gives the system more freedom to implement or not implement certain non-standard features.

			Link to an explanation on how GFS works
			[http://computer.howstuffworks.com/internet/basics/google-file-system1.htm]

	== Architecture ==		== Architecture ==

Abeinges: /* Google File System */

2015-04-06T04:56:10Z

Google File System

← Older revision		Revision as of 04:56, 6 April 2015
Line 115:		Line 115:
	# it finally communicates with the the chunk servers to perform the operation		# it finally communicates with the the chunk servers to perform the operation

	The system is geared towards appends and sequential reads. This is why the master server responds with multiple server addresses for each chunk - the client can then request a small piece from each server, increasing the data throughput linearly with the number of servers. Writes, in general, are in the form of a special ''append'' system call. When appending, there is no chance that two clients will want to write to the same location at the same time. This helps avoid any potential synchronization issues. If there are multiple appends to the same file at the same time, the chunk servers are free to order them as they wish (chunks on each server are not guaranteed to be byte-for-byte identical). While a problem in the general sense, this is good enough for Google's needs.		The system is geared towards appends and sequential reads. This is why the master server responds with multiple server addresses for each chunk - the client can then request a small piece from each server, increasing the data throughput linearly with the number of servers. Writes, in general, are in the form of a special ''append'' system call. When appending, there is no chance that two clients will want to write to the same location at the same time. This helps avoid any potential synchronization issues. If there are multiple appends to the same file at the same time, the chunk servers are free to order them as they wish (chunks on each server are not guaranteed to be byte-for-byte identical). Changes may also be applied multiple times. These issues are left for the application using GFS to resolve themselves. While a problem in the general sense, this is good enough for Google's needs.

	== Redundancy ==		== Redundancy ==

Abeinges: /* Operation */

2015-04-06T04:54:02Z

Operation

← Older revision		Revision as of 04:54, 6 April 2015
Line 106:		Line 106:

	Master and Chunk server communication consists of		Master and Chunk server communication consists of
	# checking whether there is any chunk-server is down.		# checking whether there any chunk-server is down
	# checking if any file is corrupted.		# checking if any file is corrupted
	# deleting stale chunks		# deleting stale chunks

	When a client ~~want~~ to do some operations on the chunks		When a client wants to do some operations on the chunks
	# it first asks the master server for the list of servers that store the parts of a file it wants to access		# it first asks the master server for the list of servers that store the parts of a file it wants to access
	# it receives a list of chunk servers, with multiple servers for each chunk		# it receives a list of chunk servers, with multiple servers for each chunk

Abeinges: /* Plan 9 */

2015-04-06T04:51:02Z

Plan 9

Ameti073: /* Similaritieis with the UNIX */

2015-02-11T03:23:25Z

Similaritieis with the UNIX

← Older revision		Revision as of 03:23, 11 February 2015
Line 53:		Line 53:

	== Similaritieis with the UNIX ==		== Similaritieis with the UNIX ==
	<ul><li>~~sdfsdf~~</li></ul>		<ul>
			<li>shell</li>
			<li>Various C compilers</li>
			</ul>

	== Unique Features ==		== Unique Features ==

Ameti073: /* Similaritieis with the UNIX */

2015-02-11T03:22:09Z

Similaritieis with the UNIX

← Older revision		Revision as of 03:22, 11 February 2015
Line 53:		Line 53:

	== Similaritieis with the UNIX ==		== Similaritieis with the UNIX ==
	<li>sdfsdf</li>		<ul><li>sdfsdf</li></ul>

	== Unique Features ==		== Unique Features ==

Ameti073: /* Unix Compatibility */

2015-02-11T03:21:37Z

Unix Compatibility

← Older revision		Revision as of 03:21, 11 February 2015
Line 50:		Line 50:

	The commands, libraries and system calls are similar to that of Unix and therefore a casual user cannot distinguish between these two. The problems in UNIX were too deep to fix but still the various ideas were brought along. The problems addressed badly by UNIX were improved. Old tools were dropped and others were polished and reused.		The commands, libraries and system calls are similar to that of Unix and therefore a casual user cannot distinguish between these two. The problems in UNIX were too deep to fix but still the various ideas were brought along. The problems addressed badly by UNIX were improved. Old tools were dropped and others were polished and reused.


			== Similaritieis with the UNIX ==
			<li>sdfsdf</li>

	== Unique Features ==		== Unique Features ==

Aimbot: Formatted and added to the GFS section.

2015-02-07T15:56:24Z

Formatted and added to the GFS section.

@@ Line 87: / Line 87: @@
 = Google File System =
-It is scalable file system for large distributed data intensive applications. The design is driven by providing previous applications workloads and technical environments, both current and anticipated.
+It is scalable, distributed file system for large, data intensive applications. It is crafted to Google's unique needs as a search engine company.
-The architecture of the Google file system consists of a single master, multiple chunk-servers and multiple clients. These chunk-servers store the data or file in unit of named chunks. Each chunk is identified by globally unique 64 bit chunk handle assigned by master at the end of the time of chunk creation. For more reliability and availability chunks are replicated  on more chunk servers.  The master maintains all the file system meta data which include the name space, chunk location and also the access control information.
+Unlike most filesystems, GFS must be implemented by individual applications and is not part of the kernel. While this introduces some technical overhead, it gives the system more freedom to implement or not implement certain non-standard features.
-Master and Chunk-Server Communication:
+== Architecture ==
-Operation of GFS:
+The architecture of the Google file system consists of a single master, multiple chunk-servers and multiple clients. Chunk servers are used to store the data in uniformly sized chunks. Each chunk is identified by globally unique 64 bit handle assigned by master at the time of creation. The chunks are split into 64KB blocks, each with its own hashsum for data integrity checks. The chunks are replicated between servers, 3 by default. The master maintains all the file system metadata which includes the namespace and chunk location.
-a) Client communicate with master to get the matadata.
-b) client get chunk location from matadata.
+Each of the chunks is 64 MB large (contrast this to the typical filesystem sectors of 512 or 4096 bytes) as the system is meant to hold enormous amount of data - namely the internet. The large chunk size is also important for the scalability of the system - the larger the chunk size, the less metadata the master server has to store for any given amount of data. With the current size, the master server is able to store the entirety of the metadata in memory, increasing performance by a significant margin.
-c) Communicate with the one of that chunk-server to retrieve the data to perform operations on it.

Aimbot: Formatted and edited Plan 9 section

2015-02-07T15:23:36Z

Formatted and edited Plan 9 section

← Older revision		Revision as of 15:23, 7 February 2015
Line 37:		Line 37:
	= Plan 9 =		= Plan 9 =

	Plan 9 is a general purpose, ~~multiuser~~ and mobile computing environment physically distributed across machines.		Plan 9 is a general purpose, multi-user and mobile computing environment physically distributed across machines. Development of the system began in the late 1980s. The system was built at Bell Labs - the birth place of Unix. The original Unix OS had no support for networking, and there were many attempts over the years by others to create distributed systems with Unix compatibility. Plan 9, however, is distributed systems done following the original Unix philosophy.
	~~The Plan 9~~ began in late 1980s. The ~~aims of this~~ system ~~are:~~
	~~1) To~~ built ~~a system that should be centrally administered~~
	~~2) Cost effective using cheap modern microcomputers~~.
	The ~~distribution itself is transparent~~ to ~~most programs~~.
	~~This property~~ is ~~made possible by 2 properties:~~
	~~1) A per process group name space~~
	~~2) A uniform access to all~~ the ~~resources by representing them as a file~~.

	~~It is quite similar to the Unix yet so different.~~ The ~~commands, libraries and~~ system ~~calls are similar to~~ that ~~of Unix and therefore a casual user cannot distinguish between these two~~. ~~The problems in UNIX were too deep to fix but still the various ideas were brought along. The problems addressed badly by UNIX were improved. Old tools were dropped and others were polished and reused~~.		The goals of this system were:
			# To built a distributed system that can be centrally administered.
			# Be cost effective using cheap, modern microcomputers.

	~~What actually distinguishes Plan 9~~ is ~~its '''organization'''~~.		The distribution itself is transparent to most programs. This property is made possible by 2 properties:
			# A per process group namespace.
			# Uniform access to most resources by representing them as a file.

	Plan 9 is divided along the lines of service function.		== Unix Compatibility ==
	* CPU services and terminals use same kernel
	* Users may choose to run programs locally or remotely on CPU servers		The commands, libraries and system calls are similar to that of Unix and therefore a casual user cannot distinguish between these two. The problems in UNIX were too deep to fix but still the various ideas were brought along. The problems addressed badly by UNIX were improved. Old tools were dropped and others were polished and reused.
	*~~Gives~~ the user ~~a choice to~~ choose whether they want distributed or centralized.
			== Unique Features ==

			What actually distinguishes Plan 9 is its '''organization'''. Plan 9 is divided along the lines of service function.
			* CPU services and terminals use same kernel.
			* Users may choose to run programs locally or remotely on CPU servers.
			* It lets the user choose whether they want a distributed or centralized system.

	The design of Plan 9 is based on 3 principles:		The design of Plan 9 is based on 3 principles:
	1) Resources are named and accessed like files in hierarchical file system.		# Resources are named and accessed like files in hierarchical file system.
	2) Standard protocol 9P		# Standard protocol 9P.
	3) Disjoint hierarchical provided by different services are joined together into single private ~~hierarchal~~ file name space.		# Disjoint hierarchical provided by different services are joined together into single private hierarchical file name space.

			=== Virtual Namespaces ===

			In a virtual namespace, a user boots a terminal or connects to a CPU server and then a new process group is created. Processes in group can either add to or rearrange their name space using two system calls - mount and bind.

	~~Another concept in Plan 9 is the '''Virtual Name Space'''~~
	~~In a '''Virtual Name Space''', a user boots a terminal or connects to a CPU server and then a new process group is created.~~
	~~Processes in group can either add to or rearrange their name space using two system calls- [[Mount]] and [[Bind]]~~
	* '''Mount''' is used to attach new file system to a point in name space.		* '''Mount''' is used to attach new file system to a point in name space.
	*'''Bind''' is used to attach a kernel resident file system to name space and also arrange pieces of name space.		* '''Bind''' is used to attach a kernel resident (existing, mounted) file system to name space and also arrange pieces of name space.
			* There is also '''unbind''' which undoes the effects of the other two calls.

	~~The plan~~ 9 ~~provides mechanism to customize one's view of the system with the help of the software rather than the hardware~~.		Namespaces in Plan 9 are on a per-process basis. While everything had a way reference resources with a unique name, using mount and bind, every process could also build their namespace as they saw fit.
	~~It is built for the traditional system but it can be extended to the other~~ resources.

	'''Parallel Programming~~'''~~		Since most resources are in the form of files (and folders), the term ''namespace'' really only refers to the filesystem layout.

			=== Parallel Programming ===
	The parallel programming has two aspects:		The parallel programming has two aspects:
	* Kernel provides simple process model and carefully designed system calls for synchronization.		* Kernel provides simple process model and carefully designed system calls for synchronization.
	*Programming language supports concurrent programming.		* Programming language supports concurrent programming.

			== Legacy ==

	''~~'Implementation of Name Spaces~~'''		Even though Plan 9 is no longer developed, the good ideas from the system still exist today. For example, the ''/proc'' virtual filesystem which displays current process information in the form of files exists in moden Linux kernels.
	~~User processes construct name specs using three system calls- mount, bind, unmount.~~
	~~Mount- System call attaches a tree served by a file server to~~ the ~~current name specs~~
	~~Bind-Duplicates pieces~~ of ~~existing name specs at another point~~
	~~Unmount- Allows components to be removed~~.

← Older revision		Revision as of 04:51, 6 April 2015
Line 40:		Line 40:

	The goals of this system were:		The goals of this system were:
	# To ~~built~~ a distributed system that can be centrally administered.		# To build a distributed system that can be centrally administered.
	# Be cost effective using cheap, modern microcomputers.		# To be cost effective using cheap, modern microcomputers.

	The distribution itself is transparent to most programs. This ~~property~~ is made possible by 2 properties:		The distribution itself is transparent to most programs. This is made possible by two properties:
	# A per process group namespace.		# A per process group namespace.
	# Uniform access to most resources by representing them as a file.		# Uniform access to most resources by representing them as a file.
Line 78:		Line 78:
	* There is also '''unbind''' which undoes the effects of the other two calls.		* There is also '''unbind''' which undoes the effects of the other two calls.

	Namespaces in Plan 9 are on a per-process basis. While everything had a way reference resources with a unique name, using mount and bind, every process could ~~also~~ build ~~their~~ namespace as they saw fit.		Namespaces in Plan 9 are on a per-process basis. While everything had a way reference resources with a unique name, using mount and bind every process could build a custom namespace as they saw fit.

	Since most resources are in the form of files (and folders), the term ''namespace'' really only refers to the filesystem layout.		Since most resources are in the form of files (and folders), the term ''namespace'' really only refers to the filesystem layout.

	=== Parallel Programming ===		=== Parallel Programming ===
	~~The parallel~~ programming ~~has~~ two ~~aspects~~:		Parallel programming was supported in two ways:
	* Kernel provides simple process model and carefully designed system calls for synchronization.		* Kernel provides simple process model and carefully designed system calls for synchronization.
	* Programming language supports concurrent programming.		* Programming language supports concurrent programming.
Line 89:		Line 89:
	== Legacy ==		== Legacy ==

	Even though Plan 9 is no longer developed, the good ideas from the system still exist today. For example, the ''/proc'' virtual filesystem which displays current process information in the form of files exists in ~~moden~~ Linux kernels.		Even though Plan 9 is no longer developed, the good ideas from the system still exist today. For example, the ''/proc'' virtual filesystem which displays current process information in the form of files exists in modern Linux kernels.


	= Google File System =		= Google File System =