DistOS 2014W Lecture 10 - Revision history

36chambers: /* Segue on drives and sequential access following GFS section */

2014-04-24T16:04:32Z

Segue on drives and sequential access following GFS section

← Older revision		Revision as of 16:04, 24 April 2014
Line 118:		Line 118:
	*** They do erasure in bulk, in the '''background'''.		*** They do erasure in bulk, in the '''background'''.
	*** Used to be we needed specialized FS for [http://en.wikipedia.org/wiki/Memory_Technology_Device MTDs] to get better performance; though now we have better micro-controllers in some embedded systems to abstract away the hardware.		*** Used to be we needed specialized FS for [http://en.wikipedia.org/wiki/Memory_Technology_Device MTDs] to get better performance; though now we have better micro-controllers in some embedded systems to abstract away the hardware.
	* Architectures that start big, often end up in the smallest things.		* Architectures that start big, often end up in the smallest things, such as in small storage devices.


	== Lookups vs hashing ==		== Lookups vs hashing ==
	One key aspect in the Ceph design is the attempt to replace communication with computation by using hashing based mechanism CRUSH. Following line from Anil epitomizes the general approach that is followed in the field of Computer Science “If one abstraction does not work stick another one in”.		One key aspect in the Ceph design is the attempt to replace communication with computation by using hashing based mechanism CRUSH. Following line from Anil epitomizes the general approach that is followed in the field of Computer Science “If one abstraction does not work stick another one in”.

36chambers: /* GFS */

2014-04-24T15:59:45Z

GFS

← Older revision		Revision as of 15:59, 24 April 2014
Line 17:		Line 17:
	* Mostly seeking (sequentially) through entire file.		* Mostly seeking (sequentially) through entire file.
	* Paper from 2003, mentions still using 100BASE-T links.		* Paper from 2003, mentions still using 100BASE-T links.
	* Data-heavy, metadata light. Contacting the metadata server is a rare event.		* Data-heavy, metadata light (unlike Ceph). Contacting the metadata server is a rare event.
	* Consider hardware failures as normal operating conditions:		* Consider hardware failures as normal operating conditions:
	** uses commodity hardware		** uses commodity hardware

36chambers at 15:56, 24 April 2014

2014-04-24T15:56:17Z

← Older revision		Revision as of 15:56, 24 April 2014
Line 21:		Line 21:
	** uses commodity hardware		** uses commodity hardware
	** All the replication (!)		** All the replication (!)
	** Data checksumming		** Data checksumming (few file systems do checksums)
	* Performance degrades for small random access workload; use other filesystem.		* Performance degrades for small random access workload; use other filesystem.
	* Path of least resistance to scale, not to do something super CS-smart.		* Path of least resistance to scale, not to do something super CS-smart.

Sandy.boon: /* Ceph */

2014-04-02T19:40:32Z

Ceph

← Older revision		Revision as of 19:40, 2 April 2014
Line 102:		Line 102:

	* Storage is composed of variable-length atoms		* Storage is composed of variable-length atoms

			*A very fun video to learn about ceph and OSD file systems -
			https://www.youtube.com/watch?v=C3lxGuAWEWU

	= Class Discussion =		= Class Discussion =

Cdelahou: Clean up some more and added info about the interface and random accesss stuff

2014-02-23T19:50:56Z

Clean up some more and added info about the interface and random accesss stuff

← Older revision		Revision as of 19:50, 23 February 2014
Line 6:		Line 6:
	GFS is a distributed file system designed specifically for Google' needs and they made two assumption while designing GFS:		GFS is a distributed file system designed specifically for Google' needs and they made two assumption while designing GFS:

	# Most of the Data is written in the form of appends ( write at the end of a file)		# Most of the Data is written in the form of appends ( write at the end of a file).
	# Data read from the files is read in a streaming sort of way ( read lot of data in the form of sequential access).		# Data read from the files is read in a streaming sort of way ( read lot of data in the form of sequential access).

	Because of this, they decided to lay emphasis on better performance for sequential access. These two assumption are also the reason because of which they chose to keep the chunk size so huge (64 MB). You can easily read large blocks if you get rid of random access.		Because of this, they decided to lay emphasis on better performance for sequential access. These two assumption are also the reason because of which they chose to keep the chunk size so huge (64 MB). You can easily read large blocks if you get rid of random access. Once data is written, it is rarely written '''over''' using random access.

	* Very different design because of the workload that it is designed for:		* Very different design because of the workload that it is designed for:
	** Because of the number of small files that have to be indexed for the web, it is no longer practical to have a file system that stores these individually. Too much overhead. Punts problem to userspace, incl. record delimitation.		** Because of the number of small files that have to be indexed for the web, it is no longer practical to have a file system that stores these individually. Too much overhead. Easier to store millions of objects as large files. Punts problem to userspace, incl. record delimitation.
	* Don't care about latency, surprising considering it's Google, the guys who change the TCP IW standard recommendations for latency.		* Don't care about latency
			** surprising considering it's Google, the guys who change the TCP IW standard recommendations for latency.
	* Mostly seeking (sequentially) through entire file.		* Mostly seeking (sequentially) through entire file.
	* Paper from 2003, mentions still using 100BASE-T links.		* Paper from 2003, mentions still using 100BASE-T links.
Line 25:		Line 26:
	* Google used to re-index every month, swapping out indexes. Now, it's much more online. GFS is now just a layer to support a more dynamic layer.		* Google used to re-index every month, swapping out indexes. Now, it's much more online. GFS is now just a layer to support a more dynamic layer.
	* The paper seems to lack any mention of security. This FS probably could only exist on a trusted network.		* The paper seems to lack any mention of security. This FS probably could only exist on a trusted network.
			* Implements interface similar to POSIX, but not the full standard.
			** '''create, delete, open, close, read, write'''
			** Unique operations too: '''snapshot''' which is low cost file duplication and '''record append'''

	== How other filesystems compare to GFS and Ceph ==		== How other filesystems compare to GFS and Ceph ==

Cdelahou: woops. sequential => random

2014-02-23T19:39:36Z

woops. sequential => random

← Older revision		Revision as of 19:39, 23 February 2014
Line 9:		Line 9:
	# Data read from the files is read in a streaming sort of way ( read lot of data in the form of sequential access).		# Data read from the files is read in a streaming sort of way ( read lot of data in the form of sequential access).

	Because of this, they decided to lay emphasis on better performance for sequential access. These two assumption are also the reason because of which they chose to keep the chunk size so huge (64 MB). You can easily read large blocks if you get rid of ~~sequential~~ access.		Because of this, they decided to lay emphasis on better performance for sequential access. These two assumption are also the reason because of which they chose to keep the chunk size so huge (64 MB). You can easily read large blocks if you get rid of random access.

	* Very different design because of the workload that it is designed for:		* Very different design because of the workload that it is designed for:

Cdelahou: Added line about EBOFS

2014-02-23T19:25:44Z

Added line about EBOFS

← Older revision		Revision as of 19:25, 23 February 2014
Line 78:		Line 78:
	* Near-Posix like interface: selectively extend interface while relaxing consistency semantics.		* Near-Posix like interface: selectively extend interface while relaxing consistency semantics.
	** ex: ''readdirplus'' is an extension which optimizes for a common sequence of operations: ''readdir'' followed by multiple ''stats''. This requires brief caching to improve performance which may let small concurrent changes to go unnoticed.		** ex: ''readdirplus'' is an extension which optimizes for a common sequence of operations: ''readdir'' followed by multiple ''stats''. This requires brief caching to improve performance which may let small concurrent changes to go unnoticed.
	* ~~Unlike GFS, the~~ Object Storage Devices (OSDs) have some intelligence, and autonomously distribute the data, rather than being controlled by a master.		* Object Storage Devices (OSDs) have some intelligence (unlike GFS), and autonomously distribute the data, rather than being controlled by a master.
			** Uses EBOFS (instead of ext3). Implemented in user space to avoid dealing with kernel issues. Aggressively schedules disk writes.
	** Uses hashing in the distribution process to '''uniformly''' distribute data		** Uses hashing in the distribution process to '''uniformly''' distribute data
	** The actual algorithm for distributing data is as follows:		** The actual algorithm for distributing data is as follows:

Cdelahou: More on distributed meta data

2014-02-23T19:09:51Z

Cdelahou: POSIX w/relaxing

2014-02-23T18:49:25Z

POSIX w/relaxing

← Older revision		Revision as of 18:49, 23 February 2014
Line 72:		Line 72:

	* Unlike GFS, distributes metadata, not just for read-only copies		* Unlike GFS, distributes metadata, not just for read-only copies
	* Near-Posix like interface: selectively extend interface while relaxing consistency semantics.		* Near-Posix like interface: selectively extend interface while relaxing consistency semantics.
			** ex: ''readdirplus'' is an extension which optimizes for a common sequence of operations: ''readdir'' followed by multiple ''stats''. This requires brief caching to improve performance which may let small concurrent changes to go unnoticed.
	* Unlike GFS, the Object Storage Devices (OSDs) have some intelligence, and autonomously distribute the data, rather than being controlled by a master.		* Unlike GFS, the Object Storage Devices (OSDs) have some intelligence, and autonomously distribute the data, rather than being controlled by a master.
	** Uses hashing in the distribution process to '''uniformly''' distribute data		** Uses hashing in the distribution process to '''uniformly''' distribute data

Cdelahou: Added more overview information.

2014-02-23T18:32:33Z

Added more overview information.

← Older revision		Revision as of 18:32, 23 February 2014
Line 65:		Line 65:
	* Ceph is crazy and tries to do everything		* Ceph is crazy and tries to do everything
	* GFS was very specifically designed to work in a limited scenario, under certain specific conditions, whereas CEPH is sort of generic solution- for how to build a scalable distributed file system		* GFS was very specifically designed to work in a limited scenario, under certain specific conditions, whereas CEPH is sort of generic solution- for how to build a scalable distributed file system

			* Achieves high performance, relaibility and availabily through three design features: decoupled data and metadata, dynamically distributed meta data, reliable autonomic distributed object storage.
			** Decoupled data and meta data: Metadata operations (open, close) happen to metadata clusters, clients interact directly with OSD's for IO.
			** Distributed Meta Data: Meta data operations make up a lot of work load. Ceph distributes this workload to many Meta Data Servers (MDS) to maintail a file hierarchy.
			** automic object storage: OSD's organise amongst themselves, taking advantage of their onboard CPU and Memory. Ceph delegates datamigration, replication, failure detection, recovery, to the cluster of OSDs.

	* Unlike GFS, distributes metadata, not just for read-only copies		* Unlike GFS, distributes metadata, not just for read-only copies
	* Unlike GFS, the OSDs have some intelligence, and autonomously distribute the data, rather than being controlled by a master.		* Near-Posix like interface: selectively extend interface while relaxing consistency semantics.
			* Unlike GFS, the Object Storage Devices (OSDs) have some intelligence, and autonomously distribute the data, rather than being controlled by a master.
	** Uses hashing in the distribution process to '''uniformly''' distribute data		** Uses hashing in the distribution process to '''uniformly''' distribute data
	** The actual algorithm for distributing data is as follows:		** The actual algorithm for distributing data is as follows:
Line 85:		Line 92:

	* Storage is composed of variable-length atoms		* Storage is composed of variable-length atoms


	= Class Discussion =		= Class Discussion =

← Older revision		Revision as of 19:09, 23 February 2014
Line 71:		Line 71:
	** automic object storage: OSD's organise amongst themselves, taking advantage of their onboard CPU and Memory. Ceph delegates datamigration, replication, failure detection, recovery, to the cluster of OSDs.		** automic object storage: OSD's organise amongst themselves, taking advantage of their onboard CPU and Memory. Ceph delegates datamigration, replication, failure detection, recovery, to the cluster of OSDs.

	* Unlike GFS, ~~distributes metadata, not just for read-only copies~~		* Distributed Meta Data
			** Unlike GFS
			** Clusters of MDSes.
			** Utilizes Dynamic Subtree partitioning: Dynamically mapped subtrees of directories to MDSes. Workloads for every subtree are monitored. Subtrees assigned to MDSes accordingly, in a coarse way.

	* Near-Posix like interface: selectively extend interface while relaxing consistency semantics.		* Near-Posix like interface: selectively extend interface while relaxing consistency semantics.
	** ex: ''readdirplus'' is an extension which optimizes for a common sequence of operations: ''readdir'' followed by multiple ''stats''. This requires brief caching to improve performance which may let small concurrent changes to go unnoticed.		** ex: ''readdirplus'' is an extension which optimizes for a common sequence of operations: ''readdir'' followed by multiple ''stats''. This requires brief caching to improve performance which may let small concurrent changes to go unnoticed.