DistOS-2011W BigTable - Revision history

Mdpless2: /* References */

2011-03-03T17:11:26Z

References

← Older revision		Revision as of 17:11, 3 March 2011
Line 243:		Line 243:

	=References=		=References=

	~~Apache Hadoop. Apache. (February 23, 2011) [http://hadoop.apache.org/]~~

	~~Apache Hadoop Wiki. Apache. (February 23, 2011) [http://wiki.apache.org/hadoop/]~~

	Bhandarkar, M. Practical Problem Solving With Apache-Hadoop. Yahoo Inc. (February 11, 2011) [http://www.slideshare.net/hadoop/practical-problem-solving-with-apache-hadoop-pig]		Bhandarkar, M. Practical Problem Solving With Apache-Hadoop. Yahoo Inc. (February 11, 2011) [http://www.slideshare.net/hadoop/practical-problem-solving-with-apache-hadoop-pig]
Line 257:		Line 253:

	Distributed Systems. Google Code University. (February 23, 2011) [http://code.google.com/edu/parallel/]		Distributed Systems. Google Code University. (February 23, 2011) [http://code.google.com/edu/parallel/]

			Hadoop. Apache. (February 23, 2011) [http://hadoop.apache.org/]

	Hadoop. Wikipedia. (February 23, 2011) [http://en.wikipedia.org/wiki/Hadoop]		Hadoop. Wikipedia. (February 23, 2011) [http://en.wikipedia.org/wiki/Hadoop]

			Hadoop Wiki. Apache. (February 23, 2011) [http://wiki.apache.org/hadoop/]

			HBase. Apache (February 25, 2011) [http://hbase.apache.org/]

Mdpless2: /* Conclusion */

2011-03-03T17:09:16Z

Conclusion

← Older revision		Revision as of 17:09, 3 March 2011
Line 240:		Line 240:
	=Conclusion=		=Conclusion=

	The installation of a Hadoop gave me a great deal of insight into the workings and design behind the proprietary software used in the Bigtable implementation. Using the various references, tutorials, and documentations I found, I was able to set up a working example of a master/slave computation on a single node as well as set up a cluster running on a single machine across multiple Java processes. Future work that I can do is set up a fully-distributed mini-cluster with applications and files utilizing the processing power and disk space of the nodes. Working directly ~~with a Bigtable implementation like~~ Hadoop has given me a greater understanding of the possible uses of such a system. It is also a very effective way to learn about the reasons behind some of the design decisions that were taken by the development teams. Lastly, using Hadoop was easy-to-use and allowed one to get a system running in a relatively short period of time.		The installation of a Hadoop gave me a great deal of insight into the workings and design behind the proprietary software used in the Bigtable implementation. Using the various references, tutorials, and documentations I found, I was able to set up a working example of a master/slave computation on a single node as well as set up a cluster running on a single machine across multiple Java processes. Future work that I can do is set up a fully-distributed mini-cluster with applications and files utilizing the processing power and disk space of the nodes. Working directly Hadoop has given me a greater understanding of the possible uses of such a system like Bigtable. It is also a very effective way to learn about the reasons behind some of the design decisions that were taken by the development teams. Adhering to the distributed transparencies were key to these systems, especially scalability and fault tolerance. Lastly, using Hadoop was easy-to-use and allowed one to get a system running in a relatively short period of time.

	=References=		=References=

Mdpless2: /* Evaluation of Installation */

2011-03-03T17:06:30Z

Evaluation of Installation

← Older revision		Revision as of 17:06, 3 March 2011
Line 211:		Line 211:
	=Evaluation of Installation=		=Evaluation of Installation=
	==Setup==		==Setup==
	Overall, the configuration and setup of a Hadoop cluster is fairly simple. Difficulties tend to arise on the deployment of applications across the system as was my case. ~~Interestingly, running~~		Overall, the configuration and setup of a Hadoop cluster is fairly simple. Difficulties tend to arise on the deployment of applications across the system as was my case.

	==Deploying applications==		==Deploying applications==
			The simple application that I chose to run over the single-node cluster involved using Eclipse as the development environment to generate a jar file. This jar file was added as input over the command line to the running Hadoop implementation which executed the application over the available nodes.

			Deploying this same application over the pseudo-distributed node proved to be much more challenging and is something that I plan to conquer in my future workings with Hadoop. Creating a true-distributed cluster would also be quite interesting.

	=Discussion=		=Discussion=

Mdpless2 at 17:00, 3 March 2011

2011-03-03T17:00:53Z

← Older revision		Revision as of 17:00, 3 March 2011
Line 165:		Line 165:


	[[File:run-distr-setup.png\|alt=Running a pseudo-distributed setup alt text\|Figure 1: Running a pseudo-distributed setup]]		[[File:run-distr-setup.png\|thumb\|alt=Running a pseudo-distributed setup alt text\|Figure 5: Running a pseudo-distributed setup]]

	For the pseudo-distributed mode, I changed the configuration of the following files:		For the pseudo-distributed mode, I changed the configuration of the following files:
Line 206:		Line 206:

	The result of the distributed computation is unclear here, as the output is not updated in the cluster summary page. I managed to get the clusters up and running, but did not manage to get any meaningful computation running across the separate Java processes.		The result of the distributed computation is unclear here, as the output is not updated in the cluster summary page. I managed to get the clusters up and running, but did not manage to get any meaningful computation running across the separate Java processes.

			[[File:run-distr-namenode.png\|thumb\|alt=Viewing the statistics of applications running on the cluster alt text\|Figure 6: RViewing the statistics of applications running on the cluster]]

	=Evaluation of Installation=		=Evaluation of Installation=

Mdpless2 at 16:58, 3 March 2011

2011-03-03T16:58:33Z

← Older revision		Revision as of 16:58, 3 March 2011
Line 163:		Line 163:

	===Pseudo-distributed mode===		===Pseudo-distributed mode===


			[[File:run-distr-setup.png\|alt=Running a pseudo-distributed setup alt text\|Figure 1: Running a pseudo-distributed setup]]

	For the pseudo-distributed mode, I changed the configuration of the following files:		For the pseudo-distributed mode, I changed the configuration of the following files:

Mdpless2: /* Discussion */

2011-03-03T16:48:33Z

Discussion

← Older revision		Revision as of 16:48, 3 March 2011
Line 226:		Line 226:
	* Secure		* Secure

	Not surprisingly, many of these are components of distributed transparencies.		Not surprisingly, many of these are components of distributed transparencies. What's interesting is how integral the design for failure approach is to the system. Some failure cases have extremely low probability rates of occurring, but in a large distributed system with millions of simultaneous messages over the network, these scenarios have to be accounted for. Choosing to disregard this as a legitimate issue can easily result in a minor problem cascading across entire clusters; hindering its availability.

	As an extension to what I have tried so far, I plan to test and run applications across multiple physical nodes. Ideally I want to make use of the underlying file system and understand fully how the system behaves during reads and writes as well as how it handles node's dropping out due to connectivity issues.		As an extension to what I have tried so far, I plan to test and run applications across multiple physical nodes. Ideally I want to make use of the underlying file system and understand fully how the system behaves during reads and writes as well as how it handles node's dropping out due to connectivity issues.

Mdpless2 at 16:42, 3 March 2011

2011-03-03T16:42:06Z

← Older revision		Revision as of 16:42, 3 March 2011
Line 9:		Line 9:
	=Real World Implementations=		=Real World Implementations=

	Bigtable is a proprietary system at Google, so it is currently only used and implemented by them. There are, however, other implementations that exist in the open source world; notably Hadoop, which is Apache’s implementation of ~~Bigtable~~. It is based on the papers released by Google on their MapReduce and Google File System (GFS). Their framework allows applications to be run on large clusters of commodity hardware. Some organizations that make significant use Hadoop are: Yahoo, eBay, Facebook, Twitter, and IBM.		Bigtable is a proprietary system at Google, so it is currently only used and implemented by them. There are, however, other implementations that exist in the open source world; notably Hadoop, which is Apache’s implementation of MapReduce running on a distributed file system. It is based on the papers released by Google on their MapReduce and Google File System (GFS). Their framework allows applications to be run on large clusters of commodity hardware. Some organizations that make significant use Hadoop are: Yahoo, eBay, Facebook, Twitter, and IBM.

	In the next two sections, we discuss the features and underlying’s of the two frameworks.		In the next two sections, we discuss the features and underlying’s of the two frameworks.
Line 216:		Line 216:
	What is particularly surprising is the fact that Bigtable and Hadoop are designed to run on commodity hardware. This ultimately has resulted in huge cost savings for organizations using the systems. For instance, Google has only minimal incremental costs associated with expanding their services and adding computing power to the clusters. Particularly, since it was developed in-house, they were not forced to pay royalties or buy licenses associated with each machine.		What is particularly surprising is the fact that Bigtable and Hadoop are designed to run on commodity hardware. This ultimately has resulted in huge cost savings for organizations using the systems. For instance, Google has only minimal incremental costs associated with expanding their services and adding computing power to the clusters. Particularly, since it was developed in-house, they were not forced to pay royalties or buy licenses associated with each machine.

	Another interesting part to the design is how they achieve scalability and performance over a large amount of clusters. Each tablet load balances itself over multiple machines. When a tablet becomes the victim of high-demand, the tablet is automatically spread through to other nodes to balance the processing. Furthermore, if a node goes down to a variety of reasons, other nodes will jump in and take a tablet from the downed node. With many nodes in a cluster, the overhead to retrive and restore each tablet is shared across the whole network; each available node picks up one tablet and incurs only minimal delays.		Another interesting part to the design is how they achieve scalability and performance over a large amount of clusters. Each tablet load balances itself over multiple machines. When a tablet becomes the victim of high-demand, the tablet is automatically spread through to other nodes to balance the processing. Furthermore, if a node goes down to a variety of reasons, other nodes will jump in and take a tablet from the downed node. With many nodes in a cluster, the overhead to retrive and restore each tablet is shared across the whole network; each available node picks up one tablet and incurs only minimal delays. Clearly Bigtable and Hadoop were designed with scalability in mind, but for the system to be useful it must also be reliable. The systems are designed to be able to easily add support for larger number of users and resources. This functionality adds a great deal of complexity to the system and hence, achieving complete reliability is extremely difficult. In Google's explanation, they claim the following are key design choices:

			* Fault
			* Highly Available
			* Recoverable
			* Consistent
			* Scalable
			* Predictable Performance
			* Secure

			Not surprisingly, many of these are components of distributed transparencies.

	As an extension to what I have tried so far, I plan to test and run applications across multiple physical nodes. Ideally I want to make use of the underlying file system and understand fully how the system behaves during reads and writes as well as how it handles node's dropping out due to connectivity issues.		As an extension to what I have tried so far, I plan to test and run applications across multiple physical nodes. Ideally I want to make use of the underlying file system and understand fully how the system behaves during reads and writes as well as how it handles node's dropping out due to connectivity issues.

Mdpless2: /* Conclusion */

2011-03-03T03:26:56Z

Conclusion

← Older revision		Revision as of 03:26, 3 March 2011
Line 222:		Line 222:
	=Conclusion=		=Conclusion=

	The installation of a Hadoop gave me a great deal of insight into the workings and design behind the proprietary software used in the Bigtable implementation. Using the various references, tutorials, and documentations I found, I was able to set up a working example of a master/slave computation on a single node as well as set up a cluster running on a single machine across multiple Java processes.		The installation of a Hadoop gave me a great deal of insight into the workings and design behind the proprietary software used in the Bigtable implementation. Using the various references, tutorials, and documentations I found, I was able to set up a working example of a master/slave computation on a single node as well as set up a cluster running on a single machine across multiple Java processes. Future work that I can do is set up a fully-distributed mini-cluster with applications and files utilizing the processing power and disk space of the nodes. Working directly with a Bigtable implementation like Hadoop has given me a greater understanding of the possible uses of such a system. It is also a very effective way to learn about the reasons behind some of the design decisions that were taken by the development teams. Lastly, using Hadoop was easy-to-use and allowed one to get a system running in a relatively short period of time.

	=References=		=References=

Mdpless2: /* Discussion */

2011-03-03T02:45:16Z

Discussion

← Older revision		Revision as of 02:45, 3 March 2011
Line 214:		Line 214:
	Surprisingly, the API’s for Hadoop are easy to use and incorporate with existing problems that were designed for single-core processors. However, when developing algorithms for map/reduce, it is essential to analyze the problem from a different perspective. Reduce algorithms need to be efficient and result in a constructive output that will make future reads and writes faster.		Surprisingly, the API’s for Hadoop are easy to use and incorporate with existing problems that were designed for single-core processors. However, when developing algorithms for map/reduce, it is essential to analyze the problem from a different perspective. Reduce algorithms need to be efficient and result in a constructive output that will make future reads and writes faster.

	What was interesting~~? What was surprising? Here you can go~~ out ~~on tangents relating~~ to ~~your work~~.		What is particularly surprising is the fact that Bigtable and Hadoop are designed to run on commodity hardware. This ultimately has resulted in huge cost savings for organizations using the systems. For instance, Google has only minimal incremental costs associated with expanding their services and adding computing power to the clusters. Particularly, since it was developed in-house, they were not forced to pay royalties or buy licenses associated with each machine.

			Another interesting part to the design is how they achieve scalability and performance over a large amount of clusters. Each tablet load balances itself over multiple machines. When a tablet becomes the victim of high-demand, the tablet is automatically spread through to other nodes to balance the processing. Furthermore, if a node goes down to a variety of reasons, other nodes will jump in and take a tablet from the downed node. With many nodes in a cluster, the overhead to retrive and restore each tablet is shared across the whole network; each available node picks up one tablet and incurs only minimal delays.

			As an extension to what I have tried so far, I plan to test and run applications across multiple physical nodes. Ideally I want to make use of the underlying file system and understand fully how the system behaves during reads and writes as well as how it handles node's dropping out due to connectivity issues.

	=Conclusion=		=Conclusion=

Mdpless2: /* Conclusion */

2011-03-03T02:04:36Z

Conclusion

← Older revision		Revision as of 02:04, 3 March 2011
Line 218:		Line 218:
	=Conclusion=		=Conclusion=

	The installation of a Hadoop gave me a great deal of insight into the workings and design behind the proprietary software used in the Bigtable implementation. Using the various references, tutorials, and documentations I found, I was able to set up a working example of a master/slave computation on a single node as well as set up a cluster running on a single machine across multiple Java processes.		The installation of a Hadoop gave me a great deal of insight into the workings and design behind the proprietary software used in the Bigtable implementation. Using the various references, tutorials, and documentations I found, I was able to set up a working example of a master/slave computation on a single node as well as set up a cluster running on a single machine across multiple Java processes.

	=References=		=References=