MapReduce, Globus, BOINC - Revision history

Taisia at 19:50, 26 March 2008

2008-03-26T19:50:39Z

← Older revision		Revision as of 19:50, 26 March 2008
Line 92:		Line 92:
	***Distributed OSs trying to run the programs that run on different APIs. The systems that work, they are relaxed.		***Distributed OSs trying to run the programs that run on different APIs. The systems that work, they are relaxed.
	****Here is the model, loose compatibility by gaining scalability.		****Here is the model, loose compatibility by gaining scalability.
			*Side effects - you cant redo and undo. Functional programming model

Taisia at 19:38, 26 March 2008

2008-03-26T19:38:25Z

@@ Line 14: / Line 14: @@
 ==Notes==
 ===BOINC===
 *Premise?  Local client on your machine downloads a 'workunit', churns the data, dumps the results and downloads a new 'workunit'

Rhooper: /* Readings */

2008-03-26T19:36:47Z

Readings

← Older revision		Revision as of 19:36, 26 March 2008
Line 9:		Line 9:
	[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/mapreduce-osdi04.pdf Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simpliﬁed Data Processing on Large Clusters" (2004)]		[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/mapreduce-osdi04.pdf Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simpliﬁed Data Processing on Large Clusters" (2004)]

			Paper mentioned in class:

			[http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-183.pdf Krste Asanovíc, et al, "The Landscape of Parallel Computing Research: A View from Berkeley" (2006)]

	==Notes==		==Notes==

Emmellst at 19:32, 26 March 2008

2008-03-26T19:32:44Z

← Older revision		Revision as of 19:32, 26 March 2008
Line 45:		Line 45:

	*The most reliable systems are the one that forget communication.		*The most reliable systems are the one that forget communication.
			**The more you system tolerates bad stuff with a network, the better is scales.

			*We dont have general cluster distributed OS.

			===MapReduce===
			*The communication happens when you reduce the problem.
			**MapReduce works because there is mapping and there is reducing.
			***There is no side effects (enabling things).
			*Why is it a good fit to a thousands of machines?
			**They first had all these pieces, and if one of them does not replay, then they just do it over :)
			***You create the algorithm to fit this model, create this pieces, you have a combining function.
			****You have to have some back end that keeps track of who got work done. But you don't care if any machine fail in the middle of the computation.
			*Compare MapReduce to POSIX
			**The difference is in efficiency. MapReduce is an extension to POSIX.
			***Distributed OSs trying to run the programs that run on different APIs. The systems that work, they are relaxed.
			****Here is the model, loose compatibility by gaining scalability.

Emmellst at 19:14, 26 March 2008

2008-03-26T19:14:22Z

← Older revision		Revision as of 19:14, 26 March 2008
Line 29:		Line 29:
	*The type of problems that we most care about tend not to be THAT parallel		*The type of problems that we most care about tend not to be THAT parallel

	*So what would a ~~distrbuted~~ OS be for?		*So what would a distributed OS be for?
	**Shared communication!		**Shared communication!
	***But we don't have much in the way that works well.		***But we don't have much in the way that works well.
	*An OS typically provides a lot of services, together in one package		*An OS typically provides a lot of services, together in one package
	**We have been seeing that there are no complete packages, just pieces and parts. Why?		**We have been seeing that there are no complete packages, just pieces and parts. Why?
	**Computers are changing too fast? Same NIX OS, same ~~tcp~~/ip stack... so more of the same, why no true solution?		**Computers are changing too fast? Same NIX OS, same TCP/IP stack... so more of the same, why no true solution?
	***Communication is unreliable? Yes, but that is also nothing new		***Communication is unreliable? Yes, but that is also nothing new

	*If people found that distributed file systems were ~~succesful~~, they would be in use all the time, but they aren't. Reason? PERFORMANCE		*If people found that distributed file systems were successful, they would be in use all the time, but they aren't. Reason? PERFORMANCE

	*Take away message?		*Take away message?
	*Can't handle communication - how do you abstract access to resources when driven through a network?		*Can't handle communication - how do you abstract access to resources when driven through a network?
	**As a result, we have many many ~~specialised~~ solutions for particular workloads.		**As a result, we have many many specialized solutions for particular workloads.
	*If you are willing to not have communication between nodes, you gain a HUGE amount of computation		*If you are willing to not have communication between nodes, you gain a HUGE amount of computation.

			*The most reliable systems are the one that forget communication.

Emmellst: notes in class

2008-03-26T19:01:43Z

notes in class

← Older revision		Revision as of 19:01, 26 March 2008
Line 8:		Line 8:

	[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/mapreduce-osdi04.pdf Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simpliﬁed Data Processing on Large Clusters" (2004)]		[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/mapreduce-osdi04.pdf Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simpliﬁed Data Processing on Large Clusters" (2004)]


			==Notes==
			===BOINC===
			*Premise? Local client on your machine downloads a 'workunit', churns the data, dumps the results and downloads a new 'workunit'
			*Why are we caring?
			**Entertainment?
			**How is this an OS paradigm? What is it useful for?
			***It isn't really an OS, just a method to have your mass computation done
			***More of a distributed scheduler?
			****Not even, central scheduler, but mass computation
			***How many systems have we seen that have accomplished mass computation on millions of uncontrolled computers?
			****ummm... none?
			***As an OS?
			****An OS is something that is created to run programs
			****This is a special case allowing us to run specific programs (BUT IS IT AN OS?)
			***Useful for "embarassingly parallel programs"
			*Perfect for large scale simulation?
			**But then you need LOTS of communication, and this system does not have interconnects
			*The type of problems that we most care about tend not to be THAT parallel

			*So what would a distrbuted OS be for?
			**Shared communication!
			***But we don't have much in the way that works well.
			*An OS typically provides a lot of services, together in one package
			**We have been seeing that there are no complete packages, just pieces and parts. Why?
			**Computers are changing too fast? Same NIX OS, same tcp/ip stack... so more of the same, why no true solution?
			***Communication is unreliable? Yes, but that is also nothing new

			*If people found that distributed file systems were succesful, they would be in use all the time, but they aren't. Reason? PERFORMANCE

			*Take away message?
			*Can't handle communication - how do you abstract access to resources when driven through a network?
			**As a result, we have many many specialised solutions for particular workloads.
			*If you are willing to not have communication between nodes, you gain a HUGE amount of computation

Soma: /* Readings */

2008-03-16T15:10:15Z

Readings

← Older revision		Revision as of 15:10, 16 March 2008
Line 1:		Line 1:
	==Readings==		==Readings==

	[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/~~anderson~~-~~boinc~~.pdf ~~David P. Anderson~~, "~~BOINC: A System for Public-Resource Computing and Storage~~" (~~2004~~)]		[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/foster-grid.pdf Ian Foster and Carl Kesselman, "Computational Grids" (1998)]

	[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/foster-globus-intro.pdf Ian Foster, "Globus Toolkit Version 4: Software for Service-Oriented Systems" (2006)]		[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/foster-globus-intro.pdf Ian Foster, "Globus Toolkit Version 4: Software for Service-Oriented Systems" (2006)]

	[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/~~foster~~-~~grid~~.pdf ~~Ian Foster and Carl Kesselman~~, "~~Computational Grids~~" (~~1998~~)]		[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/anderson-boinc.pdf David P. Anderson, "BOINC: A System for Public-Resource Computing and Storage" (2004)]

	[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/mapreduce-osdi04.pdf Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simpliﬁed Data Processing on Large Clusters" (2004)]		[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/mapreduce-osdi04.pdf Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simpliﬁed Data Processing on Large Clusters" (2004)]

Soma at 15:09, 16 March 2008

2008-03-16T15:09:42Z

New page

==Readings==

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/anderson-boinc.pdf David P. Anderson, "BOINC: A System for Public-Resource Computing and Storage" (2004)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/foster-globus-intro.pdf Ian Foster, "Globus Toolkit Version 4: Software for Service-Oriented Systems" (2006)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/foster-grid.pdf Ian Foster and Carl Kesselman, "Computational Grids" (1998)]

[http://homeostasis.scs.carleton.ca/~soma/distos/2008-03-24/mapreduce-osdi04.pdf Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simpliﬁed Data Processing on Large Clusters" (2004)]