MapReduce, Globus, BOINC: Difference between revisions

Revision as of 19:32, 26 March 2008

Premise? Local client on your machine downloads a 'workunit', churns the data, dumps the results and downloads a new 'workunit'
Why are we caring?
- Entertainment?
- How is this an OS paradigm? What is it useful for?
  - It isn't really an OS, just a method to have your mass computation done
  - More of a distributed scheduler?
    - Not even, central scheduler, but mass computation
  - How many systems have we seen that have accomplished mass computation on millions of uncontrolled computers?
    - ummm... none?
  - As an OS?
    - An OS is something that is created to run programs
    - This is a special case allowing us to run specific programs (BUT IS IT AN OS?)
  - Useful for "embarassingly parallel programs"
Perfect for large scale simulation?
- But then you need LOTS of communication, and this system does not have interconnects
The type of problems that we most care about tend not to be THAT parallel

So what would a distributed OS be for?
- Shared communication!
  - But we don't have much in the way that works well.
An OS typically provides a lot of services, together in one package
- We have been seeing that there are no complete packages, just pieces and parts. Why?
  - Computers are changing too fast? Same *NIX OS, same TCP/IP stack... so more of the same, why no true solution?
  - Communication is unreliable? Yes, but that is also nothing new

If people found that distributed file systems were successful, they would be in use all the time, but they aren't. Reason? PERFORMANCE

Take away message?
Can't handle communication - how do you abstract access to resources when driven through a network?
- As a result, we have many many specialized solutions for particular workloads.
If you are willing to not have communication between nodes, you gain a HUGE amount of computation.

The most reliable systems are the one that forget communication.
- The more you system tolerates bad stuff with a network, the better is scales.

The communication happens when you reduce the problem.
- MapReduce works because there is mapping and there is reducing.
  - There is no side effects (enabling things).
Why is it a good fit to a thousands of machines?
- They first had all these pieces, and if one of them does not replay, then they just do it over :)
  - You create the algorithm to fit this model, create this pieces, you have a combining function.
    - You have to have some back end that keeps track of who got work done. But you don't care if any machine fail in the middle of the computation.
Compare MapReduce to POSIX
- The difference is in efficiency. MapReduce is an extension to POSIX.
  - Distributed OSs trying to run the programs that run on different APIs. The systems that work, they are relaxed.
    - Here is the model, loose compatibility by gaining scalability.

@@ Line 45: / Line 45: @@
 *The most reliable systems are the one that forget communication.
+**The more you system tolerates bad stuff with a network, the better is scales.
+*We dont have general cluster distributed OS.
+===MapReduce===
+*The communication happens when you reduce the problem.
+**MapReduce works because there is mapping and there is reducing.
+***There is no side effects (enabling things).
+*Why is it a good fit to a thousands of machines?
+**They first had all these pieces, and if one of them does not replay, then they just do it over :)
+***You create the algorithm to fit this model, create this pieces, you have a combining function.
+****You have to have some back end that keeps track of who got work done. But you don't care if any machine fail in the middle of the computation.
+*Compare MapReduce to POSIX
+**The difference is in efficiency. MapReduce is an extension to POSIX.
+***Distributed OSs trying to run the programs that run on different APIs. The systems that work, they are relaxed.
+****Here is the model, loose compatibility by gaining scalability.