MapReduce, Globus, BOINC

Readings

Ian Foster and Carl Kesselman, "Computational Grids" (1998)

Ian Foster, "Globus Toolkit Version 4: Software for Service-Oriented Systems" (2006)

David P. Anderson, "BOINC: A System for Public-Resource Computing and Storage" (2004)

Jeffrey Dean and Sanjay Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters" (2004)

Paper mentioned in class:

Krste Asanovíc, et al, "The Landscape of Parallel Computing Research: A View from Berkeley" (2006)

Notes

BOINC

• Premise? Local client on your machine downloads a 'workunit', churns the data, dumps the results and downloads a new 'workunit'
• Why are we caring?
  • Entertainment?
  • How is this an OS paradigm? What is it useful for?
    • It isn't really an OS, just a method to have your mass computation done
    • More of a distributed scheduler?
      • Not even, central scheduler, but mass computation
    • How many systems have we seen that have accomplished mass computation on millions of uncontrolled computers?
      • ummm... none?
    • As an OS?
      • An OS is something that is created to run programs
      • This is a special case allowing us to run specific programs (BUT IS IT AN OS?)
    • Useful for "embarassingly parallel programs"
• Perfect for large scale simulation?
  • But then you need LOTS of communication, and this system does not have interconnects
• The type of problems that we most care about tend not to be THAT parallel

• So what would a distributed OS be for?
  • Shared communication!
    • But we don't have much in the way that works well.
• An OS typically provides a lot of services, together in one package
  • We have been seeing that there are no complete packages, just pieces and parts. Why?
    • Computers are changing too fast? Same *NIX OS, same TCP/IP stack... so more of the same, why no true solution?
    • Communication is unreliable? Yes, but that is also nothing new

• If people found that distributed file systems were successful, they would be in use all the time, but they aren't. Reason? PERFORMANCE

• Take away message?
• Can't handle communication - how do you abstract access to resources when driven through a network?
  • As a result, we have many many specialized solutions for particular workloads.
• If you are willing to not have communication between nodes, you gain a HUGE amount of computation.

• The most reliable systems are the one that forget communication.
  • The more you system tolerates bad stuff with a network, the better is scales.

• We dont have general cluster distributed OS.

MapReduce

• The communication happens when you reduce the problem.
  • MapReduce works because there is mapping and there is reducing.
    • There is no side effects (enabling things).
• Why is it a good fit to a thousands of machines?
  • They first had all these pieces, and if one of them does not replay, then they just do it over :)
    • You create the algorithm to fit this model, create this pieces, you have a combining function.
      • You have to have some back end that keeps track of who got work done. But you don't care if any machine fail in the middle of the computation.
• Compare MapReduce to POSIX
  • The difference is in efficiency. MapReduce is an extension to POSIX.
    • Distributed OSs trying to run the programs that run on different APIs. The systems that work, they are relaxed.
      • Here is the model, loose compatibility by gaining scalability.