MapReduce, Globus, BOINC

Readings

Premise? Local client on your machine downloads a 'workunit', churns the data, dumps the results and downloads a new 'workunit'
Why are we caring?
- Entertainment?
- How is this an OS paradigm? What is it useful for?
  - It isn't really an OS, just a method to have your mass computation done
  - More of a distributed scheduler?
    - Not even, central scheduler, but mass computation
  - How many systems have we seen that have accomplished mass computation on millions of uncontrolled computers?
    - ummm... none?
  - As an OS?
    - An OS is something that is created to run programs
    - This is a special case allowing us to run specific programs (BUT IS IT AN OS?)
  - Useful for "embarassingly parallel programs"
Perfect for large scale simulation?
- But then you need LOTS of communication, and this system does not have interconnects
The type of problems that we most care about tend not to be THAT parallel

So what would a distributed OS be for?
- Shared communication!
  - But we don't have much in the way that works well.
An OS typically provides a lot of services, together in one package
- We have been seeing that there are no complete packages, just pieces and parts. Why?
  - Computers are changing too fast? Same *NIX OS, same TCP/IP stack... so more of the same, why no true solution?
  - Communication is unreliable? Yes, but that is also nothing new

If people found that distributed file systems were successful, they would be in use all the time, but they aren't. Reason? PERFORMANCE

Take away message?
Can't handle communication - how do you abstract access to resources when driven through a network?
- As a result, we have many many specialized solutions for particular workloads.
If you are willing to not have communication between nodes, you gain a HUGE amount of computation.