DistOS 2019F 2019-11-27
Jump to navigation
Jump to search
Readings
SCOPE & Yarn
- Chaiken et al., "SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets" (PVLDB 2008)
- Vavilapalli et al., "Apache Hadoop YARN: Yet Another Resource Negotiator" (SoCC 2013)
- Optional: Isard et al., "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks" (Eurosys 2007)
Discussion Questions
- Both papers start off with a criticism of MapReduce but then go off in very different directions. What are those directions?
- How close is SCOPE to SQL? Why didn't they just use SQL?
- What is Cosmos? How is it related to SCOPE? Dryad?
- How similar is Cosmos to other systems we have discussed?
- What were the requirements for YARN, and how were they achieved?
- Why do you think Microsoft didn't commercialize SCOPE or Cosmos? See this question comparing Cosmos/SCOPE to Data Lake Store and U-SQL, which apparently are built on YARN and WebHDFS.