DistOS 2018F 2018-10-29

Readings

• Sanjay Ghemawat et al., "The Google File System" (SOSP 2003)
• Burrows, The Chubby Lock Service for Loosely-Coupled Distributed Systems (OSDI 2006)

Notes

Lecture Notes:

Peer-to-peer file sharing Napster -> classic silicon valley, business model that makes no sense Napster said, make all music available but not going to actually make the music available, use other people’s machines. Napster maintained a central directory but files stored on individual computers. People still wanted to exchange music files. Don’t have a centralized database of all the songs. DHTs are a technology...what is a hash table...give it a string and it gives you something else...where can I download this? So if implement a distributed hash table, no one system controlling the hash table...just use that and then you have to shutdown a bunch of nodes.

ISP throttling...now they poisoned it (record companies)....download something, what the heck is that? Idea behind Tapestry.....DHT as a service....Overlay network, what is it? Network that sits on-top of another network. Have the internet, isn’t the internet good enough? Point is that you want a different topology than what you have. Network based on geographic and organizational boundaries. Overlay, redo the tapology...send a message to neighbours, neighbours defined by an overlay network. i.e. Tor...defines its own topology. Facebook....social networks....send a message out, make a post, routed to friends, neighbours and they get it. Tapology of the social graph, who connects to who. Tapestry is an overlay network but does it ignore geography, no it makes use of it. Peers are nodes that are close by....the network are the systems running Tapestry but also identifies nearby nodes vs. Further distance nodes....network topology aware. Why? In large scale, will have a lot of node additions and deletions. Peer to Peer was, lets do file sharing but that was not their goal...building distributed applications of some kind. Pond...wanted something that provided a layer of messaging to their own nodes, that’s Tapestry .... table of hosts will not work....this provides the layer to find each other and send messages to each other in an efficient way. Lets build an infrastructure for sending messages, do we build apps on top of things like this today?

Keep an eye out for....DHTs will appear but their is a fundamental issue with DHT, the Limewire problem...they do very badly with untrusted nodes....can mess everyone else up by giving bad info to the network...can stop one or two nodes but attackers have significant resources like a botnet to attack your system.

Single Tapestry Node figure 6....the OS...nothing fancy, like the regular internet except they are maintaining state using a distributed hash table... Botnets...if they hard-code the IP address, how the botnet gets taken down...one way IRC, google search...use social media like an Instagram account....comments on celebrity Instagram feeds....spies used to send messages with ads in a newspaper or number stations (ham radio). Tapestry, trust issues b/c with trusted infrastructure there are better ways of doing it.

Ceph: Ceph is crazy, very complicated .... Ceph is out there, people are building this but really? What is CRUSH? Lets assume you have data, know where to go to get the parts of the files...would have to send a lot of data and update metadata every time you made changes to the file so they said we are not going to do that. These are not blocks, they are objects...what did they mean? Basically variable length chunk of storage, not dividing into fixed or variable size...file is some number of objects in some sort of order. When open file, what objects does it exist in but does the metadata storage give the objects? No, an algorithm to generate the names of the objects.

Metadata....store in memory every file ....hot spots for metadata access....3 would be maxed out and part of the system sleeping so the tree is dynamically re-partitioned to respond to hot spots.

Can have OSDs in parallel so asking for a file....distributed among lots of nodes so high performance ....many many computers talking to many many computers.

POSIX compatible (Ceph) impressive...POSIX compatibility is painful on writes...need to coordinate (centralize writes but that is slow). Can tell Ceph to be lazy. Take home lesson from Ceph....(all trusted, POSIX in distributed OS, can do it but OMG the admin overhead).

Tapestry...take home lesson...centralize node (trust)

Compare GFS with Ceph and Chubby (politically correct FAT storage :P) with Tapestry