DistOS 2015W Session 7

From Soma-notes

Ceph

Unlike GFS that was talked about previously, this is a general purpose distributed file system. It follows the same general model of distribution as GFS and Amoeba.


Main Components

  • Client
  • Cluster of Object Storage Devices (OSD)
    • It basically stores data and metadata and clients communicate directly with it to perform IO operations
    • Data is stored in objects (variable size chunks)
  • Meta-data Server (MDS)
    • It is used to manage the file and directories. Clients interact with it to perform metadata operations like open, rename. It manages the capabilities of a client.
    • Clients do not need to access MDSs to find where data is stored, improving scalability (more on that below)


Key Features

  • Decoupled data and metadata
  • Dynamic Distributed Metadata Management
    • It distributes the metadata among multiple metadata servers using dynamic sub-tree partitioning, meaning folders that get used more often get their meta-data replicated to more servers, spreading the load. This happens completely automatically
  • Object based storage
    • Uses cluster of OSDs to form a Reliable Autonomic Distributed Object-Store (RADOS) for Ceph failure detection and recovery
  • CRUSH (Controlled, Replicated, Under Scalable, Hashing)
    • The hashing algorithm used to calculate the location of object instead of looking for them
    • This significantly reduces the load on the MDSs because each client has enough information to independently determine where things should be located.
    • Responsible for automatically moving data when ODSs are added or removed (can be simplified as location = CRUSH(filename) % num_servers)
    • The CRUSH paper on Ceph’s website can be viewed here
  • RADOS (Reliable Autonomic Distributed Object-Store) is the object store for Ceph

Chubby

It is a coarse grained lock service, made and used internally by Google, that serves multiple clients with small number of servers (chubby cell).


System Components

  • Chubby Cell
    • Handles the actual locks
    • Mainly consists of 5 servers known as replicas
    • Consensus protocol (Paxos) is used to elect the master from replicas
  • Client
    • Used by programs to request and use locks


Main Features

  • Implemented as a semi POSIX-compliant file-system with a 256KB file limit
    • Permissions are for files only, not folders, thus breaking some compatibility
    • Trivial to use by programs; just use the standard fopen() family of calls
  • Uses consensus algorithm (paxos) among a set of servers to agree on who is the master that is in charge of the metadata
  • Meant for locks that last hours or days, not seconds (thus, "coarse grained")
  • A master server can handle tens of thousands of simultaneous connections
    • This can be further improved by using caching servers as most of the traffic are keep-alive messages
  • Used by GFS for electing master server
  • Also used by Google as a nameserver

Issues

  • Due to the use of Paxos (the only algorithm for this problem), the chubby cell is limited to only 5 servers. While this limits fault-tolerance, in practice this is more than enough.
  • Since it has a file-system client interface, many programmers tend to abuse the system and need education (even inside Google)