DistOS 2014W Lecture 10

Context

Very different because of the workload that it is desgined for.
- Because of the number of small files that have to be indexed for the web, etc., it is no longer practical to have a filesystem that stores these individually. Too much overhead. Punts problem to userspace, incl. record delimitation.
Don't care about latency, surprising considering it's Google, the guys who change the TCP IW standard recommendations for latency.
Mostly seeking through entire file.
Paper from 2003, mentions still using 100BASE-T links.
Data-heavy, metadata light. Contacting the metadata server is a rare event.
Really good that they designed for unreliable hardware:
- All the replication
- Data checksumming
Performance degrades for small random access workload; use other filesystem.
Path of least resistance to scale, not to do something super CS-smart.
Google used to re-index every month, swapping out indexes. Now, it's much more online. GFS is now just a layer to support a more dynamic layer.

Structure of GFS does match some other modern systems:
- Hard drives are like parallel tapes, very suited for streaming.
- Flash devices are log-structured too, but have an abstracting firmware. You want to do erasure in bulk, in the background. Used to be we needed specialized FS for MTDs to get better performance; though now we have better microcontrollers in some embedded systems to abstract away the hardware.
Architectures that start big, often end up in the smallest things.

Data and metadata are held together.
- Doesn't account for different access patterns:
  - Data → big, long transfers
  - Metadata → small, low latency
- Can't scale separately
By design, a file is a fraction of the size of a server
- Huge files spread over many servers not even in the cards for NFS
- Meant for small problems, not web-scale
  - Google has a copy of the publicly accessible internet
    - Their strategy is to copy the internet to index it
    - Insane → insane filesystem
Designed for lower latency
Designed for POSIX semantics; how the requirements that lead to the ‘standard’ evolved
Even mainframes, scale-up solutions, ultra-reliable systems, with data sets bigger than RAM don't have this scale.
Reliability was a property of the host, not the network
Point-to-point access; much less load-balancing, even in AFS
- Single point of entry, single point of failure, bottleneck

Ceph is crazy and tries to do everything
Unlike GFS, distributes metadata, not just for read-only copies
Unlike GFS, the OSDs have some intelligence, and autonomously distribute the data, rather than being controlled by a master.
- Uses hashing in the distribution process to uniformly distribute data
- The actual algorithm for distributing data is as follows:
  
  <math>file + offset → hash(object ID) → CRUSH(placement group) → OSD</math>
- Each client has knowledge of the entire storage network.
- Tracks failure groups (same breaker, switch, etc.), hot data, etc.
- Number of replicas is changeable on the fly, but the placement group is not
  - For example, if every client on the planet is accessing the same file, you can scale out for that data.
- You don't ask where to go, you just go, which makes this very scalable
CRUSH is sufficiently advanced to be called magic.
- <math>O(log n)</math> of the size of the data
- CPUs stupidly fast, so the above is of minimal overhead, whereas the network, despite being fast, has latency, etc. Computation scales much better than communication.
Storage is composed of variable-length atoms