https://homeostasis.scs.carleton.ca/wiki/index.php?action=history&feed=atom&title=DistOS_2021F_2021-12-09 2026-05-13T04:07:27Z Revision history for this page on the wiki MediaWiki 1.42.1 https://homeostasis.scs.carleton.ca/wiki/index.php?title=DistOS_2021F_2021-12-09&diff=23581&oldid=prev 2021-12-10T00:15:47Z

<p>Created page with &quot;==Notes== &lt;pre&gt; Lecture 22 ---------- Final exam review - themes of the class - key papers &amp; concepts - potential questions Format of final exam - 3 hours - essay que...&quot;</p> <p><b>New page</b></p><div>==Notes==<br /> <br /> &lt;pre&gt;<br /> Lecture 22<br /> ----------<br /> <br /> Final exam review<br /> <br /> - themes of the class<br /> - key papers &amp; concepts<br /> - potential questions<br /> <br /> Format of final exam<br /> - 3 hours<br /> - essay questions, 3 or 4 out of more<br /> (haven&#039;t created final yet)<br /> - open book, open note, open internet<br /> - just, NO COLLABORATION<br /> - and please, no plagiarism<br /> - all words should be your own<br /> - wasn&#039;t an issue on the midterm<br /> <br /> <br /> <br /> Themes of the class<br /> - distributed OSs can&#039;t just copy single-system OSs<br /> because the abstractions don&#039;t scale<br /> - UNIX-like systems work well for individual systems<br /> - people know how to program them, so we want<br /> to keep them as much as possible<br /> - but to scale, we need to approach problems from<br /> different angles, make use of different abstractions<br /> <br /> Why don&#039;t single system OSs scale?<br /> Key assumptions are violated<br /> - process memory is strongly consistent<br /> and access to it is roughly uniform<br /> - low latency<br /> - files are consistent<br /> - system all works or is completely broken<br /> - partial failures not a concern<br /> - so, communication is consistent and reliable<br /> <br /> This all changes when systems are connected by networks<br /> - network is *unreliable* and insecure<br /> - lose packets or entire connections<br /> - individual hosts fail independently yet<br /> distributed application should keep running<br /> - has to, because otherwise nothing would work<br /> <br /> (You know video for all classes are available through<br /> brightspace right?)<br /> <br /> Note that older attempts at distributed OSs dealt with distribution but not with reliability<br /> - allowed for whole system to go down when individual ones did<br /> <br /> Later there was replication to enable reliability<br /> <br /> But only when we had jobs that needed to be bigger than<br /> one host did we really scale things<br /> - the web<br /> <br /> (Scientific computing has always needed lots of computing power, but it was a specialized discipline with very different kinds of production requirements.)<br /> <br /> So, what are the strategies to make a platform for distributed applications?<br /> - keep UNIX, but in the form of containers<br /> - ephemeral, often immutable hosts<br /> - orchestration to manage containers<br /> - distributed data stores (filesystems, key-value,<br /> relational databases) built on immutability<br /> as much as possible<br /> - record append files<br /> - immutable tablets<br /> - immutable objects/chunks<br /> - relax consistency as much as possible<br /> - where we need consistency, need special mechanisms<br /> - use consensus algorithms (Paxos, etc)<br /> as little as possible to avoid bottlenecks<br /> - use synchonized time/order of events<br /> - make use of domain-specific regularities<br /> to optimize<br /> - replicate data and metadata for reliability &amp;<br /> performance<br /> - replication leads to consistency issues,<br /> which are solved through a combination of<br /> immutability and consensus<br /> - if you want performance, assume system is trusted<br /> - nodes can fail, but any bad behavior is strictly<br /> bounded<br /> - if you can&#039;t trust systems centrally because<br /> they are controlled by mutually non-trusting parties,<br /> you have to address the byzantine general&#039;s problem<br /> - and solutions tend to be expensive!<br /> - this is where you get cryptocurrencies and<br /> proof of work<br /> - these systems are best suited for embarassingly<br /> parallel tasks, e.g., ones that fit the mapreduce<br /> model or are serving independent customers concurrently<br /> - great for major online app/service providers,<br /> Facebook, Google, Amazon, etc<br /> - however, we can do computations that require more coordination - but we have to optimize every specific type separately (i.e., we can optimize for ML but it won&#039;t help with doing large-scale simulations)<br /> <br /> Single system OSs got us used to having a single infrastructure for everything<br /> - in a distributed world, the infrastructure has to mirror<br /> application requirements<br /> - except, maybe, when we do lots of engineering<br /> (think Ceph and Spanner)<br /> - and even there, we have to be careful, will need<br /> to do app-specific tweaking<br /> <br /> <br /> The best essays will<br /> - have a clear argument<br /> - logical arguments<br /> - many specific examples drawing upon multiple papers<br /> from the class<br /> - you can bring in other sources, but focus<br /> should be on what we covered<br /> <br /> Remember you&#039;re demostrating that you know the class material<br /> &lt;/pre&gt;</div>

Soma