https://homeostasis.scs.carleton.ca/wiki/index.php?action=history&feed=atom&title=DistOS_2023W_2023-02-27 2026-04-08T03:23:07Z Revision history for this page on the wiki MediaWiki 1.42.1 https://homeostasis.scs.carleton.ca/wiki/index.php?title=DistOS_2023W_2023-02-27&diff=24366&oldid=prev 2023-02-27T17:56:40Z

<p>Created page with &quot;==Notes== &lt;pre&gt; Web Scale --------- * Midterm grading is ongoing, hopefully will be finished this week * Proposal deadline extended to Friday, will try to give you some material this week to help - I&#039;ve been ignoring some of you on Teams, I will be replying today Up to this point in the class, we&#039;ve really been focused on distributed systems for running classic UNIX-like workloads - individual developer/engineer working at a workstation on their stuff Key problems...&quot;</p> <p><b>New page</b></p><div>==Notes==<br /> &lt;pre&gt;<br /> Web Scale<br /> ---------<br /> <br /> * Midterm grading is ongoing, hopefully will be finished this week<br /> * Proposal deadline extended to Friday, will try to give you some material this week to help<br /> - I&#039;ve been ignoring some of you on Teams, I will be replying today<br /> <br /> <br /> Up to this point in the class, we&#039;ve really been focused on distributed systems for running classic UNIX-like workloads<br /> - individual developer/engineer working at a workstation on their stuff<br /> <br /> Key problems here are<br /> - making small files available to any workstation<br /> - authentication at a workstation (Kerberos, out of scope)<br /> - running jobs on remote computers<br /> - process migration is nice but not so common a need in practice<br /> <br /> We&#039;ve had hints at larger problems<br /> - DSM can be used for large-scale scientific applications<br /> <br /> Scientific applications will consume as many resources as you throw at them<br /> - but mostly they&#039;ve been addressed through specialized solutions<br /> - classic method is distributed apps built on MPI (a message passing API)<br /> - look up &quot;Beowulf clusters&quot; for classic implementations<br /> <br /> These aren&#039;t so &quot;distributed OS&quot;-like, because it is like coding in assembly language - not much abstraction beyond what UNIX+networking provides<br /> <br /> But the web came in the 1990&#039;s and changed everything<br /> <br /> Why did the web change the computing landscape? There were new problems to solve!<br /> <br /> Today it is commonplace to have web applications that are accessed by millions of people concurrently.<br /> * Initially web applications ran on a single computer with just a web server process.<br /> * Then, that process was connected to a backend database (MySQL) and Perl scripts (CGI).<br /> * This was all great for a modest number of users. But then the world got on the world wide web and we neeeded to SCALE<br /> <br /> The first company to really try to scale with the growth of the web was Google<br /> - before, companies bought the biggest, most expensive computers they could and load balanced between them carefully<br /> - google realized that you could use lots of cheap computers if you could deal with their cheapness in software (i.e., distributing workload, dealing with failures)<br /> - and this really mattered for search engines<br /> <br /> Search engines were the first hard problem because to index the web you had to download it, and the web was growing exponentially in size<br /> - but how do you make an application layer for software to run on<br /> that takes advantage of many many computers?<br /> <br /> Papers for the second half of the semester are to give you an idea of how large-scale systems are built to support web applications. We need:<br /> - filesystems<br /> - databases<br /> - computation/data processing<br /> - plumbing to support the above (coordination services)<br /> <br /> <br /> Do these things make a &quot;distributed operating system&quot;?<br /> - yes and no<br /> - yes: real applications are built on top of these abstractions<br /> - no: abstractions aren&#039;t general purpose, need different ones<br /> for different applications (an OS normally has one set of abstractions that<br /> everyone uses)<br /> <br /> Why can&#039;t we just have one unifying distributed OS then?<br /> - communication and fault tolerance<br /> - communication: it is always expensive, and how to minimize it<br /> depends on the application. (Parallel is always hard)<br /> - really, we turn most of the app into being embarassingly parallel<br /> (requiring no coordination), and the remainder becomes the<br /> hard part that requires special engineering<br /> &lt;/pre&gt;</div>

Soma