Soma-notes - User contributions [en]

DistOS 2015W Session 11

2015-04-06T05:13:01Z

Abeinges: /* Spanner */

==BigTable==
* Google System used for storing data of various Google Products, for instance Google Analytics, Google Finance, Orkut, Personalized Search, Writely, Google Earth and many more
* Big table is
** Sparse
** Persistant
** Muti dimensional Sorted Map
*It is indexed by
** Row Key: Every read or write of data under single row key is atomic. Each row range is called Tablet. Select Row key to get good locality for data access.
** Column Key: Grouped into sets called Column Families. Forms basic unit of Access Control.All data stored is of same type.Syntax used: ''family:qualifier''
** Time Stamp:Each cell consists of multiple versions of same data which are indexed by Timestamps.In order to avoid collisions, Timestamps need to be generated by applications.
* Big Table '''API''': Provides functions for
** Creating and Deleting
*** Tables
*** Column Families
**Changing Cluster
**Changing Table
**Column Family metadata like Access Control Rights.
** Set of wrappers which allow Big Data to be used both as
*** Input source
***Output Target
*The timestamp mechanism in BIG table helps clients to access recent versions of data with simple accessing aspects of using row and column.
*Parallel computation and cluster management system makes BIG table flexible and highly scalable.

== Dynamo==
* Amazon's Key Value Store
*Availability is the buzz word for Dynamo. Dynamo=Availability
*Shifted Computer Science paradigm from caring about the consistency to availability.
*Sacrifices consistency under certain failure scenarios.
*Treats failure handling as normal case without impact on availability and performance.
*Data is partitioned and replicated using consistent hashing and consistency is facilitated by use of object versioning.
* This system has certain requirements such as:
** Query Model: Simple read and write operations to data item that are uniquely identified by a key.
**ACID properties: Atomicity, Consistency, Isolation, Durability.
**Efficiency: System needs to function on a commodity hardware infrastructure.
* Service Level Agreements(SLA): They are a negotiated contract between a client and a service regarding characteristics related to systems. They are used in order to guarantee that in a bounded time period, an application can deliver it's functionality.
* System Architecture: It consists of ''System Interface'', ''Partitioning Algorithm'', ''Replication'',''Data Versioning''.
* Successfully handles
** Server Failure
** Data Centre Failure
** Network Partitions
* Allows service owners to customize their own storage systems according to their storage systems to meet the desired performance, durability and consistency SLAs.
* Building block for highly available applications.

==Cassandra==
* Facebook's storage system to fulfil needs of the Inbox Search Problem
*Partitions data across the cluster using consistent hashing.
*Distributed multi dimensional map indexed by a key
* In it's data model:
** Columns grouped together into sets called column families. Column Families further of 2 types:
***Simple column families
***Super column families
* API consists of :
** Insert
**Get
** Delete
* System Architecture consists of :
** Partitioning: Takes place using consistent hashing
**Replication: Each item replicated at n hosts where "n" is the replication factor configured per system.
** Membership: Cluster membership is based on Scuttle butt which is a highly efficient anti-entropy Gossip based mechanism.The Membership further has sub part such as:
***Failure Detection
**Bootstrapping
** Scaling the cluster
*It can run cheap commodity hardware and handle high throughput
*Its multiple usable structure makes it very scalable

=Spanner=
* Google's scalable, multi version, globally distributed database.
* Has been built on top of the Google's Big table.
*Provided data consistency and Supports SQL like Interface.
* Uses a separate high-reliability time service to guarantee the correctness properties around concurrency control.
** The timestamps are utilized.
*It shares data across machines and migrates data automatically across machines
*Data Control Functions in spanner controls latency and performance

DistOS 2015W Session 9

2015-04-06T05:11:13Z

Abeinges: /* Naiad */

== BOINC ==

*Public Resource Computing Platform
*Gives scientists the ability to use large amounts of computation resources.
*The clients do not connect directly with each other but instead they talk to a central server located at Berkley
*The goals of Boinc are:
:*1) reduce the barriers of entry
:*2) Share resources among autonomous projects
:*3) Support diverse applications
:*4) Reward participants.

*It can run as applications in common language with no modifications
A BOINC application can be identified by a single master URL, which serves as the homepage as well as the directory of the servers.

== SETI@Home ==

*Uses public resource computing to analyze radio signals to find extraterrestrial intelligence
*Need good quality telescope to search for radio signals, and lots of computational power, which was unavailable locally
*It has not yet found extraterrestrial intelligence, but its has established credibility of public resource computing projects
*Originally custom, now uses BOINC as a backbone for the project
*Uses relational database to store information on a large scale, further it uses a multi-threaded server to distribute work to clients
*Quality of data in this architecture is untrustworthy, the main incentive to use it, however, is that it is a cheap and easy way of scaling the work exponentially.
*Provided social incentives to encourage users to join the system.
*This computation model still exists but not in the legitimate world.
*Formed a good concept of public resource computing and a distributed computing by providing a platform independent framework

== MapReduce ==

*A programming model presented by Google to do large scale parallel computations
*Uses the <code>Map()</code> and <code>Reduce()</code> functions from functional style programming languages
:*Map (Filtering)
::*Takes a function and applies it to a bunch of keys to produce values
* Hides parallelization, fault tolerance, locality optimization and load balancing
:*Reduce (Summary)
::*Accumulates results from the data set using a given function
* Very easy to use and understand, with many classic problems fitting this pattern
* Otherwise quite constrained in what exactly can be done
* Uses hashing to distribute similar keys to similar machines, but otherwise spread the load

== Naiad ==

*A programming model similar to <code>MapReduce</code> but with streaming capabilities so that data results are almost instantaneous
*A distributed system for executing data parallel cyclic dataflow programs offering high throughput and low latency
*Aims to provide a general purpose system which will fulfill the requirements and the will also support wide variety of high level programming models.
*Highly used for parallel execution of data
*Provides the functionality of checkpoint and restoring
*A complex framework that can be the backend for simpler models of computation like LINQ or MapReduce to be built on top of.
*Real Time Applications:
:*Batch iterative Machine Learning:
VW, an open source distributed machine learning performs iteration in 3 phases: each process updates local state; processes independently training on local data; and process jointly performed global average which is All Reduce.
:*Streaming Acyclic Computation
When compared to a system called [http://research.microsoft.com/apps/pubs/default.aspx?id=163832 Kineograph] ( also done by Microsoft ), which processes twitter handles and provides counts of the occurrence of hashtags as well as links between popular tags, was written using Naiad in 26 lines of code and ran close to 2X faster.
* Naiad paper won the best paper award in SOSP 2013, check-out this link in Microsoft Research website http://research.microsoft.com/en-us/projects/naiad/ . Down in this page you can see some videos that explains naiad including Derek's Murray presentation at SOSP 2013.

DistOS 2015W Session 9

2015-04-06T05:09:34Z

Abeinges: /* MapReduce */

== BOINC ==

*Public Resource Computing Platform
*Gives scientists the ability to use large amounts of computation resources.
*The clients do not connect directly with each other but instead they talk to a central server located at Berkley
*The goals of Boinc are:
:*1) reduce the barriers of entry
:*2) Share resources among autonomous projects
:*3) Support diverse applications
:*4) Reward participants.

*It can run as applications in common language with no modifications
A BOINC application can be identified by a single master URL, which serves as the homepage as well as the directory of the servers.

== SETI@Home ==

*Uses public resource computing to analyze radio signals to find extraterrestrial intelligence
*Need good quality telescope to search for radio signals, and lots of computational power, which was unavailable locally
*It has not yet found extraterrestrial intelligence, but its has established credibility of public resource computing projects
*Originally custom, now uses BOINC as a backbone for the project
*Uses relational database to store information on a large scale, further it uses a multi-threaded server to distribute work to clients
*Quality of data in this architecture is untrustworthy, the main incentive to use it, however, is that it is a cheap and easy way of scaling the work exponentially.
*Provided social incentives to encourage users to join the system.
*This computation model still exists but not in the legitimate world.
*Formed a good concept of public resource computing and a distributed computing by providing a platform independent framework

== MapReduce ==

*A programming model presented by Google to do large scale parallel computations
*Uses the <code>Map()</code> and <code>Reduce()</code> functions from functional style programming languages
:*Map (Filtering)
::*Takes a function and applies it to a bunch of keys to produce values
* Hides parallelization, fault tolerance, locality optimization and load balancing
:*Reduce (Summary)
::*Accumulates results from the data set using a given function
* Very easy to use and understand, with many classic problems fitting this pattern
* Otherwise quite constrained in what exactly can be done
* Uses hashing to distribute similar keys to similar machines, but otherwise spread the load

== Naiad ==

*A programming model similar to <code>MapReduce</code> but with streaming capabilities so that data results are almost instantaneous
*A distributed system for executing data parallel cyclic dataflow programs offering high throughput and low latency
*Aims to provide a general purpose system which will fulfill the requirements and the will also support wide variety of high level programming models.
*Highly used for parallel execution of data
*Provides the functionality of checkpoint and restoring
*Real Time Applications:
:*Batch iterative Machine Learning:
VW, an open source distributed machine learning performs iteration in 3 phases: each process updates local state; processes independently training on local data; and process jointly performed global average which is All Reduce.
:*Streaming Acyclic Computation
When compared to a system called [http://research.microsoft.com/apps/pubs/default.aspx?id=163832 Kineograph] ( also done by Microsoft ), which processes twitter handles and provides counts of the occurrence of hashtags as well as links between popular tags, was written using Naiad in 26 lines of code and ran close to 2X faster.
* Naiad paper won the best paper award in SOSP 2013, check-out this link in Microsoft Research website http://research.microsoft.com/en-us/projects/naiad/ . Down in this page you can see some videos that explains naiad including Derek's Murray presentation at SOSP 2013.

DistOS 2015W Session 9

2015-04-06T05:06:09Z

Abeinges: /* SETI@Home */

== BOINC ==

*Public Resource Computing Platform
*Gives scientists the ability to use large amounts of computation resources.
*The clients do not connect directly with each other but instead they talk to a central server located at Berkley
*The goals of Boinc are:
:*1) reduce the barriers of entry
:*2) Share resources among autonomous projects
:*3) Support diverse applications
:*4) Reward participants.

*It can run as applications in common language with no modifications
A BOINC application can be identified by a single master URL, which serves as the homepage as well as the directory of the servers.

== SETI@Home ==

*Uses public resource computing to analyze radio signals to find extraterrestrial intelligence
*Need good quality telescope to search for radio signals, and lots of computational power, which was unavailable locally
*It has not yet found extraterrestrial intelligence, but its has established credibility of public resource computing projects
*Originally custom, now uses BOINC as a backbone for the project
*Uses relational database to store information on a large scale, further it uses a multi-threaded server to distribute work to clients
*Quality of data in this architecture is untrustworthy, the main incentive to use it, however, is that it is a cheap and easy way of scaling the work exponentially.
*Provided social incentives to encourage users to join the system.
*This computation model still exists but not in the legitimate world.
*Formed a good concept of public resource computing and a distributed computing by providing a platform independent framework

== MapReduce ==

*A programming model presented by Google to do large scale parallel computations
*Uses the <code>Map()</code> and <code>Reduce()</code> functions from functional style programming languages
:*Map (Filtering)
::*Takes a function and applies it to all elements of the given data set
* Hides parallelization, fault tolerance, locality optimization and load balancing
:*Reduce (Summary)
::*Accumulates results from the data set using a given function

== Naiad ==

*A programming model similar to <code>MapReduce</code> but with streaming capabilities so that data results are almost instantaneous
*A distributed system for executing data parallel cyclic dataflow programs offering high throughput and low latency
*Aims to provide a general purpose system which will fulfill the requirements and the will also support wide variety of high level programming models.
*Highly used for parallel execution of data
*Provides the functionality of checkpoint and restoring
*Real Time Applications:
:*Batch iterative Machine Learning:
VW, an open source distributed machine learning performs iteration in 3 phases: each process updates local state; processes independently training on local data; and process jointly performed global average which is All Reduce.
:*Streaming Acyclic Computation
When compared to a system called [http://research.microsoft.com/apps/pubs/default.aspx?id=163832 Kineograph] ( also done by Microsoft ), which processes twitter handles and provides counts of the occurrence of hashtags as well as links between popular tags, was written using Naiad in 26 lines of code and ran close to 2X faster.
* Naiad paper won the best paper award in SOSP 2013, check-out this link in Microsoft Research website http://research.microsoft.com/en-us/projects/naiad/ . Down in this page you can see some videos that explains naiad including Derek's Murray presentation at SOSP 2013.

DistOS 2015W Session 7

2015-04-06T05:04:30Z

Abeinges: /* Issues */

= Ceph =
Unlike GFS that was talked about previously, this is a general purpose distributed file system. It follows the same general model of distribution as GFS and Amoeba.

== Main Components ==
* Client

* Cluster of Object Storage Devices (OSD)
** It basically stores data and metadata and clients communicate directly with it to perform IO operations
** Data is stored in objects (variable size chunks)

* Meta-data Server (MDS)
** It is used to manage the file and directories. Clients interact with it to perform metadata operations like open, rename. It manages the capabilities of a client.
** Clients '''do not''' need to access MDSs to find where data is stored, improving scalability (more on that below)

== Key Features ==
* Decoupled data and metadata

* Dynamic Distributed Metadata Management

** It distributes the metadata among multiple metadata servers using dynamic sub-tree partitioning, meaning folders that get used more often get their meta-data replicated to more servers, spreading the load. This happens completely automatically

* Object based storage
** Uses cluster of OSDs to form a Reliable Autonomic Distributed Object-Store (RADOS) for Ceph failure detection and recovery

* CRUSH (Controlled, Replicated, Under Scalable, Hashing)
** The hashing algorithm used to calculate the location of object instead of looking for them
** This significantly reduces the load on the MDSs because each client has enough information to independently determine where things should be located.
** Responsible for automatically moving data when ODSs are added or removed (can be simplified as ''location = CRUSH(filename) % num_servers'')
** The CRUSH paper on Ceph’s website can be [http://ceph.com/papers/weil-crush-sc06.pdf viewed here]

* RADOS (Reliable Autonomic Distributed Object-Store) is the object store for Ceph

= Chubby =
It is a coarse grained lock service, made and used internally by Google, that serves multiple clients with small number of servers (chubby cell).

== System Components ==
* Chubby Cell
** Handles the actual locks
** Mainly consists of 5 servers known as replicas
** Consensus protocol ([https://en.wikipedia.org/wiki/Paxos_(computer_science) Paxos]) is used to elect the master from replicas

* Client
** Used by programs to request and use locks

== Main Features ==
* Implemented as a semi POSIX-compliant file-system with a 256KB file limit
** Permissions are for files only, not folders, thus breaking some compatibility
** Trivial to use by programs; just use the standard ''fopen()'' family of calls

* Uses consensus algorithm (paxos) among a set of servers to agree on who is the master that is in charge of the metadata

* Meant for locks that last hours or days, not seconds (thus, "coarse grained")

* A master server can handle tens of thousands of simultaneous connections
** This can be further improved by using caching servers as most of the traffic are keep-alive messages

* Used by GFS for electing master server

* Also used by Google as a nameserver

== Issues ==
* Due to the use of Paxos (the only algorithm for this problem), the chubby cell is limited to only 5 servers. While this limits fault-tolerance, in practice this is more than enough.
* Since it has a file-system client interface, many programmers tend to abuse the system and need education (even inside Google)
* Clients need to constantly ping Chubby to verify that they still exist. This is to ensure that a server disappearing while holding a lock does not indefinitely hold that lock.
* Clients must also consequently re-verify that they hold a lock that they think they hold because Chubby may have timed them out.

DistOS 2015W Session 7

2015-04-06T05:03:56Z

Abeinges: /* Issues */

DistOS 2015W Session 7

2015-04-06T05:02:07Z

Abeinges: /* Main Features */

DistOS 2015W Session 7

2015-04-06T05:01:06Z

Abeinges: /* Ceph */

= Ceph =
Unlike GFS that was talked about previously, this is a general purpose distributed file system. It follows the same general model of distribution as GFS and Amoeba.

== Main Components ==
* Client

* Cluster of Object Storage Devices (OSD)
** It basically stores data and metadata and clients communicate directly with it to perform IO operations
** Data is stored in objects (variable size chunks)

* Meta-data Server (MDS)
** It is used to manage the file and directories. Clients interact with it to perform metadata operations like open, rename. It manages the capabilities of a client.
** Clients '''do not''' need to access MDSs to find where data is stored, improving scalability (more on that below)

== Key Features ==
* Decoupled data and metadata

* Dynamic Distributed Metadata Management

** It distributes the metadata among multiple metadata servers using dynamic sub-tree partitioning, meaning folders that get used more often get their meta-data replicated to more servers, spreading the load. This happens completely automatically

* Object based storage
** Uses cluster of OSDs to form a Reliable Autonomic Distributed Object-Store (RADOS) for Ceph failure detection and recovery

* CRUSH (Controlled, Replicated, Under Scalable, Hashing)
** The hashing algorithm used to calculate the location of object instead of looking for them
** This significantly reduces the load on the MDSs because each client has enough information to independently determine where things should be located.
** Responsible for automatically moving data when ODSs are added or removed (can be simplified as ''location = CRUSH(filename) % num_servers'')
** The CRUSH paper on Ceph’s website can be [http://ceph.com/papers/weil-crush-sc06.pdf viewed here]

* RADOS (Reliable Autonomic Distributed Object-Store) is the object store for Ceph

= Chubby =
It is a coarse grained lock service, made and used internally by Google, that serves multiple clients with small number of servers (chubby cell).

== System Components ==
* Chubby Cell
** Handles the actual locks
** Mainly consists of 5 servers known as replicas
** Consensus protocol ([https://en.wikipedia.org/wiki/Paxos_(computer_science) Paxos]) is used to elect the master from replicas

* Client
** Used by programs to request and use locks

== Main Features ==
* Implemented as a semi POSIX-compliant file-system with a 256KB file limit
** Permissions are for files only, not folders, thus breaking some compatibility
** Trivial to use by programs; just use the standard ''fopen()'' family of calls

* Uses consensus algorithm among a set of servers to agree on who is the master that is in charge of the metadata

* Meant for locks that last hours or days, not seconds (thus, "coarse grained")

* A master server can handle tens of thousands of simultaneous connections
** This can be further improved by using caching servers as most of the traffic are keep-alive messages

* Used by GFS for electing master server

* Also used by Google as a nameserver

== Issues ==
* Due to the use of Paxos (the only algorithm for this problem), the chubby cell is limited to only 5 servers. While this limits fault-tolerance, in practice this is more than enough.
* Since it has a file-system client interface, many programmers tend to abuse the system and need education (even inside Google)

DistOS 2015W Session 6

2015-04-06T04:57:24Z

Abeinges: /* Group 1 */

==Midterm==

The midterm from last year [http://homeostasis.scs.carleton.ca/~soma/distos/2015w/comp4000-2014w-midterm.pdf is now available].

==Group 1==

'''Team:''' Kirill, Jamie, Alexis, Veena, Khaled, Hassan

{| class="wikitable"
|-
!
! FARSITE
! OceanStore
|-
! Fault Tolerance
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
| Used Byzantine Fault Tolerance Algorithm - Did not manage well
|-
! Cryptography
| Trusted Certificates
| A strong cryptographic algorithm on read-only operations
|-
! Implementation
| Did not mention what programming they used, but it was based on Windows. They did not implement the file system
| Implemented in Java
|-
! Scalability
| Scalable to a University or large corporations, maximum 105
| Worldwide scalability, maximum 1010
|-
! File Usage
| Was designed for general purpose files
| Was designed for small file sizes
|-
! Scope
| All clients sharing the available resources
| Transient centralized service
|-
! Object Model
| Didn't use the object model
| Used the object model
|}

==Group 2==
'''Team''': Apoorv, Ambalica, Ashley, Eric, Mert, Shivjot

<table style="width: 100%;" border="1" cellpadding="4" cellspacing="0" class="wikitable">
<tr>
<td>'''Farsite'''</td>
<td>'''OceanStore'''</td>
</tr>
<tr>
<td>Implemented Content Leases</td>
<td>Update Model handled data consistency, no Leases</td>
</tr>
<tr>
<td>Single tier, peer to peer model</td>
<td>Two tier, server client model</td>
</tr>
<tr>
<td>Scope of ten to the five</td>
<td>Global Scope (ten to the ten)</td>
</tr>
<tr>
<td>Cryptographic public, private key security</td>
<td>Read and write privileges</td>
</tr>
<tr>
<td>Randomized data replication</td>
<td>Nomadic Data concept</td>
</tr>
</table>

==Group 3==
'''Team''': DANY, MOE, DEEP, SAMEER, TROY

----
'''
== FARSITE ==
'''

'''Security'''
• Cascading certificates system through directory hierarchy
• Keys
• Three types of certificates
• CFS required to authorized certificate
• Because directory groups only modify their shared state via a Byzantine-fault-tolerant protocol, we trust the group not to make
an incorrect update to directory metadata. This metadata includes an access control list (ACL) of public keys of all users
who are authorized writers to that directory and to files in it
• Both file content and user-sensitive metadata (meaning file and directory names) are encrypted for privacy.

'''System Architecture'''
• Client Monitor, directory group, file host
• When space runs out in directory group, delegate’s ownership to sub tree to other delegate group.

----
'''
== OCEANSTORE ==
'''

'''Security'''
• GUID and ACLs used for write, encryption used for reads.
• To prevent unauthorized reads, it encrypts
all data in the system that is not completely public and distributes the encryption key to those users with read permission

DistOS 2015W Session 5

2015-04-06T04:56:10Z

Abeinges: /* Google File System */

= The Clouds Distributed Operating System =
It is a distributed OS running on a set of computers that are interconnected by a group of network. It basically unifies different computers into a single component.

The OS is based on 2 patterns:
* Message Based OS
* Object Based OS

== Object Thread Model ==

The structure of this is based on object thread model. It has set of objects which are defined by the class. Objects respond to messages. Sending message to object causes object to execute the method and then reply back.

The system has '''active objects''' and '''passive objects'''.

# Active objects are the objects which have one or more processes associated with them and further they can communicate with the external environment.
# Passive objects are those that currently do not have an active thread executing in them.

The content of the Clouds data is long lived. Since the memory is implemented as a single-level store, the data exists forever and can survive system crashes and shut downs.

== Threads ==

The threads are the logical path of execution that traverse objects and executes code in them. The Clouds thread is not bound to a single address space. Several threads can enter an object simultaneously and execute concurrently. The nature of the Clouds object prohibits a thread from accessing any data outside the current address space in which it is executing.

== Interaction Between Objects and Threads ==

# Inter object interfaces are procedural
# Invocations work across machine boundaries
# Objects in clouds unify concept of persistent storage and memory to create address space, thus making the programming simpler.
# Control flow achieved by threads invoking objects.

== Clouds Environment ==

# Integrates set of homogeneous machines into one seamless environment
# There are three logical categories of machines- Compute Server, User Workstation and Data server.

= Plan 9 =

Plan 9 is a general purpose, multi-user and mobile computing environment physically distributed across machines. Development of the system began in the late 1980s. The system was built at Bell Labs - the birth place of Unix. The original Unix OS had no support for networking, and there were many attempts over the years by others to create distributed systems with Unix compatibility. Plan 9, however, is distributed systems done following the original Unix philosophy.

The goals of this system were:
# To build a distributed system that can be centrally administered.
# To be cost effective using cheap, modern microcomputers.

The distribution itself is transparent to most programs. This is made possible by two properties:
# A per process group namespace.
# Uniform access to most resources by representing them as a file.

== Unix Compatibility ==

The commands, libraries and system calls are similar to that of Unix and therefore a casual user cannot distinguish between these two. The problems in UNIX were too deep to fix but still the various ideas were brought along. The problems addressed badly by UNIX were improved. Old tools were dropped and others were polished and reused.

== Similaritieis with the UNIX ==
<ul>
<li>shell</li>
<li>Various C compilers</li>
</ul>

== Unique Features ==

What actually distinguishes Plan 9 is its '''organization'''. Plan 9 is divided along the lines of service function.
* CPU services and terminals use same kernel.
* Users may choose to run programs locally or remotely on CPU servers.
* It lets the user choose whether they want a distributed or centralized system.

The design of Plan 9 is based on 3 principles:
# Resources are named and accessed like files in hierarchical file system.
# Standard protocol 9P.
# Disjoint hierarchical provided by different services are joined together into single private hierarchical file name space.

=== Virtual Namespaces ===

In a virtual namespace, a user boots a terminal or connects to a CPU server and then a new process group is created. Processes in group can either add to or rearrange their name space using two system calls - mount and bind.

* '''Mount''' is used to attach new file system to a point in name space.
* '''Bind''' is used to attach a kernel resident (existing, mounted) file system to name space and also arrange pieces of name space.
* There is also '''unbind''' which undoes the effects of the other two calls.

Namespaces in Plan 9 are on a per-process basis. While everything had a way reference resources with a unique name, using mount and bind every process could build a custom namespace as they saw fit.

Since most resources are in the form of files (and folders), the term ''namespace'' really only refers to the filesystem layout.

=== Parallel Programming ===
Parallel programming was supported in two ways:
* Kernel provides simple process model and carefully designed system calls for synchronization.
* Programming language supports concurrent programming.

== Legacy ==

Even though Plan 9 is no longer developed, the good ideas from the system still exist today. For example, the ''/proc'' virtual filesystem which displays current process information in the form of files exists in modern Linux kernels.

= Google File System =

It is scalable, distributed file system for large, data intensive applications. It is crafted to Google's unique needs as a search engine company.

Unlike most filesystems, GFS must be implemented by individual applications and is not part of the kernel. While this introduces some technical overhead, it gives the system more freedom to implement or not implement certain non-standard features.

== Architecture ==

The architecture of the Google file system consists of a single master, multiple chunk-servers and multiple clients. Chunk servers are used to store the data in uniformly sized chunks. Each chunk is identified by globally unique 64 bit handle assigned by master at the time of creation. The chunks are split into 64KB blocks, each with its own hashsum for data integrity checks. The chunks are replicated between servers, 3 by default. The master maintains all the file system metadata which includes the namespace and chunk location.

Each of the chunks is 64 MB large (contrast this to the typical filesystem sectors of 512 or 4096 bytes) as the system is meant to hold enormous amount of data - namely the internet. The large chunk size is also important for the scalability of the system - the larger the chunk size, the less metadata the master server has to store for any given amount of data. With the current size, the master server is able to store the entirety of the metadata in memory, increasing performance by a significant margin.

== Operation ==

Master and Chunk server communication consists of
# checking whether there any chunk-server is down
# checking if any file is corrupted
# deleting stale chunks

When a client wants to do some operations on the chunks
# it first asks the master server for the list of servers that store the parts of a file it wants to access
# it receives a list of chunk servers, with multiple servers for each chunk
# it finally communicates with the the chunk servers to perform the operation

The system is geared towards appends and sequential reads. This is why the master server responds with multiple server addresses for each chunk - the client can then request a small piece from each server, increasing the data throughput linearly with the number of servers. Writes, in general, are in the form of a special ''append'' system call. When appending, there is no chance that two clients will want to write to the same location at the same time. This helps avoid any potential synchronization issues. If there are multiple appends to the same file at the same time, the chunk servers are free to order them as they wish (chunks on each server are not guaranteed to be byte-for-byte identical). Changes may also be applied multiple times. These issues are left for the application using GFS to resolve themselves. While a problem in the general sense, this is good enough for Google's needs.

== Redundancy ==

GFS is built with failure in mind. The system expects that at any time, there is some server or disk that is malfunctioning. The system deals with the failures as follows.

=== Chunk Servers ===

By default, chunks are replicated to three servers. This exact number depends on the application in question doing the write. When a chunk server finds that some of its data is corrupt, it grabs the data from other servers to repair itself{{Citation needed}}.

=== Master Server ===

For efficiency, there is only a single live master server at a time. While not making the system completely distributed, it avoid many synchronization problems and suits Google's needs. At any point in time, there are multiple read-only master servers that copy metadata from the currently live master. Should it go down, they will serve any read operations from the clients, until one of the hot spares is promoted to being the new live master server.

DistOS 2015W Session 5

2015-04-06T04:54:02Z

Abeinges: /* Operation */

= The Clouds Distributed Operating System =
It is a distributed OS running on a set of computers that are interconnected by a group of network. It basically unifies different computers into a single component.

The OS is based on 2 patterns:
* Message Based OS
* Object Based OS

== Object Thread Model ==

The structure of this is based on object thread model. It has set of objects which are defined by the class. Objects respond to messages. Sending message to object causes object to execute the method and then reply back.

The system has '''active objects''' and '''passive objects'''.

# Active objects are the objects which have one or more processes associated with them and further they can communicate with the external environment.
# Passive objects are those that currently do not have an active thread executing in them.

The content of the Clouds data is long lived. Since the memory is implemented as a single-level store, the data exists forever and can survive system crashes and shut downs.

== Threads ==

The threads are the logical path of execution that traverse objects and executes code in them. The Clouds thread is not bound to a single address space. Several threads can enter an object simultaneously and execute concurrently. The nature of the Clouds object prohibits a thread from accessing any data outside the current address space in which it is executing.

== Interaction Between Objects and Threads ==

# Inter object interfaces are procedural
# Invocations work across machine boundaries
# Objects in clouds unify concept of persistent storage and memory to create address space, thus making the programming simpler.
# Control flow achieved by threads invoking objects.

== Clouds Environment ==

# Integrates set of homogeneous machines into one seamless environment
# There are three logical categories of machines- Compute Server, User Workstation and Data server.

= Plan 9 =

Plan 9 is a general purpose, multi-user and mobile computing environment physically distributed across machines. Development of the system began in the late 1980s. The system was built at Bell Labs - the birth place of Unix. The original Unix OS had no support for networking, and there were many attempts over the years by others to create distributed systems with Unix compatibility. Plan 9, however, is distributed systems done following the original Unix philosophy.

The goals of this system were:
# To build a distributed system that can be centrally administered.
# To be cost effective using cheap, modern microcomputers.

The distribution itself is transparent to most programs. This is made possible by two properties:
# A per process group namespace.
# Uniform access to most resources by representing them as a file.

== Unix Compatibility ==

The commands, libraries and system calls are similar to that of Unix and therefore a casual user cannot distinguish between these two. The problems in UNIX were too deep to fix but still the various ideas were brought along. The problems addressed badly by UNIX were improved. Old tools were dropped and others were polished and reused.

== Similaritieis with the UNIX ==
<ul>
<li>shell</li>
<li>Various C compilers</li>
</ul>

== Unique Features ==

What actually distinguishes Plan 9 is its '''organization'''. Plan 9 is divided along the lines of service function.
* CPU services and terminals use same kernel.
* Users may choose to run programs locally or remotely on CPU servers.
* It lets the user choose whether they want a distributed or centralized system.

The design of Plan 9 is based on 3 principles:
# Resources are named and accessed like files in hierarchical file system.
# Standard protocol 9P.
# Disjoint hierarchical provided by different services are joined together into single private hierarchical file name space.

=== Virtual Namespaces ===

In a virtual namespace, a user boots a terminal or connects to a CPU server and then a new process group is created. Processes in group can either add to or rearrange their name space using two system calls - mount and bind.

* '''Mount''' is used to attach new file system to a point in name space.
* '''Bind''' is used to attach a kernel resident (existing, mounted) file system to name space and also arrange pieces of name space.
* There is also '''unbind''' which undoes the effects of the other two calls.

Namespaces in Plan 9 are on a per-process basis. While everything had a way reference resources with a unique name, using mount and bind every process could build a custom namespace as they saw fit.

Since most resources are in the form of files (and folders), the term ''namespace'' really only refers to the filesystem layout.

=== Parallel Programming ===
Parallel programming was supported in two ways:
* Kernel provides simple process model and carefully designed system calls for synchronization.
* Programming language supports concurrent programming.

== Legacy ==

Even though Plan 9 is no longer developed, the good ideas from the system still exist today. For example, the ''/proc'' virtual filesystem which displays current process information in the form of files exists in modern Linux kernels.

= Google File System =

It is scalable, distributed file system for large, data intensive applications. It is crafted to Google's unique needs as a search engine company.

Unlike most filesystems, GFS must be implemented by individual applications and is not part of the kernel. While this introduces some technical overhead, it gives the system more freedom to implement or not implement certain non-standard features.

== Architecture ==

The architecture of the Google file system consists of a single master, multiple chunk-servers and multiple clients. Chunk servers are used to store the data in uniformly sized chunks. Each chunk is identified by globally unique 64 bit handle assigned by master at the time of creation. The chunks are split into 64KB blocks, each with its own hashsum for data integrity checks. The chunks are replicated between servers, 3 by default. The master maintains all the file system metadata which includes the namespace and chunk location.

Each of the chunks is 64 MB large (contrast this to the typical filesystem sectors of 512 or 4096 bytes) as the system is meant to hold enormous amount of data - namely the internet. The large chunk size is also important for the scalability of the system - the larger the chunk size, the less metadata the master server has to store for any given amount of data. With the current size, the master server is able to store the entirety of the metadata in memory, increasing performance by a significant margin.

== Operation ==

Master and Chunk server communication consists of
# checking whether there any chunk-server is down
# checking if any file is corrupted
# deleting stale chunks

When a client wants to do some operations on the chunks
# it first asks the master server for the list of servers that store the parts of a file it wants to access
# it receives a list of chunk servers, with multiple servers for each chunk
# it finally communicates with the the chunk servers to perform the operation

The system is geared towards appends and sequential reads. This is why the master server responds with multiple server addresses for each chunk - the client can then request a small piece from each server, increasing the data throughput linearly with the number of servers. Writes, in general, are in the form of a special ''append'' system call. When appending, there is no chance that two clients will want to write to the same location at the same time. This helps avoid any potential synchronization issues. If there are multiple appends to the same file at the same time, the chunk servers are free to order them as they wish (chunks on each server are not guaranteed to be byte-for-byte identical). While a problem in the general sense, this is good enough for Google's needs.

== Redundancy ==

GFS is built with failure in mind. The system expects that at any time, there is some server or disk that is malfunctioning. The system deals with the failures as follows.

=== Chunk Servers ===

By default, chunks are replicated to three servers. This exact number depends on the application in question doing the write. When a chunk server finds that some of its data is corrupt, it grabs the data from other servers to repair itself{{Citation needed}}.

=== Master Server ===

For efficiency, there is only a single live master server at a time. While not making the system completely distributed, it avoid many synchronization problems and suits Google's needs. At any point in time, there are multiple read-only master servers that copy metadata from the currently live master. Should it go down, they will serve any read operations from the clients, until one of the hot spares is promoted to being the new live master server.

DistOS 2015W Session 5

2015-04-06T04:51:02Z

Abeinges: /* Plan 9 */

= The Clouds Distributed Operating System =
It is a distributed OS running on a set of computers that are interconnected by a group of network. It basically unifies different computers into a single component.

The OS is based on 2 patterns:
* Message Based OS
* Object Based OS

== Object Thread Model ==

The structure of this is based on object thread model. It has set of objects which are defined by the class. Objects respond to messages. Sending message to object causes object to execute the method and then reply back.

The system has '''active objects''' and '''passive objects'''.

# Active objects are the objects which have one or more processes associated with them and further they can communicate with the external environment.
# Passive objects are those that currently do not have an active thread executing in them.

The content of the Clouds data is long lived. Since the memory is implemented as a single-level store, the data exists forever and can survive system crashes and shut downs.

== Threads ==

The threads are the logical path of execution that traverse objects and executes code in them. The Clouds thread is not bound to a single address space. Several threads can enter an object simultaneously and execute concurrently. The nature of the Clouds object prohibits a thread from accessing any data outside the current address space in which it is executing.

== Interaction Between Objects and Threads ==

# Inter object interfaces are procedural
# Invocations work across machine boundaries
# Objects in clouds unify concept of persistent storage and memory to create address space, thus making the programming simpler.
# Control flow achieved by threads invoking objects.

== Clouds Environment ==

# Integrates set of homogeneous machines into one seamless environment
# There are three logical categories of machines- Compute Server, User Workstation and Data server.

= Plan 9 =

Plan 9 is a general purpose, multi-user and mobile computing environment physically distributed across machines. Development of the system began in the late 1980s. The system was built at Bell Labs - the birth place of Unix. The original Unix OS had no support for networking, and there were many attempts over the years by others to create distributed systems with Unix compatibility. Plan 9, however, is distributed systems done following the original Unix philosophy.

The goals of this system were:
# To build a distributed system that can be centrally administered.
# To be cost effective using cheap, modern microcomputers.

The distribution itself is transparent to most programs. This is made possible by two properties:
# A per process group namespace.
# Uniform access to most resources by representing them as a file.

== Unix Compatibility ==

The commands, libraries and system calls are similar to that of Unix and therefore a casual user cannot distinguish between these two. The problems in UNIX were too deep to fix but still the various ideas were brought along. The problems addressed badly by UNIX were improved. Old tools were dropped and others were polished and reused.

== Similaritieis with the UNIX ==
<ul>
<li>shell</li>
<li>Various C compilers</li>
</ul>

== Unique Features ==

What actually distinguishes Plan 9 is its '''organization'''. Plan 9 is divided along the lines of service function.
* CPU services and terminals use same kernel.
* Users may choose to run programs locally or remotely on CPU servers.
* It lets the user choose whether they want a distributed or centralized system.

The design of Plan 9 is based on 3 principles:
# Resources are named and accessed like files in hierarchical file system.
# Standard protocol 9P.
# Disjoint hierarchical provided by different services are joined together into single private hierarchical file name space.

=== Virtual Namespaces ===

In a virtual namespace, a user boots a terminal or connects to a CPU server and then a new process group is created. Processes in group can either add to or rearrange their name space using two system calls - mount and bind.

* '''Mount''' is used to attach new file system to a point in name space.
* '''Bind''' is used to attach a kernel resident (existing, mounted) file system to name space and also arrange pieces of name space.
* There is also '''unbind''' which undoes the effects of the other two calls.

Namespaces in Plan 9 are on a per-process basis. While everything had a way reference resources with a unique name, using mount and bind every process could build a custom namespace as they saw fit.

Since most resources are in the form of files (and folders), the term ''namespace'' really only refers to the filesystem layout.

=== Parallel Programming ===
Parallel programming was supported in two ways:
* Kernel provides simple process model and carefully designed system calls for synchronization.
* Programming language supports concurrent programming.

== Legacy ==

Even though Plan 9 is no longer developed, the good ideas from the system still exist today. For example, the ''/proc'' virtual filesystem which displays current process information in the form of files exists in modern Linux kernels.

= Google File System =

It is scalable, distributed file system for large, data intensive applications. It is crafted to Google's unique needs as a search engine company.

Unlike most filesystems, GFS must be implemented by individual applications and is not part of the kernel. While this introduces some technical overhead, it gives the system more freedom to implement or not implement certain non-standard features.

== Architecture ==

The architecture of the Google file system consists of a single master, multiple chunk-servers and multiple clients. Chunk servers are used to store the data in uniformly sized chunks. Each chunk is identified by globally unique 64 bit handle assigned by master at the time of creation. The chunks are split into 64KB blocks, each with its own hashsum for data integrity checks. The chunks are replicated between servers, 3 by default. The master maintains all the file system metadata which includes the namespace and chunk location.

Each of the chunks is 64 MB large (contrast this to the typical filesystem sectors of 512 or 4096 bytes) as the system is meant to hold enormous amount of data - namely the internet. The large chunk size is also important for the scalability of the system - the larger the chunk size, the less metadata the master server has to store for any given amount of data. With the current size, the master server is able to store the entirety of the metadata in memory, increasing performance by a significant margin.

== Operation ==

Master and Chunk server communication consists of
# checking whether there is any chunk-server is down.
# checking if any file is corrupted.
# deleting stale chunks

When a client want to do some operations on the chunks
# it first asks the master server for the list of servers that store the parts of a file it wants to access
# it receives a list of chunk servers, with multiple servers for each chunk
# it finally communicates with the the chunk servers to perform the operation

The system is geared towards appends and sequential reads. This is why the master server responds with multiple server addresses for each chunk - the client can then request a small piece from each server, increasing the data throughput linearly with the number of servers. Writes, in general, are in the form of a special ''append'' system call. When appending, there is no chance that two clients will want to write to the same location at the same time. This helps avoid any potential synchronization issues. If there are multiple appends to the same file at the same time, the chunk servers are free to order them as they wish (chunks on each server are not guaranteed to be byte-for-byte identical). While a problem in the general sense, this is good enough for Google's needs.

== Redundancy ==

GFS is built with failure in mind. The system expects that at any time, there is some server or disk that is malfunctioning. The system deals with the failures as follows.

=== Chunk Servers ===

By default, chunks are replicated to three servers. This exact number depends on the application in question doing the write. When a chunk server finds that some of its data is corrupt, it grabs the data from other servers to repair itself{{Citation needed}}.

=== Master Server ===

For efficiency, there is only a single live master server at a time. While not making the system completely distributed, it avoid many synchronization problems and suits Google's needs. At any point in time, there are multiple read-only master servers that copy metadata from the currently live master. Should it go down, they will serve any read operations from the clients, until one of the hot spares is promoted to being the new live master server.

DistOS 2015W Session 4

2015-04-06T04:45:54Z

Abeinges: /* Andrew File System */

= Andrew File System =
AFS (Andrew File System) was set up as a direct response to NFS. Essentially universities found issues when they tried to scale NFS in a way that would allow them to share files amongst their staff effectively. AFS was more scalable than NFS because read-write operations happened locally before they were committed to the server (data store).

Since AFS copies files locally when they were opened and only sends the data back when they are closed, all operations between opening and closing the file are very fast and do not need to access the network. NFS works with files remotely, so there is no data to transfer when opening/closing the file, making those operations instant.

There are several problems with this design, however:
* The local system must have enough space to temporarily store the file.
* Opening and closing the files requires a lot of bandwidth for large files. To read even a single byte, the entire file must be retrieved (later versions remedied this).
* If the close operation fails, the system will not have the updated version of the file. Many programs are designed around local filesystems, and therefore don't even check the return value of the close operation (as this is unlikely to fail on a local FS), giving users the false impression that everything went well.

Given all this, AFS was suitable for working with small files, not large ones, limiting its usefulness. It is also notoriously annoying to set up as it is geared towards university-sized networks, further limiting its success.

= Amoeba Operating System =

=== Capablities: ===
* Pointer to the object
* Capability assigning right to perform to some operation to the object ticket
* Communicate wide area network
* a kind of ticket or key that allows the holder of the capa- bility to perform some (not neces- sarily all)
* Each user process owns some collection of capabilities, which together define the set of objects it may access and the types of operations that my ne performed on each
* After the server has performed the operation, it sends back a reply message that unblocks the client
* Sending messages, blocking and accepting forms the remote procedure call that can be encapsulate using to make entire remote operation look like local procedure
* Second field: used by the sever to identify which of its objects is being addressed server port and object number identify object which operation to performed
* Generates 48-bit random number
* The third field is the right field which contains a bit map telling which operation the holder of the capability may performed
* X11 Window management

=== Thread Management: ===
* Same process have multiple thread and each process has its own registered counter and stack
* Behave like process
* It can synchronized using mutex semaphore
* File: Multiple thread,
* Blocked when there's multiple threads
* Buttlet thread the mutex
* The careful reader may have noticed that user process can pull 813kbytes/sec

= Unique features =

== Pool processors ==
Pool processors are group of CPUs that are dynamically allocated as user needs. When a program is executed, any of the available processor run.

== Supported architectures ==
Many different processor architectures are supported including:
* i80386 (Pentium)
* 68K
* SPARC

= The V Distributed System =

* First tent in V design: High Performance communication is the most critical facility for distributed systems.
* Second; The protocols, not the software, define the system.
* Third; a relatively small operating system kernel can implement the basic protocols and services providing a simple network-transparent process, address space & communication model.

=== Ideas that significantly affected the design ===
* Shared Memory.
* Dealing with group of entities same as they deal w/individual entities.
* Efficient file caching mechanism using the virtual memory caching mechanism.

=== Design Decisions ===
* Designed for a cluster of workstations with high speed network access ( only really supports LAN ).
* Abstract the physical architecture of the participating workstations, by defining common protocols providing well-defined interfaces.

V was run on LAN, and its developers developed a really fast IPC protocol which allowed for it to be a fasted distributed operating system in a small geographic area. Aside from the IPC protocols, V also implemented RPC calls in the background.

V uses the strong consistency model. This model can cause issues with concurrency because in V files are a memory space. Thus two different users accessing the same file and in fact accessing the same memory location. This could result in issues unless there is an effective implementation to deal with multiple versions, etc.

VMTP protocol was used for communication. It supports request-respond behavior. Besides, it provides transparency, group communication facility and flow control. It is pretty much like TCP.

DistOS 2015W Session 3

2015-04-06T04:42:05Z

Abeinges: /* Unix */

= Multics =

'''Team:''' Sameer, Shivjot, Ambalica, Veena

It came into being in the 1960s and it completely vanished in 2000s. It was started by Bell, General Electric and MIT but Bell backed out of the project in 1969.
Multics is a time sharing OS which provides multitasking and multiprogramming. It is not a distributed OS but it a centralized system which was written in machine-specific assembly.

It provides the following features:
# Utility Computing
# Access Control Lists
# Single level storage
# Dynamic linking
#* Sharded libraries or files can be loaded and linked to Random Access Memory at run time
# Hot swapping
# Multiprocessing System
# Ring oriented Security
#* It provides number of levels of authorization within the computer system
#* Still present in some form today, inside both processors (like x86) and operating systems

= Unix =

Unix in its original conception is a small, minimal API system designed by two guys from Bell Labs. It was essentially an OS that was optimized for the needs of programmers but not much beyond that. The UNIX OS ran on one computer, and terminals ran from that one computer. Thus it is not a distributed operating system as it is centralized and implements time sharing. In fact, it didn't even have support for networking in the first version.

The C language was created specifically for Unix, as the creators wanted to create a machine-agnostic language for the operating system.

Most features from Unix are still available in present day Unix-based systems. For example, the shell, with its piping capabilities, is still used today in its original form.

= NFS =

NFS is a protocol for working with distributed file systems transparently using RPC. These connections are not secure. Sun wanted to encrypt these RPC connections but encryption would result in government regulations that Sun wanted to avoid in order to sell NFS over seas.

<blockquote>
Sun wanted to secure their NFS with encryption, but at the time encryption was regulated like munitions in the United States. Exporting any product that had encryption was impossible, but Sun needed those sales abroad. To avoid these regulations, Sun decided to sell the insecure NFS version of the system.
</blockquote>

'''Team''': Mert

= Locus =

# Not scalable
#* The synchronization algorithms were so slow, they only managed to run it on five computers
#* Every computer stores a copy of every file
#* Also used CAS to manage files
# Not efficient with abstractions
#* Trying to distribute files and processes
#Allowed for process migration
#Transparency
#* It provided network transparency to “disguise” its distributed context.
#Dynamic reconfiguration. (It adapts topology changes.)

Locus has lots of similarities with the today's systems. It uses replication and partitioning which are employed in cloud and distributed systems.

'''Team''': Eric

= Sprite =

''' Team''': Jamie, Hassan, Khaled

Sprite had the following Design Features:
# Network Transparency
# Process Migration, file transfer between computers
#* User could initiate a process migration to an idle machine, and if the machine was no longer idle; due to it being used by another user, the system would take care of the process migration to another machine
# Handling Cache Consistency
#* Sequential file sharing ==> By using a version number for each file
#* Concurrent write sharing ==> Disable cache to clients, enable write-blocking and other methods
# Implemented a caching system that sped up performance
# Implemented a log structured file system
#* They realized that with increasing amounts of RAM in computers which can be used for caching, writes to the disk were the main bottleneck, not reads.
#* Log structured file-systems are optimized for writes, as changes to previous data are appended at the current position.
#* This allows for very fast, sequential writes.
#* Example: SSD (Solid-state disks)

The main features to take away from the Sprite system is that it implemented a log structured file system, and implemented caching to increase performance.

DistOS 2015W Session 3

2015-04-06T04:40:44Z

Abeinges: /* Sun NFS */

= Multics =

'''Team:''' Sameer, Shivjot, Ambalica, Veena

It came into being in the 1960s and it completely vanished in 2000s. It was started by Bell, General Electric and MIT but Bell backed out of the project in 1969.
Multics is a time sharing OS which provides multitasking and multiprogramming. It is not a distributed OS but it a centralized system which was written in machine-specific assembly.

It provides the following features:
# Utility Computing
# Access Control Lists
# Single level storage
# Dynamic linking
#* Sharded libraries or files can be loaded and linked to Random Access Memory at run time
# Hot swapping
# Multiprocessing System
# Ring oriented Security
#* It provides number of levels of authorization within the computer system
#* Still present in some form today, inside both processors (like x86) and operating systems

= Unix =

Unix in its original conception is a small, minimal API system designed by two guys from Bell Labs. It was essentially a OS that would be easy to grasp for an programmer but not much beyond that. The UNIX OS ran on one computer, and terminals ran from that one computer. Thus it is not a distributed operating system as it is centralized and implements time sharing. In fact, it didn't even have support for networking in the first version.

The C language was created specifically for Unix, as the creators wanted to create a machine-agnostic language for the operating system.

Most features from Unix are still available in present day Unix-based systems. For example, the shell, with its piping capabilities, is still used today in its original form.

= NFS =

NFS is a protocol for working with distributed file systems transparently using RPC. These connections are not secure. Sun wanted to encrypt these RPC connections but encryption would result in government regulations that Sun wanted to avoid in order to sell NFS over seas.

<blockquote>
Sun wanted to secure their NFS with encryption, but at the time encryption was regulated like munitions in the United States. Exporting any product that had encryption was impossible, but Sun needed those sales abroad. To avoid these regulations, Sun decided to sell the insecure NFS version of the system.
</blockquote>

'''Team''': Mert

= Locus =

# Not scalable
#* The synchronization algorithms were so slow, they only managed to run it on five computers
#* Every computer stores a copy of every file
#* Also used CAS to manage files
# Not efficient with abstractions
#* Trying to distribute files and processes
#Allowed for process migration
#Transparency
#* It provided network transparency to “disguise” its distributed context.
#Dynamic reconfiguration. (It adapts topology changes.)

Locus has lots of similarities with the today's systems. It uses replication and partitioning which are employed in cloud and distributed systems.

'''Team''': Eric

= Sprite =

''' Team''': Jamie, Hassan, Khaled

Sprite had the following Design Features:
# Network Transparency
# Process Migration, file transfer between computers
#* User could initiate a process migration to an idle machine, and if the machine was no longer idle; due to it being used by another user, the system would take care of the process migration to another machine
# Handling Cache Consistency
#* Sequential file sharing ==> By using a version number for each file
#* Concurrent write sharing ==> Disable cache to clients, enable write-blocking and other methods
# Implemented a caching system that sped up performance
# Implemented a log structured file system
#* They realized that with increasing amounts of RAM in computers which can be used for caching, writes to the disk were the main bottleneck, not reads.
#* Log structured file-systems are optimized for writes, as changes to previous data are appended at the current position.
#* This allows for very fast, sequential writes.
#* Example: SSD (Solid-state disks)

The main features to take away from the Sprite system is that it implemented a log structured file system, and implemented caching to increase performance.

DistOS 2015W Session 3

2015-04-06T04:37:44Z

Abeinges: /* Multics */

= Multics =

'''Team:''' Sameer, Shivjot, Ambalica, Veena

It came into being in the 1960s and it completely vanished in 2000s. It was started by Bell, General Electric and MIT but Bell backed out of the project in 1969.
Multics is a time sharing OS which provides multitasking and multiprogramming. It is not a distributed OS but it a centralized system which was written in machine-specific assembly.

It provides the following features:
# Utility Computing
# Access Control Lists
# Single level storage
# Dynamic linking
#* Sharded libraries or files can be loaded and linked to Random Access Memory at run time
# Hot swapping
# Multiprocessing System
# Ring oriented Security
#* It provides number of levels of authorization within the computer system
#* Still present in some form today, inside both processors (like x86) and operating systems

= Unix =

Unix in its original conception is a small, minimal API system designed by two guys from Bell Labs. It was essentially a OS that would be easy to grasp for an programmer but not much beyond that. The UNIX OS ran on one computer, and terminals ran from that one computer. Thus it is not a distributed operating system as it is centralized and implements time sharing. In fact, it didn't even have support for networking in the first version.

The C language was created specifically for Unix, as the creators wanted to create a machine-agnostic language for the operating system.

Most features from Unix are still available in present day Unix-based systems. For example, the shell, with its piping capabilities, is still used today in its original form.

= Sun NFS =

The Sun NFS OS implemented networking using RPC connections. These connections are not secure. Sun wanted to encrypt these RPC connections but encryption would result in government regulations that Sun wanted to avoid in order to sell NFS over seas.

<blockquote>
Sun wanted to secure their NFS with encryption, but at the time encryption was regulated like munitions in the United States. Exporting any product that had encryption was impossible, but Sun needed those sales abroad. To avoid these regulations, Sun decided to sell the insecure NFS version of the system.
</blockquote>

'''Team''': Mert

= Locus =

# Not scalable
#* The synchronization algorithms were so slow, they only managed to run it on five computers
#* Every computer stores a copy of every file
#* Also used CAS to manage files
# Not efficient with abstractions
#* Trying to distribute files and processes
#Allowed for process migration
#Transparency
#* It provided network transparency to “disguise” its distributed context.
#Dynamic reconfiguration. (It adapts topology changes.)

Locus has lots of similarities with the today's systems. It uses replication and partitioning which are employed in cloud and distributed systems.

'''Team''': Eric

= Sprite =

''' Team''': Jamie, Hassan, Khaled

Sprite had the following Design Features:
# Network Transparency
# Process Migration, file transfer between computers
#* User could initiate a process migration to an idle machine, and if the machine was no longer idle; due to it being used by another user, the system would take care of the process migration to another machine
# Handling Cache Consistency
#* Sequential file sharing ==> By using a version number for each file
#* Concurrent write sharing ==> Disable cache to clients, enable write-blocking and other methods
# Implemented a caching system that sped up performance
# Implemented a log structured file system
#* They realized that with increasing amounts of RAM in computers which can be used for caching, writes to the disk were the main bottleneck, not reads.
#* Log structured file-systems are optimized for writes, as changes to previous data are appended at the current position.
#* This allows for very fast, sequential writes.
#* Example: SSD (Solid-state disks)

The main features to take away from the Sprite system is that it implemented a log structured file system, and implemented caching to increase performance.

DistOS 2015W Session 2

2015-04-06T04:36:36Z

Abeinges: /* Discussion: Easy on one computer, hard on multiple computers */

= Reading Response Discussion =

== Mother of All Demos: ==

'''Team:''' Jason, Kirill, Sravan, Agustin, Hassan Ambalica, Apoorv, Khaled

* 1968, lead by Doug Inglebard
* Initial public display of many modern technologies
* One computer with multiple remote terminals
* Video conferencing
* Computer mouse (and coined the term)
* Word processing, rudimentary copy and paste
* Dynamic file linking (hypertext)
* Revision control/version control/source control
* Collaborative real-time editor
** User privilege control in that user can provide read-only access, read-write privilege to file
* Corded keyboard
** Marco keyboard that allows messages quickly, while using mouse at same time
* Not really a distributed operating system, but great start because multiple users at different terminals could share same resources

== Early Internet ==

== Alto ==

'''Team:''' Tory, Veena, Sameer, Mert, Deep, Nameet, Moe

* Initially developed around 1973 by Xerox, upgraded over next few years
* High speed network connectivity (3 Mbps @ 1km distance)
* Connected up to 256 machines
* Protocol similar to cross between UDP/TCP, before TCP invented
* Allowed sharing of printers
* Also allowed distribution of files across computers (redundancy/reliability)
* Sort of early cloud
* Allowed for remote debugging, store error logs
* Allowed machines to use processing power of others
* Much of the time it would be idling, which is amazing at a time when computers cost a fortune

= Professor lead discussion 1 =

A true distributed operating system does not actually exist, its more of a dream.

You'll find in the history of trying to achieve this dream of a distributed operating system there has always been a road block as a result of a technical issue. People would come up with a solution to this technical issue but then they would find that some other technical issue would crop up. For example 'The Mother of All Demos' tried to build a distributed operating system but they had no networking. They had to point a television camera at the computer monitor to be able to demo their concept. The early internet came along and dealt with the issue of no networking. However the early internet itself had other technical issues.

A common buzz word during the early days of development was '''time sharing''', which is an old term for multiple processes running simultaneously. However, at the time it referred to multiple users sharing the CPU cycles on a single computer. Today, a single user's many processes using a single CPU is much more common.

= Discussion: Easy on one computer, hard on multiple computers =

{| class="wikitable" width="100%"
|+ '''Instant Messaging'''
|-
! Why is it easy?
! Why is it hard?
|-
| Can't be out sync
| Synchronization issues
* Server-client modal
** If two or more servers they get requests same time, they have to figure out how to synchronize the data
* Peer-to-Peer
** Each peer has to figure out how to sync each message
|-
| Only one data store/source
|-
|}

{| class="wikitable" width="100%"
|+ '''Mortal Kombat/Twitch Gaming'''
|-
! Why is it easy?
|
* Full control of all software
* All possible users are physically co-located
* Latency minimized
! Why is it hard?
|
* Games are fundamentally low-latency. Networking is fundamental high-latency.
* Input prediction doesn't work well on twitchy games.
* Have to handle lying or faulty clients.
* Have to handle users finding each other.
|-
|}

{| class="wikitable" width="100%"
|+ '''Photo Album'''
|-
! Why is it easy?
! Why is it hard?
|-
|}

=== Commonalities ===
* Synchronization
* Bandwidth
* Reliability/Fault Tolerance
* Interoperability
* Discovery
* Routing
** In modern system this issue has been mostly abstracted away
** Classic example would be Wireless Ad-hoc networking

The above list is not hard on a single system because all the cores have equal access to all the resources. Also as a result of excellent engineering modern single systems can very effective deal with any errors that may result.

DistOS 2015W Session 2

2015-04-06T04:36:00Z

Abeinges: /* Discussion: Easy on one computer, hard on multiple computers */

= Reading Response Discussion =

== Mother of All Demos: ==

'''Team:''' Jason, Kirill, Sravan, Agustin, Hassan Ambalica, Apoorv, Khaled

* 1968, lead by Doug Inglebard
* Initial public display of many modern technologies
* One computer with multiple remote terminals
* Video conferencing
* Computer mouse (and coined the term)
* Word processing, rudimentary copy and paste
* Dynamic file linking (hypertext)
* Revision control/version control/source control
* Collaborative real-time editor
** User privilege control in that user can provide read-only access, read-write privilege to file
* Corded keyboard
** Marco keyboard that allows messages quickly, while using mouse at same time
* Not really a distributed operating system, but great start because multiple users at different terminals could share same resources

== Early Internet ==

== Alto ==

'''Team:''' Tory, Veena, Sameer, Mert, Deep, Nameet, Moe

* Initially developed around 1973 by Xerox, upgraded over next few years
* High speed network connectivity (3 Mbps @ 1km distance)
* Connected up to 256 machines
* Protocol similar to cross between UDP/TCP, before TCP invented
* Allowed sharing of printers
* Also allowed distribution of files across computers (redundancy/reliability)
* Sort of early cloud
* Allowed for remote debugging, store error logs
* Allowed machines to use processing power of others
* Much of the time it would be idling, which is amazing at a time when computers cost a fortune

= Professor lead discussion 1 =

A true distributed operating system does not actually exist, its more of a dream.

You'll find in the history of trying to achieve this dream of a distributed operating system there has always been a road block as a result of a technical issue. People would come up with a solution to this technical issue but then they would find that some other technical issue would crop up. For example 'The Mother of All Demos' tried to build a distributed operating system but they had no networking. They had to point a television camera at the computer monitor to be able to demo their concept. The early internet came along and dealt with the issue of no networking. However the early internet itself had other technical issues.

A common buzz word during the early days of development was '''time sharing''', which is an old term for multiple processes running simultaneously. However, at the time it referred to multiple users sharing the CPU cycles on a single computer. Today, a single user's many processes using a single CPU is much more common.

= Discussion: Easy on one computer, hard on multiple computers =

{| class="wikitable" width="100%"
|+ '''Instant Messaging'''
|-
! Why is it easy?
! Why is it hard?
|-
| Can't be out sync
| Synchronization issues
* Server-client modal
** If two or more servers they get requests same time, they have to figure out how to synchronize the data
* Peer-to-Peer
** Each peer has to figure out how to sync each message
|-
| Only one data store/source
|-
|}

{| class="wikitable" width="100%"
|+ '''Mortal Kombat/Twitch Gaming'''
|-
! Why is it easy?
* Full control of all software
* All possible users are physically co-located
* Latency minimized
! Why is it hard?
* Games are fundamentally low-latency. Networking is fundamental high-latency.
* Input prediction doesn't work well on twitchy games.
* Have to handle lying or faulty clients.
* Have to handle users finding each other.
|-
|}

{| class="wikitable" width="100%"
|+ '''Photo Album'''
|-
! Why is it easy?
! Why is it hard?
|-
|}

=== Commonalities ===
* Synchronization
* Bandwidth
* Reliability/Fault Tolerance
* Interoperability
* Discovery
* Routing
** In modern system this issue has been mostly abstracted away
** Classic example would be Wireless Ad-hoc networking

The above list is not hard on a single system because all the cores have equal access to all the resources. Also as a result of excellent engineering modern single systems can very effective deal with any errors that may result.

DistOS 2015W Session 12

2015-04-06T04:30:42Z

Abeinges: /* F4 */

=Haystack=
* Facebook's Photo Application Storage System.
* Previous Fb photo storage based on NFS design. The reason why NFS didn't work is that it took 3 file-system accesses per logical photo read. Haystack only needs one access.
*Main goals of Haystack:
** High throughput with low latency. It uses one disk operation to provide these.
**Fault tolerance
**Cost effective
**Simple
*Facebook stored all images in haystack with a CDN in front to cache hot data. Haystack still needs to be fast since accessing non-cached data is still common.
*Haystack reduces the memory used for ''filesystem metadata''
*It has 2 types of metadata:
**''Application metadata''
**''File System metadata''
* The architecture consists of 3 components:
**Haystack Store
**Haystack Directory
**Haystack Cache
*Pitchfork and bulk sync were used to tolerate faults. theTfault tolerance works in a very profound way to make haystack feasible and reliable

=Comet=
*Introduced the concept of distributed shared memory (DSM). In a DSM, RAMs from multiple servers would appear as if they are all belonging to one server, allowing better scalability for caching.
*client and server model maintain consistency using DSM
*Comet model works by offloading the computation intensive process from the mobile to only one server.
*The offloading process works by passing the computation intensive process to the server and hold it on the mobile device. Once the process on the server completes, it returns the results and the handle back to the mobile device. In other words, the process does not get physically offloaded to the server but instead it runs on the server and stopped on the mobile device.

=F4=
* Warm Blob Storage System.
** Warm Blob is a store for large quantities of immutable data that isn't frequently accessed, but must still be available.
** Built to reduce the overhead of haystack for old data that doesn't need to be quite as available. Generally data that is a few months old is moved from Haystack to Warm Blob.
** F4 reduce the space usage of Haystack from a replication factor of 3.6 to 2.8 or 2.1 using Reed Solomon coding and XOR coding respectively but still provides consistency.
** Less robust to data center failures as a result.
*Reed Solomon coding basically use(10,4) which means 10 data and 4 parity blocks in a stripe, and can thus tolerate losing up to 4 blocks which means it can tolerate 4 rack failure and use 1.4 expansion factor.Two copies of this would be 2* 1.4= 2.8 effective replication factor.
*XOR coding use(2,1) across three data center and use 1.5 expansion factor which gives 1.5*1.4= 2.1 effective replication factor.
*The caching mechanism provides the reduction in load on storage system and it makes BLOB scaleable.

=Sapphire=
*Represents a building block towards building this global distributed systems. The main critique to it is that it didn’t present a specific use case upon which their design is built upon.
*Sapphire does not show their scalability boundaries. There is no such distributed system model that can be “one size fits all”, most probably it will break in some large scale distributed application.
*Reaching this global distributed system that address all the distributed OS use cases will be a cumulative work of many big bodies and building it block by block and then this system will evolve by putting all these different building blocks together. In other words, reaching a global distributed system will come from a “bottom up not top down approach” [Somayaji, 2015].
*The concept of separate application logic from deployment logic helps programmers in making a flexible system. The other important part that makes it as a scalable system was that it is object based and could be integrated with any object oriented language.

DistOS 2015W Session 12

2015-04-06T04:24:35Z

Abeinges: /* Haystack */

=Haystack=
* Facebook's Photo Application Storage System.
* Previous Fb photo storage based on NFS design. The reason why NFS didn't work is that it took 3 file-system accesses per logical photo read. Haystack only needs one access.
*Main goals of Haystack:
** High throughput with low latency. It uses one disk operation to provide these.
**Fault tolerance
**Cost effective
**Simple
*Facebook stored all images in haystack with a CDN in front to cache hot data. Haystack still needs to be fast since accessing non-cached data is still common.
*Haystack reduces the memory used for ''filesystem metadata''
*It has 2 types of metadata:
**''Application metadata''
**''File System metadata''
* The architecture consists of 3 components:
**Haystack Store
**Haystack Directory
**Haystack Cache
*Pitchfork and bulk sync were used to tolerate faults. theTfault tolerance works in a very profound way to make haystack feasible and reliable

=Comet=
*Introduced the concept of distributed shared memory (DSM). In a DSM, RAMs from multiple servers would appear as if they are all belonging to one server, allowing better scalability for caching.
*client and server model maintain consistency using DSM
*Comet model works by offloading the computation intensive process from the mobile to only one server.
*The offloading process works by passing the computation intensive process to the server and hold it on the mobile device. Once the process on the server completes, it returns the results and the handle back to the mobile device. In other words, the process does not get physically offloaded to the server but instead it runs on the server and stopped on the mobile device.

=F4=
* Warm Blob Storage System.
** Warm Blob is a immutable data that gets cool very rapidly.
** F4 reduce the space usage by 3.6 to 2.8 or 2.1 replication factor using Reed Solomon coding and XOR coding respectively but still provides consistency.
*Reed Solomon coding basically use(10,4) which means 10 data and 4 parity blocks in a stripe, and can thus tolerate losing up to 4 blocks whch means it can tolerate 4 rack failure and use 1.4 expansion factor.Two copies of this would be 2* 1.4= 2.8 effective replication factor.
*XOR coding use(2,1) across three data center and use 1.5 expansion factor which gives 1.5*1.4= 2.1 effective replication factor
*The caching mechanism provides the reduction in load on storage system and it makes BLOB scalable
*The concept of hot and warm storage is used to make it simple and modular

=Sapphire=
*Represents a building block towards building this global distributed systems. The main critique to it is that it didn’t present a specific use case upon which their design is built upon.
*Sapphire does not show their scalability boundaries. There is no such distributed system model that can be “one size fits all”, most probably it will break in some large scale distributed application.
*Reaching this global distributed system that address all the distributed OS use cases will be a cumulative work of many big bodies and building it block by block and then this system will evolve by putting all these different building blocks together. In other words, reaching a global distributed system will come from a “bottom up not top down approach” [Somayaji, 2015].
*The concept of separate application logic from deployment logic helps programmers in making a flexible system. The other important part that makes it as a scalable system was that it is object based and could be integrated with any object oriented language.

WebFund 2013W: Tasks 1

2013-03-12T21:47:08Z

Abeinges:

Your first set of tasks are the following:
* Setup an account on [https://github.com/ github].
* Validate your student email with github to get a free student account: https://github.com/edu
* Setup an account on the class wiki (optional).
* Tell the TA your current ideas on your term project and who you would like to work with.
* Download and run/install a copy of node.js. A local copy of the standalone 32-bit windows executable [http://homeostasis.scs.carleton.ca/~soma/webfund-2013w/node.exe here]. Other versions are [http://nodejs.org/download/ here].
* Run the node.js hello world program as shown in the "Basic HTTP server" in the [http://www.nodebeginner.org/#a-basic-http-server Node Beginner Book].

You should get checked off on these tasks by a TA in tutorial.

==Hints==

* If you just run the node.exe executable by double-clicking on it, you'll get a ">" prompt. This is the node read/eval/print loop. Meaning, you type in JavaScript to it.
* If you want to run a file with javascript in it in node, start up a command prompt, cd to the directory containing node.exe and your file foo.js, and then run "node foo.js".
* In the lab or elsewhere, if you get a firewall prompt, that is normal - node is trying to listen on a port. That's exactly what firewalls are supposed to block. Allow access. Note that if you check any boxes you'll need to enter an admin password. If you just allow access it should work.

WebFund 2013W: Tasks 1

2013-03-12T21:46:35Z

Abeinges:

Your first set of tasks are the following:
* Setup an account on [https://github.com/ github].
* Validate your student email with github to get a free student account: https://github.com/edu
* Setup an account on the class wiki (optional).
* Tell the TA your current ideas on your term project and who you would like to work with.
* If you do not have a partner, post a blurb on Piazza to help find one, or reply to an existing solicitation.
* Download and run/install a copy of node.js. A local copy of the standalone 32-bit windows executable [http://homeostasis.scs.carleton.ca/~soma/webfund-2013w/node.exe here]. Other versions are [http://nodejs.org/download/ here].
* Run the node.js hello world program as shown in the "Basic HTTP server" in the [http://www.nodebeginner.org/#a-basic-http-server Node Beginner Book].

You should get checked off on these tasks by a TA in tutorial.

==Hints==

* If you just run the node.exe executable by double-clicking on it, you'll get a ">" prompt. This is the node read/eval/print loop. Meaning, you type in JavaScript to it.
* If you want to run a file with javascript in it in node, start up a command prompt, cd to the directory containing node.exe and your file foo.js, and then run "node foo.js".
* In the lab or elsewhere, if you get a firewall prompt, that is normal - node is trying to listen on a port. That's exactly what firewalls are supposed to block. Allow access. Note that if you check any boxes you'll need to enter an admin password. If you just allow access it should work.

WebFund 2013W: jQuery UI

2013-02-01T20:24:43Z

Abeinges:

In this tutorial you will examine and modify a simple example page using [http://jqueryui.com/ jQuery UI].

This application uses:
* jQuery
* jQuery-UI (jQuery plugin)
* jQuery-validate (jQuery plugin)
* Bootstrap

You will need to look up the documentation on these to understand how the application works.

The file [http://homeostasis.scs.carleton.ca/~soma/webfund-2013w/demo-jquery-ui.zip demo-jquery-ui.zip] contains the example index.html file and associated libraries. Please download this file locally, unzip it, and navigate to the local file <tt>demo-jquery-ui/index.html</tt> in your browser. (Note we will not be using node.js in this tutorial). You should see something like this:

[[File:Demo-jquery-ui-screenshot.png]]

Note that:
* The country field auto-completes with a few countries.
* The Birthday field brings up a calendar.
* If you click the email me button, you get another field for entering your email address.
* If you hit submit where all the fields are "valid", they all go blank (and the URL contains the values you entered). If you try submitting an invalid form, the colors of the invalid entries are shown.

Modify this page as follows:
# Remove the loading of [http://twitter.github.com/bootstrap/ bootstrap] (JS and style sheet at top of index.hml) - what changed?
# Make an alert dialog pop up when you hit submit on a valid form that says "Form validated."
# Make the email address validate only if it has a "@".
# Print a message below the country field that says "invalid Country" if you manually type in a new country that isn't in the list.
# Add some of the other widgets that are part of [http://jqueryui.com/ jQuery UI].

WebFund 2013W: jQuery UI

2013-02-01T20:24:20Z

Abeinges:

In this tutorial you will examine and modify a simple example page using [http://jqueryui.com/ jQuery UI].

This application uses:
jQuery
jQuery-UI (jQuery plugin)
jQuery-validate (jQuery plugin)
Bootstrap

You will need to look up the documentation on these to understand how the application works.

The file [http://homeostasis.scs.carleton.ca/~soma/webfund-2013w/demo-jquery-ui.zip demo-jquery-ui.zip] contains the example index.html file and associated libraries. Please download this file locally, unzip it, and navigate to the local file <tt>demo-jquery-ui/index.html</tt> in your browser. (Note we will not be using node.js in this tutorial). You should see something like this:

[[File:Demo-jquery-ui-screenshot.png]]

Note that:
* The country field auto-completes with a few countries.
* The Birthday field brings up a calendar.
* If you click the email me button, you get another field for entering your email address.
* If you hit submit where all the fields are "valid", they all go blank (and the URL contains the values you entered). If you try submitting an invalid form, the colors of the invalid entries are shown.

Modify this page as follows:
# Remove the loading of [http://twitter.github.com/bootstrap/ bootstrap] (JS and style sheet at top of index.hml) - what changed?
# Make an alert dialog pop up when you hit submit on a valid form that says "Form validated."
# Make the email address validate only if it has a "@".
# Print a message below the country field that says "invalid Country" if you manually type in a new country that isn't in the list.
# Add some of the other widgets that are part of [http://jqueryui.com/ jQuery UI].

Github

2013-01-25T18:20:48Z

Abeinges:

When using github for a school project, it is recommended that you do not fully trust each other to do the right thing. For this reason we recommend the Review To Commit (RTC) pattern. Note RTC is a bit of a misnomer for a git-oriented workflow-- it could more accurately be called Review to Merge.

Here is a diagram of the idealized untrusted development workflow:

[[File:Diagram-1.gif|700px]]

Here is the script we went through (or at least, intended to) in class:

*Change to CarletonU-COMP2406-W2013 account perspective (top-left)
*Show all repos (right), news feed (center)
*Choose Demo-Repo (repo hereafter "mothership")
*fork to private github account (top right) (repo hereafter "fork")
*grab SSH URL
*git clone [fork's SSH URL]
*show contents of directory (note .git, .gitignore, REAMDE.md)
*create issue in mothership "need index.html"
*assign issue to self
*git checkout -b new-index
*git branch (show new branch)
*create index.html
*git status (show index.html needs to be added)
*git add .
*git status (show index.html staged for commit)
*git commit -am "adding index.html, fixes #1

added basic head
added empty body"
*git status (nothing to commit)
*git push origin new-index
*go to fork's github page
*create pull request from fork new-index to mothership master
*look at pull request in mothership
*comment on incorrect tag in github diff (pretending you're a team mate)
*fix broken tag
*git status
*git commit -am "fixing broken tag"
*git push
*Note pull request has update self with new commit
*accept pull request (pretending you're a team mate)
*Note #1 has closed itself because it was mentioned in commit with "fixes"
*git remote -v
*note "origin" is assigned to fork
*copy mothership's SSH URL
*git help remote (show how to get help on a command)
*git remote add mothership [mothership's URL]
*git checkout master
*git pull mothership (gets your changes from new-index)
*git branch -d new-index (deletes new-index)
*add "foo" to body of index.html in github (pretending you're a team mate)
*add "bar" to body of index.html locally
*git status
*git commit -am "adding bar"
*git pull mothership (merge conflict!)
*resolve merge conflict
*git commit -am "resolving merge conflict"
*add "baz" to body of index.html
*git status (index is staged for commit)
*git stash (branch is back to nothing changed)
*git stash pop (baz is back)
*git commit -am "adding baz" (oh no, I don't actually like this change)
*git reset HEAD^
*git status (baz is still there, but not commited)

Github

2013-01-25T18:20:28Z

Abeinges:

When using github for a school project, it is recommended that you do not fully trust each other to do the right thing. For this reason we recommend the Review To Commit (RTC) pattern. Note RTC is a bit of a misnomer for a git-oriented workflow-- it could more accurately be called Review to Merge.

Here is a diagram of the idealized untrusted development workflow:
[[File:Diagram-1.gif|800px]]

Here is the script we went through (or at least, intended to) in class:

*Change to CarletonU-COMP2406-W2013 account perspective (top-left)
*Show all repos (right), news feed (center)
*Choose Demo-Repo (repo hereafter "mothership")
*fork to private github account (top right) (repo hereafter "fork")
*grab SSH URL
*git clone [fork's SSH URL]
*show contents of directory (note .git, .gitignore, REAMDE.md)
*create issue in mothership "need index.html"
*assign issue to self
*git checkout -b new-index
*git branch (show new branch)
*create index.html
*git status (show index.html needs to be added)
*git add .
*git status (show index.html staged for commit)
*git commit -am "adding index.html, fixes #1

added basic head
added empty body"
*git status (nothing to commit)
*git push origin new-index
*go to fork's github page
*create pull request from fork new-index to mothership master
*look at pull request in mothership
*comment on incorrect tag in github diff (pretending you're a team mate)
*fix broken tag
*git status
*git commit -am "fixing broken tag"
*git push
*Note pull request has update self with new commit
*accept pull request (pretending you're a team mate)
*Note #1 has closed itself because it was mentioned in commit with "fixes"
*git remote -v
*note "origin" is assigned to fork
*copy mothership's SSH URL
*git help remote (show how to get help on a command)
*git remote add mothership [mothership's URL]
*git checkout master
*git pull mothership (gets your changes from new-index)
*git branch -d new-index (deletes new-index)
*add "foo" to body of index.html in github (pretending you're a team mate)
*add "bar" to body of index.html locally
*git status
*git commit -am "adding bar"
*git pull mothership (merge conflict!)
*resolve merge conflict
*git commit -am "resolving merge conflict"
*add "baz" to body of index.html
*git status (index is staged for commit)
*git stash (branch is back to nothing changed)
*git stash pop (baz is back)
*git commit -am "adding baz" (oh no, I don't actually like this change)
*git reset HEAD^
*git status (baz is still there, but not commited)

Github

2013-01-25T18:20:14Z

Abeinges:

When using github for a school project, it is recommended that you do not fully trust each other to do the right thing. For this reason we recommend the Review To Commit (RTC) pattern. Note RTC is a bit of a misnomer for a git-oriented workflow-- it could more accurately be called Review to Merge.

Here is a diagram of the idealized untrusted development workflow:
[[File:Diagram-1.gif|400px]]

Here is the script we went through (or at least, intended to) in class:

*Change to CarletonU-COMP2406-W2013 account perspective (top-left)
*Show all repos (right), news feed (center)
*Choose Demo-Repo (repo hereafter "mothership")
*fork to private github account (top right) (repo hereafter "fork")
*grab SSH URL
*git clone [fork's SSH URL]
*show contents of directory (note .git, .gitignore, REAMDE.md)
*create issue in mothership "need index.html"
*assign issue to self
*git checkout -b new-index
*git branch (show new branch)
*create index.html
*git status (show index.html needs to be added)
*git add .
*git status (show index.html staged for commit)
*git commit -am "adding index.html, fixes #1

added basic head
added empty body"
*git status (nothing to commit)
*git push origin new-index
*go to fork's github page
*create pull request from fork new-index to mothership master
*look at pull request in mothership
*comment on incorrect tag in github diff (pretending you're a team mate)
*fix broken tag
*git status
*git commit -am "fixing broken tag"
*git push
*Note pull request has update self with new commit
*accept pull request (pretending you're a team mate)
*Note #1 has closed itself because it was mentioned in commit with "fixes"
*git remote -v
*note "origin" is assigned to fork
*copy mothership's SSH URL
*git help remote (show how to get help on a command)
*git remote add mothership [mothership's URL]
*git checkout master
*git pull mothership (gets your changes from new-index)
*git branch -d new-index (deletes new-index)
*add "foo" to body of index.html in github (pretending you're a team mate)
*add "bar" to body of index.html locally
*git status
*git commit -am "adding bar"
*git pull mothership (merge conflict!)
*resolve merge conflict
*git commit -am "resolving merge conflict"
*add "baz" to body of index.html
*git status (index is staged for commit)
*git stash (branch is back to nothing changed)
*git stash pop (baz is back)
*git commit -am "adding baz" (oh no, I don't actually like this change)
*git reset HEAD^
*git status (baz is still there, but not commited)

Github

2013-01-25T18:19:10Z

Abeinges: Created page with "When using github for a school project, it is recommended that you do not fully trust each other to do the right thing. For this reason we recommend the Review To Commit (RTC)..."

When using github for a school project, it is recommended that you do not fully trust each other to do the right thing. For this reason we recommend the Review To Commit (RTC) pattern. Note RTC is a bit of a misnomer for a git-oriented workflow-- it could more accurately be called Review to Merge.

Here is a diagram of the idealized untrusted development workflow:
[[File:Diagram-1.gif]]

Here is the script we went through (or at least, intended to) in class:

*Change to CarletonU-COMP2406-W2013 account perspective (top-left)
*Show all repos (right), news feed (center)
*Choose Demo-Repo (repo hereafter "mothership")
*fork to private github account (top right) (repo hereafter "fork")
*grab SSH URL
*git clone [fork's SSH URL]
*show contents of directory (note .git, .gitignore, REAMDE.md)
*create issue in mothership "need index.html"
*assign issue to self
*git checkout -b new-index
*git branch (show new branch)
*create index.html
*git status (show index.html needs to be added)
*git add .
*git status (show index.html staged for commit)
*git commit -am "adding index.html, fixes #1

added basic head
added empty body"
*git status (nothing to commit)
*git push origin new-index
*go to fork's github page
*create pull request from fork new-index to mothership master
*look at pull request in mothership
*comment on incorrect tag in github diff (pretending you're a team mate)
*fix broken tag
*git status
*git commit -am "fixing broken tag"
*git push
*Note pull request has update self with new commit
*accept pull request (pretending you're a team mate)
*Note #1 has closed itself because it was mentioned in commit with "fixes"
*git remote -v
*note "origin" is assigned to fork
*copy mothership's SSH URL
*git help remote (show how to get help on a command)
*git remote add mothership [mothership's URL]
*git checkout master
*git pull mothership (gets your changes from new-index)
*git branch -d new-index (deletes new-index)
*add "foo" to body of index.html in github (pretending you're a team mate)
*add "bar" to body of index.html locally
*git status
*git commit -am "adding bar"
*git pull mothership (merge conflict!)
*resolve merge conflict
*git commit -am "resolving merge conflict"
*add "baz" to body of index.html
*git status (index is staged for commit)
*git stash (branch is back to nothing changed)
*git stash pop (baz is back)
*git commit -am "adding baz" (oh no, I don't actually like this change)
*git reset HEAD^
*git status (baz is still there, but not commited)

Fundamentals of Web Applications (Winter 2013)

2013-01-25T18:08:14Z

Abeinges: /* Git */

==Administration==

The course outline for this course is [http://www.scs.carleton.ca/courses/course_outline.php?Term=Winter&Year=2013&Number=COMP%202406 here].

Course discussions will be on Piazza. You can sign up [https://piazza.com/carleton.ca/winter2013/comp24062006 here]. Note that Piazza has a "groups" function that can help you find partners for your project. '''Also note that piazza asks for your carleton.ca email address, so you can't directly sign up with any anonymous email address.''' If you wish to sign up using an anonymous/throw away email address, please email Prof. Somayaji or a TA and they can enroll that email address manually.

You should get an account on this wiki so you can add to it. Email Prof. Somayaji to get one with your preferred username and email address to which a password should be sent. (Note this is not a requirement.)

==Resources==

===JavaScript===

'''You should go out and learn the basics of JavaScript on your own.''' While we will discuss the language in class, much of that discussion will make more sense if you've exposed yourself to the language. You should also get basic exposure to standard web technologies.

The easiest way to get started with JavaScript and get basic understanding of web technologies is to go through the interactive lessons on [http://codeacademy.com Code Academy]. I suggest you go through their JavaScript, Web Fundamentals, and jQuery tracks. They shouldn't take you very long to do given that you already know how to program.

You should get access to ''JavaScript: The Good Parts'' by Douglas Crockford ([http://it-ebooks.info/book/274/.do free PDF download]). Read it. It is available through [http://shop.oreilly.com/product/9780596517748.do O'Reilly], [http://my.safaribooksonline.com/9780596517748 Safari Books Online], and the regular online bookstores. There is even [https://www.inkling.com/store/book/javascript-good-parts-douglas-crockford-1st/ an interactive version] which includes an embedded JavaScript interpreter. You can get access to Safari Books Online through the Carleton Library (four concurrent users only) or partial access by becoming a member of [https://www.computer.org IEEE Computer Society].

Crockford also has a lot of online resources on JavaScript, including videos of talks he's given that cover much of the content in his book. Look at his [http://javascript.crockford.com/ JavaScript page] and this [http://yuiblog.com/crockford/ page of his videos].

Another good book is [http://eloquentjavascript.net/ Eloquent JavaScript: A Modern Introduction to Programming] by Marijn Haverbeke. A version of this book is available online for free. The for-sale version is apparently updated and edited.

And of course the standard reference for JavaScript is [http://shop.oreilly.com/product/9780596805531.do JavaScript: The Definitive Guide] by David Flanagan. It is a big book, but it is comprehensive.

===Node===

You will be building your application in node.js this term. A good, relatively comprehensive book is [http://shop.oreilly.com/product/0636920024606.do Learning Node] by Shelley Powers. This book is recommended but not required. A quick way to get started with node.js is [http://www.nodebeginner.org/ The Node Beginner Book] by Manuel Kiessling.

===Git===
There is now a [[github]] organization for the course [https://github.com/CarletonU-COMP2406-W2013 here]. The TAs will add your team to this organization. Please check in your code and documentation regularly to github so the TAs may monitor your progress.

When your team is added, your TA will do the following:
* Setup a repository named after your team. This initial repository will have a README and will ignore temporary files generated by node.js. You will have full administrative access to this repository.
* Your team will include your TA as a member (along with you and your teammate).

==Lectures==

<table style="width: 100%;" border="1" cellpadding="4" cellspacing="0">
<tr valign="top">
<th>
Date
</th>
<th>
Topic
</th>
</tr>
<tr>
<td>
Jan. 8

</td>
<td>
Introduction

</td>
</tr>
<tr>
<td>
Jan. 10

</td>
<td>
[[WebFund 2013W: Symbols|Symbols]]

</td>
</tr>
<tr>
<td>
Jan. 15

</td>
<td>
[[WebFund 2013W Lecture 3|Lecture 3]]

</td>
</tr>
<tr>
<td>
Jan. 17

</td>
<td>
[[WebFund 2013W Lecture 4|Lecture 4]]

</td>
</tr>
<tr>
<td>
Jan. 22

</td>
<td>
[[WebFund 2013W Lecture 5|Lecture 5]]

</td>
</tr>
<tr>
<td>
Jan. 24

</td>
<td>
[[WebFund 2013W Lecture 6|Lecture 6]]

</td>
</tr>
<tr>
<td>
Jan. 29

</td>
<td>
[[WebFund 2013W Lecture 7|Lecture 7]]

</td>
</tr>
<tr>
<td>
Jan. 31

</td>
<td>
[[WebFund 2013W Lecture 8|Lecture 8]]

</td>
</tr>
</table>

==Tutorials/Weekly Tasks==

Each week you will get a progress grade from 0-4, given to you by a TA. If you are being diligent, you should be able to get 4's every week. The easiest way to get your grade is to come to tutorial and meet with your TA; alternately, you can meet a TA in their office hours or, at their discretion, discuss things with them online.

Initially you can talk to any TA to get your progress grade. Once groups have been formed, however, you will have an assigned TA who will be tracking your progress throughout the semester.

Below is a schedule with the tasks you need to accomplish each week for everyone. Note you need to accomplish the task before your next tutorial. So, if you attend the Monday tutorials, you need to show progress before the following Monday.

After February 1st, all milestones are between you and your TA and will follow those outline in your proposal. Milestone deliverables and precise due dates may be revised at the discretion of your TA.

<table style="width: 100%;" border="1" cellpadding="4" cellspacing="0">
<tr valign="top">
<th>
Date
</th>
<th>
Task/Milestone
</th>
</tr>
<tr>
<td>
Jan. 11,14

</td>
<td>
[[WebFund 2013W: Tasks 1|Setup Accounts, Run node.js]]

</td>
</tr>
<tr>
<td>
Jan. 18,21

</td>
<td>
[[WebFund 2013W: Task 2|Project Partners, Blog Example]]

</td>
</tr>
<tr>
<td>
Jan. 25, 28

</td>
<td>
[[WebFund 2013W: Web Develper Tools|Web Developer Tools (optional)]]

</td>
</tr>
<tr>
<td>
Feb. 1, 1 PM

</td>
<td>
[[WebFund 2013W: Proposal|Project Proposal Due]]

</td>
</tr>
</table>

==Humor/Quotes==

[http://xkcd.com/327/ Bobby Tables], a funny example of SQL injection.

"Scalability is always the answer in this class." -Alexis

"First axiom of this class is web apps suck." -[[User:soma |Professor Somayaji]]

File:Diagram-1.gif

2013-01-25T15:56:57Z

Abeinges: Diagram of idealized Github workflow for one user, when working with an untrusted team.

Diagram of idealized Github workflow for one user, when working with an untrusted team.

WebFund 2013W: Tasks 1

2013-01-11T19:50:02Z

Abeinges:

Your first set of tasks are the following:
* Setup an account on [https://piazza.com/carleton.ca/winter2013/comp24062006 Piazza].
* Setup an account on [https://github.com/ github].
* Validate your student email with github to get a free student account: https://github.com/edu
* Setup an account on the class wiki (optional).
* Tell the TA your current ideas on your term project and who you would like to work with.
* If you do not have a partner, post a blurb on Piazza to help find one, or reply to an existing solicitation.
* Download and run/install a copy of node.js. A local copy of the standalone 32-bit windows executable [http://homeostasis.scs.carleton.ca/~soma/webfund-2013w/node.exe here]. Other versions are [http://nodejs.org/download/ here].
* Run the node.js hello world program as shown in the "Basic HTTP server" in the [http://www.nodebeginner.org/#a-basic-http-server Node Beginner Book].