Soma-notes - User contributions [en]

WebFund 2016W Lecture 24

2016-04-09T03:34:26Z

LeeCroft:

==Video==

The video for the lecture given on April 7, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec24-07Apr2016.mp4 is now available].

==Notes==

===In Class===

<pre>
Lecture 24
----------

Modern web pages are a mess because they draw content
(and code) from many sources
* Including that content is easy, you just make a few
changes, typically saying to load some JavaScript

Is there any barrier between that external code
and your client-side code?
* not really
* there are technologies for separating content, and
* JavaScript provides some separation (e.g. lexical scoping)
* But everyone shares the DOM

So if you control any of the content in a page,
you can potentially change any of it
* Imagine censoring comments

If you want others to incorporate your stuff into their
pages, what do you offer?

Maybe offer some sort of API...but how?

Maybe just give a link to my animated GIFs?
- but a link to which one?
- random every time?

Why not just give them JSON + JavaScript?
* Just give JavaScript, it will load the JSON on its own

So when anyone talks about "Web Services"
* Make an API out of HTTP requests (GET, POST, ...)

Might be accessed by
* a web page from the developer
* 3rd party web page
* mobile apps
* any other program connected to the Internet

It doesn't matter to the server.
THIS is the key benefit of a "RESTful" API

Used to be 2 ways to do a "Web Service":
* REST (e.g., use regular HTTP GET and POST)
* SOAP

(at least) Two schools of thought for code talking to
code:
* make everything a function call
* do something weird for network communication

SOAP is just a way to do function calls over HTTP

How do you do a function call over HTTP?
- prepare arguments
- send arguments
- invoke remote function
- retrieve result
- store result

What about
* types?
- have to communicate that
* but think how complicated data structures can get
* Solution: XML Schemas for data

SOAP is an example of RPC: remote procedure calls
but over HTTP

RPC turns local code into network code, but
local code isn't ready to face the network
* security
* efficiency

So why SOAP?
* the old ways of doing RPC were blocked by firewalls

There's a "winpopup" service in Windows
* pops up a window
* as requested by a remote host

REST came along when "web natives" (programmers)
realized they didn't need all that infrastructure

Easy to see new things as "mostly old things"
But really you need to see it as its own thing
to really master it

What is the web?

* Publishing
* e-commerce (mail order catalogs)
* Information access (fine-grained)
* Multimedia
- realplayer => youtube
- CDs and videos
* VoIP (telephony), telegrams

* Social media
- gossip

* "Uberization"
- use a mobile app to connect people for business

Some of the progress in web technologies today is to
make "platforms"

Platforms are really technology looking for a purpose

Weird thing about the web is it is a "platform" that
nobody controls
* Good: allows innovation
* Bad: evolutionary cruft

Programmers HATE evolutionary cruft
- it doesn't make sense!
- its a mess!

Many, many investors don't like the web as a platform
- they want to control the platform

iOS as a platform
* apple gets 30%

Google takes a cut from the Play store as well

Platforms are a play at a monopoly in a space

Two directions
</pre>

===Student Notes===

====Logistics====
*Final exam on the 12th, 2 hours long, with similar format to the midterm
*Solution for Assignment #6 will be posted on Sunday, but can be submitted until the end of Saturday
*A study session will be on the coming Monday on 11th at [http://carleton.ca/ims/rooms/steacie-building-103/ Steacie Building 103]
**No set agenda
**Focus on exam-storage
**Attend and study beforehand, come with questions
**If you just sit and listen, better to just study by yourself for 2 hours
**If good questions are asked, will stick around around 10-12

====End of the Line====
*What did I leave out, what's up ahead?
*Tuesday we talked about scalability, but today we'll talk about what you're likely to encounter now and what to expect in the near future.
*Let's pick a modern [http://www.arstechnica.com/ website], and see how it compares to websites we've built.
**There's a reader comment section. What's happening?
***The comments are loaded in the background, so the server doesn't load them all at the start.
***Let's look at the page source - it seems to have comments in it - no it's a list of stories, but I didn't expect it to be in the DOM.
***Do we load pages all at once or in pieces? There are a lot of pieces and parts on the page, some images, JavaScript, GIFs, JSON.
**A lot of the pieces being requested have nothing to do with the main site. Will your webpage ever look like this? Does ArsTechnica even create and publish all this content?
***Where does this complexity come from? There's icons, banner ads, sponsored stories that are clickbait to gather click impressions.
***Almost no work to include these - only a few JavaScript scripts to include and let them run to grab someone else's content.
***e.g. code for share buttons comes from Twitter and other social media sites.
***You can combine content from other people & create a big mess, because it's easy.
***The embedded code does whatever it wants, and it's inherently not separate from your own content as they share the DOM.
***Compromised code originating from ad servers can do almost anything. Let's say people are revealing Google's secrets. Instead of blocking connections, just have ad code check the DOM for "google" or "secrets" in the main content and erase them or block them off with ads
**The Web is not intended to be this mishmash of content

====Functionality====
*What if you want your stuff incorporated into other people's pages - how do they make use of your functionality?
**APIs are interfaces to code that we haven't covered in this class, but we're on the web and not running on a machine directly
*You could provide just images and give them links, but if something dynamic is needed with parameters and controls like a twitter feed, then it needs to be data-driven
**Need JSON+JS to be loaded on the browser to do background GET/POSTs, and we've already seen this
*Web services are a buzzword for API over HTTP requests: GET/POST depending on what's important, loaded by browsers or other clients subject to interpretation at the end of the wire.
**Facebook has a connect API for apps to authenticate people on phones and when people log in to Facebook on Desktop browsers. How do they look different? They're the same API because it's a login specific to Facebook users. Wrapping/delivering code with HTTP means it can apply to everyone on the web through the remote service's API
*REST is Representational State Transfer, a successor to Simple Object Access Protocol (SOAP). They both are a way to access functionality over HTTP.
*Older inter-process communications are through function calls on specific network protocols, but that's an abomination because arguments need to be determined, formatted, sent, and retrieved. There's packaging and unpackaging involved, and data structures to parse.
**SOAP's solution was to use XML as a successor to SGML, a language for describing data rather than documents (which is what HTML does). It's a complicated way to send a dictionary + grammar book at once.
***Remote procedure calls on the network has been around a long time, but it's a dumb idea because of lack of trust in unknown remote code. Security + Efficiency a problem on the web.
***RPC bypasses firewalls that blocked other code like the Windows popup messaging functionality that was designed for local networks with considerations to convenience and not security.
***Programs trying remote access, made use of HTTP to make things simple. We know REST has stopped trying to describe the whole universe inside schemas and just use GET/POSTs. We've already built websites under the same networked architecture.

*Humans try relate new concepts with the familiar, describing the first cars as horseless carriages.
**The internet is due for change and we don't know who's going to start it, what direction it's going, and what form it's going to take on in the future.
***The functionality of displaying documents on the web supplants published books, e-commerce to replace mail-order catalogues, multimedia to consolidate cassettes and cds, and telephony to communicate better than telegrams.
***However, social media has no pre-internet equivalent to imitate except gossiping. It is a new format that evolved from webpages enabling comments that are similar to academic annotations.
***What's next? New usage of the web include ''Uberization'' to bring people closer to businesses.
***Uber and Facebook have a purpose. Is the "Internet of Things" useful for smart web-capable fridges and other appliances? It's more of a buzzword that nobody has made intensive use of, unlike how driverless cars can be conceptualized and implemented with a purpose.
**Efforts are made to create "platforms", things to be built on for other services
***What really is a platform? Technology looking for a purpose and/or users
***The web is becoming the universal platform that nobody can control.
**The free nature of the web enables innovation and evolution past its designed standards, however it is rife with legacy technology. Sometimes they're senseless, vestigial cruft that's no longer being maintained but still used out of convenience. They can be ignored without maintenance or scrapped and started over, but it's dangerous for programmers to assume they know how other people will use the code out of hubris.

====Future Platforms====
*People that are investors in companies dislike the web, since they have no control over it.
*iOS has a 30% of everything on iTunes, and so does Google and Microsoft over their respective app stores.
**Microsoft is transitioning from a OS-oriented company to a service-oriented company (Business + Azure cloud services)
*The web is an uncontrolled environment among other platforms have been acquired with monopolies.
**It will be acquired and turned stale into a legacy platform or supersede all others to become ''the'' platform.
**Developers can and should choose their own technology, to become involved in choosing what direction they want the web to go.
**Why become involved in the development of code? Open-source developments have increased in prominence, with Linux based OS taking on greater marketshare over Windows. It has co-evolved with the web through many programmers to become a better foundation for building technologies.
**We don't know what's coming next (augmented reality), but choosing the right platforms are the only way to develop infrastructure towards the next generation. The web is going to stick around, so good luck studying.

WebFund 2016W Lecture 23

2016-04-07T05:03:26Z

LeeCroft:

==Video==

The video from the lecture given on April 5, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec23-05Apr2016.mp4 is now available].

==Notes==

===In class===

<pre>
Lecture 23
----------

Scalability

* You replicate your web application, should be
"embarassingly parallel" (no direct interaction)

* Communication between servers happens through the
backend database

Why not have the web servers talk directly to each other?
- you then have to figure out how to do
synchronization/concurrency right
- that's what databases are for!

So how in the world do you scale up databases?

First answer: use a minimal solution
- only get the functionality that you want

First rule of scalability
- you can't do everything at scale

So, you have to choose what you will do

Why are sacrifices necessary?

latency versus bandwidth

bandwith: bits transferred per second on average
latency: time to get first bit of response after request

Consider a large truck full of hard disks driving
across Canada.
- very, very high bandwidth
- very, very high latency as well!
(2 weeks to get first bit of response)

Ideally, you want high bandwidth and low latency
- bandwidth you get through parallelism
- latency has to be engineered

A "supercomputer" is one with low-latency memory access,
for LOTS of memory
- so it has to have fast interconnects
- thus, accesses to different nodes aren't much
slower than local accesses

Challeng for large web apps is having the database
answer queries with low latency

But some amount of latency is inevitable
- speed of light is finite

So if you want fast access to your webserver worldwide
- you need to replicate across the globe
- be close to your clients

NoSQL databases became popular because of latency
concerns
- you needed to be as fast as possible,
- so strip it to the bone

Use an in-memory key-value store if it is sufficient
- lowest latency
- least functionality

If you have to, use an SQL database
- highest latency
- most functionality

Or use something in between (MongoDB)

Once you choose the type of database, you OPTIMIZE
- minimize I/O and computation required per access
(read or write)
- example: query optimization
- how you form the query
- how database is organized

Count the number of web pages that have the word
"amazing" in them

How?
- first, need a database with a copy of the web pages
- then, you could do linear search through all
of the web pages...

I ask this because a web search is a massive challenge
in query optimization

- need to limit scope as early as possible in query
- organize data so queries are quick to be answered
- precompute as much as possible

The best you can do is table lookup. So have the right
tables ready!

Key tool is making an INDEX
- table of search term and pointers to data

E.g., you have a table of customers sorted by ID
- have an index of names, so a table of names versus
IDs
</pre>

===Student Notes===

====Counting Log Entries====

*Exercise related to Assignment 6: The following is to help you in getting started on adding in single-page functionality for the web app.
**How do we get the data related to the number of log files and the number of entries from the database? Ask the database how many log files are there?
**What do we want to do? Count the number of entries. Use the count method.
***<code>db.logs.count()</code> counts records
**How many distinct values are there?
***<code>db.logs.distinct(“file”)</code> counts how many files we have
**The placeholder information is currently filled in on the server
***How? First go to views. The Jade file already has <code>numfiles</code> and <code>numentries</code> variables which are passed in to the template through the call to <code>render()</code> in the routes. How do we update them?
**What problems do we run in to when using the <code>count()</code> and <code>distinct()</code> methods?
***Reloading the page is one problem
***The real problem is with the callbacks
****In order to do this, we need to do 2 separate database operations to get the values. We can try to combine them, however, this entails nesting one call to the database within the other callback, and therefore is messy.
****Instead, we can make them each as a separate route. Set up the following in index.js:

<code><pre>
router.get(‘/count’, function(req,res){
function reportCount(err,count){
if (err) {
res.send(-1);
}
else {
res.send(count);
}
}

logsCollection.count({}, reportCount);
});
</pre></code>

*A problem with this code is that the count is interpreted as a response code
*We can change the code to fix this:

<code><pre>
router.get(‘/count’, function(req,res){
function reportCount(err,count){
if err {
res.sendStatus(500);}
} else {
res.send({count: count});
}
}

logsCollection.count({}, reportCount);
});
</pre></code>

*And the route for file count:

<code><pre>
router.get(‘/storeFiles’, function(req,res) {
function reportStoredFiles(err,files){
if err {
res.sendStatus(500);
} else{
res.send(files); //to get number of files res.send({count: files.length});
}
}

logsCollection.distinct(“file”, reportStoredFiles);
});
</pre></code>

*This code allows us to get these values to the client
*Currently, we do not have any client-side code to do anything with the values
*We want to somehow update the page to reflect the correct values
*We can do this on the server-side before sending the page to the client but this can be a pain so let’s change this so that we do an AJAX request from the browser and update the DOM
*We will need a client-side script to do this
*We can reference exam-storage to see how to link our scripts in the Jade templates
**In layout.jade, we want to link the jquery script since we will be using it
**From account.jade, we can see where our main client-side script was linked. We will need to do something like this for index.jade
**Make sure that the scripts are stored somewhere in the public directory and that the paths in the links are correct
*We can add some default text into index.jade to display if we do not have the actual file and log entry counts
*Then we need to write our query.js script
**The functions in this script are used to update the number of logs and number of entries shown in the browser. So, when you reload the page after uploading a file, the number of logs and entries also get updated accordingly.
***<code>updateStats()</code> is used to make the AJAX requests to the server to get the updated values
***<code>updatesStatsText()</code> is used to update the DOM with the new values

<code><pre>
$(function() {
var numEntries = 0;
var numFiles = 0;
var stats = $(“#stats”);

function updatesStatsText() {
stats.html(“Currently we have “ + numEntries + “ log entries in “ + numFiles + “log files.”);
}

function updateStats() {
var numUpdated = 0;
$.getJSON(“/ count”,function(v){
numEntries = v.count;
numUpdated++;

if(numUpdated >= 2){
updateStatsText();
}
});

$.getJSON(“/storedFileCount”, function(v){
numFiles = v.count;
numUpdated++;

if(numUpdated >= 2){
updateStatsText(); });
}
});
}

updateStats();
}
</pre></code>

*With the assignment, everything should show up at the bottom of the page.
*This means upating the DOM in a similar fashion to what we have done here

====Scalability====

*When replicating your web application, your code should be “embarrassingly parallel” (no direct interaction)
*Communication between servers happens through the backend database
*Why not have the web servers talk directly to each other?
**You have to figure out how to do synchronization/concurrency right
**That's what databases are for! Let it deal with the problem
*So, how in the world do you scale up databases?
*First answer: use a minimal solution
**Only get the functionality that you want
*Basic/first rule of scalability
**You cant do everything at scale
*So you have to choose what you will do. Have to make some sacrifices
*Why are sacrifices necessary?
*Latency vs Bandwidth
**Bandwidth: bits transferred per second on average
**Latency: time to get first bit of response after request
*Consider a large truck full of hard disks driving across Canada. What’s the bandwidth of this truck? Low bandwidth or high bandwidth?
**It’s very very high bandwidth takes two weeks to travel across the country, but when it arrives, a lot of data has been transferred
**Very very high latency as well! (2 weeks to get first bit of response)
*Ideally you want HIGH BANDWIDTH and LOW LATENCY
**Bandwidth you get through parallelism
**Latency has to be engineered.
*Latency is hard so,
**A “supercoumputer” is one with low latency memory access, for lots of memory.
***So it has to have fast interconnects between the nodes. Fast meaning low latency, not high bandwidth.
***Thus, accesses to different nodes aren't much slower than local accesses
*Challenge for large web apps is having the database answer queries with low latency
*If a user goes to your website the page should be loaded fast. Even if it's a bit slow, they will be unhappy. So, you need the low latency.
*But some amount of latency is inevitable (laws of physics, Einstein)
**The speed of light is finite. A nanosecond in wire is a short piece of wire. A microsecond a pretty good size spool. Milliseconds is thousand times larger.
**How far do signals go?
***Microprocessors operate on the nanoseconds
***Networks on the milliseconds
***Ping time to get to the other side of the globe, 100s of millisecond
****If it's more than 50 milliseconds or half a second, people will notice
*So if you want fast access to your web server worldwide
**You need to replicate across the globe
**Be close to your clients
*In general, the more the functionality a database will have, the longer it will take
*NoSQL databases became popular because of latency concerns
**You needed to be as fast as possible
**So strip it to the bone
*Use an in-memory key value store if it is sufficient
**Lowest latency
**Least functionality
*If you have to, use an SQL database
**Higher latency
**Most functionality
*Or use something in between (MongoDB)
*Once you choose the type of database, you optimize
**Minimize I/O and computation require per access (read or write)
**Example: query optimization
**How you form the query
**How the database is organized

<br>

*Why Google is getting big bucks? How do you query the entire Web?
**Google is good at query optimization
*Google has a copy of everything on the Web. Their Web crawlers do this for them
*How do you make a query of the entire Web?
**Use a very big database!
**The interface to query through Google is just a regular website, it's just working with very large data sets!
*How can we count the number of web pages that have the word that have the word “amazing” in them?
**First you need a database with a copy of all the web pages
**Then you could do linear search through all the of the pages of the web (this is not an efficient solution)
*I ask this because a web search is a massive challenge in query optimization
**We need to limit scope as early as possible in query
**Organize data so queries can be answered quickly
**Precompute as much as possible
*The best you can do is table lookup. So have the right tables ready!
*When you do a Google search, it is picking out answer from answers it already figured out before
*A key tool is making an INDEX
**Table of search terms and pointers to data
**e.g. If you have a table of customers sorted by ID
***Have an index of names, so a table of names vs IDs
**An index is a table that is very short, one column for the indexed attribute and the second for the ID
**Is this what Google is doing? No, they are doing much fancier things but don’t tell anyone because of layers and layers of proprietary stuff
*Microsoft has about 100 millions lines of code, Linux kernel is 20 millions lines of code, Google is estimated at about 2 billion lines of code. This is not surprising, there is a lot of functionality
*Look up MapReduce. You can use it to find all web pages with “amazing”. All you do is you have a cluster of machines with data across them and you tell each machine to count the number of times amazing appears in the web pages you have on that machine and then accumulate altogether to be a single count. It can also handle failures during a query. Data is being stored in special replicated way, so that if there is one particular node that is failing, others can take over the job. It is functional programming at scale.

==Code==

* [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/analyzeLogs-filecount.zip analyzeLogs-filecount]

Note this version has node_modules removed; copy this directory from analyzeLogs-sol or run "npm install".

WebFund 2016W Lecture 22

2016-04-04T03:06:12Z

LeeCroft: /* And now the web */

==Video==

The video for the lecture given on March 31, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec22-31Mar2016.mp4 is now available].

==Notes==

===In Class===

<pre>
Lecture 22
------------
Web Security

Security in isolated systems
- no networking

What is the threat model?

Threat model
- what attacks are you trying to defend against
- what attacks are you NOT trying to defend against

Computer in an isolated room
- ignore physical attacks, e.g. sledgehammers

How to compromise this computer?
- someone bad starts typing
- someone inserts malicious media/device
- parking lot flashdrive attack
(solution: epoxy)
- electromagnetic emissions
(live in a faraday cage)

Computer on a network
- all local attacks
- attacker can send arbitrary network data to system
- attacker can listen in on network traffic
- attacker can change outgoing traffic

Basic defense: network crypto
- encrypt to hide, sign data to protect integrity,
verify authenticity

Who will access?
- authenticated, authorized users
- unauthorized individuals/systems
- attackers (can be either of the above)

How can an attacker be "authorized"?
- social engineering
(can you please reset my password?)
- brute force (try all the passwords)
- technical compromise of credentials (keylogger)
- authorized user turns/is malicious
(insider attack)

And now the web

Who can access your application?
- ANYBODY
- for public web apps, ANYBODY is an authorized user

So just from this we can see that web security is going
to be hard.

The other half of the nightmare: the browser

Ideally, to secure systems you isolate components
- build big walls

Your web browser has walls between pages...but there are
BIG HOLES (on purpose)

- cookies, in particular, are shared
- this is bad because the scope of cookies is
determined by DNS (domain name system)

DNS has almost no security
- there is DNSSEC but nobody uses it, yet at least

You can better protect cookies by using HSTS
- force HTTPS everywhere on a page/domain
- protects against mixes of http/https content and
downgrade attacks

Why do we have cookies in the first place?
- because HTTP was supposed to be stateless

Also...

Cross-site Scripting (XSS) attacks
- not necessarily cross-site, not necesarily scripting
- "web injection attacks"
- see anywhere attacker can inject content into a page
- (ads)
- comments
- social media
- "user-generated content"
- failure to sanitize untrusted input that is inserted
into a web page
(adding a comment to a page)
- otherwise, you can insert arbitrary HTML, CSS,
and JavaScript into the page

Cross-site request forgery
- makes use of cookies automatically being sent
to domain, no matter where the link came from
- can stop using tokens in URLs, or checking
Referer/Origin headers (and a few other ways)

Same-origin policy
- data should only be loaded from the originating
server
- but, you can load almost anything else from
anywhere
- images
- sound
- javascript

So what about JSON?
- when you load it with AJAX, you have same origin
policy
- but if you just treated it as a script...
- and what if you were logged in
- bank sending data as JSON

JSON files are valid JavaScript but they don't specify
the name to give to the data structure
- so you can't access it, unless

To build an object, you have to call the object's
constructor
- why not replace the constructor with our own method?
e.g., override Array
- this is solved in regular browsers for JSON by
only using a clean JavaScript environment for
JSON.parse()

This doesn't help you for code, but who cares about
code, right?
- many websites dynamically generate JavaScript and put
personal information into it

Solution? Don't mix code and data!

WebAssembly
-----------

* Many, many people hate JavaScript
* Lots of code that isn't JavaScript
* But people want everything to run on the web
in a browser

Past efforts at having alternative languages all
had a basic problem
- they couldn't see the DOM
- e.g., Java applets, Flash, Silverlight

And, what about all those other languages?
- C, C++??!
- how about games?

Unreal runs in a web page
- there's WebGL which is pretty fast

So, why can't we compile code and have it run in the
browser?

Two old solutions:
- NaCl (Google)
- asm.js (Mozilla)

They joined up and are now making WebAssembly

This will be good for functionality and bad for
security.
- security won't go well because code isn't
designed for the web
- but it will be cool to play games in a browser at
full speed :-)
</pre>

===Student Notes===

====Administrative====

*There is a study session on April 11 morning.
*Students can come to tutorial next week and consider it as extended office hours

====Web Security====

*There are certain fundamental things of the web that make it very hard to secure
*When considering security, we want to consider a threat model
**What attacks are you trying to defend against?
**What attacks are you NOT trying to defend against?

=====Computer in an isolated room=====

*Ignore physical attacks, e.g. sledgehammers
*If we talking about software level, how can the computer be compromised?
**Someone bad starts typing
**Someone inserts malicious media/device
**Parking lot flashdrive attack (a flashdrive is left in a parking lot for someone to pick up and plug in)
**(solution: epoxy, no one can pug anything in)
**Electromagnetic emissions
***(solution: live in a Faraday cage, almost completely isolated from any EM transmission)

=====Computer on a network=====

*Threats:
**All local attacks (all the physical attack will be there too)
**An attacker can send arbitrary network data to system
**An attacker can listen in on network traffic
**An attacker can change outgoing traffic
**(they can listen anything on the computer, can listen anything going out of the computer, can send arbitrary network data to the computer, can change data and leaving the computer going elsewhere)
*How to protect against that?
*Basic defense: network crypto
**Encrypt to hide, sign data to protect integrity, verify authenticity
*We need to distinguish between who is trying to access the computer
*Who will access?
**Authenticated/authorized users
**Unauthorized individuals/systems
**Attackers (can be either of the above)
*How can an attacker be “authorized” ?
**Social engineering (can you please reset my password?)
**Brute force (try all the passwords)
**Technical compromise of credentials (keylogger)
**Authorized user turns/is malicious (insider attack)(most dangerous, because they are not only authorized, they also know the system)

=====And now the web=====

*Why is the web so nasty?
*Who can access your application?
**ANYBODY
**For most public web apps, ANYBODY is an authorized user
**There are different levels of insider attacks. Just because of you have account on gmail doesn’t mean you have access to the inside system. You don’t have full access, but you have some access, that is potentially dangerous.
*So from this we can see that web security is going to be hard
*The other half of the nightmare is the browser
*Ideally, to secure systems you isolate components
**Build big walls
*Your web browser has walls between pages... but there are BIG HOLES (on purpose)
*The tabs in the browser are not so isolated (e.g Log in to an account on one tab then refresh another tab on the same page... it will be logged in as well). They are connected.
*Cookies are being shared between the pages, it is a shared data source
*This is bad because the scope of cookies is determined by DNS (Domain Name System)
*DNS has almost no security
*There is DNSSEC but nobody uses it, yet at least
**If you want to know more about this, you can find the article on [usenix.org usenix.org]
*You can better protect cookies by using HSTS
**This forces HTTPS everywhere on a page/domain
**Protects against mixes of HTTP/HTTPS communications and downgrade attacks
*Why we do we have cookies in the first place?
**Because HTTP was supposed to be stateless
**The cookies are used instead to create a state between the client and server
*If we could get past these problems, would we have proper security?
**Definitely not
*There are other issues...

======Cross-Site Scripting (XSS) Attacks======

*Not necessarily cross-site, and not necessarily scripting
*More like “web injection attacks”
*This can happen anywhere an attacker can inject content into a page
**Ads
**Comments
**Social media
**User-generated content
*Problems can occur when there is a failure to sanitize untrusted input that is inserted into a web page
**This allows users to insert arbitrary HTML, CSS, and JavaScript into the page

======Cross-Site Request Forgery======

*Cross-site scripting is more straight-forward, cross-site request forgery is a little bit weird
*This involves having a link from a page sent by one server cause a request to be sent off to another server
*If you are logged in at the other server, this can cause things to happen which you didn't expect or want to happen
**For example, while logged in to the course wiki, a link from a page on a different website may cause a page to be deleted from the wiki by sending an appropriate request
*This makes use of cookies automatically being sent to a server, no matter where the link came from
*How can we stop this?
**One way is to try to verify where the link came from
**In the browser tools, go to the network tab and check the referer header
***This header indicates where the request came from
**So in principle, to stop a cross-site request forgery, we can check this header in the requests received on the server
***If it came from one of our pages, then we can trust the request
***Otherwise we don’t
**That does not actually work. It kind of works, but not always
**Why? Because the Referer header can often be scrapped by a proxy
**An alternative solution is to randomize your links
**Instead of using a predictable url, stick a random string in the middle of it
***The string is connected to the client's cookie so that the attacker can’t guess it

======Same-Origin Policy======

*Data should only loaded from the originating server but, you can load almost anything else from anywhere
**Images
**Sound
**JavaScript
*So what about JSON?
**when you load it with AJAX, you have the same-origin policy
**But what if you just treated it as a script? (why not just load it in the page?)
**And what if you were logged in?
***Consider a bank sending data as JSON (Could I from another tab just grab your bank account data as a JSON data in the background and see what exactly what you have? It doesn’t work.)
*JSON files are valid JavaScript structure (JSON is a subset of JavaScript)
*JSON files do not specify a name to give to the data structure so you can’t access it by name
*However, to build the object in the first place, its constructor must be called
**Why not replace the constructor with our own method?
***e.g., override Array
**This is solved in regular browsers for JSON by only using a clean JavaScript environment for JSON.parse()
**But what about code?
***This doesn’t help you for code, but who cares about code, right?
**Turns out many websites dynamically generate JavaScript and put personal information into it (different people have different scripts)
***Solution? Don’t mix code and data! (treat the JSON separately, let the code just be loaded on the system, don’t mix it with anything else)

====Web Assembly====

*Many, many people hate JavaScript
**There is a lot of code that isn’t JavaScript but people want everything to run on the web in a browser (JavaScript is the language of the web)
*Past efforts at having alternative languages all had a basic problem
**They couldn’t see the DOM
***e.g., Java applets, Flash, Silverlight
*And, what about all those other languages?
**C,C++?
**How about games?
***Unreal runs in a web page because there’s WebGL which is pretty fast (WebGL is a subset of openGL)
*So why can’t we compile code and have it run in the browser?
*Two old solutions:
**NaCl (Google)
***Native Client is a sandbox for running compiled C and C++ code in the browser efficiently and securely, independent of the user’s operating system
**asm.js (Mozilla)
***This uses a compiler that takes arbitrary programs and compiles them to JavaScript
***asm.js is perfectly legal JavaScript, it is tightly restricted JavaScript
***You cannot read asm.js code, it looks like binary
*Google and Mozilla joined up and are now making WebAssembly
**You can compile arbitrary code to run inside of the browser with WebAssembly
**This will be good for functionality and bad for security
***Security won’t go well because the code isn’t designed for the web
***But it will be cool to play games in a browser at full speed

WebFund 2016W Lecture 22

2016-04-04T03:05:52Z

LeeCroft: /* And now the web */

==Video==

The video for the lecture given on March 31, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec22-31Mar2016.mp4 is now available].

==Notes==

===In Class===

<pre>
Lecture 22
------------
Web Security

Security in isolated systems
- no networking

What is the threat model?

Threat model
- what attacks are you trying to defend against
- what attacks are you NOT trying to defend against

Computer in an isolated room
- ignore physical attacks, e.g. sledgehammers

How to compromise this computer?
- someone bad starts typing
- someone inserts malicious media/device
- parking lot flashdrive attack
(solution: epoxy)
- electromagnetic emissions
(live in a faraday cage)

Computer on a network
- all local attacks
- attacker can send arbitrary network data to system
- attacker can listen in on network traffic
- attacker can change outgoing traffic

Basic defense: network crypto
- encrypt to hide, sign data to protect integrity,
verify authenticity

Who will access?
- authenticated, authorized users
- unauthorized individuals/systems
- attackers (can be either of the above)

How can an attacker be "authorized"?
- social engineering
(can you please reset my password?)
- brute force (try all the passwords)
- technical compromise of credentials (keylogger)
- authorized user turns/is malicious
(insider attack)

And now the web

Who can access your application?
- ANYBODY
- for public web apps, ANYBODY is an authorized user

So just from this we can see that web security is going
to be hard.

The other half of the nightmare: the browser

Ideally, to secure systems you isolate components
- build big walls

Your web browser has walls between pages...but there are
BIG HOLES (on purpose)

- cookies, in particular, are shared
- this is bad because the scope of cookies is
determined by DNS (domain name system)

DNS has almost no security
- there is DNSSEC but nobody uses it, yet at least

You can better protect cookies by using HSTS
- force HTTPS everywhere on a page/domain
- protects against mixes of http/https content and
downgrade attacks

Why do we have cookies in the first place?
- because HTTP was supposed to be stateless

Also...

Cross-site Scripting (XSS) attacks
- not necessarily cross-site, not necesarily scripting
- "web injection attacks"
- see anywhere attacker can inject content into a page
- (ads)
- comments
- social media
- "user-generated content"
- failure to sanitize untrusted input that is inserted
into a web page
(adding a comment to a page)
- otherwise, you can insert arbitrary HTML, CSS,
and JavaScript into the page

Cross-site request forgery
- makes use of cookies automatically being sent
to domain, no matter where the link came from
- can stop using tokens in URLs, or checking
Referer/Origin headers (and a few other ways)

Same-origin policy
- data should only be loaded from the originating
server
- but, you can load almost anything else from
anywhere
- images
- sound
- javascript

So what about JSON?
- when you load it with AJAX, you have same origin
policy
- but if you just treated it as a script...
- and what if you were logged in
- bank sending data as JSON

JSON files are valid JavaScript but they don't specify
the name to give to the data structure
- so you can't access it, unless

To build an object, you have to call the object's
constructor
- why not replace the constructor with our own method?
e.g., override Array
- this is solved in regular browsers for JSON by
only using a clean JavaScript environment for
JSON.parse()

This doesn't help you for code, but who cares about
code, right?
- many websites dynamically generate JavaScript and put
personal information into it

Solution? Don't mix code and data!

WebAssembly
-----------

* Many, many people hate JavaScript
* Lots of code that isn't JavaScript
* But people want everything to run on the web
in a browser

Past efforts at having alternative languages all
had a basic problem
- they couldn't see the DOM
- e.g., Java applets, Flash, Silverlight

And, what about all those other languages?
- C, C++??!
- how about games?

Unreal runs in a web page
- there's WebGL which is pretty fast

So, why can't we compile code and have it run in the
browser?

Two old solutions:
- NaCl (Google)
- asm.js (Mozilla)

They joined up and are now making WebAssembly

This will be good for functionality and bad for
security.
- security won't go well because code isn't
designed for the web
- but it will be cool to play games in a browser at
full speed :-)
</pre>

===Student Notes===

====Administrative====

*There is a study session on April 11 morning.
*Students can come to tutorial next week and consider it as extended office hours

====Web Security====

*There are certain fundamental things of the web that make it very hard to secure
*When considering security, we want to consider a threat model
**What attacks are you trying to defend against?
**What attacks are you NOT trying to defend against?

=====Computer in an isolated room=====

*Ignore physical attacks, e.g. sledgehammers
*If we talking about software level, how can the computer be compromised?
**Someone bad starts typing
**Someone inserts malicious media/device
**Parking lot flashdrive attack (a flashdrive is left in a parking lot for someone to pick up and plug in)
**(solution: epoxy, no one can pug anything in)
**Electromagnetic emissions
***(solution: live in a Faraday cage, almost completely isolated from any EM transmission)

=====Computer on a network=====

*Threats:
**All local attacks (all the physical attack will be there too)
**An attacker can send arbitrary network data to system
**An attacker can listen in on network traffic
**An attacker can change outgoing traffic
**(they can listen anything on the computer, can listen anything going out of the computer, can send arbitrary network data to the computer, can change data and leaving the computer going elsewhere)
*How to protect against that?
*Basic defense: network crypto
**Encrypt to hide, sign data to protect integrity, verify authenticity
*We need to distinguish between who is trying to access the computer
*Who will access?
**Authenticated/authorized users
**Unauthorized individuals/systems
**Attackers (can be either of the above)
*How can an attacker be “authorized” ?
**Social engineering (can you please reset my password?)
**Brute force (try all the passwords)
**Technical compromise of credentials (keylogger)
**Authorized user turns/is malicious (insider attack)(most dangerous, because they are not only authorized, they also know the system)

=====And now the web=====

*Why is the web so nasty?
*Who can access your application?
**ANYBODY
**For most public web apps, ANYBODY is an authorized user
**There are different levels of insider attacks. Just because of you have account on gmail doesn’t mean you have access to the inside system. You don’t have full access, but you have some access, that is potentially dangerous.
*So from this we can see that web security is going to be hard
*The other half of the nightmare is the browser
*Ideally, to secure systems you isolate components
**Build big walls
*Your web browser has walls between pages... but there are BIG HOLES (on purpose)
*The tabs in the browser are not so isolated (e.g Log in to an account on one tab then refresh another tab on the same page... it will be logged in as well). They are connected.
*Cookies are being shared between the pages, it is a shared data source
*This is bad because the scope of cookies is determined by DNS (Domain Name System)
*DNS has almost no security
*There is DNSSEC but nobody uses it, yet at least
**If you want to know more about this, you can find the article on [usenix.org usenix.org]
*You can better protect cookies by using HSTS
**This forces HTTPS everywhere on a page/domain
**Protects against mixes of HTTP/HTTPS communications and downgrade attacks
*Why we do we have cookies in the first place?
**Because HTTP was supposed to be stateless
**The cookies are used instead to create a state between the client and server

<br>

*If we could get past these problems, would we have proper security?
**Definitely not
*There are other issues...

======Cross-Site Scripting (XSS) Attacks======

*Not necessarily cross-site, and not necessarily scripting
*More like “web injection attacks”
*This can happen anywhere an attacker can inject content into a page
**Ads
**Comments
**Social media
**User-generated content
*Problems can occur when there is a failure to sanitize untrusted input that is inserted into a web page
**This allows users to insert arbitrary HTML, CSS, and JavaScript into the page

======Cross-Site Request Forgery======

*Cross-site scripting is more straight-forward, cross-site request forgery is a little bit weird
*This involves having a link from a page sent by one server cause a request to be sent off to another server
*If you are logged in at the other server, this can cause things to happen which you didn't expect or want to happen
**For example, while logged in to the course wiki, a link from a page on a different website may cause a page to be deleted from the wiki by sending an appropriate request
*This makes use of cookies automatically being sent to a server, no matter where the link came from
*How can we stop this?
**One way is to try to verify where the link came from
**In the browser tools, go to the network tab and check the referer header
***This header indicates where the request came from
**So in principle, to stop a cross-site request forgery, we can check this header in the requests received on the server
***If it came from one of our pages, then we can trust the request
***Otherwise we don’t
**That does not actually work. It kind of works, but not always
**Why? Because the Referer header can often be scrapped by a proxy
**An alternative solution is to randomize your links
**Instead of using a predictable url, stick a random string in the middle of it
***The string is connected to the client's cookie so that the attacker can’t guess it

======Same-Origin Policy======

*Data should only loaded from the originating server but, you can load almost anything else from anywhere
**Images
**Sound
**JavaScript
*So what about JSON?
**when you load it with AJAX, you have the same-origin policy
**But what if you just treated it as a script? (why not just load it in the page?)
**And what if you were logged in?
***Consider a bank sending data as JSON (Could I from another tab just grab your bank account data as a JSON data in the background and see what exactly what you have? It doesn’t work.)
*JSON files are valid JavaScript structure (JSON is a subset of JavaScript)
*JSON files do not specify a name to give to the data structure so you can’t access it by name
*However, to build the object in the first place, its constructor must be called
**Why not replace the constructor with our own method?
***e.g., override Array
**This is solved in regular browsers for JSON by only using a clean JavaScript environment for JSON.parse()
**But what about code?
***This doesn’t help you for code, but who cares about code, right?
**Turns out many websites dynamically generate JavaScript and put personal information into it (different people have different scripts)
***Solution? Don’t mix code and data! (treat the JSON separately, let the code just be loaded on the system, don’t mix it with anything else)

====Web Assembly====

*Many, many people hate JavaScript
**There is a lot of code that isn’t JavaScript but people want everything to run on the web in a browser (JavaScript is the language of the web)
*Past efforts at having alternative languages all had a basic problem
**They couldn’t see the DOM
***e.g., Java applets, Flash, Silverlight
*And, what about all those other languages?
**C,C++?
**How about games?
***Unreal runs in a web page because there’s WebGL which is pretty fast (WebGL is a subset of openGL)
*So why can’t we compile code and have it run in the browser?
*Two old solutions:
**NaCl (Google)
***Native Client is a sandbox for running compiled C and C++ code in the browser efficiently and securely, independent of the user’s operating system
**asm.js (Mozilla)
***This uses a compiler that takes arbitrary programs and compiles them to JavaScript
***asm.js is perfectly legal JavaScript, it is tightly restricted JavaScript
***You cannot read asm.js code, it looks like binary
*Google and Mozilla joined up and are now making WebAssembly
**You can compile arbitrary code to run inside of the browser with WebAssembly
**This will be good for functionality and bad for security
***Security won’t go well because the code isn’t designed for the web
***But it will be cool to play games in a browser at full speed

WebFund 2016W Lecture 22

2016-04-04T03:05:26Z

LeeCroft: /* Computer in an isolated room */

==Video==

The video for the lecture given on March 31, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec22-31Mar2016.mp4 is now available].

==Notes==

===In Class===

<pre>
Lecture 22
------------
Web Security

Security in isolated systems
- no networking

What is the threat model?

Threat model
- what attacks are you trying to defend against
- what attacks are you NOT trying to defend against

Computer in an isolated room
- ignore physical attacks, e.g. sledgehammers

How to compromise this computer?
- someone bad starts typing
- someone inserts malicious media/device
- parking lot flashdrive attack
(solution: epoxy)
- electromagnetic emissions
(live in a faraday cage)

Computer on a network
- all local attacks
- attacker can send arbitrary network data to system
- attacker can listen in on network traffic
- attacker can change outgoing traffic

Basic defense: network crypto
- encrypt to hide, sign data to protect integrity,
verify authenticity

Who will access?
- authenticated, authorized users
- unauthorized individuals/systems
- attackers (can be either of the above)

How can an attacker be "authorized"?
- social engineering
(can you please reset my password?)
- brute force (try all the passwords)
- technical compromise of credentials (keylogger)
- authorized user turns/is malicious
(insider attack)

And now the web

Who can access your application?
- ANYBODY
- for public web apps, ANYBODY is an authorized user

So just from this we can see that web security is going
to be hard.

The other half of the nightmare: the browser

Ideally, to secure systems you isolate components
- build big walls

Your web browser has walls between pages...but there are
BIG HOLES (on purpose)

- cookies, in particular, are shared
- this is bad because the scope of cookies is
determined by DNS (domain name system)

DNS has almost no security
- there is DNSSEC but nobody uses it, yet at least

You can better protect cookies by using HSTS
- force HTTPS everywhere on a page/domain
- protects against mixes of http/https content and
downgrade attacks

Why do we have cookies in the first place?
- because HTTP was supposed to be stateless

Also...

Cross-site Scripting (XSS) attacks
- not necessarily cross-site, not necesarily scripting
- "web injection attacks"
- see anywhere attacker can inject content into a page
- (ads)
- comments
- social media
- "user-generated content"
- failure to sanitize untrusted input that is inserted
into a web page
(adding a comment to a page)
- otherwise, you can insert arbitrary HTML, CSS,
and JavaScript into the page

Cross-site request forgery
- makes use of cookies automatically being sent
to domain, no matter where the link came from
- can stop using tokens in URLs, or checking
Referer/Origin headers (and a few other ways)

Same-origin policy
- data should only be loaded from the originating
server
- but, you can load almost anything else from
anywhere
- images
- sound
- javascript

So what about JSON?
- when you load it with AJAX, you have same origin
policy
- but if you just treated it as a script...
- and what if you were logged in
- bank sending data as JSON

JSON files are valid JavaScript but they don't specify
the name to give to the data structure
- so you can't access it, unless

To build an object, you have to call the object's
constructor
- why not replace the constructor with our own method?
e.g., override Array
- this is solved in regular browsers for JSON by
only using a clean JavaScript environment for
JSON.parse()

This doesn't help you for code, but who cares about
code, right?
- many websites dynamically generate JavaScript and put
personal information into it

Solution? Don't mix code and data!

WebAssembly
-----------

* Many, many people hate JavaScript
* Lots of code that isn't JavaScript
* But people want everything to run on the web
in a browser

Past efforts at having alternative languages all
had a basic problem
- they couldn't see the DOM
- e.g., Java applets, Flash, Silverlight

And, what about all those other languages?
- C, C++??!
- how about games?

Unreal runs in a web page
- there's WebGL which is pretty fast

So, why can't we compile code and have it run in the
browser?

Two old solutions:
- NaCl (Google)
- asm.js (Mozilla)

They joined up and are now making WebAssembly

This will be good for functionality and bad for
security.
- security won't go well because code isn't
designed for the web
- but it will be cool to play games in a browser at
full speed :-)
</pre>

===Student Notes===

====Administrative====

*There is a study session on April 11 morning.
*Students can come to tutorial next week and consider it as extended office hours

====Web Security====

*There are certain fundamental things of the web that make it very hard to secure
*When considering security, we want to consider a threat model
**What attacks are you trying to defend against?
**What attacks are you NOT trying to defend against?

=====Computer in an isolated room=====

*Ignore physical attacks, e.g. sledgehammers
*If we talking about software level, how can the computer be compromised?
**Someone bad starts typing
**Someone inserts malicious media/device
**Parking lot flashdrive attack (a flashdrive is left in a parking lot for someone to pick up and plug in)
**(solution: epoxy, no one can pug anything in)
**Electromagnetic emissions
***(solution: live in a Faraday cage, almost completely isolated from any EM transmission)

=====Computer on a network=====

*Threats:
**All local attacks (all the physical attack will be there too)
**An attacker can send arbitrary network data to system
**An attacker can listen in on network traffic
**An attacker can change outgoing traffic
**(they can listen anything on the computer, can listen anything going out of the computer, can send arbitrary network data to the computer, can change data and leaving the computer going elsewhere)
*How to protect against that?
*Basic defense: network crypto
**Encrypt to hide, sign data to protect integrity, verify authenticity
*We need to distinguish between who is trying to access the computer
*Who will access?
**Authenticated/authorized users
**Unauthorized individuals/systems
**Attackers (can be either of the above)
*How can an attacker be “authorized” ?
**Social engineering (can you please reset my password?)
**Brute force (try all the passwords)
**Technical compromise of credentials (keylogger)
**Authorized user turns/is malicious (insider attack)(most dangerous, because they are not only authorized, they also know the system)

=====And now the web=====

*Why is the web so nasty?
*Who can access your application?
**ANYBODY
**For most public web apps, ANYBODY is an authorized user
**There are different levels of insider attacks. Just because of you have account on gmail doesn’t mean you have access to the inside system. You don’t have full access, but you have some access, that is potentially dangerous.
*So from this we can see that web security is going to be hard

<br>

*The other half of the nightmare: the browser
*Ideally, to secure systems you isolate components
**Build big walls
*Your web browser has walls between pages... but there are BIG HOLES (on purpose)
*The tabs in the browser are not so isolated (e.g Log in to an account on one tab then refresh another tab on the same page... it will be logged in as well). They are connected.
*Cookies are being shared between the pages, it is a shared data source
*This is bad because the scope of cookies is determined by DNS (Domain Name System)
*DNS has almost no security
*There is DNSSEC but nobody uses it, yet at least
**If you want to know more about this, you can find the article on [usenix.org usenix.org]
*You can better protect cookies by using HSTS
**This forces HTTPS everywhere on a page/domain
**Protects against mixes of HTTP/HTTPS communications and downgrade attacks
*Why we do we have cookies in the first place?
**Because HTTP was supposed to be stateless
**The cookies are used instead to create a state between the client and server

<br>

*If we could get past these problems, would we have proper security?
**Definitely not
*There are other issues...

======Cross-Site Scripting (XSS) Attacks======

*Not necessarily cross-site, and not necessarily scripting
*More like “web injection attacks”
*This can happen anywhere an attacker can inject content into a page
**Ads
**Comments
**Social media
**User-generated content
*Problems can occur when there is a failure to sanitize untrusted input that is inserted into a web page
**This allows users to insert arbitrary HTML, CSS, and JavaScript into the page

======Cross-Site Request Forgery======

*Cross-site scripting is more straight-forward, cross-site request forgery is a little bit weird
*This involves having a link from a page sent by one server cause a request to be sent off to another server
*If you are logged in at the other server, this can cause things to happen which you didn't expect or want to happen
**For example, while logged in to the course wiki, a link from a page on a different website may cause a page to be deleted from the wiki by sending an appropriate request
*This makes use of cookies automatically being sent to a server, no matter where the link came from
*How can we stop this?
**One way is to try to verify where the link came from
**In the browser tools, go to the network tab and check the referer header
***This header indicates where the request came from
**So in principle, to stop a cross-site request forgery, we can check this header in the requests received on the server
***If it came from one of our pages, then we can trust the request
***Otherwise we don’t
**That does not actually work. It kind of works, but not always
**Why? Because the Referer header can often be scrapped by a proxy
**An alternative solution is to randomize your links
**Instead of using a predictable url, stick a random string in the middle of it
***The string is connected to the client's cookie so that the attacker can’t guess it

======Same-Origin Policy======

*Data should only loaded from the originating server but, you can load almost anything else from anywhere
**Images
**Sound
**JavaScript
*So what about JSON?
**when you load it with AJAX, you have the same-origin policy
**But what if you just treated it as a script? (why not just load it in the page?)
**And what if you were logged in?
***Consider a bank sending data as JSON (Could I from another tab just grab your bank account data as a JSON data in the background and see what exactly what you have? It doesn’t work.)
*JSON files are valid JavaScript structure (JSON is a subset of JavaScript)
*JSON files do not specify a name to give to the data structure so you can’t access it by name
*However, to build the object in the first place, its constructor must be called
**Why not replace the constructor with our own method?
***e.g., override Array
**This is solved in regular browsers for JSON by only using a clean JavaScript environment for JSON.parse()
**But what about code?
***This doesn’t help you for code, but who cares about code, right?
**Turns out many websites dynamically generate JavaScript and put personal information into it (different people have different scripts)
***Solution? Don’t mix code and data! (treat the JSON separately, let the code just be loaded on the system, don’t mix it with anything else)

====Web Assembly====

*Many, many people hate JavaScript
**There is a lot of code that isn’t JavaScript but people want everything to run on the web in a browser (JavaScript is the language of the web)
*Past efforts at having alternative languages all had a basic problem
**They couldn’t see the DOM
***e.g., Java applets, Flash, Silverlight
*And, what about all those other languages?
**C,C++?
**How about games?
***Unreal runs in a web page because there’s WebGL which is pretty fast (WebGL is a subset of openGL)
*So why can’t we compile code and have it run in the browser?
*Two old solutions:
**NaCl (Google)
***Native Client is a sandbox for running compiled C and C++ code in the browser efficiently and securely, independent of the user’s operating system
**asm.js (Mozilla)
***This uses a compiler that takes arbitrary programs and compiles them to JavaScript
***asm.js is perfectly legal JavaScript, it is tightly restricted JavaScript
***You cannot read asm.js code, it looks like binary
*Google and Mozilla joined up and are now making WebAssembly
**You can compile arbitrary code to run inside of the browser with WebAssembly
**This will be good for functionality and bad for security
***Security won’t go well because the code isn’t designed for the web
***But it will be cool to play games in a browser at full speed

WebFund 2016W Lecture 22

2016-04-04T03:05:01Z

LeeCroft:

==Video==

The video for the lecture given on March 31, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec22-31Mar2016.mp4 is now available].

==Notes==

===In Class===

<pre>
Lecture 22
------------
Web Security

Security in isolated systems
- no networking

What is the threat model?

Threat model
- what attacks are you trying to defend against
- what attacks are you NOT trying to defend against

Computer in an isolated room
- ignore physical attacks, e.g. sledgehammers

How to compromise this computer?
- someone bad starts typing
- someone inserts malicious media/device
- parking lot flashdrive attack
(solution: epoxy)
- electromagnetic emissions
(live in a faraday cage)

Computer on a network
- all local attacks
- attacker can send arbitrary network data to system
- attacker can listen in on network traffic
- attacker can change outgoing traffic

Basic defense: network crypto
- encrypt to hide, sign data to protect integrity,
verify authenticity

Who will access?
- authenticated, authorized users
- unauthorized individuals/systems
- attackers (can be either of the above)

How can an attacker be "authorized"?
- social engineering
(can you please reset my password?)
- brute force (try all the passwords)
- technical compromise of credentials (keylogger)
- authorized user turns/is malicious
(insider attack)

And now the web

Who can access your application?
- ANYBODY
- for public web apps, ANYBODY is an authorized user

So just from this we can see that web security is going
to be hard.

The other half of the nightmare: the browser

Ideally, to secure systems you isolate components
- build big walls

Your web browser has walls between pages...but there are
BIG HOLES (on purpose)

- cookies, in particular, are shared
- this is bad because the scope of cookies is
determined by DNS (domain name system)

DNS has almost no security
- there is DNSSEC but nobody uses it, yet at least

You can better protect cookies by using HSTS
- force HTTPS everywhere on a page/domain
- protects against mixes of http/https content and
downgrade attacks

Why do we have cookies in the first place?
- because HTTP was supposed to be stateless

Also...

Cross-site Scripting (XSS) attacks
- not necessarily cross-site, not necesarily scripting
- "web injection attacks"
- see anywhere attacker can inject content into a page
- (ads)
- comments
- social media
- "user-generated content"
- failure to sanitize untrusted input that is inserted
into a web page
(adding a comment to a page)
- otherwise, you can insert arbitrary HTML, CSS,
and JavaScript into the page

Cross-site request forgery
- makes use of cookies automatically being sent
to domain, no matter where the link came from
- can stop using tokens in URLs, or checking
Referer/Origin headers (and a few other ways)

Same-origin policy
- data should only be loaded from the originating
server
- but, you can load almost anything else from
anywhere
- images
- sound
- javascript

So what about JSON?
- when you load it with AJAX, you have same origin
policy
- but if you just treated it as a script...
- and what if you were logged in
- bank sending data as JSON

JSON files are valid JavaScript but they don't specify
the name to give to the data structure
- so you can't access it, unless

To build an object, you have to call the object's
constructor
- why not replace the constructor with our own method?
e.g., override Array
- this is solved in regular browsers for JSON by
only using a clean JavaScript environment for
JSON.parse()

This doesn't help you for code, but who cares about
code, right?
- many websites dynamically generate JavaScript and put
personal information into it

Solution? Don't mix code and data!

WebAssembly
-----------

* Many, many people hate JavaScript
* Lots of code that isn't JavaScript
* But people want everything to run on the web
in a browser

Past efforts at having alternative languages all
had a basic problem
- they couldn't see the DOM
- e.g., Java applets, Flash, Silverlight

And, what about all those other languages?
- C, C++??!
- how about games?

Unreal runs in a web page
- there's WebGL which is pretty fast

So, why can't we compile code and have it run in the
browser?

Two old solutions:
- NaCl (Google)
- asm.js (Mozilla)

They joined up and are now making WebAssembly

This will be good for functionality and bad for
security.
- security won't go well because code isn't
designed for the web
- but it will be cool to play games in a browser at
full speed :-)
</pre>

===Student Notes===

====Administrative====

*There is a study session on April 11 morning.
*Students can come to tutorial next week and consider it as extended office hours

====Web Security====

*There are certain fundamental things of the web that make it very hard to secure
*When considering security, we want to consider a threat model
**What attacks are you trying to defend against?
**What attacks are you NOT trying to defend against?

=====Computer in an isolated room=====

*Ignore physical attacks, e.g. sledgehammers
*If we talking about software level, how can the computer be compromised?
**Someone bad starts typing
**Someone inserts malicious media/device
**Parking lot flashdrive attack (a flashdrive is left in a parking lot for someone to pick up and plug in)
**(solution: epoxy, no one can pug anything in)
**electromagnetic emissions
***(solution: live in a Faraday cage, almost completely isolated from any EM transmission)

=====Computer on a network=====

*Threats:
**All local attacks (all the physical attack will be there too)
**An attacker can send arbitrary network data to system
**An attacker can listen in on network traffic
**An attacker can change outgoing traffic
**(they can listen anything on the computer, can listen anything going out of the computer, can send arbitrary network data to the computer, can change data and leaving the computer going elsewhere)
*How to protect against that?
*Basic defense: network crypto
**Encrypt to hide, sign data to protect integrity, verify authenticity
*We need to distinguish between who is trying to access the computer
*Who will access?
**Authenticated/authorized users
**Unauthorized individuals/systems
**Attackers (can be either of the above)
*How can an attacker be “authorized” ?
**Social engineering (can you please reset my password?)
**Brute force (try all the passwords)
**Technical compromise of credentials (keylogger)
**Authorized user turns/is malicious (insider attack)(most dangerous, because they are not only authorized, they also know the system)

=====And now the web=====

*Why is the web so nasty?
*Who can access your application?
**ANYBODY
**For most public web apps, ANYBODY is an authorized user
**There are different levels of insider attacks. Just because of you have account on gmail doesn’t mean you have access to the inside system. You don’t have full access, but you have some access, that is potentially dangerous.
*So from this we can see that web security is going to be hard

<br>

*The other half of the nightmare: the browser
*Ideally, to secure systems you isolate components
**Build big walls
*Your web browser has walls between pages... but there are BIG HOLES (on purpose)
*The tabs in the browser are not so isolated (e.g Log in to an account on one tab then refresh another tab on the same page... it will be logged in as well). They are connected.
*Cookies are being shared between the pages, it is a shared data source
*This is bad because the scope of cookies is determined by DNS (Domain Name System)
*DNS has almost no security
*There is DNSSEC but nobody uses it, yet at least
**If you want to know more about this, you can find the article on [usenix.org usenix.org]
*You can better protect cookies by using HSTS
**This forces HTTPS everywhere on a page/domain
**Protects against mixes of HTTP/HTTPS communications and downgrade attacks
*Why we do we have cookies in the first place?
**Because HTTP was supposed to be stateless
**The cookies are used instead to create a state between the client and server

<br>

*If we could get past these problems, would we have proper security?
**Definitely not
*There are other issues...

======Cross-Site Scripting (XSS) Attacks======

*Not necessarily cross-site, and not necessarily scripting
*More like “web injection attacks”
*This can happen anywhere an attacker can inject content into a page
**Ads
**Comments
**Social media
**User-generated content
*Problems can occur when there is a failure to sanitize untrusted input that is inserted into a web page
**This allows users to insert arbitrary HTML, CSS, and JavaScript into the page

======Cross-Site Request Forgery======

*Cross-site scripting is more straight-forward, cross-site request forgery is a little bit weird
*This involves having a link from a page sent by one server cause a request to be sent off to another server
*If you are logged in at the other server, this can cause things to happen which you didn't expect or want to happen
**For example, while logged in to the course wiki, a link from a page on a different website may cause a page to be deleted from the wiki by sending an appropriate request
*This makes use of cookies automatically being sent to a server, no matter where the link came from
*How can we stop this?
**One way is to try to verify where the link came from
**In the browser tools, go to the network tab and check the referer header
***This header indicates where the request came from
**So in principle, to stop a cross-site request forgery, we can check this header in the requests received on the server
***If it came from one of our pages, then we can trust the request
***Otherwise we don’t
**That does not actually work. It kind of works, but not always
**Why? Because the Referer header can often be scrapped by a proxy
**An alternative solution is to randomize your links
**Instead of using a predictable url, stick a random string in the middle of it
***The string is connected to the client's cookie so that the attacker can’t guess it

======Same-Origin Policy======

*Data should only loaded from the originating server but, you can load almost anything else from anywhere
**Images
**Sound
**JavaScript
*So what about JSON?
**when you load it with AJAX, you have the same-origin policy
**But what if you just treated it as a script? (why not just load it in the page?)
**And what if you were logged in?
***Consider a bank sending data as JSON (Could I from another tab just grab your bank account data as a JSON data in the background and see what exactly what you have? It doesn’t work.)
*JSON files are valid JavaScript structure (JSON is a subset of JavaScript)
*JSON files do not specify a name to give to the data structure so you can’t access it by name
*However, to build the object in the first place, its constructor must be called
**Why not replace the constructor with our own method?
***e.g., override Array
**This is solved in regular browsers for JSON by only using a clean JavaScript environment for JSON.parse()
**But what about code?
***This doesn’t help you for code, but who cares about code, right?
**Turns out many websites dynamically generate JavaScript and put personal information into it (different people have different scripts)
***Solution? Don’t mix code and data! (treat the JSON separately, let the code just be loaded on the system, don’t mix it with anything else)

====Web Assembly====

*Many, many people hate JavaScript
**There is a lot of code that isn’t JavaScript but people want everything to run on the web in a browser (JavaScript is the language of the web)
*Past efforts at having alternative languages all had a basic problem
**They couldn’t see the DOM
***e.g., Java applets, Flash, Silverlight
*And, what about all those other languages?
**C,C++?
**How about games?
***Unreal runs in a web page because there’s WebGL which is pretty fast (WebGL is a subset of openGL)
*So why can’t we compile code and have it run in the browser?
*Two old solutions:
**NaCl (Google)
***Native Client is a sandbox for running compiled C and C++ code in the browser efficiently and securely, independent of the user’s operating system
**asm.js (Mozilla)
***This uses a compiler that takes arbitrary programs and compiles them to JavaScript
***asm.js is perfectly legal JavaScript, it is tightly restricted JavaScript
***You cannot read asm.js code, it looks like binary
*Google and Mozilla joined up and are now making WebAssembly
**You can compile arbitrary code to run inside of the browser with WebAssembly
**This will be good for functionality and bad for security
***Security won’t go well because the code isn’t designed for the web
***But it will be cool to play games in a browser at full speed

WebFund 2016W Lecture 20

2016-03-31T14:04:47Z

LeeCroft: /* Exam Storage */

==Video==

The video for the lecture given on March 24, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec20-24Mar2016.mp4 is now available].

==Student Notes==

===Assignment 5 – Last Minute Question===

*How should the download functionality display the query result?
**Display as a page as a text file. Do a <code>res.send()</code> of a big string with each log entry terminated by a new line. You should be sending the file as plain text.

===Assignment 6 Hints===

*This will involve making an application that works in a similar fashion to exam-storage (a single-page app)
*You will be doing the same things that are already implemented in assignment 5 but you need to change it to work from a single page

===Exam Storage===

*A little bit more complicated than interactive-demo.
*The file structure of the application is modified to clarify the division between what runs on the client and what runs on the server
**public: contains everything that goes to the client directly (CSS and JavaScript files)
**server: contains all the JavaScript files that will run on the server
**start.js is a little JavaScript file that you will use to start the server (this is replacing bin/www)
**views contains all the templates for generating html
**keys: cryptographic keys
*Note that you can use <code>npm rebuild</code> to recompile any binary dependencies in the application modules (in case something is not working on your machine)

====start.js====

*The app.js file is now located in the server directory
**The call to <code>require()</code> in start.js is therefore changed to: <code>var app = require(‘./server/app’);</code>
**We then set the port, create the server and it start listening on port 3000.

====app.js====

*Things are fairly similar to our previous applications
**We are specifying the ssl keys and setting up the same modules we've previously used including one for sessions
**<code>app.use(express.static(‘public’));</code> is serving everything from public directory

====routes.js ====

*Starts with setting up a connection to the database.
**The users collection contains documents which have a username and a hashed password
*Many routes defined in here are similar to what we have done before
*The <code>storeSaltedpassword()</code> function in the /register is something new
**This stores the username and hashed and salted password in the users collection
**The hashing and salting is done using bcrypt
*In the /login route, we are again using bcrypt to compare a password that a user has entered with a hashed version stored in the database
**Why do we have the same error message for both incorrect password and invalid username?
***If someone is trying to break into your system, you don’t want them to know that they have come across the valid account and they are trying to figure out the password. You just want them to know that the username and the password is not the right one. Therefore, we use the same error string for both username and password.

====Password Hashing====

*When you log in to the virtual machine and type in the password, it is stored in the virtual machine
**Go to cd /etc there is a file called passwd which gives you a file that contains all of the accounts on the system
**In old school UNIX installations, the hash of the password is shown instead of 'x' in the file, but not here if you check the stored passwords stored in virtual machine
**To show the hash of the password in the file, you can type in the command <code>sudo shadowconfig off</code>, and then you should be able to see the file with the hash of your password instead of the 'x'
*For any file you create, you can compute a sha1sum (hash) of that file which is a big alphanumeric string
**If you change one bit in the file (input), it will make a huge change in the output
**Sha1sum should no longer be used since it is no longer considered secure but you can use sha256 instead
*Bcrypt is what we are using to hash passwords before storing them in the database
**Bcrypt is a hash function but it is a bit weird. It is slow by design and is memory intensive.
**Storing your passwords using bcrypt is really good today but in general, it is a good idea to use some kind of framework of something like Passport.
***These types of frameworks take care of making decisions that you might not know how to make and they help avoid using outdated and insecure technology
***Passport is a middleware authentication module for Node.js
*Why do we want to use hashing for passwords in the first place? Why do we not want to store them in the plain text?
**We want to avoid a breach in security caused by an attacker obtaining the stored passwords
**The hashing approach should have an irreversibility property that is essential for password storage, has adequate pre-computation defenses, and performance defenses needed to prevent dictionary attacks
**Just by looking at a hashed string, no one should be able to guess the original value and get your password
***They will actually have to play around with the hashing function or will have to plug in random values and try every single possibility which should hopefully take them a very long time
*In our web application, it is important to do the hashing on the server as opposed to on the client's machine even though doing this on the client would reduce the computational burden for the server
**This is because we cannot trust anything that is sent to us from the client
**Rather than computing the hash function and then sending us the resultant hash, the client could iterate over hash values and send them to us
***This would allow the client to avoid the need to spend time actually running the hash function

====Salting====

*Let’s say user 1 and user 2 have the same password, they will also have the same resultant hash
**We don’t want that as this makes things easier for an attacker
**What we will do is that we use the same password for both but add a string called salt1 and salt2 for users 1 and 2 respectively
**Now they have completely different hashes but the password is still the same
**Every user should have a different hash
*Salting is also the defeat to precomputation
**An attacker could use a precomputed dictionary of hash values allowing them to quickly look up the input that was used to create the hash
**By hashing a salted password, we are able to render the lookup tables useless

====account.js====

*<code>updateFileList()</code> is used to update the files displayed on the page
**It first sends an ajax GET request for /getFileStats to the server to get the updated file list
***When the server receives this request, it first checks to see if the user is logged in
***If the user is logged in, it queries the database to get their uploaded files
****When doing a query, we can optionally specify a projection object
****The projection is used to specify which information (document attributes) you want to retrieve in the query
****We specify the query object as <code>{content:0}</code> meaning that we do not want to include the content attribute in the documents that we retrieve
**In the <code>doUpdateFileList()</code> callback function, we update the DOM with the data received in the response
*<code>updateFileList()</code> is called when the page is initially loaded and every time a file is uploaded
**This means that the content on the page will always be up to date
*When a user requests a file download, how do we know which file to give them?
**We use the <code>downloadFile()</code> function to create and return a function which will be used as a callback function for the click event of the link to download a specific file
**By passing the ID of the file into <code>downloadFile()</code> we are able to create a callback function specific to each file so that we know which one to download when it gets called for the click event
*Notice that the href attribute for the download links is set to #
**This causes the link to take the user to the top of the page
**This is useful when we want a link that will not cause the user to go to a different page
*<code>saveDownloadedFile()</code> is used as a callback function for the POST request that is sent to the server when a file is requested for download from within a download click event callback function
**This function handles saving the file that gets sent to the client from the server
**It relies on the <code>saveAs()</code> function which lets us download file in the folder/location of our choice without changing the page.
*The reason why you want to manipulate things on the browser instead of on the server and the differences involved in making this change are important parts of exam-storage that you should understand

===Types of Questions for the Exam===

*Why are we using https?
**To protect data going between the browser and the server, just to make sure everything is encrypted there
*Why are we using bcrypt?
**To make sure that the password is not stored in plain text on the server
*What if in routes.js I just compare the password and the user.password instead of doing bcrypt?

===Next Week===

*High level topics which aren’t included in the final
*How do you make web applications pretty without being a web designer?

==Code==

In this lecture we discussed [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/exam-storage.zip exam-storage], the code for [[WebFund 2016W: Tutorial 10|Tutorial 10]]

WebFund 2016W Lecture 21

2016-03-30T19:20:08Z

LeeCroft:

==Video==

The video for the lecture given on March 29, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec21-29Mar2016.mp4 is now available].

==Notes==

===In Class===

<pre>
Lecture 21
----------

* interactive-graphs-sol will be posted after lecture
* study session on April 11th, time TBD (final is on April 12th) but will be in the morning.
(Earliest is 10 AM).

What did I leave out?

First, what did we cover?
- JavaScript
- Some HTML, CSS
- clients versus servers
- HTTP
- synchronous versus async code
- callbacks
- Node, Jade, Express
- browser developer tools
- closures, lexical scoping
- a little networking
- databases
- noSQL, MongoDB
- queries
- a little crypto (SSL, certificates, password hashing)
- a little security
(cross-site scripting, input validation
- client side versus server side JavaScript
- jQuery
- DOM basics
- basic AJAX

Left out...
- much more to HTML, CSS
- much more to the DOM
- more to AJAX, client-server communication
(e.g., WebSockets)
- SQL
- transactions
- indexing *
- PHP
- language + HTML templates
- instead we learned Node + Jade
- Meteor, AngularJS, ...
client-side JS libraries and frameworks
- other server-side languages & frameworks
- ASP.NET, .NET, Java-based frameworks,
Ruby on Rails
- every programming language has multiple
web frameworks

So, is this how the "big players" do the web?
- facebook, google, etc.

They have
- more servers
- challenge is how to *use* them

HTTP is inherently stateless. How do you scale stateless?
- replicate, load balance
- to store state, make state store bigger
- need bigger database!

Big websites have some mix of:
- scalable databases
- replicate application servers
- caching for static content

E.g., ecommerce
- while shopping, users get static content
- easy to scale
- but purchases have to hit the database
- and have to generate writes

To scale you have to engineer the system
- but you don't need to change your basic application

Security?
- input validation/sanitization
- don't let the user talk to the browser, server,
or database directly - no code injection
- use reasonable crypto
- communications security (SSL/TLS)
- authentication (passwords, etc)
- security policy
- what is allowed, what is not
- how to enforce

Networking?
- latency versus bandwidth*
- IPv6
- firewalls
- NAT

Cloud computing
- remote servers where there are LOTS of remote servers
- challenge is in coordinating applications across
all of those servers
- basic tools is virtualization
- easier to manage a virtual server than a real one
- Give me 10,000 servers for 2 hours
- cloud servers run web apps, almost entirely

Amazon is the real "parent" of cloud computing
- they got into the cloud because of Christmas
- they had servers that were doing nothing 11 months
out of the year
- so why not rent them out
- they provide more and more infrastructure for
building large web applications, based on their
internal tech

Networking

The Internet is a network of networks
SCS network is connected to the Carleton backbone
which is connected to the Unicentre's network. The
backbone also connects to Carleton's ISPs (ISPs)

Firewalls on the Internet block all kinds of network traffic.
Mainly for 'security reasons'

Now, arbitrary network traffic gets routed through HTTP.
Because it can reach all of the Internet
- other protocols get squeezed into HTTP

WebRTC

Learn networking but realize modern networks just ship HTTP

* Databases, indexes
* Security
- web security issues
* The future

Posting Assignment 5 solutions now
</pre>

===Student Notes===

====Announcements====

*There might be an exam review on April 7th. There will also be a review session on April 11th (time TBD, probably AM). Final is on April 12th (9:00 AM).
*Solutions for Assignment 5 (Interactive Graphs) have been posted on the course website.

====What We Covered in the Course====

*First, what did we cover in the course so far?
**JavaScript
**Some HTML, CSS
**Clients versus servers
**HTTP
**Synchronous versus asynchronous code (callbacks)
**Node, Jade, Express
**Browser developer tools
**Closures, lexical scoping
**A little bit of networking
**Databases (noSQL, MongoDB)
***Queries
**A little bit of cryptography (SSL, certificates, password hashing)
**A little security (cross-site scripting, input validation)
**Client-side versus server-side JavaScript (jQuery)
**DOM basics
**Basic AJAX
*This is a good amount of material, especially given that COMP 2406 is an introductory class. However, it is only a small portion of the technologies upon which the modern web is based.

====What did we leave out?====

*There's much more to HTML and CSS
**The stuff we left out wasn't essential to an introductory class like 2406, and can be learned easily through online resources like MDN or Code Academy. You might have to learn them on your own if you intend to work on an actual web development project. Thanks to technologies like Express, however, you don't really have to know a lot of HTML and CSS to get a basic website up and running.
*SQL
**The way databases are accessed
**Transactions
**Indexing (how to make database access faster)
*PHP: language + HTML templates
**There are a lot of serious web applications, both proprietary and open source, that use PHP. If you understand Java and JavaScript well, PHP isn't very hard to learn. It's basically a language combined with HTML templates.
**Instead, we learned Node + Jade to generate HTML
*Meteor, AngularJS, and other client-side JS libraries and frameworks
**There are a lot of JS libraries: low-level as well as high-level
**There are JS libraries to do almost everything under the sun
*Other server-side languages and frameworks
**E.g. ASP.NET, .NET, Java-based frameworks, Ruby on Rails
**There's even a COBOL-based server-side language
**Learning each of these frameworks takes time and energy, but it might be worthwhile if you end up working with them.
**Many fundamental concepts will be the same across the different languages
*There's much more to the DOM
**The DOM is a very complicated data structure; there's much more to learn than what we covered in class.
*There's more to AJAX
**AJAX provides a way for the client and server to talk in the background

====Scaling====

*So, is this how the "big players" do web development?
**They have more servers
**The challenge is how to use the powerful servers
**Scalability becomes a key issue
*The big players have to be able to serve a massive number of people at the same time. Not only do they have to have more servers, they also need to utilize the additional hardware efficiently. The tough part is figuring out how to make use of the extra computational resources.
*HTTP is inherently stateless. How do you scale stateless machines?
**Replicate, load-balance
***These things can be done automatically for you if you use the cloud. For example: AWS.
*To store state, you'd need a bigger database
**The database has to be able to scale to whatever load you put on it
**Companies like Oracle sell big, scalable databases
*Big websites have some mix of:
**Scalable databases
***Databases must be designed for concurrency
**Replication of application on servers (many instances running)
***Allows you to serve large number of clients concurrently
**Caching for static content
***Preemptive loading of some content to reduce dynamic database access
**The state of your app is not stored on server, it is stored in the database
**Good databases are designed for concurrency
**Good databases are designed to optimize write performance and store data reliably.
**However, as a web designer, your job is not to reinvent the wheel. Pick the proper DB option for your particular app.
**Your app should try to minimize DB accesses in order to increase performance
*E.g.: e-commerce
**While shopping, you're getting static content. That's because the static parts are easy to scale.
**However, purchases have to hit the database and have to invoke writes. Writes are much more expensive than reads.
*To scale up, you don't need to change your basic application. But you'll have to engineer the system so that it can handle a large amount of simultaneous database queries and writes.

====Security====

*Input validation/sanitization
**Don't let users talk to the browser, server, or database directly (no code injection)
*Use good cryptography
**Communications security (SSL/TLS)
**Authentication (passwords, etc.)
*Security policy (what is allowed, what isn't allowed)
*Ways of enforcing the security policy
**Policy specification and enforcement are both difficult

====Cloud Computing====

*Remote servers that handle a lot of data and processing for clients
*The challenge is to coordinate applications across all the servers
*Basic tool is virtualization
**Easier to manage a virtual server than a real server
*It's easy to forget that cloud servers run big web apps, almost entirely

====Amazon====

*Amazon is the real "parent" of cloud computing
*They got into the cloud because of Christmas
**They had servers that were doing nothing 11 months out of the year
**Why not rent them out?
**That's how Amazon got into the cloud computing business!
**They provide more and more infrastructure for building large web apps, based on their internal technologies.
**They sell a scalable cloud computing service called AWS Elastic Beanstalk
***"elastic" refers to the scalability aspect of the service
***Handles load-balancing and instance replication automatically

====Networking====

*Latency versus badwidth
**Latency: how many hops to destination?
**Bandwidth: max total download/upload speed
*IPv6
*Firewalls
*NAT
*The internet is a "network of networks". For instance, the SCS network is connected to the Carleton backbone, which is connected to the Unicentre's network. The backbone is also connected to Carleton's ISPs.
*Firewalls on the internet block all kinds of network traffic, mainly due to "security concerns". However, firewalls always let web traffic through.
*Now, arbitrary network traffic gets routed through HTTP. This is mainly done for security. This means that other protocols can be squeezed into HTTP. Learn networking, but realize that modern networks just ship all kind of data within HTTP.

====Topics for Next Lecture====

*Databases, indexes
*Security (web security issues)

WebFund 2016W Lecture 20

2016-03-28T18:41:20Z

LeeCroft:

==Video==

The video for the lecture given on March 24, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec20-24Mar2016.mp4 is now available].

==Student Notes==

===Assignment 5 – Last Minute Question===

*How should the download functionality display the query result?
**Display as a page as a text file. Do a <code>res.send()</code> of a big string with each log entry terminated by a new line. You should be sending the file as plain text.

===Assignment 6 Hints===

*This will involve making an application that works in a similar fashion to exam-storage (a single-page app)
*You will be doing the same things that are already implemented in assignment 5 but you need to change it to work from a single page

===Exam Storage===

*A little bit more complicated than interactive-demo.
*The file structure of the application is modified to clarify the division between what runs on the client and what runs on the server
**public: contains everything that goes to the client directly (CSS and JavaScript files)
**server: contains all the JavaScript files that will run on the server
**start.js is a little JavaScript file that you will use to start the server (this is replacing bin/www)
**views contains all the templates for generating html
**keys: cryptographic keys
*Note that you can use <code>npm rebuild<code> to recompile any binary dependencies in the application modules (in case something is not working on your machine)

====start.js====

*The app.js file is now located in the server directory
**The call to <code>require()</code> in start.js is therefore changed to: <code>var app = require(‘./server/app’);</code>
**We then set the port, create the server and it start listening on port 3000.

====app.js====

*Things are fairly similar to our previous applications
**We are specifying the ssl keys and setting up the same modules we've previously used including one for sessions
**<code>app.use(express.static(‘public’));</code> is serving everything from public directory

====routes.js ====

*Starts with setting up a connection to the database.
**The users collection contains documents which have a username and a hashed password
*Many routes defined in here are similar to what we have done before
*The <code>storeSaltedpassword()</code> function in the /register is something new
**This stores the username and hashed and salted password in the users collection
**The hashing and salting is done using bcrypt
*In the /login route, we are again using bcrypt to compare a password that a user has entered with a hashed version stored in the database
**Why do we have the same error message for both incorrect password and invalid username?
***If someone is trying to break into your system, you don’t want them to know that they have come across the valid account and they are trying to figure out the password. You just want them to know that the username and the password is not the right one. Therefore, we use the same error string for both username and password.

====Password Hashing====

*When you log in to the virtual machine and type in the password, it is stored in the virtual machine
**Go to cd /etc there is a file called passwd which gives you a file that contains all of the accounts on the system
**In old school UNIX installations, the hash of the password is shown instead of 'x' in the file, but not here if you check the stored passwords stored in virtual machine
**To show the hash of the password in the file, you can type in the command <code>sudo shadowconfig off</code>, and then you should be able to see the file with the hash of your password instead of the 'x'
*For any file you create, you can compute a sha1sum (hash) of that file which is a big alphanumeric string
**If you change one bit in the file (input), it will make a huge change in the output
**Sha1sum should no longer be used since it is no longer considered secure but you can use sha256 instead
*Bcrypt is what we are using to hash passwords before storing them in the database
**Bcrypt is a hash function but it is a bit weird. It is slow by design and is memory intensive.
**Storing your passwords using bcrypt is really good today but in general, it is a good idea to use some kind of framework of something like Passport.
***These types of frameworks take care of making decisions that you might not know how to make and they help avoid using outdated and insecure technology
***Passport is a middleware authentication module for Node.js
*Why do we want to use hashing for passwords in the first place? Why do we not want to store them in the plain text?
**We want to avoid a breach in security caused by an attacker obtaining the stored passwords
**The hashing approach should have an irreversibility property that is essential for password storage, has adequate pre-computation defenses, and performance defenses needed to prevent dictionary attacks
**Just by looking at a hashed string, no one should be able to guess the original value and get your password
***They will actually have to play around with the hashing function or will have to plug in random values and try every single possibility which should hopefully take them a very long time
*In our web application, it is important to do the hashing on the server as opposed to on the client's machine even though doing this on the client would reduce the computational burden for the server
**This is because we cannot trust anything that is sent to us from the client
**Rather than computing the hash function and then sending us the resultant hash, the client could iterate over hash values and send them to us
***This would allow the client to avoid the need to spend time actually running the hash function

====Salting====

*Let’s say user 1 and user 2 have the same password, they will also have the same resultant hash
**We don’t want that as this makes things easier for an attacker
**What we will do is that we use the same password for both but add a string called salt1 and salt2 for users 1 and 2 respectively
**Now they have completely different hashes but the password is still the same
**Every user should have a different hash
*Salting is also the defeat to precomputation
**An attacker could use a precomputed dictionary of hash values allowing them to quickly look up the input that was used to create the hash
**By hashing a salted password, we are able to render the lookup tables useless

====account.js====

*<code>updateFileList()</code> is used to update the files displayed on the page
**It first sends an ajax GET request for /getFileStats to the server to get the updated file list
***When the server receives this request, it first checks to see if the user is logged in
***If the user is logged in, it queries the database to get their uploaded files
****When doing a query, we can optionally specify a projection object
****The projection is used to specify which information (document attributes) you want to retrieve in the query
****We specify the query object as <code>{content:0}</code> meaning that we do not want to include the content attribute in the documents that we retrieve
**In the <code>doUpdateFileList()</code> callback function, we update the DOM with the data received in the response
*<code>updateFileList()</code> is called when the page is initially loaded and every time a file is uploaded
**This means that the content on the page will always be up to date
*When a user requests a file download, how do we know which file to give them?
**We use the <code>downloadFile()</code> function to create and return a function which will be used as a callback function for the click event of the link to download a specific file
**By passing the ID of the file into <code>downloadFile()</code> we are able to create a callback function specific to each file so that we know which one to download when it gets called for the click event
*Notice that the href attribute for the download links is set to #
**This causes the link to take the user to the top of the page
**This is useful when we want a link that will not cause the user to go to a different page
*<code>saveDownloadedFile()</code> is used as a callback function for the POST request that is sent to the server when a file is requested for download from within a download click event callback function
**This function handles saving the file that gets sent to the client from the server
**It relies on the <code>saveAs()</code> function which lets us download file in the folder/location of our choice without changing the page.
*The reason why you want to manipulate things on the browser instead of on the server and the differences involved in making this change are important parts of exam-storage that you should understand

===Types of Questions for the Exam===

*Why are we using https?
**To protect data going between the browser and the server, just to make sure everything is encrypted there
*Why are we using bcrypt?
**To make sure that the password is not stored in plain text on the server
*What if in routes.js I just compare the password and the user.password instead of doing bcrypt?

===Next Week===

*High level topics which aren’t included in the final
*How do you make web applications pretty without being a web designer?

==Code==

In this lecture we discussed [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/exam-storage.zip exam-storage], the code for [[WebFund 2016W: Tutorial 10|Tutorial 10]]

WebFund 2016W Lecture 19

2016-03-28T17:16:17Z

LeeCroft:

==Video==

The video from the lecture given on March 22, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec19-22Mar2016.mp4 is now available].

==Student Notes==

===OpenStack===

*There is no need to use the course VM to set up of connect to an instance
**This is intended as a substitute for the VM
*There are various options to get your web application on the instance:
**WINSCP
**wget [URL]
**Others...
*You can create multiple sessions to the same instance with SSH

===Assignment 5===

*Extended until Sunday (March 27) at 11:55 pm
*Extra functionality (2% bonus) will be at the TA’s discretion
*If things are running slowly for you, try testing with files no larger than 1000 entries
*If database has too many entries, drop them
**To drop: <code>db.logs.drop()</code>
***This is done from the Mongo terminal client
*“Show” shows the entries in a scrollable textbox

===MongoDB Storage===

*By default, documents in MongoDB can be up to 16 megabytes
*For what we are doing, nothing should ever exceed this size since we are breaking files up into multiple documents
*GridFS can be used to store larger documents
**It will split these larger documents into chunks sized 255 kB each

===Assignment 6===

*This will involve making assignment 5 into a single-page web appliction
*Look at exam-storage to see an example of how to make a single-page application using client-side code

===Tutorial 9===

*The base application analyzes text entered by the user and counts word frequencies to display in a graph
*Division between client and server:
**Server: dumb web server (one route to render index)
**Client: everything (all the functionality, ie. interact.js)
*In index.jade, we can see that some client-side scripts are linked and many of the HTML elements have IDs
**These IDs are used in the client-side script to easily access the elements using jQuery
*It is important to understand the control flow when clicking on the update button
**The click event for the button causes a function to be called
**In this function, the user-entered data is taken from the page using jQuery
**Based on the settings specified by this data, words frequencies are counted and the graph is created
*What does the map method do?
**It processes each element of an array, passing that element in to a function
**The return value of the function is put into a new array

===Other Notes===

*If VM runs out of space, things may break
**Be careful, clean up space
**See the forums for a way to minimize the space MongoDB takes up

==Code==

In this lecture we discussed [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/interactive-graphs.zip interactive-graphs], the code for [[WebFund 2016W: Tutorial 9|Tutorial 9]].

WebFund 2016W Lecture 20

2016-03-26T02:33:42Z

LeeCroft:

==Video==

The video for the lecture given on March 24, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec20-24Mar2016.mp4 is now available].

==Student Notes==

===OpenStack===

*There is no need to use the course VM to set up of connect to an instance
**This is intended as a substitute for the VM
*There are various options to get your web application on the instance:
**WINSCP
**wget [URL]
**Others...
*You can create multiple sessions to the same instance with SSH

===Assignment 5===

*Extended until Sunday (March 27) at 11:55 pm
*Extra functionality (2% bonus) will be at the TA’s discretion
*If things are running slowly for you, try testing with files no larger than 1000 entries
*If database has too many entries, drop them
**To drop: <code>db.logs.drop()</code>
***This is done from the Mongo terminal client
*“Show” shows the entries in a scrollable textbox

===MongoDB Storage===

*By default, documents in MongoDB can be up to 16 megabytes
*For what we are doing, nothing should ever exceed this size since we are breaking files up into multiple documents
*GridFS can be used to store larger documents
**It will split these larger documents into chunks sized 255 kB each

===Assignment 6===

*This will involve making assignment 5 into a single-page web appliction
*Look at exam-storage to see an example of how to make a single-page application using client-side code

===Tutorial 9===

*The base application analyzes text entered by the user and counts word frequencies to display in a graph
*Division between client and server:
**Server: dumb web server (one route to render index)
**Client: everything (all the functionality, ie. interact.js)
*In index.jade, we can see that some client-side scripts are linked and many of the HTML elements have IDs
**These IDs are used in the client-side script to easily access the elements using jQuery
*It is important to understand the control flow when clicking on the update button
**The click event for the button causes a function to be called
**In this function, the user-entered data is taken from the page using jQuery
**Based on the settings specified by this data, words frequencies are counted and the graph is created
*What does the map method do?
**It processes each element of an array, passing that element in to a function
**The return value of the function is put into a new array

===Other Notes===

*If VM runs out of space, things may break
**Be careful, clean up space
**See the forums for a way to minimize the space MongoDB takes up

==Code==

In this lecture we discussed [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/exam-storage.zip exam-storage], the code for [[WebFund 2016W: Tutorial 10|Tutorial 10]]

WebFund 2016W Lecture 18

2016-03-21T02:17:21Z

LeeCroft:

==Video==

The video for the lecture given on March 17, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec18-17Mar2016.mp4 is now available].

==Notes==

===In Class===

(This is lecture 18, not 17!)

<pre>
Lecture 18
----------

Symmetric cryptography
- everyone uses the same key
- think secret decoder rings
- or, file encrypted with AES
- secret has to be shared between sender and receiver

Public-key cryptography
- but what if we don't have a shared secret?
- your password isn't good enough, and
- websites don't remember your password
- and how would you send them your password in
the first place
- instead, we split the key into two parts
- a public key
- a private key
- whatever one does, the other can undo

- if you encrypt with a public key and decrypt with a
private key
- private one-way communication
- e.g., you download Anil's public key and send
Anil a secret message
- if you encrypt with a private key and decrypt with
the public key
- that's a digital signature

First rule of cryptography
- friends don't let friends implement their own crypto
- algorithms OR code!

Why? You'll miss protections against attacks.
Example: timing attacks
- can extract secrets by watching execution time
- exploits the fact that some numbers take longer to
multiply & other operations

Everyone else messes up too
- so be prepared to update/replace all crypto-related
code
</pre>

===Student Notes===

====Additional Notes on OpenStack Instances====

*When selecting a floating IP address, the subnet must correspond to the network that you chose for your instance i.e. If you pick comp2406-net-31 you should use a 134.117.31 address and if you pick comp2406-net-26, use a 134.117.26 address. Mismatching will result in an error
*<code>sudo setcap ‘cap_net_bind_service=rep’ /usr/bin/node.js</code> is used to allow the user to bind to a privileged port

====HTTPS====

*HTTP is HyperText Transfer Protocol
*HTTPS is a secure version of HTTP which uses SSL/TLS
**SSL - Secure Sockets Layer
**TLS - Transport Layer Security
**The current version of SSL is TLS
**They are security protocols (cryptographic protocols) that encrypt communications to provide a secure connection over a network
*Normally in a URL, we don't see HTTP because it is the default so it is omitted
*With HTTPS, we will see the protocol written in the URL
*We also see a lock icon with HTTPS
**It means that the data you are sending is secure and the website you’re accessing has been digitally certified
**When you click on a lock icon, you can view information about the certificate that the server has
*If you want to enable SSL/TLS support in a Node application, you can change HTTP to HTTPS (change <code>var http = require(‘http’);</code> to <code>var http = require(‘https’);</code> in bin/www)
**For the code to work, we have to specify some options (key and certificate) for the HTTPS server
*You can get it working by modifying the code:

In bin/www:
<code><pre>
//use the app objects and put all the configurations into app, If require http is defined.
if (app.locals.sslOptions){
var http = require(‘https’);
}
else{
var http-require(‘http’);
}

//modify the ‘HTTP server’ part of your code with sslOptions.
//create an object in the app file.
if (app.locals.sslOptions){
var server = http.createServer(app.locals.sslOptions, app);
}
else{
var server = http.createServer(app);
}
</pre></code>

In app.js, we specify the HTTPS options by loading the key and certificate files:
<code><pre>
var fs = require('fs');

//put this after app is defined but before it is exported
app.locals.sslOptions = {
key: fs.readFileSync(‘keys/comp2406-test-key.pem’),
cert: fs.readFileSync(‘keys/comp2406-test-cert.pem’)
}
</pre></code>

=====Certificates=====

*Certificates are a form of digital authentication for a web server
*They contain information about the server and the authority that has signed the certificate
*Companies such as GlobalSign and Verisign are certificate authorities which are relied upon to sign trustworthy certificates
*If you do not want to pay a trusted certificate authority to sign your server's certificate, you can sign it yourself
**This gives you a usable certificate but it will produce a warning when a browser communicates with your server since it will receive a certificate signed by an untrusted source
**In order to get our web application to work properly with SSL/TLS, we have to create a key and use it to sign a certificate
**An alternative to get a trusted certificate is to get a free temporary certificate (from Let's Encrypt) and update it by using a script to run it in order to obtain new ones upon expiry

=====Symmetric Cryptography=====

*With symmetric cryptography, everyone uses the same key i.e. if you want to send a secret message to another person, you have to have the same secret decoder ring or files encrypted with AES
**AES is a symmetric block cipher in which the same key is used to encrypt and decrypt a document. It is a block cipher because it encrypts data in blocks as opposed to the whole thing
*The key to symmetric cryptography is that the secret has to be shared between the sender and receiver to keep it a secret i.e. instead of making the whole document secret, just make a key to keep it secure
**The difficulty is in initially sharing this secret in a secure fashion

=====Public-key Cryptography=====

*But what if we don’t have a shared secret?
*Passwords aren’t good enough
*Websites don’t remember your password
**How would you send them your password in the first place (in a secure fashion)?
*Instead, we split the key into two parts: public key and private key
*In public key cryptography, you have two keys, a matched pair of public and private keys are opposite (inverters) because what you do with one can be undone with the other
**If you encrypt with a public key and decrypt with a private key, it’s a private one way communication
***For example, you can send the professor a secret message after downloading his public key and using it for encryption
**If you encrypt with a private key and decrypt with the public key, that’s a digital signature
*This is what is used for HTTPS

=====Creating Our Key and Certificate=====

*We can use OpenSSL to create a key and to use that key to sign a certificate
*Randomness is an important factor in the creation of our key
**We want this done in a non-deterministic fashion so that nobody can guess what our key is
**OpenSSL uses Linux drivers which are designed to gather random data from events such as:
***User actions on the keyboard and mouse
***CPU temperatures
***Network activity

====First Rule of Cryptography====

*Friends don’t let friends implement their own cryptography, algorithms or code because it’s hard to do it and be sure you’re secure
*Why?
**You’ll miss protections against attacks. E.g. Timing attacks.
***In a timing attack, the attacker can extract secrets by watching execution time
***This exploits the fact that some numbers take longer to multiply than others
*Even professionally developped crypto systems will have bugs, so be prepared to update/replace all crypto-related code

==Code==

[http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/analyzeLogs-ssl-template.zip analyzeLogs-ssl-template.zip]

WebFund 2016W Lecture 18

2016-03-21T02:12:49Z

LeeCroft:

==Video==

The video for the lecture given on March 17, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec18-17Mar2016.mp4 is now available].

==Notes==

===In Class===

(This is lecture 18, not 17!)

<pre>
Lecture 18
----------

Symmetric cryptography
- everyone uses the same key
- think secret decoder rings
- or, file encrypted with AES
- secret has to be shared between sender and receiver

Public-key cryptography
- but what if we don't have a shared secret?
- your password isn't good enough, and
- websites don't remember your password
- and how would you send them your password in
the first place
- instead, we split the key into two parts
- a public key
- a private key
- whatever one does, the other can undo

- if you encrypt with a public key and decrypt with a
private key
- private one-way communication
- e.g., you download Anil's public key and send
Anil a secret message
- if you encrypt with a private key and decrypt with
the public key
- that's a digital signature

First rule of cryptography
- friends don't let friends implement their own crypto
- algorithms OR code!

Why? You'll miss protections against attacks.
Example: timing attacks
- can extract secrets by watching execution time
- exploits the fact that some numbers take longer to
multiply & other operations

Everyone else messes up too
- so be prepared to update/replace all crypto-related
code
</pre>

===Student Notes===

====Additional Notes on OpenStack Instances====

*When selecting a floating IP address, the subnet must correspond to the network that you chose for your instance i.e. If you pick comp2406-net-31 you should use a 134.117.31 address and if you pick comp2406-net-26, use a 134.117.26 address. Mismatching will result in an error
*<code>sudo setcap ‘cap_net_bind_service=rep’ /usr/bin/node.js</code> is used to allow the user to bind to a privileged port

====HTTPS====

*HTTP is HyperText Transfer Protocol
*HTTPS is a secure version of HTTP which uses SSL/TLS
**SSL - Secure Sockets Layer
**TLS - Transport Layer Security
**The current version of SSL is TLS
**They are security protocols (cryptographic protocols) that encrypt communications to provide a secure connection over a network
*Normally in a URL, we don't see HTTP because it is the default so it is omitted
*With HTTPS, we will see the protocol written in the URL
*We also see a lock icon with HTTPS
**It means that the data you are sending is secure and the website you’re accessing has been digitally certified
**When you click on a lock icon, you can view information about the certificate that the server has
*If you want to enable SSL/TLS support in a Node application, you can change HTTP to HTTPS (change <code>var http = require(‘http’);</code> to <code>var http = require(‘https’);</code> in bin/www)
**For the code to work, we have to specify some options (key and certificate) for the HTTPS server
*You can get it working by modifying the code:

In bin/www:
<code><pre>
//use the app objects and put all the configurations into app, If require http is defined.
if (app.locals.sslOptions){
var http = require(‘https’);
}
else{
var http-require(‘http’);
}

//modify the ‘HTTP server’ part of your code with sslOptions.
//create an object in the app file.
if (app.locals.sslOptions){
var server = http.createServer(app.locals.sslOptions, app);
}
else{
var server = http.createServer(app);
}
</pre></code>

In app.js, we specify the HTTPS options by loading the key and certificate files:
<code><pre>
var fs = require('fs');

//put this after app is defined but before it is exported
app.locals.sslOptions = {
key: fs.readFileSync(‘keys/comp2406-test-key.pem’),
cert: fs.readFileSync(‘keys/comp2406-test-cert.pem’)
}
</pre></code>

=====Certificates=====

*Certificates are a form of digital authentication for a web server
*They contain information about the server and the authority that has signed the certificate
*Companies such as GlobalSign and Verisign are certificate authorities which are relied upon to sign trustworthy certificates
*If you do not want to pay a trusted certificate authority to sign your server's certificate, you can sign it yourself
**This gives you a usable certificate but it will produce a warning when a browser communicates with your server since it will receive a certificate signed by an untrusted source
**In order to get our web application to work properly with SSL/TLS, we have to create a key and use it to sign a certificate
**An alternative to get a trusted certificate is to get a free temporary certificate (from Let's Encrypt) and update it by using a script to run it in order to obtain new ones upon expiry

=====Symmetric Cryptography=====

*With symmetric cryptography, everyone uses the same key i.e. if you want to send a secret message to another person, you have to have the same secret decoder ring or files encrypted with AES
**AES is a symmetric block cipher in which the same key is used to encrypt and decrypt a document. It is a block cipher because it encrypts data in blocks as opposed to the whole thing
*The key to symmetric cryptography is that the secret has to be shared between the sender and receiver to keep it a secret i.e. instead of making the whole document secret, just make a key to keep it secure
**The difficulty is in initially sharing this secret in a secure fashion

=====Public-key Cryptography=====

*But what if we don’t have a shared secret?
*Passwords aren’t good enough
*Websites don’t remember your password
**How would you send them your password in the first place (in a secure fashion)?
*Instead, we split the key into two parts: public key and private key
*In public key cryptography, you have two keys, a matched pair of public and private keys are opposite (inverters) because what you do with one can be undone with the other
**If you encrypt with a public key and decrypt with a private key, it’s a private one way communication
***For example, you can send the professor a secret message after downloading his public key and using it for encryption
**If you encrypt with a private key and decrypt with the public key, that’s a digital signature
*This is what is used for HTTPS

====First Rule of Cryptography====

*Friends don’t let friends implement their own cryptography, algorithms or code because it’s hard to do it and be sure you’re secure
*Why?
**You’ll miss protections against attacks. E.g. Timing attacks.
***In a timing attack, the attacker can extract secrets by watching execution time
***This exploits the fact that some numbers take longer to multiply than others
*Even professionally developped crypto systems will have bugs, so be prepared to update/replace all crypto-related code

==Code==

[http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/analyzeLogs-ssl-template.zip analyzeLogs-ssl-template.zip]

WebFund 2016W Lecture 18

2016-03-21T02:12:34Z

LeeCroft:

==Video==

The video for the lecture given on March 17, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec18-17Mar2016.mp4 is now available].

==Notes==

===In Class===

(This is lecture 18, not 17!)

<pre>
Lecture 18
----------

Symmetric cryptography
- everyone uses the same key
- think secret decoder rings
- or, file encrypted with AES
- secret has to be shared between sender and receiver

Public-key cryptography
- but what if we don't have a shared secret?
- your password isn't good enough, and
- websites don't remember your password
- and how would you send them your password in
the first place
- instead, we split the key into two parts
- a public key
- a private key
- whatever one does, the other can undo

- if you encrypt with a public key and decrypt with a
private key
- private one-way communication
- e.g., you download Anil's public key and send
Anil a secret message
- if you encrypt with a private key and decrypt with
the public key
- that's a digital signature

First rule of cryptography
- friends don't let friends implement their own crypto
- algorithms OR code!

Why? You'll miss protections against attacks.
Example: timing attacks
- can extract secrets by watching execution time
- exploits the fact that some numbers take longer to
multiply & other operations

Everyone else messes up too
- so be prepared to update/replace all crypto-related
code
</pre>

===Student Notes===

====Additional Notes on OpenStack Instances====

*When selecting a floating IP address, the subnet must correspond to the network that you chose for your instance i.e. If you pick comp2406-net-31 you should use a 134.117.31 address and if you pick comp2406-net-26, use a 134.117.26 address. Mismatching will result in an error
*<code>sudo setcap ‘cap_net_bind_service=rep’ /usr/bin/node.js</code> is used to allow the user to bind to a privileged port

====HTTPS====

*HTTP is HyperText Transfer Protocol
*HTTPS is a secure version of HTTP which uses SSL/TLS
**SSL - Secure Sockets Layer
**TLS - Transport Layer Security
**The current version of SSL is TLS
**They are security protocols (cryptographic protocols) that encrypt communications to provide a secure connection over a network
*Normally in a URL, we don't see HTTP because it is the default so it is omitted
*With HTTPS, we will see the protocol written in the URL
*We also see a lock icon with HTTPS
**It means that the data you are sending is secure and the website you’re accessing has been digitally certified
**When you click on a lock icon, you can view information about the certificate that the server has
*If you want to enable SSL/TLS support in a Node application, you can change HTTP to HTTPS (change <code>var http = require(‘http’);</code> to <code>var http = require(‘https’);</code> in bin/www)
**For the code to work, we have to specify some options (key and certificate) for the HTTPS server
*You can get it working by modifying the code:

In bin/www:
<code><pre>
//use the app objects and put all the configurations into app, If require http is defined.
if (app.locals.sslOptions){
var http = require(‘https’);
}
else{
var http-require(‘http’);
}

//modify the ‘HTTP server’ part of your code with sslOptions.
//create an object in the app file.
if (app.locals.sslOptions){
var server = http.createServer(app.locals.sslOptions, app);
}
else{
var server = http.createServer(app);
}
</pre></code>

In app.js, we specify the HTTPS options by loading the key and certificate files:
<code><pre>
var fs = require('fs');

//put this after app is defined but before it is exported
app.locals.sslOptions = {
key: fs.readFileSync(‘keys/comp2406-test-key.pem’),
cert: fs.readFileSync(‘keys/comp2406-test-cert.pem’)
}
</pre></code>

=====Certificates=====

*Certificates are a form of digital authentication for a web server
*They contain information about the server and the authority that has signed the certificate
*Companies such as GlobalSign and Verisign are certificate authorities which are relied upon to sign trustworthy certificates
*If you do not want to pay a trusted certificate authority to sign your server's certificate, you can sign it yourself
**This gives you a usable certificate but it will produce a warning when a browser communicates with your server since it will receive a certificate signed by an untrusted source
**In order to get our web application to work properly with SSL/TLS, we have to create a key and use it to sign a certificate
**An alternative to get a trusted certificate is to get a free temporary certificate (from Let's Encrypt) and update it by using a script to run it in order to obtain new ones upon expiry

=====Symmetric Cryptography=====

*With symmetric cryptography, everyone uses the same key i.e. if you want to send a secret message to another person, you have to have the same secret decoder ring or files encrypted with AES
**AES is a symmetric block cipher in which the same key is used to encrypt and decrypt a document. It is a block cipher because it encrypts data in blocks as opposed to the whole thing
*The key to symmetric cryptography is that the secret has to be shared between the sender and receiver to keep it a secret i.e. instead of making the whole document secret, just make a key to keep it secure
**The difficulty is in initially sharing this secret in a secure fashion

=====Public-key Cryptography=====

*But what if we don’t have a shared secret?
*Passwords aren’t good enough
*Websites don’t remember your password
**How would you send them your password in the first place (in a secure fashion)?
*Instead, we split the key into two parts: public key and private key
*In public key cryptography, you have two keys, a matched pair of public and private keys are opposite (inverters) because what you do with one can be undone with the other
**If you encrypt with a public key and decrypt with a private key, it’s a private one way communication
***For example, you can send the professor a secret message after downloading his public key and using it for encryption
**If you encrypt with a private key and decrypt with the public key, that’s a digital signature
*This is what is used for HTTPS

===First Rule of Cryptography===

*Friends don’t let friends implement their own cryptography, algorithms or code because it’s hard to do it and be sure you’re secure
*Why?
**You’ll miss protections against attacks. E.g. Timing attacks.
***In a timing attack, the attacker can extract secrets by watching execution time
***This exploits the fact that some numbers take longer to multiply than others
*Even professionally developped crypto systems will have bugs, so be prepared to update/replace all crypto-related code

==Code==

[http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/analyzeLogs-ssl-template.zip analyzeLogs-ssl-template.zip]

WebFund 2016W Lecture 18

2016-03-21T02:07:28Z

LeeCroft:

==Video==

The video for the lecture given on March 17, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec18-17Mar2016.mp4 is now available].

==Notes==

===In Class===

(This is lecture 18, not 17!)

<pre>
Lecture 18
----------

Symmetric cryptography
- everyone uses the same key
- think secret decoder rings
- or, file encrypted with AES
- secret has to be shared between sender and receiver

Public-key cryptography
- but what if we don't have a shared secret?
- your password isn't good enough, and
- websites don't remember your password
- and how would you send them your password in
the first place
- instead, we split the key into two parts
- a public key
- a private key
- whatever one does, the other can undo

- if you encrypt with a public key and decrypt with a
private key
- private one-way communication
- e.g., you download Anil's public key and send
Anil a secret message
- if you encrypt with a private key and decrypt with
the public key
- that's a digital signature

First rule of cryptography
- friends don't let friends implement their own crypto
- algorithms OR code!

Why? You'll miss protections against attacks.
Example: timing attacks
- can extract secrets by watching execution time
- exploits the fact that some numbers take longer to
multiply & other operations

Everyone else messes up too
- so be prepared to update/replace all crypto-related
code
</pre>

===Student Notes===

====Additional Notes on Openstack Instances====

*When selecting a floating IP address, the subnet must correspond to the network that you chose for your instance i.e. If you pick comp2406-net-31 you should use a 134.117.31 address and if you pick comp2406-net-26, use a 134.117.26 address. Mismatching will result in an error
*<code>sudo setcap ‘cap_net_bind_service=rep’ /usr/bin/node.js</code> is used to allow the user to bind to a privileged port

====HTTPS====

*HTTP is HyperText Transfer Protocol
*HTTPS is a secure version of HTTP which uses SSL/TLS
**SSL - Secure Sockets Layer
**TLS - Transport Layer Security
**The current version of SSL is TLS
**They are security protocols (cryptographic protocols) that encrypt communications to provide a secure connection over a network
*Normally in a URL, we don't see HTTP because it is the default so it is omitted
*With HTTPS, we will see the protocol written in the URL
*We also see a lock icon with HTTPS
**It means that the data you are sending is secure and the website you’re accessing has been digitally certified
**When you click on a lock icon, you can view information about the certificate that the server has
*If you want to enable SSL/TLS support in a Node application, you can change HTTP to HTTPS (change <code>var http = require(‘http’);</code> to <code>var http = require(‘https’);</code> in bin/www)
**For the code to work, we have to specify some options (key and certificate) for the HTTPS server
*You can get it working by modifying the code:

In bin/www:
<code><pre>
//use the app objects and put all the configurations into app, If require http is defined.
if (app.locals.sslOptions){
var http = require(‘https’);
}
else{
var http-require(‘http’);
}

//modify the ‘HTTP server’ part of your code with sslOptions.
//create an object in the app file.
if (app.locals.sslOptions){
var server = http.createServer(app.locals.sslOptions, app);
}
else{
var server = http.createServer(app);
}
</pre></code>

In app.js, we specify the HTTPS options by loading the key and certificate files:
<code><pre>
var fs = require('fs');

//put this after app is defined but before it is exported
app.locals.sslOptions = {
key: fs.readFileSync(‘keys/comp2406-test-key.pem’),
cert: fs.readFileSync(‘keys/comp2406-test-cert.pem’)
}
</pre></code>

=====Certificates=====

*Certificates are a form of digital authentication for a web server
*They contain information about the server and the authority that has signed the certificate
*Companies such as GlobalSign and Verisign are certificate authorities which are relied upon to sign trustworthy certificates
*If you do not want to pay a trusted certificate authority to sign your server's certificate, you can sign it yourself
**This gives you a usable certificate but it will produce a warning when a browser communicates with your server since it will receive a certificate signed by an untrusted source
**In order to get our web application to work properly with SSL/TLS, we have to create a key and use it to sign a certificate
**An alternative to get a trusted certificate is to get a free temporary certificate (from Let's Encrypt) and update it by using a script to run it in order to obtain new ones upon expiry

=====Symmetric Cryptography=====

*With symmetric cryptography, everyone uses the same key i.e. if you want to send a secret message to another person, you have to have the same secret decoder ring or files encrypted with AES
**AES is a symmetric block cipher in which the same key is used to encrypt and decrypt a document. It is a block cipher because it encrypts data in blocks as opposed to the whole thing
*The key to symmetric cryptography is that the secret has to be shared between the sender and receiver to keep it a secret i.e. instead of making the whole document secret, just make a key to keep it secure
**The difficulty is in initially sharing this secret in a secure fashion

=====Public-key Cryptography=====

*But what if we don’t have a shared secret?
*Passwords aren’t good enough
*Websites don’t remember your password
**How would you send them your password in the first place (in a secure fashion)?
*Instead, we split the key into two parts: public key and private key
*In public key cryptography, you have two keys, a matched pair of public and private keys are opposite (inverters) because what you do with one can be undone with the other
**If you encrypt with a public key and decrypt with a private key, it’s a private one way communication
***For example, you can send the professor a secret message after downloading his public key and using it for encryption
**If you encrypt with a private key and decrypt with the public key, that’s a digital signature

=====First Rule of Cryptography=====

*Friends don’t let friends implement their own cryptography, algorithms or code because it’s hard to do it and be sure you’re secure
*Why?
**You’ll miss protections against attacks. E.g. Timing attacks.
***In a timing attack, the attacker can extract secrets by watching execution time
***This exploits the fact that some numbers take longer to multiply than others
*Even professionally developped crypto systems will have bugs, so be prepared to update/replace all crypto-related code

==Code==

[http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/analyzeLogs-ssl-template.zip analyzeLogs-ssl-template.zip]

WebFund 2016W Lecture 17

2016-03-19T19:49:37Z

LeeCroft:

==Video==

The video from the lecture given on March 15, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec17-15Mar2016.mp4 is now available].

==Student Notes==

===Administrative===

*Assignment 5 is due on March 24th
**The marking scheme has been updated for the assignment (now has 20 marks)
***2 for uploading logs
***4 for storing logs in MongoDB properly
***6 for listing entries in response to a query
***2 for allowing such listings to be downloaded IN THE ORDER they were in the original file
***6 for graphing daily frequency of logs
**This is worth 2 assignments since we now have less assignments
*Assignment 6 is due on April 7th (last day of class)
**This is worth 2 assignments since we have less assignments
*The solutions for our assignment 4 will be posted soon
*Solutions for assignments 5 & 6 will be posted by the end of the semester

===Assignment 5===

*The template we’re given should give you everything you need to start the assignment
*Each output type includes instructions with what needs to be displayed
**‘Download’ displays the log entries as text (similar to queryLogs in web application form)
**‘Show’ will have similar functionality to ‘Download’ but you’ll be putting the logs onto a text box you can scroll through
**‘Visualize’ can be done in a similar fashion to the graph tutorial
***Chart.js is a graphing script ([http://www.chartjs.org/docs/ http://www.chartjs.org/docs/]) that allows you to visualize your data
***For our assignment, we are setting up all of the data needed for the graph on the server and then passing it into the Jade template to be rendered into a script tag
***The code in the script tag will be run as client-side code in the web browser to build the graph
*In the <code>doQuery()</code> function, you can see that our query types are listed here (visualize, show and download)
**For ‘Download’, you just send the logs as plain text
**For ‘Show’, we use the template given for Show.jade
**For ‘Visualize’, we specify the data as a result of calling <code>analyzeSelected()</code>
***The <code>analyzeSelected()</code> function has to take the logs and analyze them and to generate the values and counts
***To get the logs you need to implement the <code>getLogs()</code> function which is included in the template
***The <code>analyzeSelected()</code> function does produces a string of JavaScript code with the data object in JSON format
****This is passed in to the Visualize.jade template

===Introduction to Instances===

*Access the SCS OpenStack web portal at: [https://openstack.scs.carleton.ca/horizon/auth/login/?next=/horizon/ https://openstack.scs.carleton.ca/horizon/auth/login/?next=/horizon/]
*You can only access the SCS OpenStack machines when you’re on the Carleton Network (CU-Wireless)
**If you’re attempting to access OpenStack outside the network, you’ll need a VPN (Virtual Private Network) for Carleton’s network
*Log in using:
**Username: your MyCarletonOne username
**Password: your MyCarletonOne username (all lowercase) and your student number (no spaces between the two)
*To create/launch your own instance in the COMP2406 subcategory
**You’re going to have to use Google Chrome!
**Make sure you are in the COMP2406 project (You can change at the top of the page, to the right of the OpenStack logo)
**Go to Project → Compute → Instances
**Click the Launch Instance button to bring up the new instance interface
***You can name your own instance (ex// anilsomayaji-comp2406-1)
***Instance Boot Source : Boot from snapshot
***Instance Snapshot: COMP2406-W16
***Go to the Access & Security tab
****Check ping, ssh, https and default.
****Ignore the Key Pair option
***Go to the Networking tab
****Under Selected networks, add the “comp2406-net-26” available network (or “comp2406-net-31”)
***After following this procedure, you’ll be able to launch to create an instance!
**Find you instance on the list of instances and under the Actions menu, select Allocate Floating IP
***Select an IP address with a subnet matching the network that you joined (either 134.117.26.xx or 134.117.31.xx)
*Now you can connect to your instance via ssh
**In a terminal, enter <code>ssh [floating IP address] -l student</code>
**The password is tneduts!
*You are now connected to your own virtual machine! You can use it to run a web server!
*Transfer a web application to the instance using wget or some FTP utility
*When running your server, instead of using localhost, we can use our IP address to access our server
**We only have access to port 443 for the server on this instance but we need elevated priviledges to use it
**TO FIX: <code>sudo setcap ‘cap_net_bind_service=+ep’ /user/bin/www/nodejs</code>
*Now you can run your server with <code>PORT=443 bin/www</code> (It should now be listening on port 443)
*Though all the capabilities of the OpenStack instance can be done on the course virtual machine, using OpenStack allows you to deploy Node.js in a production environment

===Tutorial 7 - Downloading Files===

*First we need to make a download button in account.jade
**Inside the each loop, we add a form to go along with each file
**We use a hidden field in the form to store the file name as a value to be sent as form data in the body of the POST request
<code><pre>
li #{s.name}: #{s.size}
form (method=”post”, action=”/downloadFile”)
p #{s.name}: #{s.size}
input(type=”hidden”, value=s.name, name=”downloadFile”)
input(type=”submit”, value=”Download”, name=”submit”)#{s.name}
</pre></code>

*Next, in index.js, we add the code to handle this new request
<code><pre>
function downloadFile(req, res){
var theFile = req.body.downloadFile;
var allFiles = req.session.files;
var i;
var sent = false;

if(theFile){
for(i = 0; i < allFiles.length; i++){
if(allFiles[i].originalname === theFile){
response = Buffer(allFiles[i].buffer.data).toString(‘utf8’);
res.send(response);
sent = true;
break;
}
}

if(!sent)
res.send(“File not found”);
}
}

router.post(“/downloadFile”, downloadFile);
</pre></code>

*The <code>downloadFile()</code> searches through the array of uploaded files from the <code>req.session</code> object for a file whose name matches the name received in the request
**If such a file is found, it is sent in the response
**Note the use of the <code>Buffer()</code> constructor
***This is to get a round a bug where the file buffer was not preserved in its Buffer format during storage in memory

==Instructor comments==

In class I had trouble with downloading the file. Everything worked except that the downloaded file became [Object object].

The problematic line was this, in downloadFile()

response = allFiles[i].buffer.toString('utf8');

This line '''should''' work because when we stored the file in uploadText() it did. But in between (perhaps when stored as a session property) it was changed.

Before, we had:

{ '0': 123,
'1': 10,
'2': 32,
'3': 32,
...
}

But now this got changed into:

{ type: 'Buffer',
data:
[ 123,
10,
32,
32,
...
]
}

The former is of type Buffer (as reported by Buffer.isBuffer()), the latter is not.

While this is strange behavior, the fix is easy:

response = Buffer(allFiles[i].buffer.data).toString('utf8');

This is definitely a bug in node or express. If anyone wants to try isolating this bug and reporting it, let me know!

==Code==

[http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/upload-demo-class.zip upload-demo with working download button]

WebFund 2016W Lecture 17

2016-03-19T19:48:15Z

LeeCroft:

==Video==

The video from the lecture given on March 15, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec17-15Mar2016.mp4 is now available].

==Student Notes==

===Administrative===

*Assignment 5 is due on March 24th
**The marking scheme has been updated for the assignment (now has 20 marks)
***2 for uploading logs
***4 for storing logs in MongoDB properly
***6 for listing entries in response to a query
***2 for allowing such listings to be downloaded IN THE ORDER they were in the original file
***6 for graphing daily frequency of logs
**This is worth 2 assignments since we now have less assignments
*Assignment 6 is due on April 7th (last day of class)
**This is worth 2 assignments since we have less assignments
*The solutions for our assignment 4 will be posted soon
*Solutions for assignments 5 & 6 will be posted by the end of the semester

===Assignment 5===

*The template we’re given should give you everything you need to start the assignment
*Each output type includes instructions with what needs to be displayed
**‘Download’ displays the log entries as text (similar to queryLogs in web application form)
**‘Show’ will have similar functionality to ‘Download’ but you’ll be putting the logs onto a text box you can scroll through
**‘Visualize’ can be done in a similar fashion to the graph tutorial
***Chart.js is a graphing script ([http://www.chartjs.org/docs/ http://www.chartjs.org/docs/]) that allows you to visualize your data
***For our assignment, we are setting up all of the data needed for the graph on the server and then passing it into the Jade template to be rendered into a script tag
***The code in the script tag will be run as client-side code in the web browser to build the graph
*In the <code>doQuery()</code> function, you can see that our query types are listed here (visualize, show and download)
**For ‘Download’, you just send the logs as plain text
**For ‘Show’, we use the template given for Show.jade
**For ‘Visualize’, we specify the data as a result of calling <code>analyzeSelected()</code>
***The <code>analyzeSelected()</code> function has to take the logs and analyze them and to generate the values and counts
***To get the logs you need to implement the <code>getLogs()</code> function which is included in the template
***The <code>analyzeSelected()</code> function does produces a string of JavaScript code with the data object in JSON format
****This is passed in to the Visualize.jade template

===Introduction to Instances===

*Access the SCS OpenStack web portal at: [https://openstack.scs.carleton.ca/horizon/auth/login/?next=/horizon/ https://openstack.scs.carleton.ca/horizon/auth/login/?next=/horizon/]
*You can only access the SCS OpenStack machines when you’re on the Carleton Network (CU-Wireless)
**If you’re attempting to access OpenStack outside the network, you’ll need a VPN (Virtual Private Network) for Carleton’s network
*Log in using:
**Username: your MyCarletonOne username
**Password: your MyCarletonOne username (all lowercase) and your student number (no spaces between the two)
*To create/launch your own instance in the COMP2406 subcategory
**You’re going to have to use Google Chrome!
**Make sure you are in the COMP2406 project (You can change at the top of the page, to the right of the OpenStack logo)
**Go to Project → Compute → Instances
**Click the Launch Instance button to bring up the new instance interface
***You can name your own instance (ex// anilsomayaji-comp2406-1)
***Instance Boot Source : Boot from snapshot
***Instance Snapshot: COMP2406-W16
***Go to the Access & Security tab
****Check ping, ssh, https and default.
****Ignore the Key Pair option
***Go to the Networking tab
****Under Selected networks, add the “comp2406-net-26” available network (or “comp2406-net-31”)
***After following this procedure, you’ll be able to launch to create an instance!
**Find you instance on the list of instances and under the Actions menu, select Allocate Floating IP
***Select an IP address with a subnet matching the network that you joined (either 134.117.26.xx or 134.117.31.xx)
*Now you can connect to your instance via ssh
**In a terminal, enter <code>ssh [floating IP address] -l student</code>
**The password is tneduts!
*You are now connected to your own virtual machine! You can use it to run a web server!
*Transfer a web application to the instance using wget or some FTP utility
*When running your server, instead of using localhost, we can use our IP address to access our server
**We only have access to port 443 for the server on this instance but we need elevated priviledges to use it
**TO FIX: <code>sudo setcap ‘cap_net_bind_service=+ep’ /user/bin/www/nodejs</code>
*Now you can run your server with <code>PORT=443 bin/www</code> (It should now be listening on port 443)
*Though all the capabilities of the OpenStack instance can be done on the course virtual machine, using OpenStack allows you to deploy Node.js in a production environment

===Tutorial 7 - Downloading Files===

*First we need to make a download button in account.jade
**Inside the each loop, we add a form to go along with each file
**We use a hidden field in the form to store the file name as a value to be sent as form data in the body of the POST request
<code><pre>
li #{s.name}: #{s.size}
form (method=”post”, action=”/downloadFile”)
p #{s.name}: #{s.size}
input(type=”hidden”, value=s.name, name=”downloadFile”)
input(type=”submit”, value=”Download”, name=”submit”)#{s.name}
</pre></code>

*Next, in index.js, we add the code to handle this new request
<code><pre>
function downloadFile(req, res){
var theFile = req.body.downloadFile;
var allFiles = req.session.files;
var i;
var sent = false;

if(theFile){
for(i = 0; i < allFiles.length; i++){
if(allFiles[i].originalname === theFile){
response = Buffer(allFiles[i].buffer.data).toString(‘utf8’);
res.send(response);
sent = true;
break;
}
}

if(!sent)
res.send(“File not found”);
}
}

router.post(“/downloadFile”, downloadFile);
</pre></code>

*The <code>downloadFile()</code> searches through the array of uploaded files from the <code>req.session</code> object for a file whose name matches the name received in the request
**If such a file is found, it is sent in the response
**Note the use of the <code>Buffer()</code> constructor
***This is to get a round a bug where the file buffer was not preserved in its Buffer format during storage in memory

===Instructor comments===

In class I had trouble with downloading the file. Everything worked except that the downloaded file became [Object object].

The problematic line was this, in downloadFile()

response = allFiles[i].buffer.toString('utf8');

This line '''should''' work because when we stored the file in uploadText() it did. But in between (perhaps when stored as a session property) it was changed.

Before, we had:

{ '0': 123,
'1': 10,
'2': 32,
'3': 32,
...
}

But now this got changed into:

{ type: 'Buffer',
data:
[ 123,
10,
32,
32,
...
]
}

The former is of type Buffer (as reported by Buffer.isBuffer()), the latter is not.

While this is strange behavior, the fix is easy:

response = Buffer(allFiles[i].buffer.data).toString('utf8');

This is definitely a bug in node or express. If anyone wants to try isolating this bug and reporting it, let me know!

==Code==

[http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/upload-demo-class.zip upload-demo with working download button]

WebFund 2016W Lecture 15

2016-03-14T18:37:08Z

LeeCroft:

==Video==

The video from the lecture given on March 8, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec15-08Mar2016.mp4 is now available].

==Student Notes==
*Midterms are all graded (go check out CULearn)
*Midterms will be available during tutorials

===Client-Side Data Population===

*On the midterm, many people got the last question about the Jade each loop wrong
**The loop runs on the server, not in the browser
**We can look at the source code in the web browser and see that the table data is all populated
***To make the Jade-rendered HTML display nicely (with proper line break and indentations), pass the "pretty" attribute to the template:

<code><pre>res.render(‘list’, { title: ‘People listing’, items: state, pretty: true});</pre></code>

In previous examples we built the table rows using a for each loop in the Jade template. Today we will be moving the functionality to the web browser/client. This will allow our page to be more dynamic in the sense that the table will be rendered after the page has loaded. How does this happen? The list of people (in JSON) is requested from the server using an AJAX request and is then parsed and then put into the table.

*To do this we need to:
**Get the data to the browser
**Make the list from the data (using client-side JavaScript)

====Sending List Data to the Browser====

*Add a new route on the server

<code><pre>app.get(‘/items’, function(req, res){
res.send(state);
}
</pre></code>

*This will send JSON-formatted data (the array) to the requesting client (the web browser)
**We will need to use jQuery (client-side JavaScript) to make this request and handle the response
***We'll write the script for this later
*We then change the /list route to render a new Jade template

<code><pre>
app.get(‘/list’, function(req, res){
res.render(‘clientlist’, {title: “People listing”, pretty: true});
});
</pre></code>

=====clientlist.jade=====

*We can copy the list.jade template and make some modifications to it
*We will need to link jQuery and the client-side script that we will be writting

<code><pre>
block header
script(src=“jquery-1.12.0.js”)
script(src=“showlist.js”)
</pre></code>

*We can remove the body of the old table and instead use

<code><pre>
tbody#theListing (the pound sign indicates an ID)
</pre></code>

*We need to give an ID to the tbody because in showlist.js we will need to let it know where we want to add content to the table

====Processing the Data in the Browser====

*Now we need to make the showlist.js script
*In this script, we will need to get get the array of objects (in JSON) from the server by making an AJAX request
**We can use <code>$.getJSON(‘/items’, displayItems);</code> to make this request and then call the <code>displayItems</code> callback function
***What is AJAX?
****It stands for Asynchronous JavaScript and XML
****It’s a library that communicates between the server and the client. It’s great because of it’s asynchronous nature which means it can do its work without needing to refresh the page.
**Now we have an array of objects in the browser (the data has moved!)
**We need to display this data
**Once displayed they are still not in the source code! (We have updated the DOM but not the source code)

<code><pre>
$(function(){
//grab JSON from server
//insert data into DOM

function displayItems(items){
var i;
for ( i = 0; i < items.length; i++){
//Remember the id? This is DOM manipulation
$(“#theListing”).append(“<tr>” +
“<td>” + items[i].name + “</td>” +
“<td>” + items[i].city + “</td>” +
“<td>” + items[i].country + “</td>” +
“<td>” + items[i].birthday + “</td>” +
“<td>” + items[i].email + “</td>” +
“</tr>”);
}
}

$.getJSON(‘/items’, displayItems); //callback
// This is the AJAX request (as mentioned above)
// We are receiving the array of objects and calling the displayItems call back function.
// “items” is the array of objects we got from the server
});
</pre></code>

===Tutorial 6 Solution===

*Make sure you complete this tutorial (super important for next assignment)
*We will walk through it today…
*The code already has some saved data it’s displaying
*The /results route passes the stats object in the the Jade template (we need to build up this stats object dynamically)

<code><pre>
router.post(‘/add’, function(req, res){
var answer = { firstname : req.body.firstname};

// qlist holds all of the answers to the question (ex. for colours if would have “red”, “green”, etc)
var qlist = Object.keys(questions);

// for each answer to the question
qlist.forEach(function(q){
answer[q] = req.body[q];
}

answers.push(answer);

…the rest is the same…
}
</pre></code>

*Now we need to gather the answers into a stats object
*When making an object, start with the properties you know it needs to have and add the rest later

<code><pre>
function calcResultsStats(results){
var resultsStats = [];

Object.keys(questions).forEach(function(q){
var r = {};
var i;
var choice;

r.questionsName = q;
r.questionText = questions[q];
r.labels = [];
r.values = [];

var count = {}; // mapping of question answers to counts

for (i = 0; i < answers.length; i++){
choice = answers[i][q];

if( counts[choice])
counts[choice] = counts[choice] + 1;
else
counts[choice] = 1;
}

// Now we want to convert that object into two arrays
Object.keys(counts).forEach(function(c){
r.labels.push(c);
r.values.push(counts[c]);
});

resultsStats.push(r);
});

return resultsStats;
}
</pre></code>

===How Real Is All of This?===

*While the web app was running during class, you could visit http://134.117.222.110:3000/
**It’s a real web server!
*What we don't have is a domain name (we must use the IP address instead)
**Root Name Servers:
***Canada (.ca)
***Carleton (carleton.ca)
***SCS (scs.carleton.ca)
***Anil (homeostatis.scs.carleton.ca)
*To have everything set up for you, you can use a PaaS (Platform as a Service)
**Node.js Developer Centre:
<code><pre>
nam install azure
git commit -m “My first Node app”
git push azure master
</pre></code>
<ul><ul>
<li>Modulus:</li>
</ul></ul>
<code><pre>
nam install modulus
modulus deploy
</pre></code>

*Next time:
**Get Node running on OpenStack
**You won’t have to run a VM

==Code==

Full code:
* [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/examforms-jquery.zip examforms-jquery.zip]
* [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/graph-demo-sol.zip graph-demo-sol.zip]

===examforms-jquery/examforms-jquery.js===
<source lang="javascript" line>
var http = require('http');
var express = require('express');
var bodyParser = require('body-parser');
var logger = require('morgan');
var port = 3000;
var state = [];

var app = express();

app.set('view engine', 'jade');
app.set('views', __dirname);
app.use(logger('dev'));
app.use(bodyParser.urlencoded({ extended: false }));

app.use(express.static('.'));

app.get('/', function(req, res, next) {
res.render('index', { title: 'COMP 2406 Exam form demo' });
});

app.post('/add', function(req, res) {
var obj = { name: req.body.name,
city: req.body.city,
country: req.body.country,
birthday: req.body.birthday,
email: req.body.email };
state.push(obj);
res.redirect('/list');
});

app.get('/list', function(req, res) {
// res.render('list', { title: 'People Listing', items: state,
// pretty: true});
res.render('clientlist', { title: 'People Listing', pretty: true});
});

app.get('/items', function(req, res) {
res.send(state);
});

var serverUp = function() {
console.log("ExamForms listening on port " + port);
}

var serverDown = function() {
console.log("Server shutting down.");
process.exit(0);
}

var server = http.createServer(app);
server.listen(port);
server.on('listening', serverUp);
process.on('SIGINT', serverDown);
</source>

===examforms-jquery/clientlist.jade===

<source lang="html4strict" line>
extends layout

block header
script(src="jquery-1.12.0.js")
script(src="showlist.js")

block content
h1= title

div
div
table
thead
th Name
th City
th Country
th Birthday
th Email
tbody#theListing

form(method="get", action="/")
button(type="submit") Home
</source>

===examforms-jquery/showlist.js===

<source lang="javascript" line>
$(function() {
// grab JSON from server
// insert data into DOM

function displayItems(items) {
var i;

for (i=0; i<items.length; i++) {
$("#theListing").append("<tr>" +
"<td>" + items[i].name + "</td>" +
"<td>" + items[i].city + "</td>" +
"<td>" + items[i].country + "</td>" +
"<td>" + items[i].birthday + "</td>" +
"<td>" + items[i].email + "</td>" +
"</tr>"
);
}

}

$.getJSON('/items', displayItems);
});
</source>

===graph-demo/routes/index.js===

<source lang="javascript" line>
var express = require('express');
var router = express.Router();

var answers = [];
var questions = {
color: "Favorite color?",
day: "Favorite day of the week?",
dog: "Favorite dog breed?",
city: "Favorite city?"
}

function calcResultsStats(results) {
// var resultsStats = [
// {questionName: "color",
// questionText: "Favorite color?",
// labels: ["red", "blue", "yellow", "brown"],
// values: [5, 7, 2, 0]
// },
// {questionName: "day",
// questionText: "Favorite day of the week?",
// labels: ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday",
// "Saturday", "Sunday"],
// values: [0, 2, 1, 5, 70, 100, 20]
// }
// ];

var resultsStats = [];

Object.keys(questions).forEach(function(q) {
var r = {};
var i, choice;
r.questionName = q;
r.questionText = questions[q];
r.labels = [];
r.values = [];
var counts = {}; // mapping of question answers to counts

for (i = 0; i<answers.length; i++) {
choice = answers[i][q];
if (counts[choice]) {
counts[choice] = counts[choice] + 1;
} else {
counts[choice] = 1;
}
}

Object.keys(counts).forEach(function(c) {
r.labels.push(c);
r.values.push(counts[c]);
});
resultsStats.push(r);
});

return resultsStats;
}

router.get('/', function(req, res) {
res.render('index', {title: 'COMP 2406 Graphing Demo',
questions: questions});
});

router.post('/add', function(req, res) {
var answer = { firstname: req.body.firstname};

var qlist = Object.keys(questions);

qlist.forEach(function(q) {
answer[q] = req.body[q];
});

answers.push(answer);

console.log(answer);

res.redirect('/results');
});

router.get('/results', function(req, res) {
var stats = calcResultsStats(answers);

res.render('results', { title: 'Survey Results',
stats: stats,
barwidth: 60
});
});

// app.get('/render', function (req, res, next) {
// res.render('chart', {xlabel: "Months",
// ylabel: "Failure rate (%)",
// unit: '%',
// barwidth: 40,
// pretty: true,
// items: [{value: 10, bin: "Jan"},
// {value: 20, bin: "Feb"},
// {value: 40, bin: "Mar"},
// {value: 40, bin: "Apr"},
// {value: 40, bin: "May"},
// {value: 40, bin: "Jun"},
// {value: 40, bin: "Jul"},
// {value: 40, bin: "Aug"},
// {value: 40, bin: "Sep"},
// {value: 40, bin: "Oct"},
// {value: 80, bin: "Nov"},
// {value: 100, bin: "Dec"},
// ]});
// });

module.exports = router;
<source>

WebFund 2016W Lecture 16

2016-03-13T00:09:56Z

LeeCroft:

==Video==

The video for the lecture given on March 10, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec16-10Mar2016.mp4 is now available].

==Student Notes==

===Exam-forms Vulnerabilities===

*The current version of this application is vulnerable to a cross site scripting attack
*What is a cross site scripting attack?
**The user supplies data that is actually code which gets interpreted by the browser
**Example from class: <code><script>alert(“hi there”)</script></code> is typed into the city field of the form and causes a popup displaying “hi there”
*How is this different from SQL injection?
**With an SQL injection, code gets interpreted by a back-end database
*Input must be sanitized to prevent these types of attacks
*How can we do this?
**HTML characters can be escaped so that any HTML code in the input gets transformed to be interpreted purely as a string and displayed in its original format

====jQuery Version====

*Problem: takes a raw string and puts it in between tags (this is bad) so it gets interpreted by the browser
*Solution: use an escape function to transform the raw string into a sanitized version
**Include the following in your code to escape html tags

<code><pre>var _entityMap = {
“&” : “&amp”,
“<“ : “&lt”,
“>” : “&gt”,
“ “ “ : “&quot”,
“ ‘ “ : “&#39”,
“/“ : “&#x2F”
};

function escapeHtml(string) {
return String(string).replace(/[&<>"'\/]/g, function (s) {
return entityMap[s];
});
}
</pre></code>

*Problems like this are not hard to fix but they are often forgotten or overlooked because the solutions are not built in to the language

====Original Version====

*In the original version of exam-notes, we used only Jade and no client-side JavaScript to set up the HTML
*When the same cross site scripting attack is tried on this version, we can see that it is not vulnerable
**The HTML code given as input gets displayed as actual text on the web page
*Why is it not vulnerable?
**By default, Jade will escape everything for us
**For example, <code>#{item.city}</code> when given a value of <code><script>alert(“hi there”)</script></code> will be rendered as <code>&lt;script&gt;alert(&quot;hi there&quot;);&lt;/script&gt;</code>
*To prevent Jade from escaping HTML like this, use the bang symbol (!)
**For example: <code>!#{item.city}</code>

===Assignment 5===

*All functionality can be found in parts of tutorials 4-7 and past assignments
*We will start a template with the completed graph-demo code
*Template details:
**The index page is the home screen
**We won't do any logging in
**The index page will show stats of uploaded logs
***How many files
***How many entries
**The index page will have a form for uploading a file
**The index page will have a form for performing query with two radio buttons and a submit button
***The 1st radio button causes the form submission to do a visualization of the query
***The 2nd radio button causes the form submission to send the logs to the browser as plain text for download
**The query form will have fields for
***Service, message and file as regular expressions
***Month and day as plain strings

===Other Notes===
*A favicon is small icon that is displayed in the web page tab

WebFund 2016W Lecture 14

2016-03-08T16:57:51Z

LeeCroft:

==Video==

The video for the lecture given on March 3, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec14-03Mar2016.mp4 is now available].

==Notes==

===In Class===

<pre>
Lecture 14
----------
* sessions
* jQuery

To do sessions securely, you need:
(necessary but NOT sufficient)
* session cookies that cannot be guessed
- use a secret
* secure password storage
- need to be hashed at minimum
- see bcrypt
* secure communication with web server
- HTTPS (HTTP over SSL/TLS)

jQuery and client side JavaScript

jQuery is just a standard library for client-side JS
- far from the only one

I could teach you the standard browser interface
- but it is ugly and has quirks

How you build interfaces

code versus data

* when you build an interface, you have
- code that determines the behavior
- data that describes the interface appearance

* But code can change the appearance, and data can
describe behavior

* how much do you do with each?

* traditionally, you do most everything with code

* When you want end-user customizability, you do more with data
- theming

* Even when lots of the interface is in data, the code
is in charge...except on the web

* On the web, the data is king, and the code serves the
data
</pre>

===Student Notes===

*There are now fewer assignments. Assignments 5 and 6 are going to be bigger assignments since there are now less.
*For the graph-demo tutorial, you could have hard coded the solution but the idea was to use loops to make things dynamic
**Each loops in Jade templates will be important for our next assignment
*Today's goals:
**Sessions
**jQuery

====Cookies and Sessions====

=====Cookies=====

*When logging in to the sample application shown in class:
**If you login on one tab, it won't let you login on another without logging out unless we use a private browser (the new tab appears as already logged in)
**If we login to different users on both the normal and private browser, then if we check to see who is logged in via either browser, it will tell us that both users are logged in
*This is done using cookies
*Cookies are small pieces of data sent from a website and stored in the user's web browser while the user is browsing. Every time the user loads the website, the browser sends the cookie back to the server to inform it of the user's previous activity
**More generally, cookies can be given to a client from any type of server. The client never does anything with the cookie aside from sending it back to the server during future interaction
*Cookies are why the web browsers (normal and private) were logged in separately. Each had their own cookie from the server
**The normal and private browsers were seen by the server as different clients whereas two tabs from the same browser were seen as the same client so they used the same cookie
**If we remove the cookie you will not be logged in any more
*Many cookies can be received just from visiting one website
**They are used by web servers for advertisement purposes
**There are also good cookies that let websites remember who you are so you do not have to constantly re-login every time you visit the website

=====Sessions=====

*In the sample web application, we are using sessions to manage the cookies
*In app.js we need to include:
<code><pre>
var session = require('express-session');
app.use(session({secret: 'superSekretHere!',
resave: false,
saveUnitialized: false}));
</pre></code>

*The secret option is used to prevent people from stealing or guessing your secret. If you don't have a good secret people could reconstruct your cookies
*In routes/index.js, we can see the use of a <code>req.session</code> object
**This object contains session information about the client that has sent in a request
*We are using <code>req.session.username</code> to keep track of whether a user (a client) is logged in or not
**<code>req.session.username</code> at first has no value but once you submit a username (in the /login route), this value gets set
**Whether or not the value is set influences the behaviour of the web app
***When it is set, we can no longer submit a username to log in but we can logout
***This is because of the conditional statements in the route functions
****In the / route, we render the index template (for logging in) if <code>req.session.username</code> is not set, otherwise we redirect to the /users route
****Similarly, in the /users route, we render the users template if the values is set, otherwise, we redirect to the / route
*Sessions are currently being stored in memory but the sessions should ideally be stored in a database
**To do this we use MongoDB (we will see this later)
*To handle sessions securely, you need (necessary but NOT sufficient):
**Session cookies that cannot be guessed
***Use a secret
**Secure password storage
***Passwords need to be hashed at minimum
***See bcrypt
**Secure communication with the web server
***HTTPS (HTTP over SSL/TLS)

====Client-Side JavaScript and jQuery====

*In the jQuery demo, we have an interactive interface that has nothing to do with the server and it is implemented with JavaScript
**This is run completely within the browser (there is not even a server running on the machine)
**When clicking on the favourite dog breed it automatically changes
**This is done through the use of client-side JavaScript and the jQuery library
*jQuery is just a standard library for client side JavaScript and it is far from the only one
**We could learn the standard browser interface but it is ugly and has quirks
*To include a client-side script on a web page, we need to add a script tag in the head of the HTML document which points to the right script
**In the demo, we have one to link the jQuery library and another to link our own client-side script (which uses jQuery)

=====Using jQuery=====

*What is the "$" ?
**The "$" is equal to jQuery by default
*The first thing that we do at the start of the script is give jQuery a callback function
**This callback function contains all of the other code in our demo client-side script and it will run when the page has fully loaded
**Because web browsers load things on a web page incrementally, we need this callback function to force the script to wait until everything is loaded
***This ensures that we don't try to access something that is not there yet
*We are primarily using jQuery to query things
**What are we querying?
***The DOM (Document Object Model)
**Using the "#" is a way to tell jQuery that we are doing a query based on the ID of an element
**In order to allow for the demo script to easily access the favourite dog breed elements, they were given IDs in the HTML code
*Once we have access to an HTML element, we can modify it however we want
*jQuery is useful for making things nice and easy
**For example, it has a built in function for fading in and out elements
*We can access the in-browser console through the dev tools which allows us to alter what we see on our webpage in real time (using a read-eval-print loop)
**For example, typing in <code>$("#color").val("pink");</code> will set the value (in this case the text) of the element with ID "color" (in this case the text field) equal to "pink" so that "pink" will appear in the text field
*Make sure to check the jQuery documentation for more details on how it works

====Building Interfaces (Code Versus Data)====

*When you build an interface, you have
**Code that determines behavior
**Data that describes the interface appearance
*But code can change the appearance, and data can describe behavior
**How much do you do with each?
***Traditionally, you must do everything with code
***When you want end-user customizability, you do more with data
****Theming
***Even when lots of the interface is in the data, the code is in charge... except on the web
**On the web, the data is king, and the code serves the data
***The data (the HTML elements) completely define the interface
***The code (client-side JavaScript) can be used to change things but ultimately we don't need this code

==Code==

* [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/session-demo.zip session-demo.zip]
* [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/jquery-test.zip jquery-test.zip]

WebFund 2016W Lecture 13

2016-03-06T14:08:46Z

LeeCroft: /* JavaScript Closures */

==Video==

The video from the lecture given on March 1, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec13-01Mar2016.mp4 is now available].

==Notes==

===In lecture===
<pre>
Lecture 13
----------

To get help with assignment 4, look at Assignment 5 from Winter 2015.

closures

DOM
- document object model
- set of JavaScript objects for accessing the current page in the browser

client-side JS can access the DOM
(data structures representing the current web page)

server-side JS can access the operating system

client-side JS is *limited* in its access
- because such access is dangerous
- ANYONE can run JavaScript in your browser
- so, JavaScript in the browser is sandboxed

execution sandbox is an environment for running untrusted programs
- sandbox limits access and resources

Client-side JavaScript has no native way to
(well, it used to...)
- access local files
- open network connections
- run multiple threads
- access other windows/programs

Originally JavaScript was so limited it could do no background processing
- could only run when the page was loaded or when the user acted

Microsoft messed it up
- Outlook Web Access
- they built an ActiveX control which implemented XMLHttpRequest()
- but really, it was a way to do GETs and POSTs in the background
- other browsers implemented the API natively

Now you could update data in a web page without reloading the page
</pre>

===Student Notes===

====storelogline.js and storelogs.js====

*To use storelogline.js:
**Find an entry from an existing log file and copy the line
**Run storelogline.js with the copied line pasted after the script name as command line parameters
**Optionally, verify that the line was entered in the database by typing the following with the Mongo client:
<code><pre>
mongo
use log-demo
db.logs.find()
</pre></code>

=====Writing storelogs.js=====

*We can use the code from storeloglines.js to start building the storelogs.js script for tutorial 5
*To read in a file specified in the command line, you do <code>var data = fs.readFileSync(process.argv[2], ‘utf-8’);</code>
**You use <code>process.argv[2]</code> because in the terminal your third argument is going to be the file to read
*To split the data into lines, yo do <code>var lines = data.split(‘\n’);</code>
*To parse the lines and store them into an array, you can do something like:

<code><pre>
var entries = [];
int i; entry, fields, j;

for(i=0;i<lines.length; i++) {
if(lines[i] &&lines[i]!==null) {
field=lines[i].split(‘ ‘); //this is used to split the lines at index i wherever the spaces occur
console.log(split); //when this is printed you’ll see that it splits between all the spaces as well—>you can regular expression to fix this but we fixed it through the while loop
entry={};
j=0;

//Used to delete the spaces
while(j<field.length) {
if(field.lenght[j]===“ “) {
field.splice(j, 1);
}else{
j++;
}
}

console.log(field); //—>this prints the logs out without the spaces
entry.date=field[0] + “ “ + field[1]; //Than you start populating the array
entry.time=field[2];
entry.host=field[3];
entry.service=field[4].slice(0,-1); //the program will have an error here without the if statement above
entry.message=filed.slice(5).join(‘ ‘);
//to store in to the entries ( the array )
entries.push(entry);
}
}
</pre></code>

*To insert into the database you don’t need a loop to iterate. You can use the <code>collection.insert()</code> method which can take an array (entries) as the argument to insert
**API’s have documentation and they change because there’s also <code>insertOne()</code> and <code>insertMany()</code> that seem to do the same thing
**For what we want to do <code>insert()</code> will work
*For querying the database (assignment 4), there is a trick
**There are previous assignment solutions that can help you.
**It's the previous assignment from last year’s assignment 5
*Note that when using special characters like [] and () in regular expression, you will need to use a backslash to escape them

====JavaScript Closures====

*JavaScript closures are about nesting functions
**For example:
<code><pre>
var f = function(x) {
function g(y) {
return x+y;
}
return g; //The function g can see x
}

var q = f(7);
var h = f(12);
console.log(“q = “ +q(4)); //prints 11
console.log(“h(4) = “ + h(4)); //Prints 16
</pre></code>

*This gives different answers because <code>x</code> is different in two two instances of <code>g()</code>
*You cannot do things like this with C
*In examforms we have an app.get for list and we have a function in it that has a callback called <code>findCallback()</code>
**This callback function has access to the request and response objects because they were in scope at the location where the callback function was defined
**To have <code>findCallback</code> have access to request and response you just need to create different instances of <code>findCallback</code> that have their own copies of request and response, therefore one version of <code>findCallback</code> will have one request and on another call it will have another request
*Closure is a concept you need to understand because it’s very handy

====Client-Side JavaScript====

*In JavaScript we’ve been doing everything on the server side so far
*Using the browser dev tools, you can go to console and play around in it. Its almost like the Node environment in a terminal but with some missing features
*In the browser there is no <code>require()</code>
*In the browser the unit is the page but in the server its the file
*You can access objects representing the page by typing <code>document</code> or <code>window</code>
*When you load a webpage, it can load and run scripts
**By running JavaScript within the browser, you can write code that has access to the webpage (in particular, the DOM)
***The DOM (Document Object Model) is a tree-like structure of objects representing the HTML elements of the page

=====Differences Between Client-Side and Server-Side JavaScript=====

*Client-side JavaScript can access the DOM (data structure repressing the current web page)
*Server-side JavaScript can access the operating system
*In client-side JavaScript, you can’t access everything... It's *limited*
**That’s because such access is dangerous, the code could be coming from ANYONE
**ANYONE can run JavaScript in your browser and you don’t want this code messing with your system
**The browser is designed to protect your system from bad JavaScript, so JavaScript in the browser is sandboxed
***An execution sandbox is an environment for running untrusted code
***The sandbox limits access and resources
***Client-side JavaScript has no native way to:
****Access local files
****Open network connections
****Run multiple threads
****Access other windows/programs
*The model of client-side JavaScript was that it should be limited
*Originally JavaScript was so limited that it could do no background processing
**It originally could only run when the webpage was loaded or when the user acted
**It was singly threaded so you had to limit your execution
**Microsoft messed it up (in particular, the Outlook Web Access team messed it up)
***They built an ActiveX control which implemented XMLHttpRequest() —> an object that would send a GET or a POST request and would bring back a document that is assumed to be XML
***But more importantly, it was a way to do GETs and POSTs in the background
***Other browsers implemented the API natively
***Now you could update data in a web page without reloading the page
***Google really ran with this (with gmail, google maps, etc)
*Client-side JavaScript is the same JavaScript we have been working with except it has access to a webpage now but you access the things you can access the exact same way
*If you want to change how the current page looks or works you can type JavaScript directly into the browser console (it uses a read-eval-print loop which will affect the page)

==Code==

===nested.js===

<source lang="javascript" line>
var f = function(x) {
function g(y) {
return x + y;
}

return g;
}

var q = f(7);
var h = f(12);

console.log("q(4) = " + q(4));
console.log("h(4) = " + h(4));
</source>

===storeLogs.js===

[http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/storeLogs.js downloadable version]

<source lang="javascript" line>
// storeLogs.js
//
// node storeLogs.js <logfile>
//
//
// The log messages in the file should be of the format:
// <month> <day of month> <24-hour time in hh:mm:ss> <host> <service name[pid]>: Actual message
//

var mc = require('mongodb').MongoClient;
var fs = require('fs');

var data = fs.readFileSync(process.argv[2], 'utf-8');

var lines = data.split('\n');

var entries = [];

var i, j, entry, field;

for (i=0; i<lines.length; i++) {
if (lines[i] && lines[i] !== '') {
field = lines[i].split(' ');
entry = {};
j = 0;
while (j < field.length) {
if (field[j] === "") {
field.splice(j, 1);
} else {
j++;
}
}
entry.date = field[0] + " " + field[1];
entry.time = field[2];
entry.host = field[3];
entry.service = field[4].slice(0,-1);
entry.message = field.slice(5).join(' ');
entries.push(entry);
}
}

// entry.date = process.argv[2] + " " + process.argv[3];
// entry.time = process.argv[4];
// entry.host = process.argv[5];
// entry.service = process.argv[6].slice(0,-1); // drop the trailing colon
// entry.message = process.argv.slice(7).join(' ');

var db;

var reportInserted = function(err, result) {
if (err) {
throw err;
}

console.log("Inserted the following log record:");
console.log(result.ops);
db.close();
}

var connectCallback = function(err, returnedDB) {
if (err) {
throw err;
}

db = returnedDB;

db.collection('logs').insert(entries, reportInserted);
}

mc.connect('mongodb://localhost/log-demo', connectCallback);
</source>

WebFund 2016W Lecture 13

2016-03-06T14:08:19Z

LeeCroft: /* JavaScript Closures */

==Video==

The video from the lecture given on March 1, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec13-01Mar2016.mp4 is now available].

==Notes==

===In lecture===
<pre>
Lecture 13
----------

To get help with assignment 4, look at Assignment 5 from Winter 2015.

closures

DOM
- document object model
- set of JavaScript objects for accessing the current page in the browser

client-side JS can access the DOM
(data structures representing the current web page)

server-side JS can access the operating system

client-side JS is *limited* in its access
- because such access is dangerous
- ANYONE can run JavaScript in your browser
- so, JavaScript in the browser is sandboxed

execution sandbox is an environment for running untrusted programs
- sandbox limits access and resources

Client-side JavaScript has no native way to
(well, it used to...)
- access local files
- open network connections
- run multiple threads
- access other windows/programs

Originally JavaScript was so limited it could do no background processing
- could only run when the page was loaded or when the user acted

Microsoft messed it up
- Outlook Web Access
- they built an ActiveX control which implemented XMLHttpRequest()
- but really, it was a way to do GETs and POSTs in the background
- other browsers implemented the API natively

Now you could update data in a web page without reloading the page
</pre>

===Student Notes===

====storelogline.js and storelogs.js====

*To use storelogline.js:
**Find an entry from an existing log file and copy the line
**Run storelogline.js with the copied line pasted after the script name as command line parameters
**Optionally, verify that the line was entered in the database by typing the following with the Mongo client:
<code><pre>
mongo
use log-demo
db.logs.find()
</pre></code>

=====Writing storelogs.js=====

*We can use the code from storeloglines.js to start building the storelogs.js script for tutorial 5
*To read in a file specified in the command line, you do <code>var data = fs.readFileSync(process.argv[2], ‘utf-8’);</code>
**You use <code>process.argv[2]</code> because in the terminal your third argument is going to be the file to read
*To split the data into lines, yo do <code>var lines = data.split(‘\n’);</code>
*To parse the lines and store them into an array, you can do something like:

<code><pre>
var entries = [];
int i; entry, fields, j;

for(i=0;i<lines.length; i++) {
if(lines[i] &&lines[i]!==null) {
field=lines[i].split(‘ ‘); //this is used to split the lines at index i wherever the spaces occur
console.log(split); //when this is printed you’ll see that it splits between all the spaces as well—>you can regular expression to fix this but we fixed it through the while loop
entry={};
j=0;

//Used to delete the spaces
while(j<field.length) {
if(field.lenght[j]===“ “) {
field.splice(j, 1);
}else{
j++;
}
}

console.log(field); //—>this prints the logs out without the spaces
entry.date=field[0] + “ “ + field[1]; //Than you start populating the array
entry.time=field[2];
entry.host=field[3];
entry.service=field[4].slice(0,-1); //the program will have an error here without the if statement above
entry.message=filed.slice(5).join(‘ ‘);
//to store in to the entries ( the array )
entries.push(entry);
}
}
</pre></code>

*To insert into the database you don’t need a loop to iterate. You can use the <code>collection.insert()</code> method which can take an array (entries) as the argument to insert
**API’s have documentation and they change because there’s also <code>insertOne()</code> and <code>insertMany()</code> that seem to do the same thing
**For what we want to do <code>insert()</code> will work
*For querying the database (assignment 4), there is a trick
**There are previous assignment solutions that can help you.
**It's the previous assignment from last year’s assignment 5
*Note that when using special characters like [] and () in regular expression, you will need to use a backslash to escape them

====JavaScript Closures====

*JavaScript closures are about nesting functions
**For example:
<code><pre>
var f = function(x) {
function g(y) {
return x+y;
}
return g; //The function g can see x
}

var q = f(7);
var h = f(12);
console.log(“q = “ +q(4)); //prints 11
console.log(“h(4) = “ + h(4)); //Prints 16
</pre></code>

*This gives different answers because x is different in two two instances of <code>g</code>
*You cannot do things like this with C
*In examforms we have an app.get for list and we have a function in it that has a callback called <code>findCallback()</code>
**This callback function has access to the request and response objects because they were in scope at the location where the callback function was defined
**To have <code>findCallback</code> have access to request and response you just need to create different instances of <code>findCallback</code> that have their own copies of request and response, therefore one version of <code>findCallback</code> will have one request and on another call it will have another request
*Closure is a concept you need to understand because it’s very handy

====Client-Side JavaScript====

*In JavaScript we’ve been doing everything on the server side so far
*Using the browser dev tools, you can go to console and play around in it. Its almost like the Node environment in a terminal but with some missing features
*In the browser there is no <code>require()</code>
*In the browser the unit is the page but in the server its the file
*You can access objects representing the page by typing <code>document</code> or <code>window</code>
*When you load a webpage, it can load and run scripts
**By running JavaScript within the browser, you can write code that has access to the webpage (in particular, the DOM)
***The DOM (Document Object Model) is a tree-like structure of objects representing the HTML elements of the page

=====Differences Between Client-Side and Server-Side JavaScript=====

*Client-side JavaScript can access the DOM (data structure repressing the current web page)
*Server-side JavaScript can access the operating system
*In client-side JavaScript, you can’t access everything... It's *limited*
**That’s because such access is dangerous, the code could be coming from ANYONE
**ANYONE can run JavaScript in your browser and you don’t want this code messing with your system
**The browser is designed to protect your system from bad JavaScript, so JavaScript in the browser is sandboxed
***An execution sandbox is an environment for running untrusted code
***The sandbox limits access and resources
***Client-side JavaScript has no native way to:
****Access local files
****Open network connections
****Run multiple threads
****Access other windows/programs
*The model of client-side JavaScript was that it should be limited
*Originally JavaScript was so limited that it could do no background processing
**It originally could only run when the webpage was loaded or when the user acted
**It was singly threaded so you had to limit your execution
**Microsoft messed it up (in particular, the Outlook Web Access team messed it up)
***They built an ActiveX control which implemented XMLHttpRequest() —> an object that would send a GET or a POST request and would bring back a document that is assumed to be XML
***But more importantly, it was a way to do GETs and POSTs in the background
***Other browsers implemented the API natively
***Now you could update data in a web page without reloading the page
***Google really ran with this (with gmail, google maps, etc)
*Client-side JavaScript is the same JavaScript we have been working with except it has access to a webpage now but you access the things you can access the exact same way
*If you want to change how the current page looks or works you can type JavaScript directly into the browser console (it uses a read-eval-print loop which will affect the page)

==Code==

===nested.js===

<source lang="javascript" line>
var f = function(x) {
function g(y) {
return x + y;
}

return g;
}

var q = f(7);
var h = f(12);

console.log("q(4) = " + q(4));
console.log("h(4) = " + h(4));
</source>

===storeLogs.js===

[http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/storeLogs.js downloadable version]

<source lang="javascript" line>
// storeLogs.js
//
// node storeLogs.js <logfile>
//
//
// The log messages in the file should be of the format:
// <month> <day of month> <24-hour time in hh:mm:ss> <host> <service name[pid]>: Actual message
//

var mc = require('mongodb').MongoClient;
var fs = require('fs');

var data = fs.readFileSync(process.argv[2], 'utf-8');

var lines = data.split('\n');

var entries = [];

var i, j, entry, field;

for (i=0; i<lines.length; i++) {
if (lines[i] && lines[i] !== '') {
field = lines[i].split(' ');
entry = {};
j = 0;
while (j < field.length) {
if (field[j] === "") {
field.splice(j, 1);
} else {
j++;
}
}
entry.date = field[0] + " " + field[1];
entry.time = field[2];
entry.host = field[3];
entry.service = field[4].slice(0,-1);
entry.message = field.slice(5).join(' ');
entries.push(entry);
}
}

// entry.date = process.argv[2] + " " + process.argv[3];
// entry.time = process.argv[4];
// entry.host = process.argv[5];
// entry.service = process.argv[6].slice(0,-1); // drop the trailing colon
// entry.message = process.argv.slice(7).join(' ');

var db;

var reportInserted = function(err, result) {
if (err) {
throw err;
}

console.log("Inserted the following log record:");
console.log(result.ops);
db.close();
}

var connectCallback = function(err, returnedDB) {
if (err) {
throw err;
}

db = returnedDB;

db.collection('logs').insert(entries, reportInserted);
}

mc.connect('mongodb://localhost/log-demo', connectCallback);
</source>

WebFund 2016W Lecture 13

2016-03-06T14:06:51Z

LeeCroft: /* Writing storelogs.js */

==Video==

The video from the lecture given on March 1, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec13-01Mar2016.mp4 is now available].

==Notes==

===In lecture===
<pre>
Lecture 13
----------

To get help with assignment 4, look at Assignment 5 from Winter 2015.

closures

DOM
- document object model
- set of JavaScript objects for accessing the current page in the browser

client-side JS can access the DOM
(data structures representing the current web page)

server-side JS can access the operating system

client-side JS is *limited* in its access
- because such access is dangerous
- ANYONE can run JavaScript in your browser
- so, JavaScript in the browser is sandboxed

execution sandbox is an environment for running untrusted programs
- sandbox limits access and resources

Client-side JavaScript has no native way to
(well, it used to...)
- access local files
- open network connections
- run multiple threads
- access other windows/programs

Originally JavaScript was so limited it could do no background processing
- could only run when the page was loaded or when the user acted

Microsoft messed it up
- Outlook Web Access
- they built an ActiveX control which implemented XMLHttpRequest()
- but really, it was a way to do GETs and POSTs in the background
- other browsers implemented the API natively

Now you could update data in a web page without reloading the page
</pre>

===Student Notes===

====storelogline.js and storelogs.js====

*To use storelogline.js:
**Find an entry from an existing log file and copy the line
**Run storelogline.js with the copied line pasted after the script name as command line parameters
**Optionally, verify that the line was entered in the database by typing the following with the Mongo client:
<code><pre>
mongo
use log-demo
db.logs.find()
</pre></code>

=====Writing storelogs.js=====

*We can use the code from storeloglines.js to start building the storelogs.js script for tutorial 5
*To read in a file specified in the command line, you do <code>var data = fs.readFileSync(process.argv[2], ‘utf-8’);</code>
**You use <code>process.argv[2]</code> because in the terminal your third argument is going to be the file to read
*To split the data into lines, yo do <code>var lines = data.split(‘\n’);</code>
*To parse the lines and store them into an array, you can do something like:

<code><pre>
var entries = [];
int i; entry, fields, j;

for(i=0;i<lines.length; i++) {
if(lines[i] &&lines[i]!==null) {
field=lines[i].split(‘ ‘); //this is used to split the lines at index i wherever the spaces occur
console.log(split); //when this is printed you’ll see that it splits between all the spaces as well—>you can regular expression to fix this but we fixed it through the while loop
entry={};
j=0;

//Used to delete the spaces
while(j<field.length) {
if(field.lenght[j]===“ “) {
field.splice(j, 1);
}else{
j++;
}
}

console.log(field); //—>this prints the logs out without the spaces
entry.date=field[0] + “ “ + field[1]; //Than you start populating the array
entry.time=field[2];
entry.host=field[3];
entry.service=field[4].slice(0,-1); //the program will have an error here without the if statement above
entry.message=filed.slice(5).join(‘ ‘);
//to store in to the entries ( the array )
entries.push(entry);
}
}
</pre></code>

*To insert into the database you don’t need a loop to iterate. You can use the <code>collection.insert()</code> method which can take an array (entries) as the argument to insert
**API’s have documentation and they change because there’s also <code>insertOne()</code> and <code>insertMany()</code> that seem to do the same thing
**For what we want to do <code>insert()</code> will work
*For querying the database (assignment 4), there is a trick
**There are previous assignment solutions that can help you.
**It's the previous assignment from last year’s assignment 5
*Note that when using special characters like [] and () in regular expression, you will need to use a backslash to escape them

====JavaScript Closures====

*JavaScript closures are about nesting functions
*For example:
<code><pre>
var f = function(x) {
function g(y) {
return x+y;
}
return g; //The function g can see x
}

var q = f(7);
var h = f(12);
console.log(“q = “ +q(4)); //prints 11
console.log(“h(4) = “ + h(4)); //Prints 16
</pre></code>

*This gives different answers because x is different in two two instances of g
*You cannot do things like this with C
*In examforms we have an app.get for list and we have a function in it that has a callback called <code>findCallback()</code>
**This callback function has access to the request and response objects because they were in scope at the location where the callback function was defined
**To have <code>findCallback</code> have access to request and response you just need to create different instances of <code>findCallback</code> that have their own copies of request and response, therefore one version of <code>findCallback</code> will have one request and on another call it will have another request
*Closure is a concept you need to understand because it’s very handy

====Client-Side JavaScript====

*In JavaScript we’ve been doing everything on the server side so far
*Using the browser dev tools, you can go to console and play around in it. Its almost like the Node environment in a terminal but with some missing features
*In the browser there is no <code>require()</code>
*In the browser the unit is the page but in the server its the file
*You can access objects representing the page by typing <code>document</code> or <code>window</code>
*When you load a webpage, it can load and run scripts
**By running JavaScript within the browser, you can write code that has access to the webpage (in particular, the DOM)
***The DOM (Document Object Model) is a tree-like structure of objects representing the HTML elements of the page

=====Differences Between Client-Side and Server-Side JavaScript=====

*Client-side JavaScript can access the DOM (data structure repressing the current web page)
*Server-side JavaScript can access the operating system
*In client-side JavaScript, you can’t access everything... It's *limited*
**That’s because such access is dangerous, the code could be coming from ANYONE
**ANYONE can run JavaScript in your browser and you don’t want this code messing with your system
**The browser is designed to protect your system from bad JavaScript, so JavaScript in the browser is sandboxed
***An execution sandbox is an environment for running untrusted code
***The sandbox limits access and resources
***Client-side JavaScript has no native way to:
****Access local files
****Open network connections
****Run multiple threads
****Access other windows/programs
*The model of client-side JavaScript was that it should be limited
*Originally JavaScript was so limited that it could do no background processing
**It originally could only run when the webpage was loaded or when the user acted
**It was singly threaded so you had to limit your execution
**Microsoft messed it up (in particular, the Outlook Web Access team messed it up)
***They built an ActiveX control which implemented XMLHttpRequest() —> an object that would send a GET or a POST request and would bring back a document that is assumed to be XML
***But more importantly, it was a way to do GETs and POSTs in the background
***Other browsers implemented the API natively
***Now you could update data in a web page without reloading the page
***Google really ran with this (with gmail, google maps, etc)
*Client-side JavaScript is the same JavaScript we have been working with except it has access to a webpage now but you access the things you can access the exact same way
*If you want to change how the current page looks or works you can type JavaScript directly into the browser console (it uses a read-eval-print loop which will affect the page)

==Code==

===nested.js===

<source lang="javascript" line>
var f = function(x) {
function g(y) {
return x + y;
}

return g;
}

var q = f(7);
var h = f(12);

console.log("q(4) = " + q(4));
console.log("h(4) = " + h(4));
</source>

===storeLogs.js===

[http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/storeLogs.js downloadable version]

<source lang="javascript" line>
// storeLogs.js
//
// node storeLogs.js <logfile>
//
//
// The log messages in the file should be of the format:
// <month> <day of month> <24-hour time in hh:mm:ss> <host> <service name[pid]>: Actual message
//

var mc = require('mongodb').MongoClient;
var fs = require('fs');

var data = fs.readFileSync(process.argv[2], 'utf-8');

var lines = data.split('\n');

var entries = [];

var i, j, entry, field;

for (i=0; i<lines.length; i++) {
if (lines[i] && lines[i] !== '') {
field = lines[i].split(' ');
entry = {};
j = 0;
while (j < field.length) {
if (field[j] === "") {
field.splice(j, 1);
} else {
j++;
}
}
entry.date = field[0] + " " + field[1];
entry.time = field[2];
entry.host = field[3];
entry.service = field[4].slice(0,-1);
entry.message = field.slice(5).join(' ');
entries.push(entry);
}
}

// entry.date = process.argv[2] + " " + process.argv[3];
// entry.time = process.argv[4];
// entry.host = process.argv[5];
// entry.service = process.argv[6].slice(0,-1); // drop the trailing colon
// entry.message = process.argv.slice(7).join(' ');

var db;

var reportInserted = function(err, result) {
if (err) {
throw err;
}

console.log("Inserted the following log record:");
console.log(result.ops);
db.close();
}

var connectCallback = function(err, returnedDB) {
if (err) {
throw err;
}

db = returnedDB;

db.collection('logs').insert(entries, reportInserted);
}

mc.connect('mongodb://localhost/log-demo', connectCallback);
</source>

WebFund 2016W Lecture 13

2016-03-06T14:06:04Z

LeeCroft:

==Video==

The video from the lecture given on March 1, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec13-01Mar2016.mp4 is now available].

==Notes==

===In lecture===
<pre>
Lecture 13
----------

To get help with assignment 4, look at Assignment 5 from Winter 2015.

closures

DOM
- document object model
- set of JavaScript objects for accessing the current page in the browser

client-side JS can access the DOM
(data structures representing the current web page)

server-side JS can access the operating system

client-side JS is *limited* in its access
- because such access is dangerous
- ANYONE can run JavaScript in your browser
- so, JavaScript in the browser is sandboxed

execution sandbox is an environment for running untrusted programs
- sandbox limits access and resources

Client-side JavaScript has no native way to
(well, it used to...)
- access local files
- open network connections
- run multiple threads
- access other windows/programs

Originally JavaScript was so limited it could do no background processing
- could only run when the page was loaded or when the user acted

Microsoft messed it up
- Outlook Web Access
- they built an ActiveX control which implemented XMLHttpRequest()
- but really, it was a way to do GETs and POSTs in the background
- other browsers implemented the API natively

Now you could update data in a web page without reloading the page
</pre>

===Student Notes===

====storelogline.js and storelogs.js====

*To use storelogline.js:
**Find an entry from an existing log file and copy the line
**Run storelogline.js with the copied line pasted after the script name as command line parameters
**Optionally, verify that the line was entered in the database by typing the following with the Mongo client:
<code><pre>
mongo
use log-demo
db.logs.find()
</pre></code>

=====Writing storelogs.js=====

*We can use the code from storeloglines.js to start building the storelogs.js script for tutorial 5
To read in a file specified in the command line, you do <code>var data = fs.readFileSync(process.argv[2], ‘utf-8’);</code>
**You use <code>process.argv[2]</code> because in the terminal your third argument is going to be the file to read
*To split the data into lines, yo do <code>var lines = data.split(‘\n’);</code>
*To parse the lines and store them into an array, you can do something like:

<code><pre>
var entries = [];
int i; entry, fields, j;

for(i=0;i<lines.length; i++) {
if(lines[i] &&lines[i]!==null) {
field=lines[i].split(‘ ‘); //this is used to split the lines at index i wherever the spaces occur
console.log(split); //when this is printed you’ll see that it splits between all the spaces as well—>you can regular expression to fix this but we fixed it through the while loop
entry={};
j=0;

//Used to delete the spaces
while(j<field.length) {
if(field.lenght[j]===“ “) {
field.splice(j, 1);
}else{
j++;
}
}

console.log(field); //—>this prints the logs out without the spaces
entry.date=field[0] + “ “ + field[1]; //Than you start populating the array
entry.time=field[2];
entry.host=field[3];
entry.service=field[4].slice(0,-1); //the program will have an error here without the if statement above
entry.message=filed.slice(5).join(‘ ‘);
//to store in to the entries ( the array )
entries.push(entry);
}
}
</pre></code>

*To insert into the database you don’t need a loop to iterate. You can use the <code>collection.insert()</code> method which can take an array (entries) as the argument to insert
**API’s have documentation and they change because there’s also <code>insertOne()</code> and <code>insertMany()</code> that seem to do the same thing
**For what we want to do <code>insert()</code> will work
*For querying the database (assignment 4), there is a trick
**There are previous assignment solutions that can help you.
**It's the previous assignment from last year’s assignment 5
*Note that when using special characters like [] and () in regular expression, you will need to use a backslash to escape them

====JavaScript Closures====

*JavaScript closures are about nesting functions
*For example:
<code><pre>
var f = function(x) {
function g(y) {
return x+y;
}
return g; //The function g can see x
}

var q = f(7);
var h = f(12);
console.log(“q = “ +q(4)); //prints 11
console.log(“h(4) = “ + h(4)); //Prints 16
</pre></code>

*This gives different answers because x is different in two two instances of g
*You cannot do things like this with C
*In examforms we have an app.get for list and we have a function in it that has a callback called <code>findCallback()</code>
**This callback function has access to the request and response objects because they were in scope at the location where the callback function was defined
**To have <code>findCallback</code> have access to request and response you just need to create different instances of <code>findCallback</code> that have their own copies of request and response, therefore one version of <code>findCallback</code> will have one request and on another call it will have another request
*Closure is a concept you need to understand because it’s very handy

====Client-Side JavaScript====

*In JavaScript we’ve been doing everything on the server side so far
*Using the browser dev tools, you can go to console and play around in it. Its almost like the Node environment in a terminal but with some missing features
*In the browser there is no <code>require()</code>
*In the browser the unit is the page but in the server its the file
*You can access objects representing the page by typing <code>document</code> or <code>window</code>
*When you load a webpage, it can load and run scripts
**By running JavaScript within the browser, you can write code that has access to the webpage (in particular, the DOM)
***The DOM (Document Object Model) is a tree-like structure of objects representing the HTML elements of the page

=====Differences Between Client-Side and Server-Side JavaScript=====

*Client-side JavaScript can access the DOM (data structure repressing the current web page)
*Server-side JavaScript can access the operating system
*In client-side JavaScript, you can’t access everything... It's *limited*
**That’s because such access is dangerous, the code could be coming from ANYONE
**ANYONE can run JavaScript in your browser and you don’t want this code messing with your system
**The browser is designed to protect your system from bad JavaScript, so JavaScript in the browser is sandboxed
***An execution sandbox is an environment for running untrusted code
***The sandbox limits access and resources
***Client-side JavaScript has no native way to:
****Access local files
****Open network connections
****Run multiple threads
****Access other windows/programs
*The model of client-side JavaScript was that it should be limited
*Originally JavaScript was so limited that it could do no background processing
**It originally could only run when the webpage was loaded or when the user acted
**It was singly threaded so you had to limit your execution
**Microsoft messed it up (in particular, the Outlook Web Access team messed it up)
***They built an ActiveX control which implemented XMLHttpRequest() —> an object that would send a GET or a POST request and would bring back a document that is assumed to be XML
***But more importantly, it was a way to do GETs and POSTs in the background
***Other browsers implemented the API natively
***Now you could update data in a web page without reloading the page
***Google really ran with this (with gmail, google maps, etc)
*Client-side JavaScript is the same JavaScript we have been working with except it has access to a webpage now but you access the things you can access the exact same way
*If you want to change how the current page looks or works you can type JavaScript directly into the browser console (it uses a read-eval-print loop which will affect the page)

==Code==

===nested.js===

<source lang="javascript" line>
var f = function(x) {
function g(y) {
return x + y;
}

return g;
}

var q = f(7);
var h = f(12);

console.log("q(4) = " + q(4));
console.log("h(4) = " + h(4));
</source>

===storeLogs.js===

[http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/storeLogs.js downloadable version]

<source lang="javascript" line>
// storeLogs.js
//
// node storeLogs.js <logfile>
//
//
// The log messages in the file should be of the format:
// <month> <day of month> <24-hour time in hh:mm:ss> <host> <service name[pid]>: Actual message
//

var mc = require('mongodb').MongoClient;
var fs = require('fs');

var data = fs.readFileSync(process.argv[2], 'utf-8');

var lines = data.split('\n');

var entries = [];

var i, j, entry, field;

for (i=0; i<lines.length; i++) {
if (lines[i] && lines[i] !== '') {
field = lines[i].split(' ');
entry = {};
j = 0;
while (j < field.length) {
if (field[j] === "") {
field.splice(j, 1);
} else {
j++;
}
}
entry.date = field[0] + " " + field[1];
entry.time = field[2];
entry.host = field[3];
entry.service = field[4].slice(0,-1);
entry.message = field.slice(5).join(' ');
entries.push(entry);
}
}

// entry.date = process.argv[2] + " " + process.argv[3];
// entry.time = process.argv[4];
// entry.host = process.argv[5];
// entry.service = process.argv[6].slice(0,-1); // drop the trailing colon
// entry.message = process.argv.slice(7).join(' ');

var db;

var reportInserted = function(err, result) {
if (err) {
throw err;
}

console.log("Inserted the following log record:");
console.log(result.ops);
db.close();
}

var connectCallback = function(err, returnedDB) {
if (err) {
throw err;
}

db = returnedDB;

db.collection('logs').insert(entries, reportInserted);
}

mc.connect('mongodb://localhost/log-demo', connectCallback);
</source>

WebFund 2016W Lecture 12

2016-03-01T16:53:14Z

LeeCroft:

==Video==

The video for the lecture given on February 25, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec12-25Feb2016.mp4 is now available].

==Code==

Below is examforms.js modified to have MongoDB support. You can also [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/code/examforms-mongodb.zip download the full version].

<source lang="javascript" line>
var http = require('http');
var express = require('express');
var bodyParser = require('body-parser');
var logger = require('morgan');
var port = 3000;

var mc = require('mongodb').MongoClient;
var db, formsCollection;

var state = [];

var app = express();

app.set('view engine', 'jade');
app.set('views', __dirname);
app.use(logger('dev'));
app.use(bodyParser.urlencoded({ extended: false }));

var connectCallback = function(err, returnedDB) {
if (err) {
throw err;
}

db = returnedDB;

formsCollection = db.collection('forms');
}

mc.connect('mongodb://localhost/examforms', connectCallback);

app.get('/', function(req, res, next) {
res.render('index', { title: 'COMP 2406 Exam form demo' });
});

app.post('/add', function(req, res) {
var obj = { name: req.body.name,
city: req.body.city,
country: req.body.country,
birthday: req.body.birthday,
email: req.body.email };
state.push(obj);

formsCollection.insert(obj, null);
res.redirect('/list');
});

app.get('/list', function(req, res) {

var findCallback = function(err, dbstate) {
if (err) {
console.log("Couldn't query database for some reason");
return;
}
res.render('list', { title: 'People Listing', items: dbstate});
}

formsCollection.find({}).toArray(findCallback);
});

var serverUp = function() {
console.log("ExamForms listening on port " + port);
}

var serverDown = function() {
console.log("Server shutting down.");
process.exit(0);
}

var server = http.createServer(app);
server.listen(port);
server.on('listening', serverUp);
process.on('SIGINT', serverDown);
</source>

==Student Notes==

===Midterms Solutions===
*Make sure to read the questions carefully as they may use slightly different patterns from the things you have seen previously in class and in the tutorials and assignments<br><br>

<strong>1. What is the type of the values associated with all of the symbols referenced in line examforms.js:44? Specifically, what is the type of the values bound to server, on, and serverUp?</strong>

A: server: object, on: function, serverUp: function
<br>
<br>
<strong>2. If I create a line 46 in examforms.js (i.e., add a line to the end of the file) that says ’console.log(”Finished!”);’, when will finished! be printed relative to the other output of the program? Explain your reasoning.</strong>

A: “Finished!” should generally be printed first, before any other output, because all of the other output is produced by callbacks and they are normally executed after the main body of the file has been evaluated. It is acceptable to say that you don’t actually know when it will be printed since you can’t know for certain when callbacks will be invoked; however, the mainly single-threaded computation model of JavaScript makes this sort of interleaving very unlikely (unless you use something like web workers).
<br>
<br>
<em>We had an example like this from class before the Midterm:</em><br>
<strong>3. If you change the app.post() call to an app.get() call on examforms.js:19, how would the program no longer function properly? Is it possible to repair it without changing this line back to app.get()? (yes or no)</strong>

A: It would no longer function properly because the form submission generates a “POST /add” HTTP request and that HTTP request is no longer recognized by the server. Yes. (If you change the form’s method (index.jade:8) to get and make the function access parameters as passed by GETs (req.query rather than req.body) then it would be fixed.)
<br>
<br>
<strong>4. (2) As written, examforms returns a 404 error for requests to style.css. What is wrong with examforms? How could you fix it?</strong>

A: examforms has no way of serving static files. You’d either have to create a custom route that would load the style sheet manually, or you’d have to include the line normally used in express node applications to add static server support. (Specifically, you’d add app.use (express.static( dirname)); around examforms.js:14.) In other words, the line that adds functionality to serve static files is currently missing.
<br>
<br>
<strong>5. The request object (req) passed in to route handlers has a “headers” property, just as the request objects in tinywebserver.js. Given this fact, how could you change examforms so that it uses a different style sheet when receiving requests from iPhones? (Describe the general strategy, no need for all the implementation details.)</strong>

<em>//We saw this previously for assignment 3 and this question wasn’t a question of coding it to answer, you should simply understand the concepts behind it.</em>

A: We would just have to check the value of req.headers[“user-agent”] to see if it contained the string “iPhone” in each route handling function. We’d then have to either use different jade files for iPhones that used the different stylesheet, or we’d move the stylesheet from layout.jade and instead pass it in like we do the title of the page.
<br>
<br>
<strong>6. When a web browser loads a standard web page, what type of HTTP request does it make of the server? And, if the requested URL is invalid (i.e., page not found), what is the standard response code?</strong>

A: HTTP GET; 404
<br>
<br>
<strong>7. How could you change examforms so you could specify the PORT on the command line, e.g. you’d run node examforms.js 7000 to have it listen on port 7000? If no port is specified it should listen on port 3000.</strong>

<em>//Student question: Will there be any marks for talking about environment variables because it was what was learned in tutorial?<br>
//Answer: No, the question has nothing to do with environment variables. You must answer specifically what the question asked.</em>

A: Add a line to set the port variable based on the value of process.argv[2], e.g. var port = process.argv[2] || 3000;
<br>
<br>
<strong>8. What is the purpose of the Content-Type: HTTP response header?</strong>

A: It tells the browser the type of content being returned, whether it is a plain text file (text/plain), an HTML document (text/HTML), an image (image/jpeg), etc.
<br>
<br>
<strong>9. When the call to app.get on lines 15–17 of examforms.js returns, what work has it accomplished? Specifically what data (if any) has been returned to the requesting web client? Why?</strong>

<em>//Note that the function is only being defined, not run. It is only run when a request is received (which is not what has happened here)!!</em>

A: After these lines have been written a function has been defined and it has been registered as a callback for an HTTP GET request. The anonymous function has not been called; it will only be invoked when there is an HTTP GET request.
<br>
<br>
<strong>10. When is the function serverDown(), on lines 37-40 of examforms.js, called? How can a user cause this function to be called?</strong>

A: This function is called when the server receives a SIGINT signal. This signal is generated when a user presses Ctrl-C at the command line.
<br>
<br>
<strong>11. What would happen to examforms if we deleted layout.jade:6 (block header)? What about layout.jade:8 (block content)?</strong>

A: If you delete line 6, nothing would happen as this block isn’t used in any of the templates. Deleting line 8 would make the body of all pages blank.
<br>
<br>
<strong>12. The each statement on list.jade:16 is a loop. What does this loop do? Where is it executed (server or browser)?</strong>

A: This loop generates a table line for every person stored in the state array (passed to the template on examforms.js:30). It runs/ is EXECUTED on the server as it is a Jade loop. The browser simply gets a fully populated table.
<br>
<br>
<strong>13. What would happen (if anything) if you removed two spaces from the beginning of list.jade:25, button(type="submit") Home?</strong>

A: By removing these spaces the button is no longer indented under the form on line 24; thus now it is a submit button that is no longer enclosed inside of an HTML form. Thus while the button will still be present in the page, clicking it will now do nothing.
<br>
<br>
<strong>14. Do web browsers normally run/interpret Jade code? Explain briefly.</strong>

A: Web browsers do not interpret Jade code; instead, they interpret the HTML that is produced by compiling Jade templates.
<br>
<br>
<strong>15. What does res.redirect(/list); on line 26 of examforms.js do? Specifically, what response is (eventually or immediately) sent back to the browser after this line is executed? Explain briefly.</strong>

A: This line causes a 302 (permanent redirect) response to be sent back to the browser with /list as the new location. The browser then proceeds to do an HTTP GET /list which then causes the callback for /list to be run on the server (and return the corresponding page).

===Using a Database with Examforms===

*Question: We want to write records to the database, how?
*We will need to modify the code in order to give it the ability to interact with the database

====Establishing a Connection====

*First we add the code needed to establish a connection to the database:

<code><pre>
var mc = require(‘Mongodb’).MongoClient;
mc.connect(‘mongodb://localhost/examforms’, connectCallback);
</pre></code>

*This allows you to connect to the examforms database via the Mongo server running on localhost using the mongodb protocol (as specified by the first argument to the <code>connect()</code> method)
*Note that we need the Mongo server running in order for this code to run properly
*Recall that if you’re using the course VM, Mongo is installed; if using you’re own machine, you must install Mongo to be able to use it
**We can verify that Mongo is running by opening a new terminal and typing <code>mongo</code> to run the terminal client
**If the Mongo server is not running, we will see a message stating that the connection has failed
**We would get a similar error if our application tries to establish a connection while Mongo is not running
*Next, we need to implement connectCallback

<code><pre>
var connectCallback = function(err, returnedDB) {
if (err) {
throw err;
}

db = returnedDB;
formsCollection = db.collection('forms');
}
</pre></code>

*<code>db</code> and <code>formsCollection</code> should be defined beforehand at the global scope so that we will have access to them in other callback functions later

====Inserting Documents into the Database/Collection====

*Now we can add code to store the data that we receive from form submissions (currently stored in the <code>obj</code> variable)
*This code should be added inside the function defined for the /add route

<code><pre>
formsCollection.insert(obj, null); //pass null because we are being explicit and don’t want to pass anything as the callback function
res.redirect(‘/list’);
</pre></code>

*To verify if we have properly inserted anything into the database, we can use the terminal client
*Mongo has many commands that can be used to tell you the current state of your database
*The db.serverStatus() command (from the terminal client) returns an overview of the status of the database. Includes disk usage, memory use, connection, journaling, and index access
*For other administration requirements that you may run across visit [https://docs.mongodb.org/manual/administration/monitoring/ https://docs.mongodb.org/manual/administration/monitoring/]
*To check if the document was actually stored in the collection, in the terminal client type:
<code><pre>
use examforms
db.forms.find()
</pre></code>

*This will display the documents in the forms collection within the examforms database
*To clear data from the collection, you can use the <code>drop()</code> method of the collection in order to drop (remove) the entire collection

====Querying the Database/Collection====

*In order to now display the list of people that have been entered via the form, we need to be able to query the database
*We can add the following code to the function defined for the /list route

<code><pre>
formsCollection.find({}).toArray(findCallback);
</pre></code>

*The argument passed to the <code>find()</code> method is our query
**Here, the query is an empty object so every document will be matched
*The return value of the <code>find()</code> method is a cursor object used to iterate through the results
*We are calling the <code>toArray()</code> method of the cursor object to get all the results as an array
**The argument to this method is the callback function where we will have access to the array
**Since the callback function is the only place where we have access to the results, we must send a response to the client from there
***This means that the callback function must be defined within the scope of the response object
*Look into MongoDB API online for help with assignment 4, specifically <code>toArray()</code>: [https://mongodb.github.io/node-mongodb-native/api-generated/cursor.html#toarray https://mongodb.github.io/node-mongodb-native/api-generated/cursor.html#toarray]

WebFund 2016W Lecture 11

2016-02-28T21:49:07Z

LeeCroft:

==Video==

The video for the lecture given on February 23, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec11-23Feb2016.mp4 is now available].

==Notes==

===In-class Notes===

<pre>
Lecture 11
----------

Databases

/var/spool/mail
- where email is stored
- one file per user

my mail inbox in /var/spool/mail/soma

Who accesses this file?
- mail client
- mail server

What happens if the mail server delivers mail while I'm going
through my email (deleting, filing)
- unless you lock it, you lose mail, mail client will overwrite
inbox and delete new messages
- solution: lock the inbox
create a soma.lock file
(inside it has the process ID of the accessing process)

This is a bad idea that does not scale
Instead, use a database
- concurrent access
- data persistence

What kind of database?
- key-value stores
like a filesystem for very small files,
with search
- document store <-- we'll use this
values have structure
can index on values
- relational database
sets of tables
tables have rows and columns
relations between tables
example: names and addresses
- rows are individual people
- columns are first name, last name, address, city, etc.
relation example: customers and purchases
- each purchase (invoice) has item purchased, price, and
customer ID
- customer table connects customer ID and name, address, etc

We're going to use MongoDB, a document store, not a relational
database.
- stores JSON-like documents
- scriptable in JavaScript

If we were to use a relational database, we'd have to learn SQL
- Structured Query Language
- many man implementations
Oracle, MySQL (MariaDB), PostgreSQL, MS SQL Server, SQLite, etc
- you mainly need a relational database to handle transactions

Transaction
- what if a change to a database involves multiple tables?
- a transaction ensures all tables are changed or none are

Classic example: airline tickets

Tables:
- seat reservations
- payment
- customer info

When you buy a ticket, you want to change all three

A transaction ensures atomic behavior
all happen or none happen

MongoDB Relational DB
Document => row
Collection => table

Web Architecture
----------------

web browser (interprets HTML, CSS, JavaScript)

|
| HTTP
|

web server (runs arbitrary code, in this class JavaScript in node)

|
| SQL or MongoDB protocol
|

database (persistent store, allows concurrent access safely)
</pre>

===Student Notes===
====Databases====
=====Problems with Examforms=====
*Previously, we used and array (<code>var state = [];</code>) to store the data entered in the form
*The data stored in <code>state</code> is not persistent meaning it is lost when the server is shut down (stored in RAM)
*We could try storing the data in a file instead
**This quickly becomes problematic when we have multiple machines trying to work with the same file (concurrent access)

=====General Motivation=====
*<code>/var/spool/mail</code>
**The place where email is stored and one file is assigned per user
**My Inbox: <code>/var/spool/mail/soma</code>
*Who have the ability to access the file?
**Mail client
**Mail server
*What happens if the mail server delivers mail while I’m going through my email?
**(deleting, filing)?
**Unless we don’t lock it, we will lose the mail
**The mail client will overwrite inbox and delete new incoming messages
*Solution: lock the inbox
**Create a soma.lock file (Inside it has the process id of the accessing process)
**This is bad idea that does not scale

=====A Database As a Solution=====
*A database offers:
**Concurrent access
**Data persistence
*Now we know what the database does and why should we use it. But what kind of database should we use?

======Key Value Store======
*Simple association of keys with values
*Like a file system for a small files, with search

======Document Store======
*Similar to key value stores
*Values have a structure
*Able to do indexing
*We’re going to use MongoDB, a document store
**Stores JSON-like documents (actually BSON)
**Scriptable in JavaScript

======Relational Database (Gold Standard)======
*Sets of tables
*Tables have rows and columns
*Relations between tables
*Example: names and addresses
**Where rows are individual people and columns are first name, last name, address, city etc.
*Another example: customers and purchases
**Each purchase is a row in the purchases table (the row has item purchased, price and customer ID)
**The customer table connects name, address etc via the customer ID
*If we were to use a relational database, we’d have to lean SQL (Structured Query Language)
**There are a lot of implementations:
***Oracle
***MySQL(MariaDB)
***PostgresQL
***MS SQL Server
***SQLite
*You mainly need a relational database to handle transactions
**What if a change to a database involves a change to multiple tables?
**A transaction ensures all tables are changed or none are
**Classic example: airline tickets
***Tables:
****Seat reservations
****Payment
****Customer info
***When you buy a ticket, you would want to change all three tables
***A transaction ensures atomic behavior
***All changes happen or none happen

====Storageline.js====

=====MongoDB=====
*We will be working with MongoDB as the database that our web applications will use
*When compared to a relational database, the collections of MongoDB are like tables and the documents of MongoDB are like rows
*MongoDB is already installed and is running on the course VM
**If you are using your own machine, you will have to install it
**You can run MongoDB by typing <code>mongod</code> in the terminal
*MongoDB listens on 27017 port by default
*You can run a Mongo client from the terminal by typing <code>mongo</code>
**From here you can interact with the database

=====Interacting With the Database from a Script=====
*We use the npm module <code>mongodb</code> to make our web application interact as a client with the MongoDB server
*The Mongo server (MongoDB) must be running in order for the application to be able to communicate with it (the same goes for the terminal client)
*In this script, we are saving log file entries into the database
**Logs are records left by the system and other applications
**We will be using a syslog-like logs in the tutorial
**Our goal is to parse and store it the log entries in a Mongo collection
*Notice the use of <code>mc.connect()</code> in the script which is used to establish a connection to the database. It takes 2 arguments:
**A string with the address of the database to access
**A callback function which runs once the connection is established (or the application has failed to establish a connection)
*From within the callback function, we can now access the collections of the database that we have connected to
*Here, we use the <code>insert()</code> method of the collection to store a log entry as a document
*Every document in a collection has to have a unique ID
**The ID is generated automatically during an insertion
**It is also known as a primary key which helps us to distinguish between documents

=====Checking the Content of a Collection=====
*An example of looking up documents in a collection might look like:
<code><pre>
mongo //launches the Mongo client
show dbs //lists all databases
use demo-log //indicates which database we wish to use
show collections //lists all collections in the current databse
db.logs.find() //shows all documents in the logs collection
</pre></code>

=====Web Architecture=====
*The database acts as a server which our web server talks to
*Our web server is thus a client of the database server
*A web browser talks to our server using HTTP
*Our server talks to the Database using SQL or a database-specific protocol (such as the MongoDB protocol)

WebFund 2016W Lecture 10

2016-02-10T15:29:35Z

LeeCroft:

==Video==

The video from the lecture given on February 9, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec10-09Feb2016.mp4 is now available].

==Student Notes==

===Assignment 3 Solutions===

*Solutions to the assignment questions have been posted on the [assignment 3 page http://homeostasis.scs.carleton.ca/wiki/index.php/WebFund_2016W:_Assignment_3]
*Below is a review of these solutions

====Q1====
*What is the type of:
**<code>line</code>: lines may be classified as an object but more specifically it is an array
**<code>rawContents</code>: rawContents is of type string
**<code>split()</code>: is a function. It is also acceptable to say that it is a method

====Q2====
*Line 24 should be printed first
*Since JavaScript is single threaded there is a high guarantee it is printed first since the program runs asynchronously
*In general, you can consider the program to do all of its main sequential work first and to then check for callback functions from asynchronous calls that must be run

====Q3====
*A one line change would be adding a line at line 42 by setting <code>status = 404;</code>
*We can also change the docroot but it's an unsafe way

====Q4====
*Just before line 92 we can add another if statement
*We check to see if the user-agent headers exists and whether it contains iPhone as a substring
**If the condition is met, we respond with 403

====Q5====
=====A=====
*By commenting out this line, <code>app</code> will no longer be defined
*Anything referencing app will produce a reference error so the program crashes on the first line that uses app

======Important Point======
*What is the difference between JavaScript's <code>require</code> and C's <code>include</code>?
**In C <code>include</code> looks similar to a <code>require</code> but in reality it’s a very different mechanism.
**The <code>include</code> in C pre-processes the included file to put all of its code into the file that included it
***Symbols are thus defined for us but there is no visible code! We cant tell what kind of symbols include has brought in
**In JavaScript the file we require is pure JavaScript code!
***The required script will run and put some value into its module.exports which is then returned via the require call
***We know what symbol we are working with because we assign the return value of the require call ourselves

=====B=====
*Setting var port = 3000;
*It works perfectly fine but we can no longer change the port using the PORT environment variable

=====C=====
*Since we change the directory, it can no longer find “views”
*This means that the server will not be able to find any templates to render

=====D=====
*Commenting out line 24
*We are no longer exporting anything from the file
*Any calls to require the file will not get an expected return value from the require call and so the server will not work properly

=====E=====
*This line acts as a placeholder for child templates to put content into the body of this template
*By removing it, the child templates have nowhere to put the content in the body

=====F=====
*When we indent the code in this way, the submit button is no longer a part of the form
*The button will therefore not cause the form to be submitted
*Essentially, the button no longer does anything

===Possible questions for the Midterm===

*What is the purpose of app.post? What if <code>app.post(‘/add’, ...)</code> were be changed to <code>app.get('/add', ...)</code>?
**Answer: The server will no longer know how to deal with any incoming POST requests for /add so it will not properly handle the form submission
***Instead, the server could handle an incoming GET request for the form submission (but there are some other problems that would need to be fixed for parsing the incoming data)
*How can we keep it a get and still make it work?
**Answer: Change the method in the form from method “post” to “get”
***POST requests have the form data in their body whereas a GET request stores it in a query strings
***So we can change <code>req.body</code> to <code>req.query</code> in the routing function to access the data properly

<br>

*What is <code>app.use()</code>?
**What happens when we comment out the line <code>app.use( ‘logger'..)</code>?
***The server no longer logs incoming request to the console
**What if we comment out <code>app.use(body.parser...)</code>?
***The program would crash because it wouldn’t be able to find <code>req.body</code> since the body of the request has no been parsed
**What is the purpose of <code>app.use(express.static...)</code>?
***Its tells the server to use a static web server to serve basic webpages and other static resources
*So <code>app.use()</code> is telling the server to use a given module to do something with the incoming requests

WebFund 2016W Lecture 6

2016-02-09T21:02:01Z

LeeCroft:

==Video==

The video from the lecture given on January 26, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec06-26Jan2016.mp4 is now available].

==Notes==

===In-class Notes===

<pre>
Lecture #6
---------

PLAN
* Assignment 2
- aliases
- headers
- other clarifications?

* Objects
- associative hashes with inheritance
- prototypes using Object.create
- "global" object
- "Object" object
- Object.prototype
- hasOwnProperty and Object.hasOwnProperty

* Function calling and "this"
- functions in regular scope
- methods
- constructors with new
- call/apply
- functions in global scope

* form demo!

------------------

If a function starts with a capital letter, it is an object
constructor that you call with new.
- why? because it keeps JavaScript programmers sort of sane

if (x.a) { blah} instead do

if (x.hasOwnProperty(a) && x.a) better, but

if (hasOwnProperty.call(x,a) && x.a) is even better

Why not tinywebserver?

* templates: otherwise, have to manually generate web pages using
variable data
* routing: otherwise, you'll be using regexp's to match each
incoming URL and then call the right function

</pre>

===Student Notes===

====Assignment 2====
*Aliases: A fake name used instead of something the real name
**In the assignment, we take one URL and replace with another URL
*Headers: Headers on the incoming request
**Stored in a headers object in the request object. The request object has lots of things, one of those is request.headers
**For the header logging, just iterate through the array of headers to log and check if it is in the request.headers object

====Objects====

=====JSON=====
*In Javascript, we can define an object using the following syntax:
<code><pre>var x = {a: 5, b: true};</pre></code>
*For JSON, we must use quotes as follows:
<code><pre>{“a”: “5”, “b”: “true”}</pre></code>

=====Dashes in Attribute Names=====
<code><pre>
//This must be quoted, otherwise it will produce an error
x.anil-memory = “long ago”;
</pre></code>

<code><pre>
//The correct syntax is shown here. Use this when the attribute name contains irregular characters
x[”anil-memory”] = “long ago”;
</pre></code>

<code><pre>
//Quotes must also be used when creating an object name with a dash during the definition of an object
var x = {a: 5, b: true, "anil-memory" : "long ago"};
</pre></code>

=====Prototypes=====
<code><pre>
//This is an example of how inheritance works in JavaScript, is uses prototypes instead of being class based
//For another example see: http://robertnyman.com/2008/10/06/javascript-inheritance-how-and-why/)
Object -> [Function Object]
Object.prototype -> {}
Object.prototype.what = “yep”;
x.what -> yep
</pre></code>

*To make use of object prototyping, use <code>Object.create()</code>
<code><pre>
//Object.create() is used in a similar fashion to the way "new" is used, it creates a new object.
//(x is going to be prototype for z)
z = Object.create(x); -> {}
z -> {}
z.whoami -> x
z[“anil-memory”] -> ‘long ago’
z.b -> true
</pre></code>

*What is "new" for? What does "new" do:
<code><pre>
var f = function(a) {this.myval = a};
var g = new f(7);
g -> {myval: 7}
</pre></code>

*What actually happened here was “this” became an empty object. When you use "new", "this" becomes and empty object with the prototype of the object it was constructed from. Don’t use "new", too confusing. Use <code>Object.create()</code> instead.
*How to tell if you are supposed to use "new": if a function starts with a capital letter, it is meant to be used as a constructor function. In this case, use "new"

=====Checking Object Properties=====
*Method for checking just its own property:
<code><pre>
x.hasOwnProperty(“what”); -> false
x.hasOwnProperty(“b”) -> true
</pre></code>
*This matters is when users are going to provide the keys for the object

=====Use of "this"=====
*When used from within a call to a method, <code>this</code> refers to the object that the method belongs to
<code><pre>
var f = function(p) { console.log("this.whoami = " + this.whoami); }
f -> [Function]
x.f = f; -> [Function]
x.f(); -> this.whoami = x;

var y = {p:2, whoami: “y”};
y.f = f;
y.f(); -> this.whoami = y
</pre></code>

*The call method calls a function using the argument provided as the value of <code>this</code>
*Example: https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Function/call
<code><pre>
var k = {k:7, whoami: “k”};
f.call(k); -> this.whoami = k
</pre></code>

*When <code>this</code> is used in a function, it refers to the global object

<code><pre>
var f = function(a) {this.myval = a};
f -> [Function]
var a = {};
a.f = f;
a.f(3);
a -> {f: [Function], myval: 3}
f(5); -> undefined
</pre></code>

*The last line of the code above sets the myval attribute of the global object to 5.
*When used not in context of an object, you get the global object (which is almost always what you do NOT want to do)

====Form Demo====

=====Setup=====

*We will be using the Express framework to set up a web application
*In a terminal, enter:
<code><pre>
mkdir testapp
cd testapp
sudo npm install express-generator –g
express –version
express
npm install
</pre></code>

*npm is the Node packet manager
**"npm install" installs all the needed files in order for programs to run, it knows how to do this because the dependencies of the application are stored inside the package.json file
*To run the application, you can enter <code>bin/www</code>

=====Folder Structure=====
*Routes: code that’s going to run for different incoming requests
**This code will be in Routes/index.js
*Views: templates
**We will be working with Jade templates
*Routes/index.js :This directs which scripts to be called for when you go to certain pages.
*bin: this contains the startup script for the server
*Public: This is where all of the static resources are stored
**We can put HTML, stylesheets, images, etc in here

WebFund 2016W Lecture 9

2016-02-09T05:42:22Z

LeeCroft:

==Video==

The video from the lecture given on February 4, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec09-04Feb2016.mp4 is now available].

==Notes==

===Agenda===

*We are now a week away from the midterm (February 11th, 2016)
*The solutions for Assignment 3 will be reviewed in class on Tuesday, February 9th
*In this class the focus was on extending the functionality of tinyweb.js to that of form-demo.js and examforms.js in lieu of introducing new topics before the midterm

===Version Control===

*Question: How many people have used CVS or Github previously? Has version control been covered in previous courses?
**Majority of class indicated that version control has not been a topic covered in previous courses.
*Pros of version control:
**The ability to create backups and branch code
**Github is a great resource for powerful collaboration, code review, and code management for open source and private projects. Public projects are always free. (https://github.com/ )
*Git was originally developed for the Linux kernel.
*Code Academy has a Github tutorial - https://www.codecademy.com/learn/learn-git

*Diff command was recommended – analyzes two files and prints the lines that different. (diff file1.js file2.js)
**e.g diff ~/Documents/Class/WebFund/2016W/code tinwebserver.js tinywebserver_solution

===tinyweb.js vs form-demo.js===
*They can both serve static files
*tinyweb.js:
**tinyweb.js serves static content from the directory specified as the options directory
**A file on disk is served for a particular URL. As a result, the content always remains the same (static content)
*form-demo.js:
**form-demo serves static content form a public directory
**Can handle form submissions... this depends upon handling dynamic content since the whole point is to change the state of the web server
**Supports jade templates
**Can run code to generate pages... meaning that there are functions that are run every time a request is made for a particular URL (dynamic content)
***e.g in form-demo/routes/index.js:

<code><pre>
router.get('/', function(req, res, next) {
res.render('index', { title: 'COMP 2406 Simple form demo' });
});
</pre></code>

===Coding Exercise - Adding More Functionality To tinyweb.js===
*To get started, it’s better to initially create data structures and then create code to support the data structures
*First step of emulating form-demo was to create the framework for the routing functions (‘/’ , ‘/add’, “/list’).
**Must follow the definition of <code>var route_get...</code>, <code>var route_post...</code> and <code>var_route_list...</code> functions in order to avoid the generation of “Type Error: the Route is not a function error.

<code><pre>
// routes is a mapping of the url’s we want to come in and the code we want to run when the url’s are run.
var routes = {
‘/’: route_index,
‘/add’ : route_add,
‘/list’ : route_list
}
</pre></code>

*The following code was copied from index.js:

<code><pre>
router.get('/', function(req, res, next) {
res.render('index', { title: 'COMP 2406 Simple form demo' });
});

router.post('/add', function(req, res) {
var obj = { name: req.body.name,
city: req.body.city,
country: req.body.country,
birthday: req.body.birthday,
email: req.body.email };
state.push(obj);
res.render('add', { title: 'Person just added', item: obj });
});

router.get('/list', function(req, res) {
res.render('list', { title: 'People Listing', items: state});
});
</pre></code>

*The above code was pasted into tinyweb-forms.js after <code>var options;</code> and before the definition of the <code>routes</code> object
*The code was adapted as shown below
**Note the <code>next</code> argument was removed from functions since this argument is not needed in our version

<code><pre>
//The purpose of these functions is to render a template using jade and respond with the rendered HTML
//We will be making modifications to achieve this

//This one has been partially changed and now provides a response with a static web page
var route_index = function(req, res) {
//res.render('index', { title: 'COMP 2406 Simple form demo' });
respond(req, res, 200, “<html><body><h1>Success!</h1></body></html>”,”text/html”); //tiny web page
}

var route_add = (function(req, res) {
var obj = { name: req.body.name,
city: req.body.city,
country: req.body.country,
birthday: req.body.birthday,
email: req.body.email };
state.push(obj);
res.render('add', { title: 'Person just added', item: obj });
}

var route_list = (function(req, res) {
res.render('list', { title: 'People Listing', items: state});
}
</pre></code>

*To get the above functions called, the code to tinyweb_forms.js was modified in the <code>request_handler</code> function just after the following if statement:

<code><pre>
if(options.aliases.hasOwnProperty(url)){
myConsole.log(“Aliasing “ + url + “ to “ + options.aliases[url]);
url = options.aliases[url];
}
</pre></code>

*The code that was added is:

<code><pre>
if (routes.hasOwnProperty(url)) {
theRoute = routes[url]; // look it up and send the info the return and response
return theRoute(request, response); // arguments used from var request_handler
}
</pre></code>

*We do not use the serve file function since we are generating content dynamically
*To make sure the code ran correctly, the tinyweb-options.json, error.html and tinyweb-forms.js files were moved to the same folder
*Furthermore, the tinyweb-options.json file was modified from

<code><pre>
{
“host”: “localhost”,
“port”: 8000,
”index”: ”index.html”,
”docroot” : “.”,
“logged-headers”: [”user-agent”, “referrer”],
”errorpage”: ”error.html”,
”aliases”: {
”/index.html”: “/hello.html”,
“/”: “/hello.html”,
“/really/cool.html”: “/notcool.html”
}
}
</pre></code>

*To:

<code><pre>
{
“host”: “localhost”,
“port”: 8000,
”index”: ”index.html”,
”docroot” : “.”,
“logged-headers”: [”user-agent”, “referrer”],
”errorpage”: ”error.html”,
”aliases”: {
”/index.html”: “/”,
“/hello.html”: “/”,
“/really/cool.html”: “/notcool.html”
}
}
</pre></code>

*In order to render the Jade template, the following line was added at the top of the script:

<code><pre>var jade = require(‘jade’);</pre></code>

*In order to load Jade in this way, you must run the following command in the folder that contains the tinyweb-forms files:

<code><pre>npm install jade</pre></code>

*A node_modules sub folder will be created
*The jade files from form-demo/views(templates) can now be moved over to the tinyweb-forms folder (main folder)
*The <code>route_index</code> function is then modified as follows to in order to use the Jade template
**Note that an async function is used to ensure this is done in a non-blocking fashion... we want the webserver to remain responsive for other incoming requests

<code><pre>
var route_index = function(req, res) {

var content, template, options;

var render_template = function(err, template){
content = jade.render(template, options);
respond(req, res, 200, content, ”text/html”);
}

options = {pretty: true,
title: 'COMP 2406 Tinyweb Forms Demo'};

return fs.readFile(“index.jade”, ”utf-8”, render_template);
}
</pre></code>

*When the code above was run errors that were generated were:
**Error #1:
***Error: Jade:1
***>1 extends layout….the “filename” option is required to use ”extends” with ””relative” paths... because it doesn’t know what file to load
**Solution #1: add a reference to index.jade

<code><pre>
var route_index = function(req, res) {

var content, template, options;

var render_template = function(err, template){
content = jade.render(template, jade_options);
respond(req, res, 200, content,”text/html”);
}

jade_options = {pretty: true,
title: 'COMP 2406 Tinyweb Forms Demo',
filename: ‘index.jade’};

return fs.readFile(jade_options.filename, ”utf-8”, render_template);
}

// The calls to fs.readFile(), jade.render() and respond() are all needed to get the functionality of the res.render() which was contained exam-forms
</pre></code>

**Error #2:
***A 404 error was generated for style.css
**Solution #2:
***Copy the stylesheet over to the form-demo folder
***Once this was done, prettier fonts were generated and the 404 error was no longer generated
**Error and Solution #3: Unexpected identifier
***Generated because there was a comma missing at the end of title: <code>'COMP 2406 Tinyweb Forms Demo',</code> in the <code>jade_options</code> object
*Next, we moved on to modifying code to redirect the /add route to /list

<code><pre>
var route_add = (function(req, res) {
// var obj = { name: req.body.name,
// city: req.body.city,
// country: req.body.country,
// birthday: req.body.birthday,
// email: req.body.email };
//state.push(obj);
//res.render('add', { title: 'Person just added', item: obj });

myConsole.log(“302” + “\t” + req.method + “\t” + req.url);

res.writeHead(302, {
“Content-Type”: “text/html”
“Location”: “/list”
});
return res.end();

respond(req, res, 200, content, “text/html”);
}

var route_list = (function(req, res) {
res.render('list', { title: 'People Listing', items: state});
}
</pre></code>

*By setting the appropriate HTTP response code and response headers, we indicate to the client's browser that it should make another request in order to achieve the redirection
*In examforms.js, <code>res.redirect(‘/list’);</code> handles this same functionality for redirection

====To Summarize====

*The changes that we made in this exercise include:
**Giving tinyweb the ability to run code to dynamically generate response for incoming requests
**Bringing in the functionality to render templates and respond with the rendered HTML
**Implementing redirection in response to some requests
*The code in this exercise does not:
**Save the state of the data (the data does not persist after the server is shut down)
**Store the state object (the people added in the form)
**Render the contents of the list

===Additional Notes===
*Don’t use globals for data unless you are thinking explicitly that the data will be shared
**e.g the <code>state</code> array used in this code may not be well-suited as a global variable
*Web servers have databases so that they can persistently store their data (don’t want the data to go away) and concurrently access the data
**This is a better alternative for storing the state data
*The code modifications made in this exercise to bridge the gap between tinyweb.js and examforms.js were very specific
**examforms.js is far more robust in terms of how it’s coded

WebFund 2016W Lecture 8

2016-02-05T15:11:11Z

LeeCroft: /* Using Jade */

==Video==

The video from the lecture given on February 2, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec08-02Feb2016.mp4 is now available].

==Notes==

===Agenda===

* Spoke/explained the solutions for [[WebFund 2016W: Assignment 2|assignment 2]].
* Answered questions about the upcoming [[WebFund 2016W: Assignment 3|assignment 3]].

===Topics (from assignment 2 & 3)===

* What does the regular expression on line 92 of tinywebserver.js do? (Answer: Prevents breaking out of the current directory by using <tt>..</tt> and <tt>~</tt>)
* What does path.normalize do? (Answer: Tries to fix a malformed path. e.g. <tt>/home//dave/</tt> becomes <tt>/home/dave/</tt>)
* What is Jade? (Answer: Templating language.)

==Student Notes==

===General===
*Code from Tutorial 4 will be used on the midterm
*Know and understand what the code does and master it

===Debugging===
*One way to deal with potential errors is to surround code with a try/catch block
*To further track down the problem, use the Node debugger
*In class, there was a problem when reading/parsing the JSON option file
**Using the debugger, we saw that everything was fine parsing it
**The issue was that the errorpage did not exist from the working directory
**Make sure to have all the required files in the correct directory
**also, make sure you are working on the correct version of your code (in class, we were modifying the wrong code while debugging)

===Jade===

====General====
*Jade is a templating language for HTML
*HTML and Jade have a one-to-one correspondence (for the most part)
*See: [http://jade-lang.com jade-lang.com] for documentation
*Jade is parsed on the server side, NOT the client side
*In our example (form-demo), form data comes from the body (object) of the post request (object)
**The bodyparser is used to parse form data from the body of the request that has been received. The parsed data is then passed into a Jade template to be rendered along with the other Jade code into HTML

====The extends keyword====
*How does the <code>extends</code> keyword work in Jade?
**We write: <code>extends JADE_FILE</code>
**This allows a template to extend a parent template. It can then override fill in pre-defined blocks of content
**Jade supports template inheritance via the <code>block</code> and <code>extends</code> keywords. A block acts as a placeholder in the parent template which can be replaced in the child templates
**Allows for reuse of Jade templates

====Using Jade====
*To render template with the Jade engine, we must tell the app to use the engine
**Specify that the app should use Jade: <code>app.set('view engine', 'jade');</code>
**Specify where to find the Jade templates: <code>app.set('views', __dirname);</code>
*To render a Jade code, you can use <code>jade.render('Jade code...', {/*Options go here*/})</code>
**These options can be found on the documentation site: [http://jade-lang.com/api/ http://jade-lang.com/api/]
*To render a template, you can use <code>jade.renderFile('Template', {/*Options go here*/})</code>
*To respond to a request with a rendered page, you can use <code>res.render('Template', {/*Data to pass to the template goes here*/});</code>
**Note, you don't specify a .jade extension
*The <code>render()</code> method will convert the jade code or template to HTML

===module.exports and require()===
*You can set what will be exported form a file using <code>module.exports = SOME_OBJECT;</code>
*<code>module.exports</code> is the data "returned" by the file so that it can be used in other .js files
*This is loaded using <code>require(FILE)</code>
**Note that this can only be used to load the same file once

===Code Modifications===
*Removing the line: <code>app.use(bodyParser.urlencoded({extended:false}));</code>
**The list page has no user people listed
**The form data wasn't parsed, so the page didn't have values to display
*removing <code>module.exports=router;</code> from index.js:
**The following line in app.js will no longer have the expected return value to store in the routes variable: <code>var routes = require('./routes/index');</code>
**This means that none of the routing will work

===HTML Button Types===
*submit -> submits form data to server(default option)
*reset -> resets all controls to initial values
*button -> no default behaviour (usually used for client-side JavaScript)

WebFund 2016W Lecture 8

2016-02-05T15:10:03Z

LeeCroft: /* Using Jade */

==Video==

The video from the lecture given on February 2, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec08-02Feb2016.mp4 is now available].

==Notes==

===Agenda===

* Spoke/explained the solutions for [[WebFund 2016W: Assignment 2|assignment 2]].
* Answered questions about the upcoming [[WebFund 2016W: Assignment 3|assignment 3]].

===Topics (from assignment 2 & 3)===

* What does the regular expression on line 92 of tinywebserver.js do? (Answer: Prevents breaking out of the current directory by using <tt>..</tt> and <tt>~</tt>)
* What does path.normalize do? (Answer: Tries to fix a malformed path. e.g. <tt>/home//dave/</tt> becomes <tt>/home/dave/</tt>)
* What is Jade? (Answer: Templating language.)

==Student Notes==

===General===
*Code from Tutorial 4 will be used on the midterm
*Know and understand what the code does and master it

===Debugging===
*One way to deal with potential errors is to surround code with a try/catch block
*To further track down the problem, use the Node debugger
*In class, there was a problem when reading/parsing the JSON option file
**Using the debugger, we saw that everything was fine parsing it
**The issue was that the errorpage did not exist from the working directory
**Make sure to have all the required files in the correct directory
**also, make sure you are working on the correct version of your code (in class, we were modifying the wrong code while debugging)

===Jade===

====General====
*Jade is a templating language for HTML
*HTML and Jade have a one-to-one correspondence (for the most part)
*See: [http://jade-lang.com jade-lang.com] for documentation
*Jade is parsed on the server side, NOT the client side
*In our example (form-demo), form data comes from the body (object) of the post request (object)
**The bodyparser is used to parse form data from the body of the request that has been received. The parsed data is then passed into a Jade template to be rendered along with the other Jade code into HTML

====The extends keyword====
*How does the <code>extends</code> keyword work in Jade?
**We write: <code>extends JADE_FILE</code>
**This allows a template to extend a parent template. It can then override fill in pre-defined blocks of content
**Jade supports template inheritance via the <code>block</code> and <code>extends</code> keywords. A block acts as a placeholder in the parent template which can be replaced in the child templates
**Allows for reuse of Jade templates

====Using Jade====
*To render template with the Jade engine, we must tell the app to use the engine
**Specify that the app should use Jade: <code>app.set('view engine', 'jade');</code>
**Specify where to find the Jade templates: <code>app.set('views', __dirname);</code>
*To render a Jade code, you can use <code>jade.render('Jade code...', {/*Options go here*/})</code>
**These options can be found on the documentation site: [http://jade-lang.com/api/ http://jade-lang.com/api/]
*To render a template, you can use <code>jade.renderFile('Template', {/*Options go here*/})</code>
**Note, you don't specify a .jade extension
*To respond to a request with a rendered page, you can use <code>res.render('file', {/*Data to pass to the template goes here*/});</code>
*The <code>render()</code> method will convert the jade code or template to HTML

===module.exports and require()===
*You can set what will be exported form a file using <code>module.exports = SOME_OBJECT;</code>
*<code>module.exports</code> is the data "returned" by the file so that it can be used in other .js files
*This is loaded using <code>require(FILE)</code>
**Note that this can only be used to load the same file once

===Code Modifications===
*Removing the line: <code>app.use(bodyParser.urlencoded({extended:false}));</code>
**The list page has no user people listed
**The form data wasn't parsed, so the page didn't have values to display
*removing <code>module.exports=router;</code> from index.js:
**The following line in app.js will no longer have the expected return value to store in the routes variable: <code>var routes = require('./routes/index');</code>
**This means that none of the routing will work

===HTML Button Types===
*submit -> submits form data to server(default option)
*reset -> resets all controls to initial values
*button -> no default behaviour (usually used for client-side JavaScript)

WebFund 2016W Lecture 8

2016-02-04T23:06:41Z

LeeCroft:

==Video==

The video from the lecture given on February 2, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec08-02Feb2016.mp4 is now available].

==Notes==

===Agenda===

* Spoke/explained the solutions for [[WebFund 2016W: Assignment 2|assignment 2]].
* Answered questions about the upcoming [[WebFund 2016W: Assignment 3|assignment 3]].

===Topics (from assignment 2 & 3)===

* What does the regular expression on line 92 of tinywebserver.js do? (Answer: Prevents breaking out of the current directory by using <tt>..</tt> and <tt>~</tt>)
* What does path.normalize do? (Answer: Tries to fix a malformed path. e.g. <tt>/home//dave/</tt> becomes <tt>/home/dave/</tt>)
* What is Jade? (Answer: Templating language.)

==Student Notes==

===General===
*Code from Tutorial 4 will be used on the midterm
*Know and understand what the code does and master it

===Debugging===
*One way to deal with potential errors is to surround code with a try/catch block
*To further track down the problem, use the Node debugger
*In class, there was a problem when reading/parsing the JSON option file
**Using the debugger, we saw that everything was fine parsing it
**The issue was that the errorpage did not exist from the working directory
**Make sure to have all the required files in the correct directory
**also, make sure you are working on the correct version of your code (in class, we were modifying the wrong code while debugging)

===Jade===

====General====
*Jade is a templating language for HTML
*HTML and Jade have a one-to-one correspondence (for the most part)
*See: [http://jade-lang.com jade-lang.com] for documentation
*Jade is parsed on the server side, NOT the client side
*In our example (form-demo), form data comes from the body (object) of the post request (object)
**The bodyparser is used to parse form data from the body of the request that has been received. The parsed data is then passed into a Jade template to be rendered along with the other Jade code into HTML

====The extends keyword====
*How does the <code>extends</code> keyword work in Jade?
**We write: <code>extends JADE_FILE</code>
**This allows a template to extend a parent template. It can then override fill in pre-defined blocks of content
**Jade supports template inheritance via the <code>block</code> and <code>extends</code> keywords. A block acts as a placeholder in the parent template which can be replaced in the child templates
**Allows for reuse of Jade templates

====Using Jade====
*To render template with the Jade engine, we must tell the app to use the engine
**Specify that the app should use Jade: <code>app.set('view engine', 'jade');</code>
**Specify where to find the Jade templates: <code>app.set('views', __dirname);</code>
*To render a template, you can use <code>jade.render('file', {/*Options go here*/})</code>
**These options can be found on the documentation site: [http://jade-lang.com/api/ http://jade-lang.com/api/]
**Note, you don't specify a .jade extension
**To respond to a request with a rendered page, you can use <code>res.render('file', {/*Data to pass to the template goes here*/});</code>
*The <code>render()</code> method will convert the jade template to HTML

===module.exports and require()===
*You can set what will be exported form a file using <code>module.exports = SOME_OBJECT;</code>
*<code>module.exports</code> is the data "returned" by the file so that it can be used in other .js files
*This is loaded using <code>require(FILE)</code>
**Note that this can only be used to load the same file once

===Code Modifications===
*Removing the line: <code>app.use(bodyParser.urlencoded({extended:false}));</code>
**The list page has no user people listed
**The form data wasn't parsed, so the page didn't have values to display
*removing <code>module.exports=router;</code> from index.js:
**The following line in app.js will no longer have the expected return value to store in the routes variable: <code>var routes = require('./routes/index');</code>
**This means that none of the routing will work

===HTML Button Types===
*submit -> submits form data to server(default option)
*reset -> resets all controls to initial values
*button -> no default behaviour (usually used for client-side JavaScript)

WebFund 2016W Lecture 7

2016-01-31T17:25:08Z

LeeCroft:

==Video==

The video for the lecture given on January 28, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec07-28Jan2016.mp4 is now available].

==Notes==

===In-class Poll===

https://pollev.com/anilsomayaji565

==Code==

<source lang="javascript" line>
var http = require('http');

var request_handler = function(request, response) {

var body_content;

console.log("I got a request for: " + request.url);

response.writeHead(404, {
"Content-Type": "text/html"
});

body_content = "You requested: " + request.url;

response.write("<html>\n<head>\n<title>Hello!</title>\n</head>" +
"<body><h1>Hello world!</h1>" +
"<p>" + body_content + "</p>" +
"<script>alert('where does this print');</script> " +
"\n</body>\n</html>\n");

return response.end();
}

var server = http.createServer(request_handler);

server.listen(8000, "localhost", function() {
return console.log("Server listening at http://localhost:8000/");
});

console.log("Finished!");
</source>

==Student Notes==

===Even Smaller Web Server===
*Can we go smaller than tiny?
*Bare bones:
**http module
**server
***Made using <code>http.createServer()</code>
***We include the port number and host in the <code>server.listen()</code> to actually start up the server
**request handler
***A function provided as an argument to the <code>createServer()</code> method which is call for every request that comes in
*With these pieces, the server can accept requests but does not yet send any responses
*We need responses so we can hard code something and put it in request handler
**We can make the server respond with a webpage (html file), using <code>response.writeHead()</code>, <code>response.write</code> and <code>response.end()</code>
*At this point we have a very basic web server that works
*We can also display what the request was, using <code>request.url</code>
*All tiny web server has over this is error handling, dealing with files and providing different responses based on the requests
**This additional functionality can be easily added
**Providing different responses based on the requests, for example, can be achieved by using the <code>request.url</code> to decide how to respond

===Client Side JavaScript===
(We will be playing with this much more after the midterm)
*We can include client-side JavaScript in a response by using the <code><script></code> tag within the HTML code that we send to the client
*This will allow for JavaScript code to be embedded in the webpage and run in the client's browser
*Example:
<code><pre><script>alert(‘where does this print?’);</script></pre></code>

===Polls===
*When are A and B added?
<code>y = f(A+B);</code>

A and B are added before <code>f()</code> is called. It needs the value of <code>A+B</code> as an argument to the function.

*In what order do these execute?
<code>writeFile(“/tmp/foo.txt”, myData, console.log(“Finished!”));</code>

Finished is printed then <code>myData</code> is written to <code>/tmp/foo.txt</code>. No callback function is given (<code>console.log()</code> returns <code>undefined</code>). It evaluates the value of all the arguments before calling the function.

===Express===
*We use an <code>app</code> object in Express to represent the server as an application
*Don’t actually call <code>createServer()</code> or explicitly write <code>server.listen()</code>
*No request handler. Instead have <code>app.get()</code> (essentially the same thing though)
*How do we use this like tiny web server to find files in directories and then just serve them?
**<code>app.use(express.static(‘.’));</code>
***This makes a static web server that just serves files from whatever directory you want
**Besides setup, tiny web server is essentially this one line in Express
*Express does some basic error handling for us

===Node Debugger===
*When running the Node debugger, it breaks on first line of program by default
*You can write <code>debugger;</code> in your code to create additional break points
**This is useful to see if your code is actually running at particular points

===Code Examples===
<code><pre>fs.readFile(“classtest”, “utf-8”, callback);
console.log(“Finished!”);
5;</pre></code>
This prints:
<code><pre>Finished!
5
Got data: I have to put something here (callback function)</pre></code>
It prints in this order because the callback function passed as an argument is not executed immediately (no parentheses means it is not being called). The application therefore continues to do other work while it waits for the callback to be called.

<code><pre>var f = function(x) { console.log(“Inside f, x = ” + x); return x + 30; };
console.log(“Running f: “ + f(3));</pre></code>
This prints:
<code><pre>Inside f, x = 3
Running f: 30</pre></code>
The argument inside the <code>console.log()</code> function gets evaluated first, resulting in <code>“Inside f, x = 3”</code> being printed before <code>“Running f: 30”</code>.

WebFund 2016W Lecture 7

2016-01-29T01:55:57Z

LeeCroft:

WebFund 2016W Lecture 5

2016-01-24T19:51:22Z

LeeCroft:

==Video==

The video from the lecture given on January 21, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec05-21Jan2016.mp4 is now available].

==Notes==

===formtest.html===

<source lang="html4strict">
<html>
<head>
<title>A simple web page</title>
</head>
<body>
<h1>A SIMPLE WEB PAGE (really)</h1>
<p>It isn't so bad.</p>
<form action="useName" method="POST">
What is your name?
<input name="theName" type="text"></input>
<input type="submit"value="Done!" />
</form>
</body>
</html>
</source>

===Student Notes===
====Assignment 1====
NOTE: The solutions for assignment 1 are posted on the wiki

*In the async version "Finished" should be output using a callback function for <code>wrtieFile()</code> so that it only runs when the work is truly complete
**Note that writing <code>fs.writeFile('output.txt', data, console.log('Finished'));</code> will not use the <code>console.log()</code> as a callback. Instead, this passes the return value of the <code>console.log()</code>, which is undefined, as the callback.
*If you want something to happen after a specific function you have to use a callback
*Values given to the arguments of callback functions come from the function that calls the callback, in other words, the function to which the callback was passed as an argument
*To find out what the specific arguments for a given callback are you can check the documentation for that function
*Always cite any sources you use!!
*Don't use fancy things that you don't understand while coding. It is a bad habit and can lead to errors that you won't know how to solve
*There might be a midterm question about the <code>filter()</code> method

====Quick Talk About Functions====
*Functions can be written in different ways
<code>function theFunction(a,b){...}</code>

<code>function(err, param){...}</code> --> anonymous function

<code>var theFunction = function(a,b){...}</code> --> this syntax is a reminder that functions are just objects
*Anonymous functions are useful for one time use
*You can make your own callback functions

====Forms====
What is a form used for?
*A form is used when you want to send information to the server
*The HTML syntax for a from is simply
<code><pre>
<form action="someAction" method="GET or POST">
What is your name? <input type="text" name="theName"/>
<input type="submit" value="Done!"/>
</form>
</pre></code>

*A form has a method attribute and an action attribute
**The method is used to specify the type of HTTP request that will be sent when the form is submitted (usually either a POST or a GET)
***POST is used to send data to the server
***GET is used to ask the server for data
**The action is similar to a path on the server where the request is sent
*Tinywebserver.js can't handle data being sent to it (can't handle POST)
*Tinywebserver.js can currently only handle GET requests
**There is no code in it that can process POST requests
**Only processes files
*A form also has nested input tags
**A self closing tag (<code><input ... /></code>) can be used when there is no content that would appear between the opening and closing tags
**An input tag (of type text) displays an input field that the user can use to enter data
**For example, after the question "What is your name?" there is an input tag, which is being used to display an input field
**The second input tag, <code><input type="submit"...></code>, is used to display a button that will submit the data to the server
**The 'name' attribute of the input tag identifies the name of the input
***This becomes part of the URL (when the method is GET) and is paired with the data that was entered in the input field
***The name of the field gets sent to the server along with the value
***Ideally we don't want this to all be in the URL
****By using a POST request instead, the data is stored elsewhere in the request when it is sent to the server
****You can see what is sent by using the developer tools on your browser and looking under 'Params'
*We need code in the Web server to handle forms
*The first way to process forms was by using Perl
**The problem with Perl is that it's not efficient
**Need to run Perl for every incoming request
**Solution was to run Perl continuously and integrate it into the server
*Then PHP was used
**People liked PHP in the beginning because it was integrated with HTML
***PHP allowed for programmatic parts to be added to HTML documents
**A lot of the modern Web uses PHP (Facebook, Wordpress)
***They have separate files that are just PHP, it is not embedded in the webpage.
*Why is PHP not a good thing?
**It is messy because there is code mixed in with the document
*The modern solution is to separate the code and the HTML by leaving place holders in the document

====Jade====
*Has PHP-like functionality
*This is a template engine that we will be using in class
*It uses HTML and its own embedded programming language
*Instead of nesting Jade uses indentation
*Instead of start and end tags Jade has only starts tags and without angle brackets
*The server runs the Jade code
*The browser only sees HTML
*Anything that can be done in HTML can be done in Jade

====Express====
*This is a Web application framework we will be using in class
*This is useful because we won't have to work from scratch
*Node.js is the platform that Express runs on

WebFund 2016W Lecture 2

2016-01-22T15:36:47Z

LeeCroft:

==Video==

The video from the lecture given on January 12, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec02-12Jan2016.mp4 is now available].

==Notes==

===In-class Notes===

<pre>
Lecture #2
----------
JavaScript

* no declared types
- strings
- objects
- arrays
- numbers (floats)
- boolean

With Node
- asynchronous (non-blocking) I/O
</pre>

===Student Notes===

* Administration Reminders
** [[Fundamentals_of_Web_Applications:_Winter_2016_Course_Outline#Course_Information|Office hours have been added.]]
** [[Fundamentals_of_Web_Applications:_Winter_2016_Course_Outline#Grading|Midterm date has been tentatively created.]]
** [[WebFund_2016W:_Assignment_1|Assignment #1 has been posted.]]

====JavaScript: As a Language====

* JavaScript has '''nothing''' to do with Java.
** Why is it even called JavaScript then? At first it wasn't; it was developed as Mocha, changed to LiveScript and then finally renamed to JavaScript due to an agreement with Sun (acquired by Oracle).<ref>[http://stackoverflow.com/a/2475528 Stack Overflow - Why is JavaScript called JavaScript, since it has nothing to do with Java?]</ref>
* The only similarity between Java/JavaScript is some of the syntax, but that can be said for a lot of languages.

* What's different about JavaScript?
** No declared data types; it's a loosely typed language.
*** Instead of int i = 0, it's simply var i = 0.
** There's seven data types<ref>[https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures MDN - JavaScript Data Structures]</ref> (examples in brackets)
*** Boolean (true, false)
*** Number (3.1337, 1337, etc. There's no integer or long, everything is a float)
*** String ("Dave")
*** Symbol
*** Object
*** Null
*** Undefined

====How should I learn JavaScript?====

* Practice! While attending the lectures is a great start, you ''will'' have to practice outside of class. This will not be graded, but will ultimately affect your performance in the course (and in the workplace).
** [https://www.codecademy.com/learn/javascript Codecademy] has a lot of good resources for beginners.
** For more advanced students, check on [https://github.com/search?l=javascript&o=desc&q=stars%3A%3E1&s=stars&type=Repositories GitHub] to see if there's any projects you're interested in contributing to. If you plan to work in web development, it's very likely you will be asked in an interview if you've contributed to any open source projects (having a good portfolio will help a ''lot'' in getting a first job).
* The Mozilla Developer Network maintains a [https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference JavaScript reference] that is recommend. ''(Note: There is no official documentation for Google Chrome's V8 engine, although there is an [http://v8.paulfryzel.com/docs/master/index.html unofficial reference].)''

====Working with Variables====

If you've previously worked with Java, the first large difference you've noticed is likely that variables are very "flexible".

* Trying to use things that have not been declared can produce some behaviour that you might not expect:
** Referencing a variable that has not been declared will produce a reference error
** Assigning a value to a variable that has not been declared will implicitly declare the variable at the global level (avoid doing this by explicitly declaring your variables)
** Accessing a property that does not exist in an object will give you <code>undefined</code>
* Similar to how in other languages, a symbol x can refer to an object which can encapsulate other object or types of data, in JavaScript, an object can hold various types of data in its properties (e.g., <tt>x.y</tt>, <tt>x.y.z</tt>, <tt>x.y.x.qwerty</tt>, etc).
* <tt>x.size</tt> (dot notation) is the same as <tt>x["size"]</tt> (array notation).
* Functions are also a special type of object and they can be stored in a variable.
* JavaScript engines use a garbage collector, so you do not need to manually deallocate memory (in C you need to). Unlike Java, you can "reuse" a variable name and use a completely different type; e.g. creating <tt>var x = 1337</tt> and then <tt>x = "Dave"</tt> later is completely valid.

====Regular Expressions (Regex)====

While this topic is technically outside of the course material, you will likely need to use it in some assignments. Regex is the same across different languages, so you will not need to relearn it all for every language.

https://developer.mozilla.org/en/docs/Web/JavaScript/Guide/Regular_Expressions

====Synchronous vs. Asynchronous====

* Synchronous (blocking) functions are where a task must be fully complete before the next task can be begin.
** Examples:
*** Waiting on a single network connection before accepting additions requests.
*** Waiting for a file to be fully loaded before displaying it in a text editor.
* Asynchronous (non-blocking) functions are where the program is allowed to continue with other tasks while another one is in process.
** Examples:
*** Accepting new network connections while another connection is already in progress.

====Node.js====

Node.js is designed to provide a server side environment of JavaScript. Instead of using Google's V8 JavaScript inside Chrome/Chromium, Node.js uses the V8 engine directly. Because Node.js is meant to be used outside of the browser, there's some slight differences.

* Contains a package manager called npm to manage dependencies (libraries).
* Has specialized libraries for networking, file system access, etc that you usually wouldn't find in a browser (mostly for security reasons). These functions have their own set of [https://nodejs.org/api/ documentation].
* Has no default graphic frontend such as WebKit (although there are libraries to provide that if you wish).

=====Why use Node.js?=====

* JavaScript makes it easier to write asynchronous functions.
* You can reuse your knowledge on client-side JavaScript to apply to server-side Node.js applications.

=====Getting started with Node.js=====

To open up Read-Eval-Print-Loop (REPL), run <tt>node</tt> in a terminal. For more details on setting your environment up, see [[WebFund_2016W:_Tutorial_1#Running_the_VM|Tutorial 1]].

====References====
<references/>

WebFund 2016W Lecture 4

2016-01-21T15:36:28Z

LeeCroft:

==Video==

The video for the lecture given on January 19, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec04-19Jan2016.mp4 is now available].

==Notes==

===In-class Notes===

<pre>
Lecture 4
---------
Web servers

What does a web server do?
Listens on port 80, 443, or other port
INPUT: HTTP requests
OUTPUT: HTTP responses & requested documents

tinywebserver.js
- written in JavaScript (Node)
- can only serve files
- understands only a few file types

So tinywebserver is how web servers worked in 1995

What changed since then is now web servers run code to generate responses

Why use frameworks?
- because we don't want to build up HTML from scratch
- and, we want logic that is abstracted away
- and, we want routing
- have someone else decide which function to call in response to requests

But for this week, bare bones

Debugging

</pre>

===Student Notes===
====Announcements====
*Assignment 1 due tomorrow (now pushed back to Thursday before class)
*Basics for tutorial 2 and assignment 2 are up!

====Web Servers====
*What does a web server do?
**Listens on port 80 (usually HTTP), 443 (usually HTTPS) or some other port.
***If the web server is listening on a non-standard port, you must specify the port in the URL (using something like “localhost:8080”)
**Input: HTTP requests
**Output: HTTP responses with requested documents
***HTTP responses always have a response code
****200: OK, used for successful request
****404: Not Found
**In other words it services requests
*Need root privileges to listen on ports <~1024
**Do not do development as root if you can, it is risky!
*Popular Web Servers: Apache, NginX

====Tiny Web Server====
*Written in JavaScript(Node)
*Can only serve files! (most modern servers can do more)
*Understands limited file types
*Similar to web servers from 1995
*To try to understand what the code is doing, we can start by looking at the bottom of the file

=====http.createServer()=====
*Apart from the loading of some modules and the definitions of some objects and functions, the first real code to run in this script is the <code>createServer()</code> method
*We are calling a method of the <code>http</code> object (loaded from a module) in order to create an object which will act as the server (which we then store in the <code>server</code> variable)
*This method takes a single argument, <code>request_handler</code>
**<code>request_handler</code> is a function which is intended to handler incoming requests
**By providing this function as the argument, we are telling the server to call the function every time a request comes in

=====server.listen()=====
*Here we are calling the <code>listen()</code> method of the <code>server</code> object that was just created
*This tells the server to start listening for incoming requests
*The first two arguments are telling the server where to listen (on port 8080 of localhost)
**This information came from the <code>options</code> object which was defined at the top of the file
*The third argument is a callback function
**This will run once the server has started listening
**It simply prints a message in the console to indicate that the server is ready

=====request_handler()=====
*This function gets called every time a request comes in (because we provided it as an argument to the <code>http.createServer()</code> method)
*There are two arguments
**<code>request</code> is an object which represents the request that was received
**<code>response</code> is an object which represents the response that will be sent
*At a high level, this function checks for the existence of a requested file and responds with either the file or an error code
*There are many nested callback functions here
**Although this makes the code more difficult to read, why is this preferable to doing things synchronously?
***This ensures that the <code>request_handler()</code> function will return very quickly
***All the bulk of the work is done asynchronously
***This allows the server to be available to handle another request again very quickly
***This is important in order to keep the server responsive for numerous requests

====MIME Types====
*MIME types are used to specify the type of data being sent via HTTP
*In Tiny Web Server, there is an object <code>MIME_TYPES</code> defined near the top of the file which contains the types of files that the server recognizes
*The <code>MIME_TYPES</code> object is used in the <code>get_mime()</code> function to determine the type of a requested file
*Similarly, the <code>MIME_TYPES</code> object is used to set the Content-Type header of the HTTP response that is sent by the server
*We can break ‘image/jpg’ by changing it to ‘imaage/jpg’
**The web browser does not know how to handle it because it does not recognize imaage
*Web browsers can deal with broken content type for some types of files
**For example, putting a jpg as png in <code>MIME_TYPES</code>, will not break the content and Firefox is smart enough to render it correctly as jpg

====Response Headers====
*Some headers of the HTTP response are set in the <code>respond()</code> function
**This is done using the <code>response.writeHead()</code> method
**We could add more headers here if we want to
*Some other headers are set for us by the code in the <code>http</code> module
*We can see these headers from within the dev tools of the web browser after a response has been received
*There are not many headers used here because Tiny Web Server is not very complex

====Frameworks and Modern Servers====
How have things improved since 1995?
*We use code to generate responses to requests
**We often use frameworks to do much of the work for us

Why use frameworks?
*Because we don’t want to build up HTML from scratch
*Abstracted away logic
*We also want routing (dispatcher): have someone else decide which function to call in response to requests

====Debugging====
*You can use the node debugger by typing <code>node debug script.js</code> in the terminal
*This will stop the program on the first line of code that is run
*You can continue execution of the code by typing <code>c</code>
*You can learn about other useful commands by typing <code>help</code>
*In particular, you can type <code>repl</code> to enter a read-evaluate-print loop from which you can inspect or modify objects or other data
*You can also write <code>debugger;</code> in the script to create a break point where the debugger will pause the execution of the program
*There is also a debugger available in the web browser
**This is for client side code and will not work for any server-side code

====HTML====
*You can use the inspect in the web browser dev tools to see the the tree-structure HTML which makes up a web page
*<code><div></code> tags are used for sections in HTML
*<code><a></code> tags are used for links (review on CodeAcademy)

WebFund 2016W Lecture 3

2016-01-18T15:24:24Z

LeeCroft:

==Video==

The video from the lecture given on January 14, 2015 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec03-14Jan2016.mp4 is now available].

==Notes==

===In-class Notes===

<pre>
Lecture 3
---------
What is the web?

Web != Internet

So, what is the Internet?!

Network of networks

What is a network?

Networks allow computers to talk to each other

WiFi, Ethernet, LTE - different standards for computer networks

Layers of networking (OSI)
* Physical layer
* Data link layer
* Network layer <----
* Application layer (and such)

Internet Protocol (IP)
- packet-based protocol (NOT a stream or continuous protocol)

Old POTS
- wire from your house to a switching station
- wires between switching stations A and B
...
...
- wires between switching station Y to Z
- wire from switching station to house

Packet switching networks multiplex physical wires over time. (Multiple communication connections over one wire by them taking turns.)

The "turn" is the packet

IP protocol is a "best effort" protocol. NO error correction or retransmission.

TCP means Transmission Control Protocol
turns packets into a reliable data data stream

Network Firewalls block "unwanted traffic"
- really, block all those protocols that aren't safe on the open Internet

TCP adds a "port" to the IP address
- the port identifies which program to talk to
- some are ephemeral (temporary)
- others are "well known", meaning protocol on that port is standardized

Port 25 is for SMTP (email)
80: HTTP
443: HTTPS
22: ssh

Web is transmitted over HTTP

Major HTTP commands
* GET: get documents from server
- can be CACHED
* POST: send form contents to server
- are not cached

You can GET almost anything
- MS Word .doc
- tiff
- PDF

But the real web is
- JPEG, GIF, PNG for images
- CSS for style sheets
- HTML for content
- JavaScript for code

.swf is flash. Flash is not a web standard. AVOID

</pre>

===Student Notes===

====The Web====

Web != Internet

We have to talk about networking...

*The internet is a network of networks (inter networks)
*What is a network?
**Networks allow computers to talk to each other
*There are many different technologies and standards used to connect computers. They work in different ways. This is partially due to frequency rights and legacy issues. Some well-known ones include:
**Ethernet
**WiFi
**LTE (Long Term Evolution)
*Layers of networking (OSI model)
**Physical Layer (e.g. wire, or frequency)
**Data Link Layer
**Network Layer
***Interconnection standard
****one standard = INTERNET PROTOCOL!!
****TCP/IP
****all layers have to transport IP (internet protocol)
**Transport Layer
**Session Layer
**Presentation Layer
**Application Layer

=====Internet Protocol (IP)=====

*Packet-based protocol (how do you interpret bits in and out)
*NOT a stream or continuous protocol like POTS: plain old telephone system
*Old POTS:
**Wire from origin home to switching station
**Wires between switching stations
**...
**Wire from switch station to destination home
*Problem: there's a 1:1 ratio of wires to connections. This is inefficient.
*Goal is to share 1 wire for 10 phone calls!
*Packet switching networks multiplex physical wires over time (Multiple connections by them taking turns)
**People have to take turns
**The "turn" is the packet
**It's actually pretty good if not all people are talking at once, then it slows down
**[https://en.wikipedia.org/wiki/Network_packet https://en.wikipedia.org/wiki/Network_packet]

=====IP Addresses=====
*Any computer that's on a network has an IP address
*We currently use IPv4 addresses but we are transitioning towards IPv6
*Packets have an IP address source and destination
*The address for localhost is 127.0.0.1
**That's the computer itself
*Domain name system (DNS) translates between IP addresses and "English" (e.g. www.google.ca)

=====TCIP/IP=====
*What happens when packets gets lots or corrupt? (electrical interference/microwave/etc)
*The IP protocol is a "best effort" protocol. NO error correction or re-transmission
*For this, we use TCP/IP, not just IP
*TCP means Transmission Control Protocol
*Turns packets into a RELIABLE data stream
*TCP is layer that takes care of re-transmitting as necessary. It'll come through reliably in the same order.

=====Firewalls and Ports=====
*A Network Firewall blocks "unwanted traffic"
**Really blocks all those protocols that aren't safe on the open internet
*Firewalls blocked everything except "The Web"
**Not that it's so great, it's just the only thing that wasn't blocked on the open internet.

*TCP adds a port to the IP addresses on both sides
*The port identifies which program to talk to
**Some ports are temporary (ephemeral)
**Others are "well known", meaning the protocol on that port is standardized
***Port 25: SMTP (email)
***Port 80: HTTP
***Port 443: HTTPS
***Port 22: SSH (secure shell)
*Modern web browsers don't show the HTTP (protocol) or port number (:80 for HTTP or :443 for HTTPS)
*Try localhost:631 //on Linux
*"Standard" ports are below 1024
*[www.iana.org www.iana.org] //these are the guys that decide the protocols for the ports
*Port 80 is hypertext transfer protocol, the web is transmitted over HTTP (protocol)
**HTTP over TCP over IP over datalink layers, over physical layers

=====HTTP=====
*Browsers (clients) communicate to servers through HTTP requests (typically GET requests)
**GET requests have request headers which consist of key-value pairs
**Responses also have headers of key-value pairs
*Over the history of the web, different companies implemented different standards
**Like IE6, Web pages would be required to send different info, web pages would break
*Normally, in Web dev, you're working a level above HTTP (you use the existing standard)
*What are the major HTTP requests:
**GET
***Client GETs resource from the server
***Can be CACHED (people in the middle can hold copies)
**POST
***POST (send) information (such as form contents) to the web server
***Are NOT CACHED
*Doing a reload on a GET is safe
**The resource is simply requested again
**This usually results in a cached copy being used
*Doing a reload on a POST isn't safe (you typically do not want to do this action twice)
**They permanently alter the server state
*You can GET almost anything!
**.doc
**.tiff
**.anything!
*Typical file types that are requested include:
**HTML for content
**CSS (style sheets)
**JavaScript for code
**JPEG, GIF, PNG for images
*.swf is flash
**We don't like flash (bad security, performance hog, new browsers don't have support for plugins)
**NOT STANDARD, AVOID
*Plugins inject arbitrary code into browsers, but "real web content" (HTML CSS, JS, Media) is ENOUGH.
**Modern JS has fully-fledged Socket Support

=====HTML Page Example=====
In a new .html file:
<code><pre>

<!DOCTYPE html> 
<html>
<head> 
<title>A simple web page</title>
</head>

<body>
<h1>A SIMPLE WEB PAGE</h1>
<p>It isn't so bad.</p>
</body>
</html>
</pre></code>

WebFund 2016W: Assignment 1

2016-01-15T02:54:57Z

LeeCroft:

This assignment is due on January 20th on cuLearn.

In this assignment you will writing two programs, <tt>words-sync.js</tt> and <tt>words-async.js</tt>. These programs will make use of synchronous and asynchronous I/O is the manner of the programs from [[WebFund 2016W: Tutorial 1|Tutorial 1]].

These programs should both behave as follows:

* They should take two arguments, an input text file and an output file to create. Thus you run them by typing "node words-sync.js document.txt wordlist.txt", with document.txt being the file to be read and wordlist.txt is the file to be created.
* The output file should contain a list of words in the input file, one word per line, with each line terminated by a newline character ('\n'). These words should be sorted in ascending order, as ordered by the standard JavaScript [https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort Array.sort() method] (with no comparison function specified). All duplicates should be removed and there should be no blank lines.
* Once the output file has been created, the program should output "Finished!" to the console.
* For the purposes of this program, words can be defined as anything matching the \w operator. In other words, to split the document into words you may call [https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/split split()] using "/\W//" as the split [https://developer.mozilla.org/en/docs/Web/JavaScript/Guide/Regular_Expressions regular expression].
* You may assume that the input file exists and is readable and the output file is writable. In other words you do not need to do any error checking (although it is appreciated).

In creating this program, don't forget that you can use standard JavaScript methods such as [https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Array/push push()], [https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Array/pop pop()], [https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Array/shift shift()], and [https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/Array/unshift unshift()]. And of course you'll need the methods from [https://nodejs.org/api/fs.html the Node fs module].

There are 10 points, allocated as follows:
* 3 points for producing the correct output given well-formed input.
* 2 points for correctly structuring the code in the synchronous version.
* 5 points for correctly structuring the code in the asynchronous version. In particular, "Finished!" should only be output when the program is truly finished!

Please submit your answers as a zip file called "<username>-comp2406-assign1.zip", where username is your MyCarletonOne username. This zip file should uncompress to a directory called "<username>-comp2406-assign1" and inside this directory should be three files: "words-sync.js", "words-async.js", and a text file "comments.txt".

"comments.txt" should:
* list any references you used to complete the assignment (documentation web sites, for example),
* list your collaborators, and
* optionally, should discuss any issues or concerns you had when completing this assignment.
Remember that while you are allowed to collaborate with others, plagiarism is not allowed. In other words you should not be copying any code or data directly from anywhere, and any assistance or inspiration should be credited. Any significant code similarity (beyond the code already given to you) will be considered plagiarism and will be reported to the Dean.

WebFund 2016W Lecture 1

2016-01-09T02:59:34Z

LeeCroft:

==Video==

The video for the lecture given on January 7, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec01-07Jan2016.mp4 is now available].

==Notes==

===In-Class Notes===

<pre>
Lecture 1
---------

web servers
- apache, nginx

web browsers (clients)
- firefox, chrome, IE, Safari

web protocols
- HTTP

When you view a web page:
- browser sends a GET request for the URL to a web server
- web server sends back a document
- browser renders document and makes more GET requests

Web pages are, at base, HTML documents with other stuff

Major web renderers
- Gecko (Firefox)
- Blink (Chrome, fork of WebKit)
- Trident (MSHTML)
- WebKit
- EdgeHTML (fork of Trident)

Web renderers interpret:
- HTML
- CSS
- JavaScript

(and they incorporate images and videos)

HTML - specifies basic structure and content of "page"
CSS - makes it pretty (fonts, layout, etc)
JavaScript - code

Read-Eval-Print loop (REPL)
</pre>

=== Student Notes ===

==== Course Logistics ====

* This course is about ''forming a conceptual model of the web''
* Notes and other content from previous years can be found on the wiki
* Anil & TA's have office hours
* Tutorials are required
** Come see a TA promptly if you miss one (do the work beforehand)
* Mark Breakdown Highlights
** Assignments (lowest grade dropped)
** Tutorials (20%!! Very important)
* Collaboration
** Don't transfer bits to each other
** Don't submit the same thing
** Cite!!

==== Strategies for succeeding ====

* Don't get discouraged! This course is targeted at a wide demographic so there might be stuff you don't understand.
* Try to keep up in your understanding of the general course content. You should not be trying to piece things together from many code snippets from web searches. The core material is all provided for you.

==== Basics of web technology ====

* Linux environment provided, but you don't have to use it
* Node
** Chrome's JavaScript engine + some extra capabilities such as networking and disk I/O
** Used for server-side development
* Important concepts:
** Web servers - Apache, Nginx
** Web browsers (clients) - Firefox, Chrome, IE, Safari, Edge
*** Browsers are more complicated than servers
*** Rendering Engines
*** Gecko (Firefox), Blink (Chrome, fork of Webkit), Trident (MSHTML), WebKit, EdgeHTML (Opera used to have it's own)
*** Web renderers interpret HTML, CSS, and JavaScript
**** HTML specifies layout
**** CSS makes it pretty
**** JavaScript is the code!
**** We use JS on the server BECAUSE it's on the client (other languages could be used instead)
** Web protocols - HTTP
* What happens when you load the wiki?
** Homeostasis server is running Ubuntu, with Apache configured to run MediaWiki with PHP
** Anil's laptop talked to his server (over HTTP), retrieved the main page
*** You can see the details of this in the web developer tools under the network tab! (press Ctrl-Shift-I)
*** Also loaded a bunch of resources
** In general, for any site:
*** Browser sends a GET request for the URL to a web server
*** Web server sends back a document
*** Browser renders document and makes more GET requests

==== Node REPL/Intro to JavaScript ====

* Most interpreted languages have a REPL (Read-Eval-Print-Loop, interactive mode)
* Node is JIT compiled
* Simple arithmetic/equality examples:
<code><pre>>2+2
4
>5 === 2
false
>5 === 5
true
>var x;
undefined
>x
undefined
>x + 3
NaN</pre></code>
* Typos are hard to find in JS (you can use a variable without defining it)
* Example: Reading a file
** <code>console.log()</code> prints to the console (ex. <code>console.log("Hi there!")</code>)
** You have to "require" the "fs" module: <code>var fs = require("fs");</code>
** To read a file (synchronously): <code>fs.readFileSync("testfile.txt");</code>, returns bytes.
*** Specify an encoding, utf-8 is pretty standard. Otherwise it doesn't know what kind of text (or even binary data!) it is
*** Specifically in our example: <code>var contents = fs.readFileSync("testfile.txt", "utf-8");</code>
** What does sync mean?
*** Synchronicity: Code is done line by line, waiting for operations to complete
*** In this example, our code waited for the readFileSync to end
* Asynchronous programming:
** You have to specify a callback function - what to do when the operation is done. For our example: <code>var callback = function(err, data) {console.log("Data read:\n" + data);}</code>
** To use this callback: <code>fs.readFile("testfile.txt", "utf-8", callback);</code>
** This will call "callback" when the data is done reading

WebFund 2016W Lecture 1

2016-01-09T02:58:45Z

LeeCroft:

==Video==

The video for the lecture given on January 7, 2016 [http://homeostasis.scs.carleton.ca/~soma/webfund-2016w/lectures/comp2406-2016w-lec01-07Jan2016.mp4 is now available].

== Topics & Readings ==

* Course Logistics
* Strategies for succeeding in 2406
* Basics of web technologies
* Node REPL/Intro to JavaScript

==Notes==

===In-Class Notes===

<pre>
Lecture 1
---------

web servers
- apache, nginx

web browsers (clients)
- firefox, chrome, IE, Safari

web protocols
- HTTP

When you view a web page:
- browser sends a GET request for the URL to a web server
- web server sends back a document
- browser renders document and makes more GET requests

Web pages are, at base, HTML documents with other stuff

Major web renderers
- Gecko (Firefox)
- Blink (Chrome, fork of WebKit)
- Trident (MSHTML)
- WebKit
- EdgeHTML (fork of Trident)

Web renderers interpret:
- HTML
- CSS
- JavaScript

(and they incorporate images and videos)

HTML - specifies basic structure and content of "page"
CSS - makes it pretty (fonts, layout, etc)
JavaScript - code

Read-Eval-Print loop (REPL)
</pre>

=== Student Notes ===

==== Course Logistics ====

* This course is about ''forming a conceptual model of the web''
* Notes and other content from previous years can be found on the wiki
* Anil & TA's have office hours
* Tutorials are required
** Come see a TA promptly if you miss one (do the work beforehand)
* Mark Breakdown Highlights
** Assignments (lowest grade dropped)
** Tutorials (20%!! Very important)
* Collaboration
** Don't transfer bits to each other
** Don't submit the same thing
** Cite!!

==== Strategies for succeeding ====

* Don't get discouraged! This course is targeted at a wide demographic so there might be stuff you don't understand.
* Try to keep up in your understanding of the general course content. You should not be trying to piece things together from many code snippets from web searches. The core material is all provided for you.

==== Basics of web technology ====

* Linux environment provided, but you don't have to use it
* Node
** Chrome's JavaScript engine + some extra capabilities such as networking and disk I/O
** Used for server-side development
* Important concepts:
** Web servers - Apache, Nginx
** Web browsers (clients) - Firefox, Chrome, IE, Safari, Edge
*** Browsers are more complicated than servers
*** Rendering Engines
*** Gecko (Firefox), Blink (Chrome, fork of Webkit), Trident (MSHTML), WebKit, EdgeHTML (Opera used to have it's own)
*** Web renderers interpret HTML, CSS, and JavaScript
**** HTML specifies layout
**** CSS makes it pretty
**** JavaScript is the code!
**** We use JS on the server BECAUSE it's on the client (other languages could be used instead)
** Web protocols - HTTP
* What happens when you load the wiki?
** Homeostasis server is running Ubuntu, with Apache configured to run MediaWiki with PHP
** Anil's laptop talked to his server (over HTTP), retrieved the main page
*** You can see the details of this in the web developer tools under the network tab! (press Ctrl-Shift-I)
*** Also loaded a bunch of resources
** In general, for any site:
*** Browser sends a GET request for the URL to a web server
*** Web server sends back a document
*** Browser renders document and makes more GET requests

==== Node REPL/Intro to JavaScript ====

* Most interpreted languages have a REPL (Read-Eval-Print-Loop, interactive mode)
* Node is JIT compiled
* Simple arithmetic/equality examples:
<code><pre>>2+2
4
>5 === 2
false
>5 === 5
true
>var x;
undefined
>x
undefined
>x + 3
NaN</pre></code>
* Typos are hard to find in JS (you can use a variable without defining it)
* Example: Reading a file
** <code>console.log()</code> prints to the console (ex. <code>console.log("Hi there!")</code>)
** You have to "require" the "fs" module: <code>var fs = require("fs");</code>
** To read a file (synchronously): <code>fs.readFileSync("testfile.txt");</code>, returns bytes.
*** Specify an encoding, utf-8 is pretty standard. Otherwise it doesn't know what kind of text (or even binary data!) it is
*** Specifically in our example: <code>var contents = fs.readFileSync("testfile.txt", "utf-8");</code>
** What does sync mean?
*** Synchronicity: Code is done line by line, waiting for operations to complete
*** In this example, our code waited for the readFileSync to end
* Asynchronous programming:
** You have to specify a callback function - what to do when the operation is done. For our example: <code>var callback = function(err, data) {console.log("Data read:\n" + data);}</code>
** To use this callback: <code>fs.readFile("testfile.txt", "utf-8", callback);</code>
** This will call "callback" when the data is done reading

WebFund 2015W Final Exam Review

2015-04-11T22:54:05Z

LeeCroft: /* Notes */

==Audio==

The audio from the exam review given on April 10, 2015 [http://homeostasis.scs.carleton.ca/~soma/webfund-2015w/lectures/comp2406-2015w-examreview-10Apr2015.m4a is now available]. (Note: the file does not play in Firefox unfortunately.)

==Notes==

The final exam will be based on [http://homeostasis.scs.carleton.ca/~soma/webfund-2015w/code/exam-notes.zip exam-notes], a simplified version of the solutions to tls-notes from Assignment 10.

There will be very little in terms of code that you must write on the exam, no more than a few lines for small implementations. While your answers can be pure code, it is better to provide written answers with high level explanations.

Review of the different contexts for functions (the main difference between these contexts is the use of the keyword 'this'):
<ul>
<li>Regular function call - keyword 'this' refers to the global object</li>
<li>Method call - keyword 'this' refers to the object for which the method was called</li>
<li>Constructor call - keyword 'this' refers to the object being constructed</li>
<li>Apply call - keyword 'this' refers to the parameter supplied to the <code>apply()</code> call indicating what the function is being applied to</li>
</ul>

<h3>Differentiation between client side and server side</h3>
On the client, HTML is rendered by the browser and used to construct the DOM. JavaScript can also be provided to be run inside the browser and interact with the DOM.

On the server, JavaScript is run as an application to provided all necessary server functionality.

<h3>Communications between client and server</h3>
The two sides (client and server) talk to each other using HTTP. HTTP is built on top of other technologies but it is essentially just structured data being sent back and forth.

A server cannot initiate communication by sending a response to a client. The client must first send a request to the server, the server may then respond to that request. Requests are sent to a server due to actions taken by a client such as trying to load a page or a user clicking on a button which submits a form. A response typically contains HTML, CSS or JavaScript code. A response may cause subsequent requests to be made in order to obtain linked resources (images, stylesheets, scripts, etc).

<h3>Older servers (static servers)</h3>
On an older web server, you would have files such as HTML documents, images, stylesheets, etc. These are static resources which could be served 'as is' by the server. A new web server such as one implemented in Node can easily serve resources as a static web server but its main benefit is the ability to run additional code when a request comes in. JavaScript can be used in this context to generate HTML or do many other things.

<h3>JavaScript execution environments</h3>
Node.js (run on the server) is an execution environment for JavaScript. Similarly, there is an execution environment for JavaScript in web browsers to allow for the client to run JavaScript code. These executions environments have differences between them. The browser environment is sandboxed so that it cannot access the rest of the computer (file I/O, networking).

JavaScript run in the browser is meant for two main purposes:
<ul>
<li>Manipulation of the DOM - Changing the structure of the page, updating the interface, etc...</li>
<li>Sending AJAX requests (asynchronous GET and POST requests) - This allows the client to communicate with the server in the background, bypassing the need to use HTML elements to trigger communications as well as the need to reload a page when receiving data.</li>
</ul>

<h3>Exam code</h3>
<h4>Differentiation between client side code and server side code</h4>
<ul>
<li>The code in index.js is all run on the server.</li>
<li>Anything in the public folder is sent directly to the client. Scripts and HTML code sent from this folder are run on the client.</li>
<li>Jade templates are rendered in HTML on the server and then sent to the client to have the HTML rendered by the browser.</li>
</ul>

<h4>What does register.js do?</h4>
First, recall that $ refers to jQuery, it represents the jQuery object. By providing a single parameter to jQuery as a function, we are defining an on-load function. This means that the function is run once the page is fully loaded. The first line in this function uses jQuery to find an HTML element with an ID of 'register' and calls to <code>on()</code> method of the element to define a function for a click event. The function finds a form element in the DOM and changes its action from /login to /register and then submits the form.

<h3>Other notes and tips</h3>
You must fully understand the definition of routes that occurs in index.js. The router methods such a <code>get()</code> and <code>post()</code> are run only once when the index.js file is run during the startup of the server. These methods register a callback function for the specified route so that whenever a request comes to the server for that route, the callback function is run. The callback function is not run until such a request comes in and so it could potentially never be run if a request never comes in for that route.

It is important to recognize that functions are treated as first class objects in JavaScript. They can be treated just as other objects can, meaning that they can even be used as parameters for another function.

Notice that the exam code does not use bcrypt. The passwords are stored in plain text. This makes the code much simpler but it is a very bad practice. This makes your database very vulnerable as all passwords are completely visible in plain text if anyone happens to get access to the database. You should never do this.

Recall that hashing and encryption are different. With encryption, you can decrypt the data back to its original format if you have the proper key. With hashing, you are producing a fixed length representation of the data based on a hashing function. You do not expect to be able to retrieve data that has been hashed, only verify that something matches what you expect.

Expect questions on the exam about what causes certain requests and what type of response you would expect the server to produce and why.

==Code==

===routes/index.js===

<source line lang="javascript">
var express = require('express');
var router = express.Router();
var mongodb = require('mongodb');
var mc = mongodb.MongoClient;
var ObjectID = mongodb.ObjectID;

var notesCollection, usersCollection;

mc.connect('mongodb://localhost/exam-notes', function(err, db) {
if (err) {
throw err;
}

notesCollection = db.collection('notes');
usersCollection = db.collection('users');
});

router.post('/register', function(req, res) {
var username = req.body.username;
var password = req.body.password;

var checkInsert = function(err, newUsers) {
if (err) {
res.redirect("/?error=Unable to add user");
} else {
res.redirect("/?error=User " + username +
" successfully registered");
}
}

var checkUsername = function(err, user) {
if (err) {
res.redirect("/?error=unable to check username");
} else if (user === null) {
var newUser = {
username: username,
password: password
};
usersCollection.update({username: username},
newUser,
{upsert: true},
checkInsert);

} else {
res.redirect("/?error=user already exists");
}
}

usersCollection.findOne({username: username}, checkUsername);
});

router.get('/', function(req, res) {
if (req.session.username) {
res.redirect("/notes");
} else {
res.render('index', { title: 'COMP 2406 Exam Notes Demo',
error: req.query.error });
}
});

router.get('/notes', function(req, res) {
var username = req.session.username;

if (username) {
res.render("notes.jade", {username: username,
title: username +"'s Notes"});
} else {
res.redirect("/?error=Not Logged In");
}
});

router.post('/login', function(req, res) {
var username = req.body.username;
var password = req.body.password;

var authenticateUser = function(err, user){
if (err || user === null || password !== user.password) {
res.redirect("/?error=invalid username or password");
} else {
req.session.username = username;
res.redirect("/notes");
}
}

usersCollection.findOne({username: username}, authenticateUser);
});

router.post('/logout', function(req, res) {
req.session.destroy(function(err){
if (err) {
console.log("Error: %s", err);
}
});
res.redirect("/");
});

router.get('/getNotes', function(req, res) {
var username = req.session.username;

var renderNotes = function(err, notes) {
if (err) {
notes = [{"title": "Couldn't get notes",
"owner": username,
"content": "Error fetching notes!"}];
}
res.send(notes);
}

if (username) {
notesCollection.find({owner: username}).toArray(renderNotes);
} else {
res.send([{"title": "Not Logged In",
"owner": "None",
"content": "Nobody seems to be logged in!"}]);
}
});

router.post('/updateNote', function(req, res) {
var username = req.session.username;
var id = req.body.id;
var title = req.body.title;
var content = req.body.content;

var checkUpdate = function(err, result) {
if (err) {
res.send("ERROR: update failed");
} else {
res.send("update succeeded");
}
}

if (username) {
if (id && title && content) {
notesCollection.update({_id: ObjectID(id)},
{$set: {title: title,
content: content}},
checkUpdate);
} else {
res.send("ERROR: bad parameters");
}
} else {
res.send("ERROR: not logged in");
}
});

router.post('/newNote', function(req, res) {
var username = req.session.username;
var newNote;

var reportInserted = function(err, notesInserted) {
if (err) {
res.send("ERROR: Could not create a new note");
} else {
res.send(notesInserted[0]._id);
}
}

if (username) {
newNote = {title: "Untitled",
owner: username,
content: "No content"};

notesCollection.insert(newNote, reportInserted);
} else {
res.send("ERROR: Not Logged In");
}
});

module.exports = router;
</source>

===public/javascripts/register.js===

<source line lang="javascript">
$(function(){
$("#register").on("click",function(){
var $form = $("form");
$form.attr("action", "/register");
$form.submit();
});
});
</source>

===public/javascripts/notes.js===

<source line lang="javascript">
$(function() {
var insertNotesIntoDOM = function(userNotes) {
var i, theNote;

$(".notes").remove();

if (userNotes.length < 1) {
$("#notesList").append('<div class="notes">No notes!</div>');
return;
}

for (i=0; i < userNotes.length; i++) {
console.log("Adding note " + i);
theNote = userNotes[i];
$("#notesList").append('<div class="notes list-group-item ' +
'list-group-item-default"> ' +
'<a href="#" ' +
'class="notes list-group-item-heading" ' +
'id="edit' + i + '">' +
'<h4>' + theNote.title + '</h4></a> ' +
'<p class="list-group-item-text">' +
theNote.content + '</p>' +
'</div>');
$("#edit" + i).click(userNotes[i], editNote);
}
}

var getAndListNotes = function() {
$.getJSON("/getNotes", insertNotesIntoDOM);
}

var updateNoteOnServer = function(event) {
var theNote = event.data;
theNote.title = $("#noteTitle").val();
theNote.content = $("#noteContent").val();
noteListing();
$.post("/updateNote",
{"id": theNote._id,
"title": theNote.title,
"content": theNote.content},
getAndListNotes);
}

var editNote = function(event) {
var theNote = event.data;

$(".notesArea").replaceWith('<div class="notesArea" ' +
'id="editNoteDiv"></div>');
$("#editNoteDiv").append('<h2>Edit Note</h2>');

$("#editNoteDiv").append('<div><input type="text" ' +
'id="noteTitle"></input></div>');
$("#noteTitle").val(theNote.title);

$("#editNoteDiv").append('<div>' +
'<textarea cols="40" rows="5" ' +
'id="noteContent" wrap="virtual">' +
'</textarea></div>');
$("#noteContent").val(theNote.content);

$("#editNoteDiv").append('<button type="button" ' +
'class="btn btn-success" ' +
'id="updateNote">' +
'Update</button>')
$("#updateNote").click(theNote, updateNoteOnServer);
}

var editNewNote = function(result) {
var theNote = {"title": "",
"content": ""};

if (result.indexOf("ERROR") === -1) {
theNote._id = result;
editNote({data: theNote});
} else {
noteListing();
}
}

var newNote = function() {
$.post("/newNote", editNewNote);
}

var noteListing = function() {
$(".notesArea").replaceWith('<div class="notesArea" ' +
'id="listNotesDiv"></div>');

$("#listNotesDiv").append('<h2>Notes</h2>');
$("#listNotesDiv").append('<div id="notesList" class="list-group well">' +
'<a class="notes list-group-item">Loading Notes...</a>' +
'</div>');

$("#listNotesDiv").append('<button id="newNote"' +
' class="btn btn-primary"' +
' type="button">' +
'New Note</button>');
$("#newNote").click(newNote);

getAndListNotes();
}

noteListing();
});
</source>

===views/layout.jade===

<source line lang="html5">
doctype html
html
head
title= title
link(rel='stylesheet', href='/libs/bootstrap/css/bootstrap.css')
link(rel='stylesheet', href='/libs/bootstrap/css/bootstrap-theme.css')
link(rel='stylesheet', href='/stylesheets/style.css')
script(src='/libs/jquery.js')
script(src='/libs/bootstrap/js/bootstrap.js')
block header
body(role="document")
block content
</source>

===views/index.jade===

<source line lang="html5">
extends layout

block header
script(src="javascripts/register.js")

block content
h1= title
p Welcome to #{title}
- if(error)
div.error #{error}
div
form(action="/login", method="post")
div.form-group
label(for="username") Username
input.form-control.limitwidth(type="text", name="username")
div.form-group
label(for="password") Password
input.form-control.limitwidth(type="password", name="password")
button.btn.btn-default(type="submit") Login
button#register.btn.btn-warning(type="button") Register
</source>

===views/notes.jade===

<source line lang="html5">
extends layout

block header
script(src="javascripts/notes.js")

block content
div.page-header
h1#pageTitle #{username}'s Account
p#pageWelcome Welcome #{username}

form(action="/logout", method="post")
button.btn.btn-info(type="submit") Logout

div.notesArea
</source>

===views/error.jade===

<source line lang="javascript">
extends layout

block content
h1= message
h2= error.status
pre #{error.stack}
</source>

WebFund 2015W Lecture 20

2015-04-10T00:43:30Z

LeeCroft:

==Video==

The video from the lecture given on Monday, March 23, 2015 [http://homeostasis.scs.carleton.ca/~soma/webfund-2015w/lectures/comp2406-2015w-lec20-23Mar2015.mp4 is now available].

==Notes==
<h3>Socat</h3>
You may be familiar with netcat which is somewhat of a swiss-army knife for dealing with network traffic. Socat is a better netcat. Socat allows you to view all of the traffic going back and forth over a TCP/IP connection.

To use socat, you must:
<ul>
<li>Download and install it.</li>
<li>Run your web server.</li>
<li>Run socat as a mirror of the web server on a different port.</li>
<li>Use a web browser to connect to the socat mirrored server.</li>
</ul>

Using socat, we can see all of the contents of requests and responses going back and forth between the client and the server.

'>' indicates the beginning of a request going to the server and shows the timestamp and how many bytes are going in. Similarly, '<' indicates the beginning of a response and has the same information.

HTTP traffic uses a text format that comes from Windows machines (which uses text output with a carriage return and a line feed) by default. With the text output in a Unix box we see '\r' for the carriage return and then an actual line break for the line feed.

If a request or response has a body then then will be an extra line break in between the head and the body.

Requests show their method at the start. This will usually be GET or POST. The responses will start with 'HTTP/1.1 followed' by the response code (eg 200 OK).

<h3>Assignment 5</h3>
We want to make queryNotes.js from storeNotes.js and fileStats.js.

fileStats.js takes in a command line argument. We can copy the code for fileStats.js into queryNotes.js. We would then create variables in which to store our expected command line arguments. The arguments can be parsed using familiar functions such as <code>length()</code>, <code>slice()</code>, <code>indexOf()</code>. By checking if a string such as "-output=" exists in an argument, we can determine whether or not it corresponds to the output argument. Because this process must be done for each argument, we would want to make it into a function.

As you write code, you may find that some parts are redundant and can be factored out into functions. The important part of working on these assignments is not the solution but how to reason your way towards the solution.

<h3>Assignment 6</h3>
To complete Question 1, you could go to views/notes.jade and change <code>p!= userNotes[i].content</code> to <code>p = userNotes[i].content</code>. However, by doing this, it will no longer be possible to include links in the notes. Instead you must write your own function to sanitize the user input.

The parsing of the links can be handled by once again using the string functions that you are familiar with. Using <code>split("[")</code> separates each segment containing a link. You can then split these segments at the "]" and use <code>indexOf(" ")</code> to find the division between the URL and the label. Once you have broken up all of the pieces, you can reassemble them into one string with HTML links included.

To update the username, you will need to create a new route to handle the changeusername request. Inside this route, you must verify that the new username is not already in use and then make the update. You must both update the owner of the notes in the MongoDB collection using the collection's <code>update()</code> method and you must update the session using <code>req.session.username = ...</code>.

<h3>Assignment 9</h3>
For Assignment 9, you need to capture a transcript of the HTTP request going out and the response returned by the ajax-notes server. You need to explain what was sent and why for both the requests and the responses. You must do this for the following routes:
<ul>
<li>GET /</li>
<li>GET /javascripts/notes.js</li>
<li>GET /getNotes</li>
<li>POST /updateNote</li>
<li>POST /newNote</li>
</ul>

WebFund 2015W Lecture 24

2015-04-07T17:14:15Z

LeeCroft:

==Audio==

The audio from the lecture given on Monday, April 6, 2015 [http://homeostasis.scs.carleton.ca/~soma/webfund-2015w/lectures/comp2406-2015w-lec24-06Apr2015.m4a is now available].

==Notes==

<h3>On-screen notes</h3>
====Administrative====

* Optional review session in SC 103 (Steacie Bldg), Friday, April 10, 2-3:30 PM
* Assignment 10 solutions will be discussed in the review session (and posted then)
* Assignment 8 and 9 solutions are posted
* Final exam is on Tuesday, April 14th in Alumni Hall, rows 1-13

====The Future====

Before the web
* The Internet started late 1960s-early 1970s
* Web is 1993
* LAN protocols: file sharing (NFS), directory services (yp), remote login (rsh), remote "desktop" (X windows)
* WAN protocols: file transfer (ftp), email (smtp), remote login (telnet), net news (UUCP)

Why the web?
* hypertext - linked information
* many older systems: MEMEX?, Xanadu, gopher, WAIS
* what was different about the web?
** presentation-oriented markup ("pretty" documents) - HTML
** IMAGES
** simple to implement renderer (browser) and network protocol (HTTP)
** stateless: scalable servers
** URLs

What's changed since the beginning of the web?
* CSS: prettier pages
* JavaScript: dynamic pages
* plugins came and went: Java, Flash, Silverlight...
* kitchen sink, aka HTML5 - video, video chat, sockets, location, touch, WebGL
** richer input and output
** in the framework of HTML, CSS, JavaScript

Why not just build separately?
* universal platforms tend to suck the oxygen out of the room

Web is the future

But won't anything else take over?
* evolution works, not revolution
* Java applets
* Flash
* Dart (now just compiled to JavaScript)

What about other technologies?
* run on the server
* translate for the browser

Other classes
* COMP 3000 (operating systems): what runs a browser or a web server
* COMP 3007 (programming paradigms): functional programming, which you kind of do in JavaScript
* COMP 3005 (databases): learn about SQL, data modelling
* COMP 3004: bigger programs, read code
* COMP 3804: see patterns underlying code

<h3>Student notes</h3>
<h4>Before the Web</h4>
Before the Web, there was the Internet. The Internet started around the late 60's / early 70's. The Web itself began around 1993.

Internet protocols before the Web were LAN (local area network) and WAN (wide area network).
LAN protocols:
<ul>
<li>File sharing (NFS)</li>
<li>Directory services (yp)</li>
<li>Remote login (rsh)
<li>Remote “desktop” (X windows)</li>
</ul>

WAN protocols:
<ul>
<li>File transfer (ftp)</li>
<li>Email (point to point) (smtp)</li>
<li>Remote login (telnet)</li>
<li>net news (UUCP)</li>
</ul>

Each of the services using these protocols required a client which knows the protocol in order to be used. Just as a web browser is a client, email requires a client, remote login requires a client, etc...

<h4>Beginnings of the Web</h4>
What was it that the Web added to all of this? Hypertext - linked information. There were many other previous systems which attempted to do this as well such as MEMEX, Xanadu, gopher, WAIS.

So what did the Web do differently from these?
<ul>
<li>The Web had more presentation-oriented markup (it made 'prettier' documents) and handled things like images.</li>
<li>It had a simpler implementation of its renderer (the borwser) and protocol (HTTP).</li>
<li>HTTP was also particularly beneficial because of the fact that it is stateless. This greatly helps the scalability of web servers since the protocol needs to do no extra work with states, it is all managed (or ignored) by the code on the server.</li>
<li>The standardization of URLs brought about uniformity and universal access of resources.</li>
</ul>

Side note:
The NeXTSTEP operating system is the OS on which modern Apple operating systems are based. The previous Apple OS was not able to browse the web without crashing and thus required NeXTSTEP system to fix this problem.

<h4>The modern Web and the future</h4>
What has changed now since the beginnings of the Web?
<ul>
<li>CSS - Providing the ability to better manage layout and make pages look nicer</li>
<li>JavaScript - Providing the ability to make dynamic pages</li>
<li>Plugins - Providing the ability to extend functionality (Things like Java, Flash, Silverlight... although these have somewhat come and gone)</li>
<li>The kitchen sink, aka HTML5 - Provides video, video chat, sockets, location, touch, WebGL... (Allows for richer input and output)</li>
</ul>

Things are now generally implemented within this framework (HTML, CSS, JavaScript) to extend functionality. This is done in the form of new tags, new functions, etc...

Things could be implemented separately and may have a better implementation if done in that way but we do not usually want to do this. Universal platforms tend to suck the oxygen out of the room. Windows was the universal platform before the Web, and before Windows, Unix sort of held this title. Having a universal platform provides simplicity and easy adaptability.

While the platform may continue to evolve and look different, the same basic pieces are likely to be used for quite some time. For example, while the communication protocol is continuing to change, it is still the same base HTTP concept.

Won't other newer technologies take over?
<ul>
<li>Evolution works, not revolution.</li>
<li>There have been technologies like Java applets and Flash but these have had issues and are proprietary software. They are mostly on the way out now.</li>
<li>Google created Dart to try to improve on JavaScript... now Dart is just compiled to JavaScript. Nobody wants to support multiple runtimes if it is not necessary.</li>
</ul>

What about other languages?
<ul>
<li>Other languages can be run on the server.</li>
<li>Code written in other languages to be run in the browser is cross-compiled. This can be done using asm.js: an extraordinarily optimizable, low-level subset of JavaScript.</li>
</ul>

If you look on Wikipedia, you can see that there are many server frameworks available. You can choose whatever matches your programming style and needs (although you may be stuck without a choice when working for a company that has already made their decision).

How does all of this relate to other courses that you will be taking (or have already taken)?
<ul>
<li>Comp 3000 (Operating systems): what runs a browser or a web server</li>
<li>Comp 3007 (Programming paradigms): functional programming, which you kind of do in
JavaScript)</li>
<li>Comp 3005 (Databases): SQL, data modelling, more sophisticated databases</li>
<li>Comp 3004: Bigger programs, reading code</li>
<li>Comp 3804: See underlying patterns in code and algorithms</li>
</ul>

WebFund 2015W Lecture 23

2015-04-07T16:09:06Z

LeeCroft:

==Audio==

The audio from the lecture given on April 1, 2015 [http://homeostasis.scs.carleton.ca/~soma/webfund-2015w/lectures/comp2406-2015w-lec23-01Apr2015.m4a is now available].

==Notes==

<h3>On-screen notes</h3>
Two topics for last two lectures:

* What are the fundamentals of web applications?
* Web applications everywhere

====Distributed Applications====

Applications that run on multiple computers
* separate memory, CPU, storage, communication (I/O)
* can apply this processes, virtual hosts, or physical hosts

Localhost is useful when you want a distributed app on one computer

* GET and POST can go essentially anywhere on the Internet
* allows for flexible combination of code and data

* web applications are the premier form of distributed applications today

* with distributed applications, you don't control all the computers "your" code runs on

====Structured Persistence====

Before this class, you worked with files

Files are unstructured (on modern systems)

On the web we use "databases"
* really, structured storage

Why structured?
* convenience
* speed for "complex" queries
* concurrency & distribution

====Front vs Back end====

* separate parts of app for...
** doing computation/storage/real work (back end)
** providing the interface (front end)

Can you do "real work" in the front end?
* Yes, but...
* Don't trust the answers

Front end is untrusted, back end is trusted

Modern web apps have many backends and can have multiple front ends (mobile, desktop)
* but "responsive design" says you should have one front end

So why put stuff in the front end?
* lower the load on the back end
* but main reason is latency (of network communication)

====Markup and Templates====

HTML, CSS: markup
Jade: template language

Markup is constraint or declarative programming

template languages output markup (are meta-markup)

====Trust====

Who do you trust?
* library, runtime authors
* content providers
* ad providers
* whatever else you add to a page

Trust means "who can screw you over"
* if they go bad how badly are you hurt

Amazing the web works at all

<h3>Student notes</h3>
<h4>Distributed applications</h4>
These are applications that run on multiple computers using separate memory, CPU, storage and communication(I/O). These computers don't have to be separate physical computers.

Our web applications these days are at least in three pieces:
<ul>
<li>One part that runs in the browser (client).</li>
<li>One part that runs as the server.</li>
<li>One part that runs as the database(s).</li>
</ul>

Then it goes up from there e.g: communications on twitter(your server interacting with twitter's)

Importance of distributed apps:
<ul>
<li>GET and POST can go essentially anywhere on the internet.
<ul>
<li>Allows for flexible combination of code and data.</li>
</ul>
</li>
<li>With distributed applications, you don’t control all the computer your code runs on. Often, you will use third party libraries to run code that is not even your own. For example, you can use features from Google Analytics in your application but it is their code running on their server; this distributes the burden on your application, making it ore distributed.</li>
<li>Web applications are the main form of distributed applications today.</li>
</ul>

Localhost is useful when you want a distributed app on your own computer. i.e: when you want to have multiple processes talk on the same system, but you want to use the same method as you would to talk to other computers - you talk to localhost.

Distributed applications typically communicate using an HTTP or HTTPS protocol as seen in assignment 9.

Nota Bene(NB)
Google makes their money from search advertisement and imbedded advertisement on other pages.
Look at the script tags of various web pages and see if you understand them.

The web is funded primarily by advertisement. The advertisement is done using JavaScript code. The JavaScript is running on untrusted browsers (machines of people that want to steal money) That's why the code is obfuscated (hard to read and figure out).

<h4>Structured persistence</h4>
Previously, you have handled persistent storage by using files. Files are unstructured(on modern systems). On the web, we use databases.

What are Databases? Databases are an organization system which enforce a structure on the stored data.

Why structured?
<ul>
<li>Convenience.</li>
<li>Speed for Complex Queries: This is primarily because files in the databases can be indexed on various attributes allowing for quick look-ups.</li>
<li>Concurrency and distribution: Key component of databases - concurrent access to data. This is not a simple task to manage. o not try to implement this yourself as you will probably do it poorly. Use an existing database suited to your needs.</li>
</ul>

<h4>Font vs back end</h4>
This is the concept of separating parts of apps into:
<ul>
<li>The back end - Where the real work is done</li>
<li>The front end - used for the interface</li>
</ul>

Can the real work be done in the front end?

Yes, but don't trust the answers. This is because the front end is running from an unknown source so it shouldn't be trusted. The integrity of your application cannot depend on it working properly.

Back End - Trusted
Front End - Untrusted

Example:
YouTube uses Google's servers as the back end to manage videos and the front end is a user interface on the web page itself. Modern web apps may have many back ends.

So why do computation/put stuff in the front end?
<ul>
<li>To lower the load on the back end.</li>
<li>The main reason is the latency (of the network communication). The front end takes less time to perform actions and events compared with back end because back computations require communication with the server(s) to return information (which induces latency).</li>
</ul>

A web application may also have multiple front ends (desktop, mobile, etc) however it is currently considered to be preferable to only have a single front end made with a responsive design such that it will adapt to whatever display it is used with.

<h4>Markup and templates</h4>
What we have worked with:
<ul>
<li>HTML</li>
<li>CSS</li>
<li>Jade (template/markup language)</li>
</ul>

There are many others which exist as well. HTML and CSS are constantly evolving. Others include XML, Latex, etc... Even WYSIWYG programs like Microsoft Word have underlying markup languages.

Markup is constraint or declarative programming.

Template languages output markup (they are meta-markup).

A markup language is a way of programming. You are declaring what something should be. You are not telling it exactly how to do it (it is not procedural), it is not object oriented but you are specifying constraints on it. So implicitly, a template language is a markup of a markup language. Markup languages are used for safety and efficiency reasons. i.e: less code to write, protection of input, etc....

<h4>Trust</h4>
When you run a web app, you have to trust various sources like: content providers, ad providers, library and runtime authors. Trust essentially refers to who can screw you over. So basically, if they go bad, how strongly does it affect you.

Embedded JavaScript scripts have access to the entire DOM so if they wanted, they could modify the entire page as they please. Companies that have their scripts embedded in many pages across the web could do some serious damage if they wanted.

WebFund 2015W Lecture 22

2015-04-03T21:22:33Z

LeeCroft:

==Video==

The video from the lecture given on Monday, March 30, 2015 [http://homeostasis.scs.carleton.ca/~soma/webfund-2015w/lectures/comp2406-2015w-lec22-30Mar2015.mp4 is now available].

==Notes==
<h3>The tls-notes application</h3>
<h4>Some changes in the application</h4>
<ul>
<li>The application now uses HTTPS as the protocol.</li>
<li>You can log in with username as in the previous ajax-notes, however now you need a password as well.</li>
<li>Visually, the app is different. The buttons look “prettier”. That is because the app uses Bootstrap.
<ul>
<li>Bootstrap is a toolkit which can be used to help improve your application.</li>
<li>In order to use Bootstrap, you will need to include a reference to the script in the head of the layout template.</li>
<li>With Bootstrap linked, you can apply CSS classes to elements in your pages to make them look nicer.</li>
</ul>
</li>
<li>There is no longer an edit button - note names are now links that go to the edit note view.</li>
<li>Change username is not working (fixing this is a requirement of the assignment).</li>
<li>There is another collection being stored in the database, the users collections.</li>
<li>Markup is basically the same with mild changes to notes.js adding divs and classes.</li>
<li>A ConvertNote function is added to notes.js in order to escape HTML tags and parse links when you insert into the notes.</li>
</ul>

Basically the assignment consists of a lot of debugging the code to fix it the problems in the version of the app provided.

<h4>Storing passwords</h4>
How do you store password on the server? – In MongoDB, within the users collection, all usernames and passwords are stored. However; the password is hashed. To check a password, you hash some input provided and compare it to the hashed value that is stored. This is important because if someone hacks into your website, you do not want them to have your password and they cannot get it without figuring out the hash.

You should never do the hashing on the client side, the server always does it. This is to prevent others from seeing what is done as this will reveal information to them about your hashing process.

Inside the /register route of the tls-notes app, we can see a checkUsername function that uses bcrypt.hash. This takes an input string and hashes it. Before you call bcrypt.hash you must call bcrypt.gensalt. The salt is used to customize the hashed password. If you did not use a salt, a hacker can look up tables of hashed passwords but with the salt it changes the resultant hash value of the input string which makes it more secure. – We will not be tested on this, it is for personal interest.

In the saveHash function in index.js you use an update function when adding a new user
because you first check if the user already exists and if they do it will never overwrite an account, it ‘updates it’.

It is important to have a secure connection when working with passwords. If the connection is not secure, anyone can see your password if they look at the data you are sending to the server. Using password hashing without using a secure connection is insufficient.

If you are interested in hashing and security against hacking – look up “rainbow tables”.

WebFund 2015W Lecture 21

2015-04-03T20:42:34Z

LeeCroft:

==Video==

The video for the lecture given on Wednesday, March 25, 2015 [http://homeostasis.scs.carleton.ca/~soma/webfund-2015w/lectures/comp2406-2015w-lec21-25Mar2015.mp4 is now available].

==Notes==

Google uses a ton of cookies when you access their server. They use these for tracking who you are and it is vital to their advertising business. Knowing who you are allows advertisers to show you products that you would buy.

Duckduckgo is another search engine which does not use any cookies!

<h3>TLS/SSL</h3>
<h4>Encryption and certificates</h4>
HTTPS:
<ul>
<li>Merely HTTP over a secured connection</li>
<li>Uses port 443 by default</li>
</ul>

With HTTPS, everything is encrypted as it is sent. A server is certified by using an SSL certificate which is digitally signed. This is a security certificate, it identifies that the server you are connecting to is recognized/trusted (not someone who may be intercepting your connection).

These certificates are verified by companies such as DigiCert, and these are called SSL Certificates. You can SAY you are facebook.com, but you can lie about that, so you need someone else to verify your identity. Unfortunately, some certificate authorities can be scams, and issue certificates that are 'spoofed'.

When you see warnings in the browser, this means that there is a problem with the certificate, as it cannot be verified. When this happens, you should check the details of the certificate.

When you are using HTTPS, everyone can see your traffic, but it is completely encrypted, and thus it looks like garbage to everyone.

<h4>HTTPS demo code</h4>
When running the TLS notes demo, the browser says there is a problem with the certificate, and that it isn't valid. It says it is self-signed and expired (usually this == BAD). In this instance, it is ok because we know that we made the certificate so we trust it.

To adapt a previous application to use HTTPS, you must change the <code>require(http)</code> to <code>require(https)</code>. This causes the server to use HTTPS as the protocol instead of HTTP. This is within the bin/www file.

For an HTTPS server, we must specify some options. These options define the public and private keys/certificates. These are loaded using the fs module and store in an options object which is used during the creation of the server.

Keys and certificates can be generated using openSSL. The public key is given out to the clients of the server for them to encrypt data. The private key is used by the server for decryption.

WebFund 2015W Lecture 19

2015-03-19T17:55:13Z

LeeCroft:

==Video==

The video for the lecture given on Wednesday, March 18, 2015 [http://homeostasis.scs.carleton.ca/~soma/webfund-2015w/lectures/comp2406-2015w-lec19-18Mar2015.mp4 is now available].

==Notes==

<h3>Assignment due dates</h3>
Assignments 5-7 can be submitted late up till Monday March 23 (with a small grade reduction).

Assignment 8 has been moved to Wednesday March 25.

<h3>Regular expressions</h3>
In order to understand what a regular expression really is, we need a bit of background information.

The Chomsky Hierarchy defines the classes of formal grammars you can create. They are broken into different classifications based on how “powerful” the language is or how much variation it has. A formal grammar is a set of production rules for strings; the rules describe what syntax is valid for the language. A formal grammar has the following things:
<ul>
<li>A finite set of nonterminal symbols indicating that some production rule can yet be applied.</li>
<li>A finite set of terminal symbols indicating that no production rule can be applied.</li>
<li>A start symbol.</li>
</ul>
<em>(Formal grammar definition referenced from Wikipedia.)</em>

The Hierarchy breaks the grammars down into the following 4 groups:
<ul>
<li>Type 3 - Regular Expression: Think of it as a a grammar that can be parsed by a little machine with finite states (finite automaton), it can’t remember a large quantity of things.</li>
<li>Type 2 - Context Free: They use a stack. This encompasses most programming languages.</li>
<li>Type 1 - Context Sensitive: What a computer can do with a fixed amount of space.</li>
<li>Type 0 - Recursively Enumerable: Anything a computer can do.</li>
</ul>

What is a finite automaton?
An example of a simple deterministic finite automaton would be a program with 3 states that can receive a 0 or 1. See the [http://en.wikipedia.org/wiki/Deterministic_finite_automaton#/media/File:DFA_example_multiplies_of_3.svg Wikipedia example].
<ul>
<li>If we are on state 0 and receive a 0, return to state 0.</li>
<li>If we are on state 0 and receive a 1, go to state 1.</li>
<li>If we are on state 1 and receive a 0, go to state 2.</li>
<li>If we are on state 1 and receive a 1, go to state 0.</li>
<li>If we are on state 2 and receive a 0, go to state 1.</li>
<li>If we are on state 2 and receive a 1, return to state 2.</li>
</ul>

Wikipedia says it is a sequence of characters that forms a search pattern. As seen from the Chomsky Hierarchy, we also know that it is a formal grammar.

Formal grammar was initially used in linguistics but adopted by Computer Science. To represent the structure of natural language you parse it using rules - the grammar. Similarly, in Computer Science, a grammar is a set of rules specifying a language, also known as syntax. The simplest languages are ones like LISP (blah(blah blah blah)). Contains lots of parenthesis, symbols, numbers, it is very easy to parse.

The most common programming languages are C-like: Java, C++, JavaScript, Perl(sort of). Not all grammars are the same; some are more powerful or have more variation than others. You will learn this if you take 3803.

While it was more common in the past to parse user input with a formal grammar, we don’t really do this anymore because user input is done with a form so we don’t have to search for the information.

<h4>Using regular expressions to parse note contents with links</h4>
Do not use regular expressions if you don’t have to, they are hard to make and are very buggy. You will probably not get the expected output as seen in the example below. It works by creating a finite automaton and evaluating the state. Regular expressions have very fast pattern matching.

Incomplete regular expression class example: This example shows us how regular expressions work by trying to parse for links in our notes:
<code><pre>
var s = “Example [http://www.google.ca link here]” + “and [http://students.carleton.ca here]
s.replace(/\[(.+) (.+)\]/g, ‘<a href=”$1”>$2</a>;

//s will now be“Example <a href=”http://www.google.ca link here] and [http://students.carleton.ca here]</a>
</pre></code>

The string is not exactly what we want, it still has some whitespace and unnecessary text in the link, avoid using regular expressions but understand how Node uses regular expressions to parse URLs in the routing functions. If you use regular expressions expect yours to be wrong, it would be much simpler and less buggy to do it the following way.

The easier way to parse the links would have been to manipulate the string to your desire using:
<ul>
<li><code>s.indexOf(‘[‘)</code> will return the index of the first open square bracket it sees.</li>
<li><code>s.split(‘[‘)</code> will cut the string into multiple strings at the points where it sees an open square bracket.</li>
</ul>

<h3>Debugging</h3>
Code is compiled and checked for syntax before being run. Server side code problems gives error messages to the terminal, client side code problems need to be inspected directly from the browser.

Server Side Code Problems Created in Index.js:
<ul>
<li>Random characters in your code causes a Reference Error: meaning your random characters don’t exist. It would, however, run if it was defined even if it doesn't do anything.</li>
<li>Missing a parenthesis gives a Syntax Error: meaning you violated the grammar of JavaScript.</li>
</ul>

Client Side Code Problems Created in notes.js:
<ul>
<li>Missing a curly brace caused part of the page to be missing. To investigate the problem we inspected the page and went to the Console to see an error message telling us we are missing a ; before statement on line X. The line that is given is usually the end of the program even though the error occurred somewhere else. A good way to debug missing braces is to use an editor that indents based off of braces and just indent lines until we find a discrepancy.</li>
</ul>

<h3>Assignment 8 hints</h3>
<ul>
<li>Error 2: Hangs, redirection failed. Redirection happens on the server side.</li>
<li>Error 3: Error status 500 means a server side problem.</li>
<li>Error 4: Why is Not Logged In appearing? This is not normal input so it is probably encoded somewhere in the program. Find these strings, now look for the error.</li>
<li>Error 5: Problem is client side because we made a new note and got a status 200 from the server. It is not rendering the interface for making a new note.</li>
<li>Error 6: JavaScript console means it is client side code, $ is used with jQuery.</li>
<li>Error 7: Client side not making the right requests</li>
<li>Error 8: Client side</li>
</ul>

<h3>Deploying web apps</h3>
Deploying Web Apps in 3 steps:
<ol>
<li>Get a domain name by going to the right registrar – is it .ca. com? This is a database that contains databases. For example Carleton’s Computer Science department contains the following hierarchy .ca -> Carleton -> scs->scs machines.</li>
<li>Get a DNS provider/server to maintain the database for your domain.</li>
<li>Get a hosting provider to have the machine that runs the code.</li>
</ol>

You can get all these 3 services from one provider; this can be a problem if you don’t like your hosting provider since you’re locked into their services to use your domain. If you purchase the domain from somewhere separate from your hosting provider, it is easy to simply switch hosting providers for your domain.

===In Lecture===

<pre>
Regular Expressions

- comes out of formal language theory, originally from linguistics, but adopted by computer scientists

- how do you represent the structure of natural languages?
- how do you "parse" languages?
- parse using rules

- in CS, a "grammar" is a set of rules specifying a "language"
- all syntax, no meaning
- is this a valid X, where X is...English? more likely C or Java
- simplest languages are things like LISP
(blah (blah blah blah) )
- most common: C-like syntax
Java, C++, JavaScript, Perl (sort of)

- not all grammars are the same
- some are more "powerful" than others
- this is the stuff of 3803

Deploying Web Apps
------------------

* get a domain name (DNS)
- go to the right registrar - is it .ca, .com, .sucks
* get a DNS provider/server
- maintain DB for your domain
* get a hosting provider (machine to run web code)

You can get all three from certain providers

Don't do your own DNS server
you *can* do your own web server
How?
- just run it on port 80
</pre>

WebFund 2015W Lecture 19

2015-03-19T17:53:34Z

LeeCroft:

==Video==

The video for the lecture given on Wednesday, March 18, 2015 [http://homeostasis.scs.carleton.ca/~soma/webfund-2015w/lectures/comp2406-2015w-lec19-18Mar2015.mp4 is now available].

==Notes==

<h3>Assignment due dates</h3>
Assignments 5-7 can be submitted late up till Monday March 23 (with a small grade reduction).

Assignment 8 has been moved to Wednesday March 25.

<h3>Regular Expressions</h3>
In order to understand what a regular expression really is, we need a bit of background information.

The Chomsky Hierarchy defines the classes of formal grammars you can create. They are broken into different classifications based on how “powerful” the language is or how much variation it has. A formal grammar is a set of production rules for strings; the rules describe what syntax is valid for the language. A formal grammar has the following things:
<ul>
<li>A finite set of nonterminal symbols indicating that some production rule can yet be applied.</li>
<li>A finite set of terminal symbols indicating that no production rule can be applied.</li>
<li>A start symbol.</li>
</ul>
<em>(Formal grammar definition referenced from Wikipedia.)</em>

The Hierarchy breaks the grammars down into the following 4 groups:
<ul>
<li>Type 3 - Regular Expression: Think of it as a a grammar that can be parsed by a little machine with finite states (finite automaton), it can’t remember a large quantity of things.</li>
<li>Type 2 - Context Free: They use a stack. This encompasses most programming languages.</li>
<li>Type 1 - Context Sensitive: What a computer can do with a fixed amount of space.</li>
<li>Type 0 - Recursively Enumerable: Anything a computer can do.</li>
</ul>

What is a finite automaton?
An example of a simple deterministic finite automaton would be a program with 3 states that can receive a 0 or 1. See the [http://en.wikipedia.org/wiki/Deterministic_finite_automaton#/media/File:DFA_example_multiplies_of_3.svg Wikipedia example].
<ul>
<li>If we are on state 0 and receive a 0, return to state 0.</li>
<li>If we are on state 0 and receive a 1, go to state 1.</li>
<li>If we are on state 1 and receive a 0, go to state 2.</li>
<li>If we are on state 1 and receive a 1, go to state 0.</li>
<li>If we are on state 2 and receive a 0, go to state 1.</li>
<li>If we are on state 2 and receive a 1, return to state 2.</li>
</ul>

Wikipedia says it is a sequence of characters that forms a search pattern. As seen from the Chomsky Hierarchy, we also know that it is a formal grammar.

Formal grammar was initially used in linguistics but adopted by Computer Science. To represent the structure of natural language you parse it using rules - the grammar. Similarly, in Computer Science, a grammar is a set of rules specifying a language, also known as syntax. The simplest languages are ones like LISP (blah(blah blah blah)). Contains lots of parenthesis, symbols, numbers, it is very easy to parse.

The most common programming languages are C-like: Java, C++, JavaScript, Perl(sort of). Not all grammars are the same; some are more powerful or have more variation than others. You will learn this if you take 3803.

While it was more common in the past to parse user input with a formal grammar, we don’t really do this anymore because user input is done with a form so we don’t have to search for the information.

<h4>Using regular expressions to parse note contents with links</h4>
Do not use regular expressions if you don’t have to, they are hard to make and are very buggy. You will probably not get the expected output as seen in the example below. It works by creating a finite automaton and evaluating the state. Regular expressions have very fast pattern matching.

Incomplete regular expression class example: This example shows us how regular expressions work by trying to parse for links in our notes:
<code><pre>
var s = “Example [http://www.google.ca link here]” + “and [http://students.carleton.ca here]
s.replace(/\[(.+) (.+)\]/g, ‘<a href=”$1”>$2</a>;

//s will now be“Example <a href=”http://www.google.ca link here] and [http://students.carleton.ca here]</a>
</pre></code>

The string is not exactly what we want, it still has some whitespace and unnecessary text in the link, avoid using regular expressions but understand how Node uses regular expressions to parse URLs in the routing functions. If you use regular expressions expect yours to be wrong, it would be much simpler and less buggy to do it the following way.

The easier way to parse the links would have been to manipulate the string to your desire using:
<ul>
<li><code>s.indexOf(‘[‘)</code> will return the index of the first open square bracket it sees.</li>
<li><code>s.split(‘[‘)</code> will cut the string into multiple strings at the points where it sees an open square bracket.</li>
</ul>

<h3>Debugging</h3>
Code is compiled and checked for syntax before being run. Server side code problems gives error messages to the terminal, client side code problems need to be inspected directly from the browser.

Server Side Code Problems Created in Index.js:
<ul>
<li>Random characters in your code causes a Reference Error: meaning your random characters don’t exist. It would, however, run if it was defined even if it doesn't do anything.</li>
<li>Missing a parenthesis gives a Syntax Error: meaning you violated the grammar of JavaScript.</li>
</ul>

Client Side Code Problems Created in notes.js:
<ul>
<li>Missing a curly brace caused part of the page to be missing. To investigate the problem we inspected the page and went to the Console to see an error message telling us we are missing a ; before statement on line X. The line that is given is usually the end of the program even though the error occurred somewhere else. A good way to debug missing braces is to use an editor that indents based off of braces and just indent lines until we find a discrepancy.</li>
</ul>

<h3>Assignment 8 hints</h3>
<ul>
<li>Error 2: Hangs, redirection failed. Redirection happens on the server side.</li>
<li>Error 3: Error status 500 means a server side problem.</li>
<li>Error 4: Why is Not Logged In appearing? This is not normal input so it is probably encoded somewhere in the program. Find these strings, now look for the error.</li>
<li>Error 5: Problem is client side because we made a new note and got a status 200 from the server. It is not rendering the interface for making a new note.</li>
<li>Error 6: JavaScript console means it is client side code, $ is used with jQuery.</li>
<li>Error 7: Client side not making the right requests</li>
<li>Error 8: Client side</li>
</ul>

<h3>Deploying web apps</h3>
Deploying Web Apps in 3 steps:
<ol>
<li>Get a domain name by going to the right registrar – is it .ca. com? This is a database that contains databases. For example Carleton’s Computer Science department contains the following hierarchy .ca -> Carleton -> scs->scs machines.</li>
<li>Get a DNS provider/server to maintain the database for your domain.</li>
<li>Get a hosting provider to have the machine that runs the code.</li>
</ol>

You can get all these 3 services from one provider; this can be a problem if you don’t like your hosting provider since you’re locked into their services to use your domain. If you purchase the domain from somewhere separate from your hosting provider, it is easy to simply switch hosting providers for your domain.

===In Lecture===

<pre>
Regular Expressions

- comes out of formal language theory, originally from linguistics, but adopted by computer scientists

- how do you represent the structure of natural languages?
- how do you "parse" languages?
- parse using rules

- in CS, a "grammar" is a set of rules specifying a "language"
- all syntax, no meaning
- is this a valid X, where X is...English? more likely C or Java
- simplest languages are things like LISP
(blah (blah blah blah) )
- most common: C-like syntax
Java, C++, JavaScript, Perl (sort of)

- not all grammars are the same
- some are more "powerful" than others
- this is the stuff of 3803

Deploying Web Apps
------------------

* get a domain name (DNS)
- go to the right registrar - is it .ca, .com, .sucks
* get a DNS provider/server
- maintain DB for your domain
* get a hosting provider (machine to run web code)

You can get all three from certain providers

Don't do your own DNS server
you *can* do your own web server
How?
- just run it on port 80
</pre>

WebFund 2015W Lecture 18

2015-03-18T00:17:17Z

LeeCroft: Created page with "==Audio== ==Notes== <h3>Assignment 7</h3> <h4>Question 1</h4> The delete button is implemented in the same way as the update button except that it will do something differ..."

==Audio==

==Notes==

<h3>Assignment 7</h3>

<h4>Question 1</h4>
The delete button is implemented in the same way as the update button except that it will do something different in the callback function provided for its click event. You must create this callback function and set it up to:
<ul>
<li>Accept a parameter containing the note</li>
<li>Check with a dialogue box to confirm the user's choice to delete note</li>
<li>Make a POST request to '/deleteNote'</li>
</ul>

You must also set up a route on the serve to handle this request. The body of the callback function for this route will be very similar to the previously used delete note route.

<h4>Question 2</h4>
To easily format a string containing the contents of a note, you can use [https://regex101.com/#javascript JavaScript regular expressions]. One potential place to do this is on the server in the /updateNotes route.

HTML elements can be escaped in exactly the same way as in the previous assignment.

For a link with a label, use:
<code><pre>
function makeLinkswLabel(str){
return str.replace(/\[(\S+)\s+([^\]]+)\]/gi,'<a href=\"$1\">$2</a>'
);
</pre></code>

For a link without a label, use:
<code><pre>
function makeLinkswoLabel(str){
return str.replace(/\[(\S+)\]/gi,'<a href=\"$l\">$1</a>');
};
</pre></code>

<h4>Question 3</h4>
This will involve making a post request to a route /newName. The rest is left up to you to figure out...