Soma-notes - User contributions [en]

User:Sdp

2014-08-07T17:16:06Z

Sdp: updates student status

== Simon David Pratt ==

Simon is an alumnus of Carleton University, having graduated with a Bachelor of Computer Science in spring of 2014. Simon will be attending the University of Waterloo in the Fall of 2014, pursuing a Master of Mathematics in Computer Science under the supervision of Timothy Chan.

I have previously taken [[Distributed OS: Winter 2014]], [[Fundamentals of Web Applications (Winter 2013)]] and [[Operating Systems (Fall 2012)]] with [[User:Soma|Professor Somayaji]].

[http://simondavidpratt.com Simon David Pratt] My homepage

[http://pr4tt.com Simon Pratt] My other home page

User:Sdp

2014-08-07T17:08:27Z

Sdp: /* Simon David Pratt */

== Simon David Pratt ==

Simon is a fourth year computer science student.

I have previously taken [[Distributed OS: Winter 2014]], [[Fundamentals of Web Applications (Winter 2013)]] and [[Operating Systems (Fall 2012)]] with [[User:Soma|Professor Somayaji]].

[http://simondavidpratt.com Simon David Pratt] My homepage

[http://pr4tt.com Simon Pratt] My other home page

DistOS 2014W Lecture 21

2014-03-27T15:16:55Z

Sdp: Added MapReduce notes

== Presentation ==

=== Marking ===

* marked mostly on presentation, not content
* basically we want to communicate the basic structure of the paper, and do so in a way that isn't boring

=== Content ===

* concrete, not "head in the clouds"
* present the area
* compare and contrast the papers
* 10 minutes talk, 5 minutes feedback
* basic argument
* basic references

=== Form ===

* show the work we've done on paper
* try to get feedback
* think of it as a rough draft
* try to get people to read the paper
* enthusiasm
* powerpoints are easier
* don't read slides
* no whole sentences on slides
* look at talks by Mark Shuttleworth

== MapReduce ==

A clever observation that a simple solution could solve most distributed problems. It's all about programming to an abstraction that is efficiently parallelizable. Note that it's not actually a simple solution, because it sits atop a mountain of code. It requires something like BigTables which requires something like GFS, which requires something like Chubby.

== Naiad ==

Where MapReduce was suited for a specific family of solutions, Naiad tries to generalize the solution to apply parallelization to a much wider family. Naiad supports MapReduce style solutions, but also many other solutions. However, the tradeoff was simplicity. It's like we took MapReduce and took away its low barrier to entry. The idea is to create a constrained graph that can easily be parallelized.

DistOS 2014W Lecture 21

2014-03-27T15:03:37Z

Sdp: added presentation notes

DistOS 2014W Lecture 20

2014-03-25T15:25:18Z

Sdp: /* Comet */ Added Soma quote

== Cassandra ==

Cassandra is essentially running a BigTable interface on top of a Dynamo infrastructure. BigTable uses GFS' built-in replication and Chubby for locking. Cassandra uses gossip algorithms: [http://dl.acm.org/citation.cfm?id=1529983 Scuttlebutt]. Apache Zookeeper is used for distributed configuration.

== Comet ==

The major idea behind Comet is triggers/callbacks. There is an extensive literature in extensible operating systems, basically adding code to the operating system to better suit my application. "Generally, extensible systems suck." -[[User:Soma]]

[https://www.usenix.org/conference/osdi10/comet-active-distributed-key-value-store The presentation video of Comet]

Cassandra
+
Cassandra is essentially running a BigTable interface on top of a Dynamo infrastructure. BigTable uses GFS' built-in replication and Chubby for locking. Cassandra uses gossip algorithms: [http://dl.acm.org/citation.cfm?id=1529983 Scuttlebutt]. Apache Zookeeper is used for distributed configuration.

−
Google developed its technology interbally and used for competitive advantage.
+
== Comet ==
−
Facebook developed its technology in open source manner.
+

−
Gpl 3 you have to provide code with binary
+
[https://www.usenix.org/conference/osdi10/comet-active-distributed-key-value-store The presentation video of Comet]
−
In AGPL addtional service also be given with source code.
+
−

+
−
Bigtable needs gfs.cassandra depends on server's file system.Anil feels cassndra cluster is easy to setup.
+
−

+
−
Bigtable is designed for batch updates.cassandra is for hqbdling realtime stuff.
+
−

+
−
Schema design is explained in inbox example.it does does not give clarity about how table will look like.Anil thinks they store lot data with messages which makes table crappy.
+
−

+
−
Cassandra is design for high speed access and online operation.
+
−

+
−
Zookeeper is similar to chhuby
+
−
Zookeeper is for node level information
+
−
Gossip is more about key partitioning
+
−

+
−
Zookeeper is for configuration of new node.
+
−

+
−
It writes in immutable way like functional programming.there is no assignment in functional programming.

DistOS 2014W Lecture 20

2014-03-25T15:14:06Z

Sdp: /* Comet */ Added high-level description of Comet

DistOS 2014W Lecture 20

2014-03-25T15:07:22Z

Sdp: /* Cassandra */ added Scuttlebutt link

DistOS 2014W Lecture 20

2014-03-25T14:56:50Z

Sdp: /* Cassandra */ Added comparison to BigTable

== Cassandra ==

Cassandra is essentially running a BigTable interface on top of a Dynamo infrastructure. BigTable uses GFS' built-in replication and Chubby for locking. Cassandra uses gossip algorithms. Apache Zookeeper is used for distributed configuration.

== Comet ==

[https://www.usenix.org/conference/osdi10/comet-active-distributed-key-value-store The presentation video of Comet]

DistOS 2014W Lecture 20

2014-03-25T14:54:24Z

Sdp: /* Comet */

== Cassandra ==

Cassandra is essentially running a BigTable interface on top of a Dynamo infrastructure.

== Comet ==

[https://www.usenix.org/conference/osdi10/comet-active-distributed-key-value-store The presentation video of Comet]

DistOS 2014W Lecture 20

2014-03-25T14:54:13Z

Sdp: Cleaned up Comet presentation link

DistOS 2014W Lecture 20

2014-03-25T14:52:56Z

Sdp: Added Cassandra summary

Cassandra is essentially running a BigTable interface on top of a Dynamo infrastructure.

The following link is a link to the presentation video of the Comet. I thought it might be usefull (posted by Mojgan).

https://www.usenix.org/conference/osdi10/comet-active-distributed-key-value-store

DistOS 2014W Lecture 7

2014-01-28T16:14:47Z

Sdp: update simon's notes at the end of class

== Project ==

We discussed moving the proposal due date back a week. We also discussed spending the class prior to that date discussing the primary papers people had chosen in order to provide preliminary feedback. Anil spent some time going through the papers from OSDI12 and discussing which ones would make good projects and why.

* Pick a primary paper.
* Find papers that cite that paper, papers it cites, etc. to collect a body of related work.
* Don't just give a history, tell a story!

== Unix and Plan 9 ==

UNIX was built as "a castrated version of Multix", which was a very complex system. Multix was, arguably, so far ahead of its time that we are only just achieving their ambitions now. Unix was much more modest, and therefore much more achievable and successful. Just enough infrastructure to avoid reinventing the wheel. Just a couple of programmers making something for their own use. Unix was not designed as product or commercial entity at all. It was licensed out because AT&T was under severe antitrust scrutiny at the time.

They wanted few, simple abstractions so they made everything a file. Berkeley promptly broke this abstraction by introducing sockets for networking. Plan 9 finally introduced networking using the right abstractions, but was too late. Arguably the reason the BSD folks didn't use the file abstraction was because of the difference in reliability. Files are generally reliable, and failures with them are catastrophic so many applications simply didn't include logic to handle such IO errors. Networks are much less reliable and applications have to be able to deal gracefully with timeouts and other errors.

== Simon's Notes ==

* project proposal
** We will discuss the primary papers we've chosen on Thursday, February 6th
* possible papers, remember to pick a topic you have some chance of understanding
** OSDI 2012
*** datacenter (filesystems for doing X, heat management, etc...)
*** web stuff
*** distributed shared memory
*** distributed network I/O infrastructure
*** distributed databases (potentially)
*** anonymity systems
** Pick a conference (usenix is pretty systems oriented, maybe Lisa), go through their papers and find something interesting
** tell a story that connects several papers in the topic you choose

* UNIX
** Relation to multics
*** Multics was a complex system which was bad because it was used less, slower, etc...
*** Multics was not for end users, it was designed to support "utility computing" wherein computation was a service to be charged for
** What?
*** Just enough infrastructure to run my programs
*** It was really just supposed to be used by programmers
*** "By programmers for programmers"
*** Software and source licensed for a nominal fee
*** "Everything is a file"
*** only difference was files that you could use seek or ones you couldn't
*** simple abstractions
** Networking
*** Berkeley folks made sockets, not files which upset the folks at Bell labs
*** Networks aren't exactly like files because they're unreliable

* Plan 9
** major ideas
*** procfs, later adopted by linux
** summary
*** a very elegant attempt to follow the philosophy "everything is a file"
*** trying too hard
** opinions
*** things that have different failure modes deserve different APIs
** niche?
*** they never found one

* Tangent about programming languages
** C was for system programming
** Java was for enterprise programming

DistOS 2014W Lecture 7

2014-01-28T16:00:54Z

Sdp: formatting

DistOS 2014W Lecture 7

2014-01-28T16:00:37Z

Sdp: fixed a bug

DistOS 2014W Lecture 7

2014-01-28T16:00:18Z

Sdp: added simon's notes to be merged in later

== Project ==

We discussed moving the proposal due date back a week. We also discussed spending the class prior to that date discussing the primary papers people had chosen in order to provide preliminary feedback. Anil spent some time going through the papers from OSDI12 and discussing which ones would make good projects and why.

* Pick a primary paper.
* Find papers that cite that paper, papers it cites, etc. to collect a body of related work.
* Don't just give a history, tell a story!

== Unix and Plan 9 ==

UNIX was built as "a castrated version of Multix", which was a very complex system. Multix was, arguably, so far ahead of its time that we are only just achieving their ambitions now. Unix was much more modest, and therefore much more achievable and successful. Just enough infrastructure to avoid reinventing the wheel. Just a couple of programmers making something for their own use. Unix was not designed as product or commercial entity at all. It was licensed out because AT&T was under severe antitrust scrutiny at the time.

They wanted few, simple abstractions so they made everything a file. Berkeley promptly broke this abstraction by introducing sockets for networking. Plan 9 finally introduced networking using the right abstractions, but was too late. Arguably the reason the BSD folks didn't use the file abstraction was because of the difference in reliability. Files are generally reliable, and failures with them are catastrophic so many applications simply didn't include logic to handle such IO errors. Networks are much less reliable and applications have to be able to deal gracefully with timeouts and other errors.

== Simon's Notes ==

* project proposal
** We will discuss the primary papers we've chosen on Thursday, February 6th
* possible papers, remember to pick a topic you have some chance of understanding
** OSDI 2012
*** datacenter (filesystems for doing X, heat management, etc...)
*** web stuff
*** distributed shared memory
*** distributed network I/O infrastructure
*** distributed databases (potentially)
*** anonymity systems
** Pick a conference (usenix is pretty systems oriented, maybe Lisa),
go through their papers and find something interesting
** tell a story that connects several papers in the topic you choose

* UNIX
** Relation to multics
*** Multics was a complex system which was bad because it was used less, slower, etc...
*** Multics was not for end users, it was designed to support "utility computing" wherein computation was a service to be charged for
** What?
*** Just enough infrastructure to run my programs
*** It was really just supposed to be used by programmers
*** "By programmers for programmers"
*** Software and source licensed for a nominal fee
*** "Everything is a file"
*** only difference was files that you could use seek or ones you couldn't
*** simple abstractions
** Networking
*** Berkeley folks made sockets, not files which upset the folks at Bell labs
*** Networks aren't exactly like files because they're unreliable

* Plan 9
** major ideas
*** procfs, later adopted by linux
** summary
*** a very elegant attempt to follow the philosophy "everything is a file"
*** trying too hard
** opinions
*** things that have different failure modes deserve different APIs

DistOS 2014W Lecture 3

2014-01-14T15:42:47Z

Sdp: /* Group 1 */

Questions to consider:
* What were the purposes envisioned for computer networks? How do those compare with the uses they are put to today?
* What sort of resources were shared? What resources are shared today?
* What network architecture did they envision? Do we still have the same architecture?
* What surprised you about this paper?
* What was unclear?

==Group 1==
* video was mostly a summary of Kahn's paper
* process migration through different zones of air traffic control
* "distributed OS" meant something different than we normally think about, because many people would log in remotely to a single machine, it is very much like cloud infrastructure that we talk about today
* alto paper makes reference to Kahn's paper, and the alto designers had the foresight to see that networks like arpanet would be necessary
* would it be useful to have a co-processor responsible for maintaining shared resources even today? Like the IMPs of the arpanet? Today, computers are usually so fast it doesn't really matter.

=== Questions ===

* What were the purposes envisioned for computer networks?
** big computation, storage, resource sharing - "having a library on a hard disk"

* How do those compare with the uses they are put to today?
** those things are being done, but mostly communication like instant messaging, email

* What sort of resources were shared?
** databases, CPU time

* What resources are shared today?
** mostly storage

* What network architecture did they envision?
** they had a checksum and acknowledge on each packet
** the IMPs were the network interface and the routers
** packet-switching

* Do we still have the same architecture?
** packet-switching definitely won
** no, now IP doesn't checksum or acknowledge, but TCP has end-to-end checksum and acknowledge
** Kahn went on to learn from the errors of arpanet to design TCP/IP
** the job of network interface and router have been decoupled

* What surprised you about this paper?
** everything
** how they were able to do this
** a network interface card and router was the size of a fridge
** high-level languages
** bootstrap protocol, bootstrapping an application
** primitive computers
** desktop publishing
** the logistics of running a cable from one university to another
** how old the idea of distributed operating systems is

* What was unclear?
** much of the more technical specifications, but we mostly skipped over those

==Group 2==

==Group 3==

==Group 4==

What were the purposes envisioned for computer networks? How do those compare with the uses they are put to today?

remote access to machines that have the resources users need: large databases, computing power, instruction set/non-portable software

Today, we use networks mostly for sharing data, although with services like Amazon AWS, we're starting to share computing resources again. We're also moving to support collaboration (e.g. Google Docs, GitHub, etc.).

What sort of resources were shared? What resources are shared today?

Computing power was the key resource being shared; today, it's access to data. (See above.)

What network architecture did they envision? Do we still have the same architecture?

Surprisingly, yes: modern networks have substantially similar architecures to the ones described in these papers.
Packet-switched networks are now ubiquitous. We no longer bother with circuit-switching even for telephony, in contrast to the assumption that non-network data would continue to use the circuit-switched common-carrier network.

What surprised you about this paper?

We were surprised by the accuracy of the predictions given how early the paper was written.

mouse in alto, resolution,
bus speed: we have networks faster than their integrated buses

What was unclear?

Nothing significant; we're looking at these with the benefit of hindsight.

DistOS 2014W Lecture 3

2014-01-14T15:41:46Z

Sdp: /* Group 1 */

Questions to consider:
* What were the purposes envisioned for computer networks? How do those compare with the uses they are put to today?
* What sort of resources were shared? What resources are shared today?
* What network architecture did they envision? Do we still have the same architecture?
* What surprised you about this paper?
* What was unclear?

==Group 1==
* video was mostly a summary of Kahn's paper
* process migration through different zones of air traffic control
* "distributed OS" meant something different than we normally think about, because many people would log in remotely to a single machine, it is very much like cloud infrastructure that we talk about today
* alto paper makes reference to Kahn's paper, and the alto designers had the foresight to see that networks like arpanet would be necessary
* would it be useful to have a co-processor responsible for maintaining shared resources even today? Like the IMPs of the arpanet? Today, computers are usually so fast it doesn't really matter.

=== Questions ===

* What were the purposes envisioned for computer networks?
** big computation, storage, resource sharing - "having a library on a hard disk"

* How do those compare with the uses they are put to today?
** those things are being done, but mostly communication like instant messaging, email

* What sort of resources were shared?
** databases, CPU time

* What resources are shared today?
** mostly storage

* What network architecture did they envision?
** they had a checksum and acknowledge on each packet
** the IMPs were the network interface and the routers
** packet-switching

* Do we still have the same architecture?
** packet-switching definitely won
** no, now IP doesn't checksum or acknowledge, but TCP has end-to-end checksum and acknowledge
** Kahn went on to learn from the errors of arpanet to design TCP/IP
** the job of network interface and router have been decoupled

* What surprised you about this paper?
** everything
** how they were able to do this
** a network interface card and router was the size of a fridge
** high-level languages
** bootstrap protocol, bootstrapping an application
** primitive computers
** desktop publishing
** the logistics of running a cable from one university to another

* What was unclear?
** much of the more technical specifications, but we mostly skipped over those

==Group 2==

==Group 3==

==Group 4==

What were the purposes envisioned for computer networks? How do those compare with the uses they are put to today?

remote access to machines that have the resources users need: large databases, computing power, instruction set/non-portable software

Today, we use networks mostly for sharing data, although with services like Amazon AWS, we're starting to share computing resources again. We're also moving to support collaboration (e.g. Google Docs, GitHub, etc.).

What sort of resources were shared? What resources are shared today?

Computing power was the key resource being shared; today, it's access to data. (See above.)

What network architecture did they envision? Do we still have the same architecture?

Surprisingly, yes: modern networks have substantially similar architecures to the ones described in these papers.
Packet-switched networks are now ubiquitous. We no longer bother with circuit-switching even for telephony, in contrast to the assumption that non-network data would continue to use the circuit-switched common-carrier network.

What surprised you about this paper?

We were surprised by the accuracy of the predictions given how early the paper was written.

mouse in alto, resolution,
bus speed: we have networks faster than their integrated buses

What was unclear?

Nothing significant; we're looking at these with the benefit of hindsight.

User:Sdp

2014-01-14T02:36:49Z

Sdp: now in fourth year

== Simon David Pratt ==

Simon is a fourth year computer science student.

I am taking [[Distributed OS: Winter 2014]] and have previously taken [[Fundamentals of Web Applications (Winter 2013)]] and [[Operating Systems (Fall 2012)]] with [[User:Soma|Professor Somayaji]].

[http://simondavidpratt.com Simon David Pratt] My homepage

[http://pr4tt.com Simon Pratt] My other home page

User:Sdp

2014-01-14T02:36:24Z

Sdp: updated to note now taking distributed OS

== Simon David Pratt ==

Simon is a third year computer science student.

I am taking [[Distributed OS: Winter 2014]] and have previously taken [[Fundamentals of Web Applications (Winter 2013)]] and [[Operating Systems (Fall 2012)]] with [[User:Soma|Professor Somayaji]].

[http://simondavidpratt.com Simon David Pratt] My homepage

[http://pr4tt.com Simon Pratt] My other home page

Fundamentals of Web Applications (Winter 2013)

2013-01-13T03:36:48Z

Sdp:

==Administration==

The course outline for this course is [http://www.scs.carleton.ca/courses/course_outline.php?Term=Winter&Year=2013&Number=COMP%202406 here].

Course discussions will be on Piazza. You can sign up [https://piazza.com/carleton.ca/winter2013/comp24062006 here]. Note that Piazza has a "groups" function that can help you find partners for your project. '''Also note that piazza asks for your carleton.ca email address, so you can't directly sign up with any anonymous email address.''' If you wish to sign up using an anonymous/throw away email address, please email Prof. Somayaji or a TA and they can enroll that email address manually.

You should get an account on this wiki so you can add to it. Email Prof. Somayaji to get one with your preferred username and email address to which a password should be sent. (Note this is not a requirement.)

==Resources==

===JavaScript===

'''You should go out and learn the basics of JavaScript on your own.''' While we will discuss the language in class, much of that discussion will make more sense if you've exposed yourself to the language. You should also get basic exposure to standard web technologies.

The easiest way to get started with JavaScript and get basic understanding of web technologies is to go through the interactive lessons on [http://codeacademy.com Code Academy]. I suggest you go through their JavaScript, Web Fundamentals, and jQuery tracks. They shouldn't take you very long to do given that you already know how to program.

You should get access to ''JavaScript: The Good Parts'' by Douglas Crockford ([http://it-ebooks.info/book/274/.do free PDF download]). Read it. It is available through [http://shop.oreilly.com/product/9780596517748.do O'Reilly], [http://my.safaribooksonline.com/9780596517748 Safari Books Online], and the regular online bookstores. I've even seen an interactive version of it which includes an embedded JavaScript interpreter. (link?) You can get access to Safari Books Online through the Carleton Library (four concurrent users only) or partial access by becoming a member of [https://www.computer.org IEEE Computer Society].

Crockford also has a lot of online resources on JavaScript, including videos of talks he's given that cover much of the content in his book. Look at his [http://javascript.crockford.com/ JavaScript page] and this [http://yuiblog.com/crockford/ page of his videos].

Another good book is [http://eloquentjavascript.net/ Eloquent JavaScript: A Modern Introduction to Programming] by Marijn Haverbeke. A version of this book is available online for free. The for-sale version is apparently updated and edited.

And of course the standard reference for JavaScript is [http://shop.oreilly.com/product/9780596805531.do JavaScript: The Definitive Guide] by David Flanagan. It is a big book, but it is comprehensive.

===Node===

You will be building your application in node.js this term. A good, relatively comprehensive book is [http://shop.oreilly.com/product/0636920024606.do Learning Node] by Shelley Powers. This book is recommended but not required. A quick way to get started with node.js is [http://www.nodebeginner.org/ The Node Beginner Book] by Manuel Kiessling.

==Lectures==

<table style="width: 100%;" border="1" cellpadding="4" cellspacing="0">
<tr valign="top">
<th>
Date
</th>
<th>
Topic
</th>
</tr>
<tr>
<td>
Jan. 8

</td>
<td>
Introduction

</td>
</tr>
<tr>
<td>
Jan. 10

</td>
<td>
[[WebFund 2013W: Symbols|Symbols]]

</td>
</tr>
</table>

==Tutorials/Weekly Tasks==

Each week you will get a progress grade from 0-4, given to you by a TA. If you are being diligent, you should be able to get 4's every week. The easiest way to get your grade is to come to tutorial and meet with your TA; alternately, you can meet a TA in their office hours or, at their discretion, discuss things with them online.

Initially you can talk to any TA to get your progress grade. Once groups have been formed, however, you will have an assigned TA who will be tracking your progress throughout the semester.

Below is a schedule with the tasks you need to accomplish each week. Note you need to accomplish the task before your next tutorial. So, if you attend the Monday tutorials, you need to show progress before the following Monday.

<table style="width: 100%;" border="1" cellpadding="4" cellspacing="0">
<tr valign="top">
<th>
Date
</th>
<th>
Tasks
</th>
</tr>
<tr>
<td>
Jan. 11,14

</td>
<td>
[[WebFund 2013W: Tasks 1|Setup Accounts, Run node.js]]

</td>
</tr>
<tr>
<td>
Jan. 18,21

</td>
<td>
Project Partners due

</td>
</tr>
</table>

==Humor/Quotes==

[http://xkcd.com/327/ Bobby Tables], a funny example of SQL injection.

"Scalability is always the answer in this class." -Alexis

"First axiom of this class is web apps suck." -[[User:soma |Professor Somayaji]]

Fundamentals of Web Applications (Winter 2013)

2013-01-13T03:35:28Z

Sdp: /* Humor/Quotes */ cleaned up the link to Soma's user page

Fundamentals of Web Applications (Winter 2013)

2013-01-10T18:20:47Z

Sdp: /* JavaScript */ fixed crockford's name

Fundamentals of Web Applications (Winter 2013)

2013-01-10T18:17:07Z

Sdp: /* Humor/Quotes */ added first axiom

User:Sdp

2013-01-10T00:54:35Z

Sdp: /* Simon David Pratt */ updated to include web applications

== Simon David Pratt ==

Simon is a third year computer science student.

I am taking [[Fundamentals of Web Applications (Winter 2013)]] and have previously taken [[Operating Systems (Fall 2012)]] with [[User:Soma|Professor Somayaji]].

[http://simondavidpratt.com Simon David Pratt] My homepage

[http://pr4tt.com Simon Pratt] My other home page

Talk:Fundamentals of Web Applications (Winter 2013)

2013-01-10T00:52:34Z

Sdp: fixed signature

Can we get some info on the project? -[[User:sdp|Simon]]

Talk:Fundamentals of Web Applications (Winter 2013)

2013-01-10T00:52:06Z

Sdp: added signature

Can we get some info on the project? -[[sdp|Simon]]

Fundamentals of Web Applications (Winter 2013)

2013-01-10T00:49:36Z

Sdp: /* Resources */ Noted that students can't sign up for piazza with an anonymous email address.

==Resources==

The course outline for this course is [http://www.scs.carleton.ca/courses/course_outline.php?Term=Winter&Year=2013&Number=COMP%202406 here].

Course discussions will be on Piazza. You can sign up [https://piazza.com/carleton.ca/winter2013/comp24062006 here]. Note that Piazza has a "groups" function that can help you find partners for your project. '''Also note that piazza asks for your carleton.ca email address, so you can't sign up with any anonymous email address.'''

You should get an account on this wiki so you can add to it. Email Prof. Somayaji to get one with your preferred username and email address to which a password should be sent. (Note this is not a requirement.)

==Lectures==

==Tutorials==

==Humor/Quotes==

[http://xkcd.com/327/ Bobby Tables], a funny example of SQL injection.

"Scalability is always the answer in this class." -Alexis

Talk:Fundamentals of Web Applications (Winter 2013)

2013-01-10T00:46:00Z

Sdp: asked for info on project

Can we get some info on the project?

Fundamentals of Web Applications (Winter 2013)

2013-01-08T18:56:48Z

Sdp: added the source of the scalability quote

The course outline for this course is [http://www.scs.carleton.ca/courses/course_outline.php?Term=Winter&Year=2013&Number=COMP%202406 here].

[http://xkcd.com/327/ Bobby Tables], a funny example of SQL injection.

"Scalability is always the answer in this class." -Alexis

Fundamentals of Web Applications (Winter 2013)

2013-01-08T18:55:32Z

Sdp: added note on scalability

Fundamentals of Web Applications (Winter 2013)

2013-01-08T18:46:38Z

Sdp: added context for bobby tables

Fundamentals of Web Applications (Winter 2013)

2013-01-08T18:45:22Z

Sdp: added bobby tables

The course outline for this course is [http://www.scs.carleton.ca/courses/course_outline.php?Term=Winter&Year=2013&Number=COMP%202406 here].

[http://xkcd.com/327/ Bobby Tables]

COMP 3000 2012 Week 8 Notes

2012-10-26T13:20:52Z

Sdp: description of malloc and sbrk

Practice questions and their associated answers:

<ol>
<li>
Q: You can create a sparse file in UNIX by writing sequences of zeros to a file. 

A: False. Sparse file means holes. Zeroes aren't holes.
You need to move the file pointer. You have to use a
seek. The data will be READ as zeros, though...
Look up truncate command and strace it. It ain't just writing zeros
</li>
<li>
Q: Are environment variables ``global variables? Specifically, if you change the value of an environment variable X in one process, what happens to the value of X in other processes? 

A:Nothing happens. Env vars aren't globals. They're passed in with Execve. Environments
variables are typically process specific.
</li>
<li>
Q: In UNIX, there are three permissions associated with the user, the user's group, and everyone else. What are those three permissions? 

A: rwx (Read, Write, Execute)
</li>
<li>
Q: If you see a zombie process during normal system operation, how can you get rid of it? 

A: Kill the parent
Why? init will own the zombie process and then kill it
</li>
<li>
Q: I can copy a file using I/O redirection as follows: cat /bin/ls > bar. If I now type ./bar, it won't run. However, if I run another command before typing ./bar, I will get
a file listing. What needed to be changed to get bar to run? 

A: chmod +x
Default non executable. Need to make it executable. (--x)
</li>
<li>
Q: System calls cannot normally be made using standard function calls. What CPU mechanism is used by regular processes to invoke kernel functionality? 

A: Software interrupt. Signals are a form of of software interrupt. 

Note:

"A system call is an entry point into the Linux kernel. Usually, system calls are not invoked directly: instead, most system calls have corresponding C library wrapper functions which perform the steps required (e.g., trapping to kernel mode) in order to invoke the system call. Thus, making a system call looks the same as invoking a normal library function." from man 2 intro 

"The call to the library function itself does not cause a switch to kernel mode (if the execution was not already in kernel mode) and is usually a normal subroutine call (using, for example, a "CALL" assembly instruction in some Instruction set architectures (ISAs)). The actual system call does transfer control to the kernel (and is more implementation-dependent and platform-dependent than the library call abstracting it). For example, in Unix-like systems, "fork" and "execve" are C library functions that in turn execute instructions that invoke the "fork" and "execve" system calls. Making the system call directly in the application code is more complicated and may require embedded assembly code to be used (in C and C++) as well as knowledge of the low-level binary interface for the system call operation, which may be subject to change over time and thus not be part of the application binary interface; the library functions are meant to abstract this away." from http://en.wikipedia.org/wiki/System_call

</li>
<li>

Q: After a fork system call, what runs first, the child process runs or the parent process? 

A: Don't know... Undefined. Can not depend on ordering

</li>
<li>

Q: Signal handlers can be called at almost any point in a process's execution. (T/F) 

A: True. That's the purpose of signals 

Register a normal C function as a signal handler 

Some are implemented by default (ex. sigkill) 

Signals can be called at any point in execution. 

"When a signal is sent, the operating system interrupts the target process's normal flow of execution. Execution can be '''interrupted during any non-atomic instruction.''' If the '''process has previously registered a signal handler, that routine is executed. Otherwise''' the default signal handler is executed." from http://en.wikipedia.org/wiki/Unix_signal

</li>
<li>

Q: System V init scripts run programs sequentially or in parallel? 

A: Sequentially (sysV). Shell script that runs every ''command'' sequentially 

Upstart does it in parallel though. This is the reason sysV is being replaced by Upstart.

</li>
<li>

Q: Without errors, what does the execve system call return? # Who calls a signal handler? 

A: On success, execve() does not return, on error -1 is returned, and errno is set appropriately. 

The program itself calls the signal handler

</li>
<li>

Q: What program is used to check the consistency of the data structures of a filesystem? 

A: fsck <- wrapper function 
fsck.FMT <- called by fsck for a specific format (ex: fsck.ext4)

</li>
<li>

Q: What allocates the storage for environment variables and command-line arguments? 

A: The kernel. The kernel is the only thing that can allocate this space.(execve) 
This also makes sense because execve is a system call.

Note that the kernel is the only thing that can change the size of a program's data segment directly (the wrapper for this system call is sbrk), but malloc is often implemented to call sbrk in advance, then divide up the space as malloc is called.

</li>
<li>

Q: A file that has a logical size of one gigabyte but only takes up 100K on disk is called what kind of file in UNIX? 

A: Sparse file.

</li>
<li>

Q: What is an easy (but not foolproof way) way to tell that you are running in a virtual environment on Linux? What command do you run to get this information? 

A: lspci, lshw. 
In /proc/, you may be able to also. 
These commands display the machine's "hardware". 

Not /dev/ because that's the abstraction.

</li>
<li>

Q:What is the purpose of the PATH environment variable? 

A: At the command line, the $PATH variable is where the shell looks for binaries/files.
The path is walked when the fullPATH/filename of the command line hasn't been specified

</li>
<li>

Q: When would you expect ltrace to output more lines than strace? When should strace output more than ltrace? 

A: Some libraries make no sys calls (strlen). Libraries that do basic string and math
operations -> no sys calls 

Programs with lots of I/O will use lot of sys calls

</li>
<li>

Q: What command is used to change the priority of a process? 

A: nice and renice 
nicess is policy value (default value and you can modify this) 
priority (dynamic) is based on niceness and the amount of time the process has run on
cpu 
Priority doesn't apply to I/O. I/O will continue despite other processing having high
priority. I/O will always slow down the system.

</li>
<li>

Q: What is the difference between a library call and a function call? 

A: The former requires a library card and a telephone.
 [[User:Spenner2|Spenner2]] ([[User talk:Spenner2|talk]]) nice
</li>
</ol>

Talk:COMP 3000 2012 Midterm Material

2012-10-26T01:54:56Z

Sdp: added yu-gi-oh reference

= Discussion =

= Only the facts =
I'll be putting up my notes of notes up here. Hopefully people can benefit from these. I'll be summing up the answers from the lab and adding a few more details here and there.

== Lab 1 ==
=== Types of shell commands ===
Built ins
Shell functions
binaries (scripts and binaries
alias

=== Shell builtins vs binaries ===
binaries exist elsewhere
faster, no forking needed
Kernel functions (IO redirection)

=== ls -l ===

1 2 3 4 5 6 7
-rwxr-xr-x 1 root root 104508 Mar 31 2012 /bin/ls

U G O
-rwxrwxrwx

To change user permission, use command: "chmod <UGO> <file/dir name>"
ex: chmod 777 file
The three digits represent User, Group or Others permissions. Represented as the sum of their permission values (read = 4, write = 2, execute = 1)

eg. 754 U = 7 = 4 + 2 + 1 = read, write and execute
G = 5 = 4 + 1 = read and execute
O = 4 = read only

eg. 623 U = 6 = 4 + 2 = read and write
G = 2 = write only
O = 3 = 2 + 1 = read and execute

Memnonic: 'You go hugo!'
Mnemonic: 'Yu-gi-oh!'

- denotes a regular file
d denotes a directory
b denotes a block special file
c denotes a character special file
l denotes a symbolic link
p denotes a named pipe

2) 1 link to /bin/ls
3) user root
4) group root
5) size in bytes
6) date last modified
7) path/filename

=== I/O redirection ===
<< keyword HEREDOC. Input until program reads specified keyword
>> append
< read from path
> write to path, blow away anything there
| pipes

=== file descriptors ===
0 stdin
1 stdout
2 stderr
3-9 other

fd redirection
#> where # is a file descriptor
a>&b redirects file descriptor a to b.

COMMAND &>> file.txt
Appends BOTH STDERR and STDOUT to file.txt

=== For loops ===

//For all files in fold that start with l, echo the title
for i in l*; do
echo $i
done

=== Bg vs fg ===
Bash waits or doesn't for return
Both write to stdout as normal

=== Shell vars ===
shell vars only apply to shell. Live in shell
environment is more global, but typically a COPY is passed on to child processes
envp is a series of keyvalue strings passed in with exec*

== Lab2 ==

=== Processes ===

ps aux
ps -e
top

run pstree.
notice that cron, upstart-socket, and others are owned by init. These are daemons.

notice that gnome display manager is a child of lightdm. Killing lightdm kills
gnome. That's why we had to log into a virtual console

=== Upstart ===

init is upstart
started by kernel. It's the only think the kernel runs in userspace

shutdown -r now
switches to run level 6

shutdown -h now
switches to run level zero

symlinks in /etc/init.d point to /lib/init/upstart-job
successful sysV to upstart transition

Upstart's entire deal is revamp the ol' sysV boot seq

traditional init process is strictly synchronous, blocking future tasks until the
current one has completed.

upstart is async. and allows more parralleissimsimsmmmsm

=== Logs ===

/var/log

use lsof to check who's accessing a file.

Rule of thumb.
if logfile is generic, then written to by rsyslog (udev, auth, etc)
if logfile has program name, written to by that program (apache)

rsyslog
daemon that handles syslog protocol
handles generic logging for system

== Lab 3 ==

=== Tracing ===

ltrace, strace
Statically linking will make ltrace output nothing.

=== writing to files ===

File * f = fopen(path, "w")
fprintf(f,"blah")
fclose(f)

=== exec* ===

the exec family of functions replace the current process image with that of
another file/binary.
Does not return value if successful, -1 on failure
Process keeps same PID

execvp() inherits the current process's environment variables
execve() needs to have the environment variables passed in as an array of pointers
to null terminated strings.

=== Other ===

kill sends signals.

used as follows:
"kill <-command> <PID of program>"

Important kill signals:
-SIGINT or -2 -> (crtl-C) sends a kill signal to program, can be ignored. SIGINT is also the default signal used by kill. (ie. kill <PID> = kill -2 <PID>)
-SIGSTP -> (crtl-Z) sends suspend program signal, to resume it use fg <pid or jobnumber> or bg <pid or jobnumber> or send SIGCONT
-SIGSTOP -> same as SIGSTP, except no shortcut keys and it is unblockable.
-SIGCONT -> Continue signal, resumes a stopped program
-SIGFPE -> floating point exception (dividing by zero)
-SIGEGV -> Segmentation fault signal.
-SIGCHLD -> Signal sent to parent when child terminates.
-SIGKILL or -9 -> Unblockable kill signal

COMP 3000 2012 Week 8 Notes

2012-10-25T23:34:06Z

Sdp: added formatting

Practice questions and their associated answers:

<ol>
<li>
Q: You can create a sparse file in UNIX by writing sequences of zeros to a file. 

A: False. Sparse file means holes. Zeroes aren't holes.
YOu need to move the file pointer. You have to use a
seek. The data will be READ as zeros, though...
Look up truncate command and strace it. It ain't just writing zeros
</li>
<li>
Q: Are environment variables ``global variables? Specifically, if you change the value of an environment variable X in one process, what happens to the value of X in other processes? 

A:Nothing happens. Env vars aren't globals. They're passed in with Execve. Environments
variables are typically process specific.
</li>
<li>
Q: In UNIX, there are three permissions associated with the user, the user's group, and everyone else. What are those three permissions? 

A: rwx (Read, Write, Execute)
</li>
<li>
Q: If you see a zombie process during normal system operation, how can you get rid of it? 

A: Kill the parent
Why? init will own the zombie process and then kill it
</li>
<li>
Q: I can copy a file using I/O redirection as follows: cat /bin/ls > bar. If I now type ./bar, it won't run. However, if I run another command before typing ./bar, I will get
a file listing. What needed to be changed to get bar to run? 

A: chmod +x
Default non executable. Need to make it executable. (--x)
</li>
<li>
Q: System calls cannot normally be made using standard function calls. What CPU mechanism is used by regular processes to invoke kernel functionality? 

A: Software interrupt. Signals are a form of of software interrupt. 

Note:

"A system call is an entry point into the Linux kernel. Usually, system calls are not invoked directly: instead, most system calls have corresponding C library wrapper functions which perform the steps required (e.g., trapping to kernel mode) in order to invoke the system call. Thus, making a system call looks the same as invoking a normal library function." from man 2 intro 

"The call to the library function itself does not cause a switch to kernel mode (if the execution was not already in kernel mode) and is usually a normal subroutine call (using, for example, a "CALL" assembly instruction in some Instruction set architectures (ISAs)). The actual system call does transfer control to the kernel (and is more implementation-dependent and platform-dependent than the library call abstracting it). For example, in Unix-like systems, "fork" and "execve" are C library functions that in turn execute instructions that invoke the "fork" and "execve" system calls. Making the system call directly in the application code is more complicated and may require embedded assembly code to be used (in C and C++) as well as knowledge of the low-level binary interface for the system call operation, which may be subject to change over time and thus not be part of the application binary interface; the library functions are meant to abstract this away." from http://en.wikipedia.org/wiki/System_call

</li>
<li>

Q: After a fork system call, what runs first, the child process runs or the parent process? 

A: Don't know... Undefined. Can not depend on ordering

</li>
<li>

Q: Signal handlers can be called at almost any point in a process's execution. (T/F) 

A: True. That's the purpose of signals 

Register a normal C function as a signal handler 

Some are implemented by default (ex. sigkill) 

Signals can be called at any point in execution. 

"When a signal is sent, the operating system interrupts the target process's normal flow of execution. Execution can be '''interrupted during any non-atomic instruction.''' If the '''process has previously registered a signal handler, that routine is executed. Otherwise''' the default signal handler is executed." from http://en.wikipedia.org/wiki/Unix_signal

</li>
<li>

Q: System V init scripts run programs sequentially or in parallel? 

A: Sequentially (sysV). Shell script that runs every ''command'' sequentially 

Upstart does it in parallel though. This is the reason sysV is being replaced by Upstart.

</li>
<li>

Q: Without errors, what does the execve system call return? # Who calls a signal handler? 

A: execve DOES NOT RETURN ANYTHING. Process image is replaced. 

The program itself calls the signal handler

</li>
<li>

Q: What program is used to check the consistency of the data structures of a filesystem? 

A: fsck <- wrapper function 
fsck.FMT <- called by fsck for a specific format (ex: fsck.ext4)

</li>
<li>

Q: What allocates the storage for environment variables and command-line arguments? 

A: The kernel. The kernel is the only thing that can allocate.(execve) 
This also makes sense because execve is a system call.

</li>
<li>

Q: A file that has a logical size of one gigabyte but only takes up 100K on disk is called what kind of file in UNIX? 

A: Sparse file.

</li>
<li>

Q: What is an easy (but not foolproof way) way to tell that you are running in a virtual environment on Linux? What command do you run to get this information? 

A: lspci, lshw. 
In /proc/, you may be able to also. 
These commands display the machine's "hardware". 

Not /dev/ because that's the abstraction.

</li>
<li>

Q:What is the purpose of the PATH environment variable? 

A: At the command line, the $PATH variable is where the shell looks for binaries/files.
The path is walked when the fullPATH/filename of the command line hasn't been specified

</li>
<li>

Q: When would you expect ltrace to output more lines than strace? When should strace output more than ltrace? 

A: Some libraries make no sys calls (strlen). Libraries that do basic string and math
operations -> no sys calls 

Programs with lots of I/O will use lot of sys calls

</li>
<li>

Q: What command is used to change the priority of a process? 

A: nice and renice 
nicess is policy value (default value and you can modify this) 
priority (dynamic) is based on niceness and the amount of time the process has run on
cpu 
Priority doesn't apply to I/O. I/O will continue despite other processing having high
priority. I/O will always slow down the system.

</li>
<li>

Q: What is the difference between a library call and a function call? 

A: The former requires a library card and a telephone.

</li>
</ol>

COMP 3000 2012 Week 7 Notes

2012-10-25T23:20:22Z

Sdp: fixed unbalanced paren

==== Linux source code spelunking ====

see linux source cross reference

* /documention
* /arch --> platform specific code
* /drives --> drivers for devices
* /fs -> file system stuff (ext4 is here... open.c)

do\_ prefix on something is a wrapper function that preprocesses args

Looking at fs/ directory now...
* All these help with the vfs
* Relationship between dentry (directory entry) and inode
* Contents are inodes, dentry is a string representingo

Filesystems
* blocks -> actual data
* inodes -> metadata (index node).

2 Types of inode:
** directories and files are inodes
** directory inode -> name to inode mapping
** file inode -> size, last modified, pointers to data blocks, permissions, etc...
*** pointers to data wihtin inodes tell you where the data is, so that you can accessit

==== How to explore the kernel from userspace instead of messing about in the source code.====

Look at the output of mount
* some kernel services are mounted to various folders (ex: proc mounted to /proc)

=== /dev and udev ===

/dev is dynamically generated by udev

Now, in /dev/, do ls -l

drwxr-xr-x 2 root root 700 Oct 5 22:23 block
drwxr-xr-x 2 root root 120 Oct 5 22:23 bsg
brw-rw---- 1 root disk 7, 7 Oct 5 22:23 loop7
crw------- 1 root root 10, 237 Oct 5 22:23 loop-control
drwxr-xr-x 2 root root 80 Oct 5 22:23 mapper
crw------- 1 root root 10, 227 Oct 5 22:23 mcelog
crw-r--r-- 1 root root 10, 62 Oct 5 22:23 rfkill
lrwxrwxrwx 1 root root 4 Oct 5 22:23 rtc -> rtc0
crw------- 1 root root 254, 0 Oct 5 22:23 rtc0
brw-rw---- 1 root disk 8, 0 Oct 5 22:23 sda
brw-rw---- 1 root disk 8, 1 Oct 5 22:23 sda1
brw-rw---- 1 root disk 8, 2 Oct 5 22:23 sda2
crw-rw---- 1 root disk 21, 2 Oct 5 22:23 sg2
crw------- 1 root root 21, 3 Oct 5 22:23 sg3
lrwxrwxrwx 1 root root 8 Oct 5 22:23 shm -> /run/shm
crw------- 1 root root 10, 231 Oct 5 22:23 snapshot
drwxr-xr-x 3 root root 180 Oct 5 22:23 snd
lrwxrwxrwx 1 root root 15 Oct 5 22:23 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root 15 Oct 5 22:23 stdout -> /proc/self/fd/1
crw-rw-rw- 1 root tty 5, 0 Oct 17 08:47 tty
crw--w---- 1 root tty 4, 0 Oct 5 22:23 tty0
crw------- 1 root root 252, 5 Oct 5 22:23 usbmon5
drwxr-xr-x 4 root root 80 Oct 5 22:23 v4l
crw-rw---- 1 root tty 7, 0 Oct 5 22:23 vcs
crw-rw-rw- 1 root root 1, 5 Oct 5 22:23 zero

Row 5: Major Number.
Row 6: Minor Number.

Every driver may control different hardware controllers. These controllers are assigned major numbers. Each device they control is assigned a minor number. Any device can be identified with a major minor number pair. (see [http://uw714doc.sco.com/en/HDK_concepts/ddT_majmin.html SCO documentation for major and minor numbers])

* d = directory
* b = block device --> drives, etc.
* c = character file --> write or output sequences of characters (printers, zero, null)

random, urandom, null, zero are all device files, but they're really just special
function in the kernel. They use the file api, you can read and write to most of
them.

in /dev. run df .

Filesystem 1K-blocks Used Available Use% Mounted on
udev 499124 8 499116 1% /dev

The file system for this is udev, dyamically generated

udev rules are set in /etc/udev, sets who can access what devices. Sets policy.

see man udev

what is udevd vs udev?
* udev is filesystem type and a command
* udevd is a daemon that monitors changes

If you kill udevd, your kernel will not be informed of new changes (plugged in
devices,etc..). They won't be listed in /dev/.
The user cannot be informed of what is actually mounted.

LOOK UP udevd

Note: initrd (initial ram disk) has it's own /dev/ folder to allow SPECIAL devices to be mounted to before the file system and the root FS's /dev/ is loaded

/dev is defined in posix standard

=== /proc ===

Kernel state is defined as special files too! We don't need need special functions,
just look at the files in /proc/
* all numbers are process IDs
* other devices like uptime
* Ownership, groups, 'n stuff are all displayed in the files

Note: all of the sizes of these files are 0. Indicates that they dynamically generated

When you call ps, all it's doing is walking the proc directory.

Proc is NOT defined in the POSIX standard. It's a linux thing.

/sys/ and /proc/ are basically the same thing, but /sys is alot more regimented, proc
is loose. <---- LOOOOOOK UP!

Ex:

strace ifconfig

You'll notice that much of what ifconfig does is reading kernel /proc/ files

NOTE:
/dev/ -> devices
/proc/ and /sys/ -> KERNEL STATE

=== LINKS ===

Hardlink: association between name and inode. Can have many per inode.

Symlink: pointer to a name, not to an inode. You can have pointer to a name that's a
has a hardlink to an actual piece of data

ex:

ls -l /bin | grep gunzip
-rwxr-xr-x 2 root root 2251 Feb 8 2012 gunzip

Two links to gunzip's inode. Meaning, uncompress points to the same inode
gunzip and uncompress binaries have the same inode

User:Sdp

2012-10-22T22:14:02Z

Sdp: Added other domain

== Simon David Pratt ==

Simon is a third year computer science student.

I'm in [[Operating Systems (Fall 2012)]]

[http://simondavidpratt.com Simon David Pratt] My homepage

[http://pr4tt.com Simon Pratt] My other home page

COMP 3000 2012 Midterm Material

2012-10-22T22:08:49Z

Sdp: added answers from lab 2

== Questions ==

Questions from Lab 1

# What Linux distribution are you using? What version? You should look at the file <tt>/etc/issue</tt> generally to find out.
# What shell do you get by default when you log in? Check by running <tt>echo $SHELL</tt>. If it is not ''bash'', then type <tt>exec bash</tt> to switch to it.
# Is your Linux environment running virtualized or on bare metal? How do you know? (If you aren't sure, say so, but explain why.)
# Who installed and configured your Linux environment? What level of access do you have to it?
# A shell command can be one of four things. What are those four things? Explain very briefly.
# On the machine you are on, in what directories are commands external to the shell stored? How can you get this information on the command line?
# Compare the commands <tt>ls /usr/bin | more</tt> and <tt>ls /usr/bin | less</tt>. How is the output similar? How are they different?
# How would you create an empty file with the filename <tt>empty file.txt</tt>? (Yes, that is a space in the filename.)
# How would you output the contents of this file?
# Some shell commands, such as ''pwd'', are both built-in and are external. What is one reason why both versions might be present?
# What are the permissions on your Linux home directory? With those permissions and your knowledge of the other accounts on the system, who has access to your home directory, and what sort of access do they have?
# Give one or more command line strings that use the following operators: <tt>></tt>, <tt><</tt>, <tt>>></tt>, <tt><<</tt>, <tt>|</tt>. Explain briefly what each operator is doing in your examples, both concretely and in terms of STDIN and STDOUT.
# Are there operators that work specifically with STDERR? Explain briefly.
# How do you use these operators with other file descriptors? Explain briefly.
# Give an example of a ''bash'' <tt>for</tt> loop and explain what it does.
# Give an example of a ''bash'' <tt>if</tt> statement and explain what it does.
# With the <tt>&</tt> you can put processes in the "background". Given that external commands are always run as separate processes from the command shell, what is the key difference between foreground and background processes?
# What are the differences between shell and environment variables? Specifically, what processes have each of them, and to what extent are they shared? HINT: look at the <tt>execve</tt> system call.

Questions from Lab 2

# How can you get a list of all of the processes running on the system from the command line? Please give the command and required arguments, if any.
# What are five processes that are running on your system as non-regular users (i.e., as users other than the one you logged in as)? What does each do, briefly? Note: please exclude all processes enclosed in [], as those are built in to the kernel.
# How can I restart the graphical login screen on Ubuntu? (Hint: use virtual terminals and Ctrl-Alt-F? combinations to access a terminal that is independent of the GUI. Also, make sure you are aren't logged into a graphical session when you try this!)
# What happens when you send a kill -9 signal to one of the <tt>getty</tt> processes that are running by default? Why?
# What starts the upstart daemon? When is it started?
# How would you "change the system runlevel" to reboot? Shut down?
# Look at the output of <tt>ls -l /etc/init.d/</tt>. You will note <tt>-></tt> for many of the entries. This arrow shows that those entries are symbolic links. Where do most of these symbolic links point to? Why?
# Install the <tt>openssh-server</tt> package in your virtual machine from the command line. What command(s) did you use?
# Note that <tt>sshd</tt> has separate entries in <tt>/etc/init</tt> and <tt>/etc/init.d</tt> differ. Why do both exist (when most services have either one or the other)?
# Log messages for the system are stored in <tt>/var/log</tt>. What are four log files that are present on your system? (Ignore the ones with numeric extensions, those are old versions.) What program wrote each of those files directly?
# In <tt>/etc/init.d/ssh</tt> there are two lines near the beginning that end with <tt>output || exit 0</tt> (should be lines 16 and 17). What do each of these lines do exactly? And what is there purpose?
# What are the equivalent lines, if any, to these <tt>|| exit 0</tt> lines in <tt>/etc/init/ssh</tt>?
# What is plymouth?
# '''BONUS: ''' Trace plymouth's behavior throughout the boot process.

Questions from Lab 3

# Compile the program hello.c (below) with <tt>gcc -O hello.c -o hello-dyn</tt> and then run it using the command <tt>ltrace ./hello-dyn</tt> . What dynamic functions does the program call?
# Compile the same program with <tt>gcc -O -static hello.c -o hello-static</tt> . Run this second binary with ltrace as before. What dynamic functions does the program now call?
# Run strace on the static and dynamically compiled versions of hello. How many system calls do they each produce?
# How can you make the output of hello.c go to the file "hello-output" by changing how it is invoked at the command line?
# How can you make the output of hello.c go to the file "hello-output" by changing its code?
#Compile run-program.c with the command ''gcc -g run-program.c -o run-program'' and use it to answer the following questions.
<ol style="list-style-type: lower-alpha;">
# Is the line "Is this line printed?" printed when you execute '''./run-program /bin/ls -a /tmp'''? Why?
# Change the program to use execve instead of execvp. What is the difference between the two functions?
</ol>
# Linux signals is a simple form of IPC that is used for a variety of purposes.
<ol style="list-style-type: lower-alpha;">
# What signal is generated when a program attempts to divide by zero? Give a simple program that generates such a signal.
# How would you send a signal to a process to have it pause execution? Resume?
</ol>
# What command can lower the priority of an already running process?
# Why do we "open" files? Specifically, to what extent is it possible to do file I/O without open and close operations - assuming you could change the UNIX API?
# What is the relationship between function calls, system calls, and library calls?
# Create ''run-program-skips.c'' that is the same as run-program.c except that it ignores every other argument to the program to be run (starting with the first non-program-name argument). Hence, ''./run-program-skip /bin/ls ignore -a ignore /tmp'' will do the same as ''./run-program /bin/ls -a /tmp''.
# Create ''sanitize-env.c'' that does the same as run-program.c except it exec's the program with all environment variables stripped except the PATH and TERM variables (assuming they are defined). Be sure to document how you tested your program.
# Create a simple program ''receive-term.c'' that uses sigaction in order to print "Ouch!" when you send the program the TERM signal and then exits. Note that your program should not consume much CPU time while waiting (i.e., do not poll for the signal). As part of your answer, give the command for sending the signal.
# Create a simple program ''nocrash.c'' that accesses an invalid pointer but then prints "Oops, my mistake" rather than crash.
# Create a program ''run-program-dots.c'' that works the same as run-program.c, except that it prints one dot every second while a given program runs. Note that it should stop printing dots once the exec'd program terminates. Your solution should use the fork(), sleep(), execve(), and sigaction() calls.
# What is the difference between the PR and NI columns in the top command? Explain in the context of running a CPU-intensive program. (Hint: you should try running a CPU intensive program or two and observe these columns...)

==Program Listings==
<code>
/* hello.c */
#include <stdio.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
pid_t p;

p = getpid();

printf("Hello, world! I am process %d.\n", (int) p);
return 0;
}
</code>

<code>
/* run-program.c */

#include <unistd.h>
#include <stdio.h>

int main( int argc, char *argv[], char *envp[] ) {

pid_t p;

p = getpid();

printf("Hello again. I am process %d.\n", (int) p);

if( argc < 2 ) {
printf( "Insufficient arguments.\n" );
return -1;
}
execvp( argv[1], argv + 1 );
printf( "Is this line printed?\n" );
return 0;
}
</code>

Questions from Lab 4

# Run the command <tt>truncate -s 1G foo</tt>. What is the logical size of foo, and how much space does it consume on disk?
# Run <tt>mkfs.ext4 foo</tt>. (Say "yes" to operating on a regular file.) What is the logical size of foo now, and how much space does it now consume on disk?
# What command do you run to check the filesystem in foo for errors?
# Run <tt>mount foo /mnt</tt>. What does this command do?
# Run <tt>df</tt>. What device is mounted on /mnt? What is this device?
# Run <tt>rsync -a -v /etc /mnt</tt>. What does this command do? Explain the arguments as well.
# Run <tt>umount /mnt</tt>. What files can you still access, and what have gone away?
# Run <tt>dd if=/dev/zero of=foo conv=notrunc count=10 bs=512</tt>. What does this command do?
# Run <tt>mount foo /mnt</tt>. What error do you get?
# What command can you run to make foo mountable again? What characteristic of the file system enables this command to work?
# Create a simple C program to generate a file consisting of the word "Hello", one thousand bytes of blank space (null bytes), and then "world\n", and then another thousand null bytes, such that the file produced consumes the maximum amount of space given its contents.
# Create a C program "mycopy-blocks.c" that takes three arguments: a block size and two filenames A and B as command line arguments. This program should copy the contents of A to B, making B a byte-for-byte copy of A, except that the length of B should be evenly divisible by the block size. The file should be padded with nulls as necessary. Zero bytes in A should consume little to no space in B, however, including the padding.
# How could loopback mounts be useful when working with virtual machines? What conditions must the VM software meet for this to work?

== Answers ==

Answers from Lab 1

<ol><li> should be: Ubuntu 12.04 LTS unless specifically stated reason otherwise
</li><li> /bin/bash *SCS gives csh as the default shell*
</li><li> bare metal (lambda) or virtualbox (virtualized)
</li><li> System administrators, level of access is restricted user

</li><li> An executable program like all those files we saw in /usr/bin. Within this category, programs can be compiled binaries such as programs written in C and C++, or programs written in scripting languages such as the shell, perl, python, ruby, etc.

A command built into the shell itself. bash supports a number of commands internally called shell builtins. The cd command, for example, is a shell builtin.

A shell function. These are miniature shell scripts incorporated into the environment. We will cover configuring the environment and writing shell functions in later chapters, but for now, just be aware that they exist.

An alias. Commands that we can define ourselves, built from other commands.
</li><li> /usr/bin and /usr/sbin and other directories. Run "echo $PATH" to see list of directories. Also, which [external command] will tell you path for specific commands, e.g. "which ls"
</li><li> less allows backward movement within the file as well as forward movement. more is a rudimentary method for printing to crt terminals, but only allows movement through the file in a forward direction. You have to press 'q' to exit less, but more quits when it reaches EOF.
</li><li> touch "empty file.txt"
</li><li> cat empty\ file.txt or cat "empty file.txt" or cat 'empty file.txt' or > 'empty file.txt'
</li><li> Some require direct access to kernel functions and therefore must be built in. Also the built in functions operate faster than non built in functions.
This is either for performance reasons -- builtins execute faster than external commands, which usually require forking off a separate process -- or because a particular builtin needs direct access to the shell internals.

</li><li> drwx------

d = a directory flag - since my home directory is a directory, there exists a d here
rwx = read, write, execute permission for the User (owner of the directory) which is my username
--- = read write execute permissions are lacking for a Group
--- = read write execute permissions are lacking for other

this means my work is private. Only myself and the root user can access my directory.

</li>
<li> cat file.txt | grep hello

the | character joins the two commands cat and grep, it enables the output from one command to be passed directly to another.

cat file.txt > error.txt

the > character redirects the standard output to the specified filename that is included in the command after it. If the file is not created it will create it, if the file exists it will overwrite the current file

cat file.txt > alreadyexistingerrorfile.txt

the >> character appends to the already existing file the output of the first command.

cat < file.txt is equivalent to cat file.txt

the < character redirects the standard input to come from the file instead

the last form of redirection is only useful within shell scripts to avoid having to code a bunch of echo statements, say for a particular menu. The document created is called a here document. It is a method of specifying a string literal in command line shells. (preserves line breaks and whitespaces / indentation within the text.)

example:
</li>
<li> present a menu
cat << alldone
What would you like to do today?
Play with shell scripts
Write wonderful awk scripts
Invent powerful sed scripts
zippity do and dah
alldone

the above code will generate the following output:

What would you like to do today?
Play with shell scripts
Write wonderful awk scripts
Invent powerful sed scripts
zippity do and dah

otherwise you would have had to code echo statements for each line.</li>

<li> STDERR does not have a dedicated redirection operator, instead to redirect stderr you have to refer to it's file descriptor. and call it using 2> </li>

<li> the other file descriptors are 0 for standard in and 1 for standard out, the descriptor number is placed right before the redirection operator.</li>

<li>
for variable [in words]; do
commands
done

variable is the name of a variable that will increment during the execution of the loop, words is an optional list of items that will be sequentially assigned to the variable, and commands are the list of commands to be executed on each iteration of the loop.

for i in A B C D; do echo $i; done
</li>

<li>
x=5
if [ $x = 5 ]; then
echo "x equals 5."
else
echo "x does not equal 5."
fi

x=5
if [ $x = 5 ]; then echo "equals 5"; else echo "does not equal 5"; fi

if commands; then
commands
[elif commands; then
commands...]
[else
commands]
fi
where commands is a set of commands. Providing an exit status that indicates success or failure.
</li>

<li> Background processes usually interact with the system and do not interact with the shell or the user, foreground processes must interact with the command shell. In the shell, background and foreground processes are the same except that the shell waits for foreground processes to finish before giving a new prompt.</li>

<li> shell variables apply only to the current shell. Environment variables apply to all possible shells running on the system. to list all env variables type printenv to list all shell variables use set
a shell variable is local to a particular instance of the shell (such as a shell script), while environment variables are "inherited" by any program you start, including another shell
That is, the new process gets its own copy of these variables, which it can read, modify, and pass on in turn to its own children. In fact, every UNIX process (not just the shell) passes its environment variables to its child processes.</li>
</ol>

Answers from Lab 2

# ps -aux ps -e Among others.
# cron - Cron is the time-based job scheduler. lightdm - Display manager. It also manages the X servers and facilitates remote logins using the XDMCP protocol. Network Manager - NetworkManager is a set of co-operative tools that make networking simple and straightforward. Whether WiFi, wired, 3G, or Bluetooth, NetworkManager allows you to quickly move from one network to another user/sbin/modemmanager. Upstart Socket Bridge - The upstart-socket-bridge queries the Upstart init(8) daemon for all job configurations which start on or stop on the socket event. It then waits for an incoming connection on each specified socket(7) and when detected emits the socket event (socket-event (7)), setting a number of environment variables for the job to query. acpid - The acpid process is designed to notify user-space programs of ACPI events. These are just some examples of acceptable answers. There are others as well.
# Make sure you are not logged into xsession, then restart lightdm with sudo service lightdm restart
# It will automatically respawn itself. There is a line in the tty script that instructs the process to automatically restart in the case of failure or termination of the process.
# The kernel starts the upstart daemon at system boot.
# To reboot use either telinit 6 or shutdown -r. To shutdown use either telinit 0 or shutdown tool with -h option and a time.
# The links point to /lib/init/upstart-job. Ubuntu is transitioning from SysVinit scripts to upstart scripts, these links are used to redirect to the upstart services.
# sudo apt-get install openssh-server.
# The init.d script is used only for chroot environments. See lines 14-26 in init.d/ssh script.
# Can check syslog process to see which logs it directly writes too. Some acceptable answers are:
## auth.log - CRON, lightdm, polkitd
## boot.log - modem-manager
## bootstrap.log - gpgv, dpkg
## dpkg.log - dpkg
## fontconfig.log - fontconfig
## kern.log - kernel
## pm-powersave.log - powerd
## pm-suspend.log - sleepd
## ufw.log - ufw
# test -x /usr/sbin/sshd || exit 0 - Check whether "/usr/sbin/sshd" exists and is executable and exit with status of 0 if not. Used to verify SSHD is installed. ( /usr/sbin/sshd -\? 2>&1 | grep -q OpenSSH ) 2>/dev/null || exit 0 - Runs SSH script with illegal option and pipes the output to grep to check for OpenSSH. Rest is piped to /dev/null essentially discarding it. This line checks to verify that the version of SSH installed is OpenSSH. If not will exit with status of 0.
# test -x /usr/sbin/sshd || { stop; exit 0; } Does not have a check for OpenSSH.
# Plymouth is a bootsplash. It provides a graphical boot animation while the boot process takes place in the background.
# not given.

COMP 3000 2012 Midterm Material

2012-10-22T22:06:02Z

Sdp: added answers for lab 1

COMP 3000 2012 Midterm Material

2012-10-22T21:54:57Z

Sdp: added questions from lab 4

Questions from Lab 1

# What is your username on this wiki? Have a TA help you create an account.
# What Linux distribution are you using? What version? You should look at the file <tt>/etc/issue</tt> generally to find out.
# What shell do you get by default when you log in? Check by running <tt>echo $SHELL</tt>. If it is not ''bash'', then type <tt>exec bash</tt> to switch to it.
# Is your Linux environment running virtualized or on bare metal? How do you know? (If you aren't sure, say so, but explain why.)
# Who installed and configured your Linux environment? What level of access do you have to it?
# A shell command can be one of four things. What are those four things? Explain very briefly.
# On the machine you are on, in what directories are commands external to the shell stored? How can you get this information on the command line?
# Compare the commands <tt>ls /usr/bin | more</tt> and <tt>ls /usr/bin | less</tt>. How is the output similar? How are they different?
# How would you create an empty file with the filename <tt>empty file.txt</tt>? (Yes, that is a space in the filename.)
# How would you output the contents of this file?
# Some shell commands, such as ''pwd'', are both built-in and are external. What is one reason why both versions might be present?
# What are the permissions on your Linux home directory? With those permissions and your knowledge of the other accounts on the system, who has access to your home directory, and what sort of access do they have?
# Give one or more command line strings that use the following operators: <tt>></tt>, <tt><</tt>, <tt>>></tt>, <tt><<</tt>, <tt>|</tt>. Explain briefly what each operator is doing in your examples, both concretely and in terms of STDIN and STDOUT.
# Are there operators that work specifically with STDERR? Explain briefly.
# How do you use these operators with other file descriptors? Explain briefly.
# Give an example of a ''bash'' <tt>for</tt> loop and explain what it does.
# Give an example of a ''bash'' <tt>if</tt> statement and explain what it does.
# With the <tt>&</tt> you can put processes in the "background". Given that external commands are always run as separate processes from the command shell, what is the key difference between foreground and background processes?
# What are the differences between shell and environment variables? Specifically, what processes have each of them, and to what extent are they shared? HINT: look at the <tt>execve</tt> system call.

Questions from Lab 2

# How can you get a list of all of the processes running on the system from the command line? Please give the command and required arguments, if any.
# What are five processes that are running on your system as non-regular users (i.e., as users other than the one you logged in as)? What does each do, briefly? Note: please exclude all processes enclosed in [], as those are built in to the kernel.
# How can I restart the graphical login screen on Ubuntu? (Hint: use virtual terminals and Ctrl-Alt-F? combinations to access a terminal that is independent of the GUI. Also, make sure you are aren't logged into a graphical session when you try this!)
# What happens when you send a kill -9 signal to one of the <tt>getty</tt> processes that are running by default? Why?
# What starts the upstart daemon? When is it started?
# How would you "change the system runlevel" to reboot? Shut down?
# Look at the output of <tt>ls -l /etc/init.d/</tt>. You will note <tt>-></tt> for many of the entries. This arrow shows that those entries are symbolic links. Where do most of these symbolic links point to? Why?
# Install the <tt>openssh-server</tt> package in your virtual machine from the command line. What command(s) did you use?
# Note that <tt>sshd</tt> has separate entries in <tt>/etc/init</tt> and <tt>/etc/init.d</tt> differ. Why do both exist (when most services have either one or the other)?
# Log messages for the system are stored in <tt>/var/log</tt>. What are four log files that are present on your system? (Ignore the ones with numeric extensions, those are old versions.) What program wrote each of those files directly?
# In <tt>/etc/init.d/ssh</tt> there are two lines near the beginning that end with <tt>output || exit 0</tt> (should be lines 16 and 17). What do each of these lines do exactly? And what is there purpose?
# What are the equivalent lines, if any, to these <tt>|| exit 0</tt> lines in <tt>/etc/init/ssh</tt>?
# What is plymouth?
# '''BONUS: ''' Trace plymouth's behavior throughout the boot process.

Questions from Lab 3

# Compile the program hello.c (below) with <tt>gcc -O hello.c -o hello-dyn</tt> and then run it using the command <tt>ltrace ./hello-dyn</tt> . What dynamic functions does the program call?
# Compile the same program with <tt>gcc -O -static hello.c -o hello-static</tt> . Run this second binary with ltrace as before. What dynamic functions does the program now call?
# Run strace on the static and dynamically compiled versions of hello. How many system calls do they each produce?
# How can you make the output of hello.c go to the file "hello-output" by changing how it is invoked at the command line?
# How can you make the output of hello.c go to the file "hello-output" by changing its code?
#Compile run-program.c with the command ''gcc -g run-program.c -o run-program'' and use it to answer the following questions.
<ol style="list-style-type: lower-alpha;">
# Is the line "Is this line printed?" printed when you execute '''./run-program /bin/ls -a /tmp'''? Why?
# Change the program to use execve instead of execvp. What is the difference between the two functions?
</ol>
# Linux signals is a simple form of IPC that is used for a variety of purposes.
<ol style="list-style-type: lower-alpha;">
# What signal is generated when a program attempts to divide by zero? Give a simple program that generates such a signal.
# How would you send a signal to a process to have it pause execution? Resume?
</ol>
# What command can lower the priority of an already running process?
# Why do we "open" files? Specifically, to what extent is it possible to do file I/O without open and close operations - assuming you could change the UNIX API?
# What is the relationship between function calls, system calls, and library calls?
# Create ''run-program-skips.c'' that is the same as run-program.c except that it ignores every other argument to the program to be run (starting with the first non-program-name argument). Hence, ''./run-program-skip /bin/ls ignore -a ignore /tmp'' will do the same as ''./run-program /bin/ls -a /tmp''.
# Create ''sanitize-env.c'' that does the same as run-program.c except it exec's the program with all environment variables stripped except the PATH and TERM variables (assuming they are defined). Be sure to document how you tested your program.
# Create a simple program ''receive-term.c'' that uses sigaction in order to print "Ouch!" when you send the program the TERM signal and then exits. Note that your program should not consume much CPU time while waiting (i.e., do not poll for the signal). As part of your answer, give the command for sending the signal.
# Create a simple program ''nocrash.c'' that accesses an invalid pointer but then prints "Oops, my mistake" rather than crash.
# Create a program ''run-program-dots.c'' that works the same as run-program.c, except that it prints one dot every second while a given program runs. Note that it should stop printing dots once the exec'd program terminates. Your solution should use the fork(), sleep(), execve(), and sigaction() calls.
# What is the difference between the PR and NI columns in the top command? Explain in the context of running a CPU-intensive program. (Hint: you should try running a CPU intensive program or two and observe these columns...)

==Program Listings==
<code>
/* hello.c */
#include <stdio.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
pid_t p;

p = getpid();

printf("Hello, world! I am process %d.\n", (int) p);
return 0;
}
</code>

<code>
/* run-program.c */

#include <unistd.h>
#include <stdio.h>

int main( int argc, char *argv[], char *envp[] ) {

pid_t p;

p = getpid();

printf("Hello again. I am process %d.\n", (int) p);

if( argc < 2 ) {
printf( "Insufficient arguments.\n" );
return -1;
}
execvp( argv[1], argv + 1 );
printf( "Is this line printed?\n" );
return 0;
}
</code>

Questions from Lab 4

# Run the command <tt>truncate -s 1G foo</tt>. What is the logical size of foo, and how much space does it consume on disk?
# Run <tt>mkfs.ext4 foo</tt>. (Say "yes" to operating on a regular file.) What is the logical size of foo now, and how much space does it now consume on disk?
# What command do you run to check the filesystem in foo for errors?
# Run <tt>mount foo /mnt</tt>. What does this command do?
# Run <tt>df</tt>. What device is mounted on /mnt? What is this device?
# Run <tt>rsync -a -v /etc /mnt</tt>. What does this command do? Explain the arguments as well.
# Run <tt>umount /mnt</tt>. What files can you still access, and what have gone away?
# Run <tt>dd if=/dev/zero of=foo conv=notrunc count=10 bs=512</tt>. What does this command do?
# Run <tt>mount foo /mnt</tt>. What error do you get?
# What command can you run to make foo mountable again? What characteristic of the file system enables this command to work?
# Create a simple C program to generate a file consisting of the word "Hello", one thousand bytes of blank space (null bytes), and then "world\n", and then another thousand null bytes, such that the file produced consumes the maximum amount of space given its contents.
# Create a C program "mycopy-blocks.c" that takes three arguments: a block size and two filenames A and B as command line arguments. This program should copy the contents of A to B, making B a byte-for-byte copy of A, except that the length of B should be evenly divisible by the block size. The file should be padded with nulls as necessary. Zero bytes in A should consume little to no space in B, however, including the padding.
# How could loopback mounts be useful when working with virtual machines? What conditions must the VM software meet for this to work?

COMP 3000 2012 Midterm Material

2012-10-22T21:54:05Z

Sdp: fixed formatting on Lab 3 questions

COMP 3000 2012 Midterm Material

2012-10-22T21:52:29Z

Sdp: added questions from Lab 3

Questions from Lab 1

# What is your username on this wiki? Have a TA help you create an account.
# What Linux distribution are you using? What version? You should look at the file <tt>/etc/issue</tt> generally to find out.
# What shell do you get by default when you log in? Check by running <tt>echo $SHELL</tt>. If it is not ''bash'', then type <tt>exec bash</tt> to switch to it.
# Is your Linux environment running virtualized or on bare metal? How do you know? (If you aren't sure, say so, but explain why.)
# Who installed and configured your Linux environment? What level of access do you have to it?
# A shell command can be one of four things. What are those four things? Explain very briefly.
# On the machine you are on, in what directories are commands external to the shell stored? How can you get this information on the command line?
# Compare the commands <tt>ls /usr/bin | more</tt> and <tt>ls /usr/bin | less</tt>. How is the output similar? How are they different?
# How would you create an empty file with the filename <tt>empty file.txt</tt>? (Yes, that is a space in the filename.)
# How would you output the contents of this file?
# Some shell commands, such as ''pwd'', are both built-in and are external. What is one reason why both versions might be present?
# What are the permissions on your Linux home directory? With those permissions and your knowledge of the other accounts on the system, who has access to your home directory, and what sort of access do they have?
# Give one or more command line strings that use the following operators: <tt>></tt>, <tt><</tt>, <tt>>></tt>, <tt><<</tt>, <tt>|</tt>. Explain briefly what each operator is doing in your examples, both concretely and in terms of STDIN and STDOUT.
# Are there operators that work specifically with STDERR? Explain briefly.
# How do you use these operators with other file descriptors? Explain briefly.
# Give an example of a ''bash'' <tt>for</tt> loop and explain what it does.
# Give an example of a ''bash'' <tt>if</tt> statement and explain what it does.
# With the <tt>&</tt> you can put processes in the "background". Given that external commands are always run as separate processes from the command shell, what is the key difference between foreground and background processes?
# What are the differences between shell and environment variables? Specifically, what processes have each of them, and to what extent are they shared? HINT: look at the <tt>execve</tt> system call.

Questions from Lab 2

# How can you get a list of all of the processes running on the system from the command line? Please give the command and required arguments, if any.
# What are five processes that are running on your system as non-regular users (i.e., as users other than the one you logged in as)? What does each do, briefly? Note: please exclude all processes enclosed in [], as those are built in to the kernel.
# How can I restart the graphical login screen on Ubuntu? (Hint: use virtual terminals and Ctrl-Alt-F? combinations to access a terminal that is independent of the GUI. Also, make sure you are aren't logged into a graphical session when you try this!)
# What happens when you send a kill -9 signal to one of the <tt>getty</tt> processes that are running by default? Why?
# What starts the upstart daemon? When is it started?
# How would you "change the system runlevel" to reboot? Shut down?
# Look at the output of <tt>ls -l /etc/init.d/</tt>. You will note <tt>-></tt> for many of the entries. This arrow shows that those entries are symbolic links. Where do most of these symbolic links point to? Why?
# Install the <tt>openssh-server</tt> package in your virtual machine from the command line. What command(s) did you use?
# Note that <tt>sshd</tt> has separate entries in <tt>/etc/init</tt> and <tt>/etc/init.d</tt> differ. Why do both exist (when most services have either one or the other)?
# Log messages for the system are stored in <tt>/var/log</tt>. What are four log files that are present on your system? (Ignore the ones with numeric extensions, those are old versions.) What program wrote each of those files directly?
# In <tt>/etc/init.d/ssh</tt> there are two lines near the beginning that end with <tt>output || exit 0</tt> (should be lines 16 and 17). What do each of these lines do exactly? And what is there purpose?
# What are the equivalent lines, if any, to these <tt>|| exit 0</tt> lines in <tt>/etc/init/ssh</tt>?
# What is plymouth?
# '''BONUS: ''' Trace plymouth's behavior throughout the boot process.

Questions from Lab 3

# Compile the program hello.c (below) with <tt>gcc -O hello.c -o hello-dyn</tt> and then run it using the command <tt>ltrace ./hello-dyn</tt> . What dynamic functions does the program call?
# Compile the same program with <tt>gcc -O -static hello.c -o hello-static</tt> . Run this second binary with ltrace as before. What dynamic functions does the program now call?
# Run strace on the static and dynamically compiled versions of hello. How many system calls do they each produce?
# How can you make the output of hello.c go to the file "hello-output" by changing how it is invoked at the command line?
# How can you make the output of hello.c go to the file "hello-output" by changing its code?
#Compile run-program.c with the command ''gcc -g run-program.c -o run-program'' and use it to answer the following questions.
<ol style="list-style-type: lower-alpha;">
# Is the line "Is this line printed?" printed when you execute '''./run-program /bin/ls -a /tmp'''? Why?
# Change the program to use execve instead of execvp. What is the difference between the two functions?

# Linux signals is a simple form of IPC that is used for a variety of purposes.
<ol style="list-style-type: lower-alpha;">
# What signal is generated when a program attempts to divide by zero? Give a simple program that generates such a signal.
# How would you send a signal to a process to have it pause execution? Resume?

# What command can lower the priority of an already running process?
# Why do we "open" files? Specifically, to what extent is it possible to do file I/O without open and close operations - assuming you could change the UNIX API?
# What is the relationship between function calls, system calls, and library calls?
# Create ''run-program-skips.c'' that is the same as run-program.c except that it ignores every other argument to the program to be run (starting with the first non-program-name argument). Hence, ''./run-program-skip /bin/ls ignore -a ignore /tmp'' will do the same as ''./run-program /bin/ls -a /tmp''.
# Create ''sanitize-env.c'' that does the same as run-program.c except it exec's the program with all environment variables stripped except the PATH and TERM variables (assuming they are defined). Be sure to document how you tested your program.
# Create a simple program ''receive-term.c'' that uses sigaction in order to print "Ouch!" when you send the program the TERM signal and then exits. Note that your program should not consume much CPU time while waiting (i.e., do not poll for the signal). As part of your answer, give the command for sending the signal.
# Create a simple program ''nocrash.c'' that accesses an invalid pointer but then prints "Oops, my mistake" rather than crash.
# Create a program ''run-program-dots.c'' that works the same as run-program.c, except that it prints one dot every second while a given program runs. Note that it should stop printing dots once the exec'd program terminates. Your solution should use the fork(), sleep(), execve(), and sigaction() calls.
# What is the difference between the PR and NI columns in the top command? Explain in the context of running a CPU-intensive program. (Hint: you should try running a CPU intensive program or two and observe these columns...)

==Program Listings==
<code>
/* hello.c */
#include <stdio.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
pid_t p;

p = getpid();

printf("Hello, world! I am process %d.\n", (int) p);
return 0;
}
</code>

<code>
/* run-program.c */

#include <unistd.h>
#include <stdio.h>

int main( int argc, char *argv[], char *envp[] ) {

pid_t p;

p = getpid();

printf("Hello again. I am process %d.\n", (int) p);

if( argc < 2 ) {
printf( "Insufficient arguments.\n" );
return -1;
}
execvp( argv[1], argv + 1 );
printf( "Is this line printed?\n" );
return 0;
}
</code>

COMP 3000 2012 Midterm Material

2012-10-22T21:49:13Z

Sdp: added questions from lab 2

COMP 3000 2012 Midterm Material

2012-10-22T21:48:10Z

Sdp: added questions from lab 1

Operating Systems (Fall 2012)

2012-10-22T21:46:50Z

Sdp: added link to compiled midterm material

This is the main page for the Fall 2012 run of Carleton University's COMP 3000, Operating Systems. The instructor for this course is [http://people.scs.carleton.ca/~soma Anil Somayaji]

==Course Outline==

The course outline for this course can be found [http://www.scs.carleton.ca/courses/course_outline.php?Number=COMP%203000&Term=Fall&Year=2012 here].

==Readings==

==Weekly Schedule==

<table style="width: 100%;" border="1" cellpadding="4" cellspacing="0">
<tr valign="top">
<th>
Week
</th>
<th>
Due

</th>
<th>
Topics
</th>
<th>
Readings
</th>
<th>
Notes
</th>
</tr>
<tr>
<td>
Sept. 7

</td>
<td> </td>
<td>
Introduction, OS Overview

</td>
<td>


</td>
<td>
[[COMP 3000 2012 Week 1 Notes|Notes]]

</td>
</tr>
<tr>
<td>Sept. 10-14
</td>
<td>
</td>
<td>UNIX Basics
</td>
<td>Ritchie & Thompson, [http://dl.acm.org/citation.cfm?id=361061 The UNIX Time-Sharing System].
</td>
<td>
[[COMP 3000 2012 Week 2 Notes|Notes]]

</td>
</tr>
<tr>
<td>Sept. 17-21
</td>
<td>[[COMP 3000 Lab 1 2012|Lab 1]]
</td>
<td>
</td>
<td>
</td>
<td>
[[COMP 3000 2012 Week 3 Notes|Notes]]

</td>
</tr>
<tr>
<td>Sept. 24-28
</td>
<td>[[COMP 3000 Lab 2 2012|Lab 2]]
</td>
<td>
</td>
<td>
</td>
<td>
[[COMP 3000 2012 Week 4 Notes|Notes]]

</td>
</tr>
<tr>
<td>Oct. 1-5
</td>
<td>[[COMP 3000 Lab 3 2012|Lab 3]] [[COMP 3000 2012 Documentation Proposal|Proposal]]
</td>
<td>
</td>
<td>
</td>
<td>
[[COMP 3000 2012 Week 5 Notes|Notes]]

</td>
</tr>
<tr>
<td>Oct. 8-12
</td>
<td>
</td>
<td> </td>
<td>
</td>
<td>
[[COMP 3000 2012 Week 6 Notes|Notes]]

</td>
</tr>
<tr>
<td>Oct. 15-19
</td>
<td>[[COMP 3000 Lab 4 2012|Lab 4]] [[COMP 3000 Demo 2012|Demo]]
</td>
<td>
</td>
<td></td>
<td>
[[COMP 3000 2012 Week 7 Notes|Notes]]

</td>
</tr>
<tr>
<td>Oct. 22-26
</td>
<td>[[COMP 3000 2012 Midterm Material | Midterm]] (Oct. 26)
</td>
<td>
</td>
<td></td>
<td>
[[COMP 3000 2012 Week 8 Notes|Notes]]

</td>
</tr>
<tr>
<td>Oct. 29-Nov.2
</td>
<td>[[COMP 3000 Lab 5 2012|Lab 5]] [[COMP 3000 Demo 2012|Demo]]
</td>
<td>
</td>
<td>
</td>
<td>
[[COMP 3000 2012 Week 9 Notes|Notes]]

</td>
</tr>
<tr>
<td>Nov. 5-9
</td>
<td>[[COMP 3000 Lab 6 2012|Lab 6]] [[COMP 3000 Demo 2012|Demo]]
</td>
<td>
</td>
<td>
</td>
<td>
[[COMP 3000 2012 Week 10 Notes|Notes]]

</td>
</tr>
<tr>
<td>Nov. 12-16
</td>
<td>[[COMP 3000 Lab 7 2012|Lab 7]] [[COMP 3000 Documentation Project|Documentation]] (Nov. 16)
</td>
<td>
</td>
<td>
</td>
<td>
[[COMP 3000 2012 Week 11 Notes|Notes]]

</td>
</tr>
<tr>
<td>Nov. 19-23
</td>
<td>[[COMP 3000 Lab 8 2012|Lab 8]]
</td>
<td>
</td>
<td>
</td>
<td>
[[COMP 3000 2012 Week 12 Notes|Notes]]

</td>
</tr>
<tr>
<td>Nov. 26-30
</td>
<td>
</td>
<td>
</td>
<td>
</td>
<td>
[[COMP 3000 2012 Week 13 Notes|Notes]]

</td>
</tr>
<tr>
<td>TBA
</td>
<td>Final Exam
</td>
<td>
</td>
<td>
</td>
<td>


</td>
</tr>
</table>

COMP 3000 2012 Week 6 Notes

2012-10-12T16:55:16Z

Sdp: created

* Sun's NFS didn't use open or close
** look into this
* signal handlers shouldn't do I/O
* open
** translates a path and options into a file descriptor
* task_struct
** represents a "task" in the kernel
** a task can be a process or a thread
** has LOTS of fields
** [http://lxr.linux.no/#linux+v3.6.1/include/linux/sched.h#L1234 task_struct on linux cross reference]
* mount
** USB into linux shows up in /media/USBNAME
* VFS layer
** Virtual File System
* resources
** [http://lxr.linux.no/+trees linux cross reference]

User:Sdp

2012-10-10T16:43:08Z

Sdp: /* Simon David Pratt */

== Simon David Pratt ==

Simon is a third year computer science student.

I'm in [[Operating Systems (Fall 2012)]]

[http://simondavidpratt.com Simon David Pratt] My homepage

User:Sdp

2012-10-10T16:42:35Z

Sdp:

== Simon David Pratt ==

Simon is a third year computer science student.

I'm in [[Operating Systems (Fall 2012)]]

[http://simondavidpratt.com Homepage]

User:Sdp

2012-10-10T16:42:25Z

Sdp: /* Simon David Pratt */

== Simon David Pratt ==

Simon is a third year computer science student.

I'm in [[Operating Systems (Fall 2012)]]

[[http://simondavidpratt.com Homepage]]

COMP 3000 2012 Week 5 Notes

2012-10-03T16:54:25Z

Sdp: fixed link

== Processes ==

* how a process starts
** fork()
*** creates a new process
*** different return values for parent and child
*** copies state, doesn't share
*** modern unix doesn't actually copy, it just pretends that it copies
**** "[...] which is the story of virtual memory." -Anil
** execve(const char *path, char *const argv[], char *const envp[])
*** when a process runs execve, it commits suicide and hands its process over to the new execution
*** note that the arrays themselves are null terminated
*** copies state, doesn't share
*** envp are environment variables
*** file descriptors stay open
**** so if you redirect file descriptors for original process, they stay redirected
*** env
**** command that dumps environment variables
* how a process ends
** exit(int return)
*** zero is normal exit
*** non-zero is abnormal
*** magic! see: [http://git.minix3.org/?p=minix.git;a=blob;f=lib/libc/stdlib/exit.c;h=c5bcd6725960f5244eed04a53659daa1b49a8194;hb=HEAD source for exit(int) in minix]
** wait(int* stat_loc)
*** blocks until a child terminates
** waitpid(pid_t pid, int* stat_loc, int options)
*** blocks until child with given pid terminates
* process information
** compare the output of strace and ltrace
*** tells you what the library is doing before it gets to the system level
* system calls
** open
*** creates a file descriptor
** close
** read
** write
** seek
** chown
*** the chgrp command uses the lchown system call
** chmod
** unlink
** mmap
*** map to memory
*** forward reference to the story of virtual memory